A First Look at dplyr

library(ascii)
library(assertthat)
library(dplyr)
library(ggplot2)
library(knitr)
library(nycflights13)
library(tidyverse)

Note: The stuff in the tidyverse tends to clobber built-in functions, which can make it hard to understand sometimes when things go wrong. If you forget to load dplyer, for instance, there's already a filter function, but it doesn't do the same thing as the one you load with dplyr so you'll get weird error messages if you try to use it. Notably I was getting this message:

Error in filter(airlines, carrier == "US") : object 'carrier' not found

Anyway, now some options for the packages.

options(asciiType="org")
theme_set(theme_minimal(base_family="Palatino"))

And a couple of helpers to print (the tibble objects dump weird characters into the org-mode output).

khead <- function(data) {
  #' Prints a table of the first lines of the dataframe
  #'
  #' @param data The data to print
  table <- kable(head(data))
  table[2] <- gsub(":", "-", table[2])
  table
}

okable <- function(data) {
  #' Prints data as an org-mode table
  #'
  #' @param data The data to print
  table <- kable(data)
  table[2] <- gsub(":", "-", table[2])
  table
}

got_all_your_data <- function(data) {
  #' Checks if there's any missing values
  #'
  #' @param data The vector of data to check
  see_if(noNA(data))
}

filter takes the dataframe as the first argument and every argument after that has to be a boolean expression based on the dataframe. If you pass in more than one condition dplyr will and them. Let's see the flights on Christmas.

christmas <- filter(flights, month==12, day==25)
plot = ggplot(data=christmas) +
  geom_bar(
    mapping=aes(x=origin, fill=carrier)
  ) +
  labs(title="Christmas Flights", x="Origin", y="Count")

ggsave("christmas_flights_counts.png")

Since the default is for the filter to and everything together, if you want to use something else (like OR) you need to create a single argument built with the OR operator (|). If you're using a single column for the OR, there's actually a (possibly) easier way that works like python's in where you check if the value is in a collection. Let's see the flights on Christmas and on Christmas Eve .

both_days <- filter(flights, month==12, day %in% c(24, 25))

plot = ggplot(data=both_days) +
  geom_bar(
    mapping=aes(x=day, fill=origin)
  ) +
  labs(title="Christmas Eve and Day Flights", x="Day", y="Count")

ggsave("christmas_day_and_eve.png")

What if we want New-Years Day and Christmas? In this case we need to keep the month with the day so we can't just use the in operator, but we can use the AND and OR operators to make it work.

and_new_years = filter(flights, (month==12 & day==25) | (month==1 & day==1))

plot = ggplot(data=and_new_years) +
  geom_bar(
    mapping = aes(x=day, fill=origin)
  ) +
  labs(title="Christmas and New Years Day", x="Date", y="Count")

ggsave("christmas_new_years.png")

plot = ggplot(data=and_new_years) +
  geom_boxplot(
    mapping = aes(x=origin, y=dep_delay, color=origin)
  ) +
  labs(title="Christmas and New Years Day Delays", x="Departure Airport", y="Delay (Minutes)")

ggsave("christmas_new_years_delay.png")

What are the shortest flights?

khead(arrange(flights, distance))

The data uses the three-letter airport codes, but there's another dataframe loaded with ggplot2 named airports that we can use to translate them to airport names.

okable(filter(airports, faa %in% c("PHL", "LGA", "EWR")))

So the shortest flights are from Newark Liberty International to La Guardia airport and the Philadelphia International Airport. I wonder why someone would take a plane to go 17 miles? I guess the traffic must be really bad, or some people just have money (and jet fuel) to burn.

To get the data to sort in descending order you have to pass in another function call (desc). Weird.

What are the longest flights?

khead(arrange(flights, desc(distance)))

You can probably guess the airports from the codes, but we'll look them up anyway.

okable(filter(airports, faa %in% c("JFK", "HNL")))

And the longest flights are from John F. Kennedy International airport to Honolulu International airport.

The basic way to get a subset of columns is to pass them in as arguments.

khead(select(flights, month, flight))

If you want to grab a section of contiguous columns you can pass in the first and last ones separated by a colon (the `:` character, not the vital organ).

khead(select(flights, dep_time:arr_time))

Like arrange select uses functions passed in to alter the behavior. Most of them are string-methods-ish.

You can also use logical AND and OR operators or a vector to combine the selections, but we'll just do a basic one-function call. Let's grab all the columns with an underscore in their name.

khead(select(flights, contains("_")))

You can also negate the selections using an exclamation point (!), so we can grab all the columns that don't have an underscore in their name like this.

khead(select(flights, !contains("_")))

That wasn't really helpful, so how about we rename some other columns?

flights <- rename(flights, destination=dest)

This is like the mutate function but it only keeps the new columns you created, not the original columns.

khead(transmute(subset,
       Time_Made_Up=dep_delay - arr_delay,
       Miles_Per_Hour = distance/(air_time/60)
       ))

The reason for summarize is that it works in tandem with groups. Instead of finding the median for the whole column, we can find the median by group.

by_month = group_by(flights, month)

okable(summarize(by_month, median_monthly_delay=median(dep_delay, na.rm=TRUE)))

Is that right? Let's look at a boxplot of the dataset and check.

plot = ggplot(data=by_month) +
  geom_boxplot(
    mapping=aes(x=month, y=dep_delay, fill=factor(month)), 
    outlier.shape=NA,
    na.rm=TRUE
  ) +
  scale_y_continuous(limits = quantile(by_month$dep_delay, c(0.1, 0.9),
                                       na.rm=TRUE)) +
  labs(title="Monthly Delays", x="Month", y="Delay (Minutes)")


ggsave("monthly_delay_boxplot.png")

So, that isn't so exciting, but it does show that the median values tend to be negative (meaning they left early) like our summary said.

dplyr adds an operator called the "pipe" (%>%) that works like a unix pipe does, sending the output of one function to another, allowing you to save some keystrokes adding temporary variables or nesting function calls.

So, here's what we're going to do.

• We're going to pass the flights dataset to the group_by function and group the data by destination
• Then we're going to pass that output to the summarize function and summarize by:
  • the counts
  • mean distance of the flight
  • and the mean arrival delay of each group
• Then we're going to pass the summarized data to filter to get the top 20 that didn't go to Honolulu.

First let's do a check for missing values.

got_all_your_data(flights$arr_delay)
got_all_your_data(flights$distance)

[1] FALSE
attr(,"msg")
[1] "data contains 9430 missing values"
[1] TRUE

There's no missing distances but there are missing arrival delays.

delays_vs_distance <- flights %>%
  group_by(destination) %>%
  summarize(
    count = n(),
    distance = mean(distance, na.rm = TRUE),
    delay = mean(arr_delay, na.rm = TRUE)
  ) %>%
  filter(count > 20, destination != "HNL")

khead(delays_vs_distance)

Note that we're using the mean here, not the median, so there's going to be more variation because of all the outliers. Also, because there are some missing values we had to tell the mean function to ignore them - otherwise the default is to return NA if any values are missing. There weren't any missing distance rows, but passing that argument in doesn't do any harm, I don't think. Now let's plot it.

plot = ggplot(data=delays_vs_distance, mapping=aes(x=distance, y=delay)) +
  geom_point(aes(size=count), alpha=0.4) +
  geom_smooth(se=FALSE) +
  labs(title="Distance Traveled vs Arrival Delay", x="Distance (Miles)",
       y="Delay (Minutes)")

ggsave("distance_vs_delay.png")

It looks like up to a point the further you travel the worse the delay, and then it starts going in the opposite direction, with longer flights having shorter delays in their arrival times.