\documentclass[a4paper]{article} \usepackage{mycourse} \newcommand{\Title}{Data Wrangling} <>= Module <- sub(".Rnw", "", current_input()) cat(paste0("\\newcommand{\\Module}{", Module, "}")) @ \begin{document} % A self paced guide using R. Suggested as a 2 hour hands on workshop. <>= @ \input{crossref.tex} \maketitle\thispagestyle{empty} \label{cha:data} \CiteDate{20180721} The \textit{business understanding} phase of a data science project aims to understand the business problem and to then liaise with the business data technicians to identify the data available. This is followed by the \textit{data understanding} phase where we work with the business data technicians to access and ingest the data into \R{}. We are then in a position to initiate our journey of discovery driven by the data. By \textit{living and breathing} the data in the context of the business problem we gain our bearings and feed our intuitions as we journey. In this chapter we present the common series of steps for the data phase of data science. As we progress through the chapter we build a \ConceptF{template} designed to be reused for other journeys. As we foreshadowed in Chapter~\ref{cha:intror} rather than delving into the intricacies of the \R{} language we immerse ourselves into using \R{} to achieve our outcomes, learning more about \R{} as we proceed. The template consists of programming code that can be reused with little or no modification on a new dataset. The intention is that to get started with a new dataset only a few lines at the top of the template need to be modified. No or only minimal change is then required for the remainder of the code. In many respects the concept of a template is a stepping stone toward writing functions in \R{}. <>= @ %----------------------------------------------------------------------- \section{Packages Used} %----------------------------------------------------------------------- \CiteDate{20180908} Packages used in this chapter include \Sexpr{Rpackage(dplyr,cite=TRUE)}, \Sexpr{Rpackage(FSelector,cite=TRUE)}, \Sexpr{Rpackage(ggplot2,cite=TRUE)}, \Sexpr{Rpackage(glue,cite=TRUE)}, \Sexpr{Rpackage(lubridate,cite=TRUE)}, \Sexpr{Rpackage(randomForest,cite=TRUE)}, \Sexpr{Rpackage(readr,cite=TRUE)}, \Sexpr{Rpackage(stringi,cite=TRUE)}, \Sexpr{Rpackage(stringr,cite=TRUE)}, \Sexpr{Rpackage(tidyr,cite=TRUE)}, \Sexpr{Rpackage(magrittr,cite=TRUE)}, and \Sexpr{Rpackage(rattle,cite=TRUE)}. <>= # Load required packages from local library into R session. library(rattle) # normVarNames(). library(readr) # Efficient reading of CSV data. library(dplyr) # Data wrangling, glimpse() and tbl_df(). library(tidyr) # Prepare a tidy dataset, gather(). library(magrittr) # Pipes %>% and %T>% and equals(). library(glue) # Format strings. library(lubridate) # Dates and time. library(FSelector) # Feature selection, information.gain(). library(stringi) # String concat operator %s+%. library(stringr) # String operations. library(randomForest) # Impute missing values with na.roughfix(). library(ggplot2) # Visualise data. @ <>= # These are dependencies that would otherwise be attached as required. library(scales) @ <>= @ %----------------------------------------------------------------------- \section{Data Source} %----------------------------------------------------------------------- \CiteDate{20180908} To begin the data phase of a project we will identify a data source. For our purposes we use the simplest of sources---a text-based \FileExtension{CSV} (comma separated value)\index{CSV file} file as a typical data source format. The \Rdataset{weatherAUS} dataset from \Sexpr{Rpackage(rattle)} will be used. A binary formatted \R{} dataset is provided by the package but the CSV file\index{CSV file} for the same dataset is available at \url{https://rattle.togaware.com/weatherAUS.csv}. We first identify and record the location of the CSV file to analyse. \R{} is capable of ingesting data directly from the Internet and so we will illustrate how to do so here. The location of the file (the so-called URL or universal resource location) will be saved as a string in a variable called \Rvariable{dspath}---the path to the dataset. The following assignment command does this for us. Simply type this into your \R{} script file within \RStudio{}. The command is then executed in \RStudio{} by clicking the \texttt{Run} button whilst the cursor is situated on the line within the script file. <>= # Note the source location of a dataset to ingest into R. dspath <- "http://rattle.togaware.com/weatherAUS.csv" @ The assignment operator \Sexpr{Rfunction('<-',show.paren=FALSE,show.package=FALSE)} will store the value on the right hand side (which is a string enclosed within quotation marks) into the computer's memory and we can later refer to it as the \R{} variable \Rvariable{dspath}---we retrieve the string simply by reference to the variable \Rvariable{dspath}. By typing the name of the variable (\Rvariable{dspath}) in the \R{} Console at the \texttt{>} prompt \R{} will respond with the value stored in the variable: <>= dspath @ If not connected to the Internet we can read the data directly from a local copy of the \FileExtension{CSV} file. The \Sexpr{Rpackage(rattle)} package (once the package has been installed) provides a smaller sample \FileName{weather.csv}. The location of the CSV file within \Sexpr{Rpackage(rattle)} is determined using \Sexpr{Rfunction(system.file)}. Knowing that \FileExtension{CSV} files are located within the \texttt{csv} sub-directory of the \Sexpr{Rpackage(rattle)} package we generate the string that identifies the file system path to \FileName{weather.csv}. \begin{knitrout} \definecolor{shadecolor}{rgb}{0.906, 0.906, 0.906}\color{fgcolor}\begin{kframe} \begin{alltt} \hlstd{dspath} \hlkwb{<-} \hlkwd{system.file}\hlstd{(}\hlstr{"csv"}\hlstd{,} \hlstr{"weather.csv"}\hlstd{,} \hlkwc{package}\hlstd{=}\hlstr{"rattle"}\hlstd{)} \hlopt{%T>%} \hlkwd{print}\hlstd{()} \end{alltt} \begin{verbatim} ## [1] "/usr/lib/R/site-library/rattle/csv/weather.csv" \end{verbatim} \end{kframe} \end{knitrout} \noindent This is the path to the CSV file on my file system. Your path may well be different depending on where your system installed the \Sexpr{Rpackage(rattle)} package. Note that this is a considerably smaller subset of the full weatherAUS dataset and ingesting this rather than the full dataset will lead to different results to those presented here. If you have separately downloaded \FileName{weatherAUS.csv} then you can identify its location. Here we identify that the downloaded file is located in the current working directory. <>= dspath <- "./weatherAUS.csv" @ %----------------------------------------------------------------------- \section{Data Ingestion} %----------------------------------------------------------------------- \CiteDate{20180721} Having identified the source of the dataset we can ingest the dataset into the memory of the computer using the function \Sexpr{Rfunction(read_csv)}. This function returns an enhanced \ConceptF{data frame}. A data frame is the basic data structure used to store a dataset within \R{} and the enhanced data frame from the tidyverse adds functionality that improves our interactions with the data frame. We set up a reference to the data frame's location in the computer's memory by assigning the result of the call to the function \Sexpr{Rfunction(read_csv)} to the \R{} variable \Rvariable{weather}. <>= dspath <- "./data/weatherAUS.csv" @ <>= # Ingest the dataset. weather <- read_csv(file=dspath) @ As a side effect of calling the function \Sexpr{Rfunction(read_csv)} helpful messages are displayed that identify the data types for each of the variables found in the ingested dataset. We should review these to ensure they match our expectations. If they don't, there are optional arguments to \Sexpr{Rfunction(read_csv)} to inform it otherwise. Note that the \Sexpr{Rpackage(rattle)} also provides a smaller \Sexpr{Rdataset(weather)} dataset as an \R{} dataset, also named \texttt{weather}. Simply by attaching the \Sexpr{Rpackage(rattle)} package from the library a variable called \Rvariable{weather} becomes available. Running the above command will replace the dataset provided by \Sexpr{Rpackage(rattle)}. Having done so we can still access the \texttt{weather} dataset provided by \Sexpr{Rpackage(rattle)} using the package prefix as in \Sexpr{Rdataset(weather,pkg="rattle")}. %----------------------------------------------------------------------- \section{Data Frame} %----------------------------------------------------------------------- \CiteDate{20180721} A data frame can essentially be thought of as a rectangular table of data consisting of rows (\ConceptF{observations}) and columns (\ConceptF{variables}). We can view the structure of such a table as it stores the weatherAUS dataset. Here we choose to display the first 10 observations of the first 6 variables. <>= # Display the table structure of the ingested dataset. weather[1:10,1:6] %>% print.data.frame() @ By choosing to pipe the data through \Sexpr{Rfunction(print.data.frame)} we request a raw display of the actual data frame. Next we select 10 random observations, using \Sexpr{Rfunction(sample_n)}, of 5 random variables, orchestrated using \Sexpr{Rfunction(select)} of a \Sexpr{Rfunction(sample)}, with the support of \Sexpr{Rfunction(ncol)}. <>= # Display a random selection of observations and variables. weather %>% sample_n(10) %>% select(sample(1:ncol(weather), 5)) %>% print.data.frame() @ Observe that this is a tabular form (i.e., it has rows and columns) and that we will generally be working with datasets in such a tabular form. %----------------------------------------------------------------------- \section{The Shape of the Dataset} %----------------------------------------------------------------------- \CiteDate{20180721} Once the dataset is loaded we want to get a basic idea of what it looks like---its shape. Being an extended data frame (what we call a tibble), we can display the data as a tibble simply by printing the data referred to by the variable name. <>= # Print the dataset in a human useful way. weather @ We observe that dataset consists of \Sexpr{comma(nrow(weather))} observations of \Sexpr{comma(ncol(weather))} variables. The enhanced nature of the data frame that representing it as a tibble brings to us is that the printout is more informative. The first few observations are shown with a subset of the variables followed by a list of all of the other variables. %----------------------------------------------------------------------- \section{A Glimpse of the Dataset} %----------------------------------------------------------------------- \CiteDate{20180721} A useful alternative to gain some insight into the dataset is through \Sexpr{Rfunction(glimpse)}. <>= # A quick view of the contents of the dataset. glimpse(weather) @ Again we receive a printed summary of the dataset, reporting on the number of observations and variables, but now the table is effectively rotated so that all variables can be listed along with their data type and a selection of their values for the first few observations. %----------------------------------------------------------------------- \section{Introducing Template Variables} % ----------------------------------------------------------------------- \CiteDate{20180721} A reference to the original dataset can be created using a template (or generic) variable\index{template variable}\index{generic variable}. The new variable will be called \Rvariable{ds} (short for dataset). <>= # Take a copy of the dataset into a generic variable. ds <- weather @ Both \Rvariable{ds} and \Rvariable{weather} will now reference the same dataset within the computer's memory. As we modify \Rvariable{ds} those modifications will only affect the data referenced by \Rvariable{ds}. Effectively, an extra copy of the dataset in the computer's memory will start to grow as we change the data from its original form. \R{} avoids making copies of datasets unnecessarily and so a simple assignment does not create a new copy. As modifications are made to one or the other copy of a dataset then extra memory\index{memory usage} will be used to store the columns that differ between the datasets. From here on we no longer refer to the dataset as \Rvariable{weather} but as \Rvariable{ds}. This allows the following analyses and processing to be rather generic---turning the \R{} code into a \Concept{template} and so requiring only minor modification when used with a different dataset assigned into \Rvariable{ds}. Often we will find that we can simply load a different dataset into memory, store it as \Rvariable{ds} and the remaining steps of our analyses and processing will essentially work unchanged. The first few steps of our template\index{template} are then captured as creating the reference to the dataset and presenting our initial view of the dataset. <>= # Prepare for a templated analysis and processing. dsname <- "weather" ds <- get(dsname) glimpse(ds) @ We are a little tricky here in recording the dataset name in the variable \Rvariable{dsname} and then using the function \Sexpr{Rfunction(get)} to make a copy of the dataset reference and link it to the generic variable\index{generic variable} \Rvariable{ds}. We could simply assign the data to \Rvariable{ds} directly as we saw above. Either way the generic variable \Rvariable{ds} refers to the same dataset. The use of \Sexpr{Rfunction(get)} allows us to be a little more generic in our template. The use of generic variables\index{generic variable} within a template\index{template} for the tasks we perform on each new dataset will have obvious advantages but we need to be careful. A disadvantage is that we may be working with several datasets and accidentally overwrite previously processed datasets referenced using the same generic variable\index{generic variable} (\Rvariable{ds}). The processing of the dataset might take some time and so accidentally losing it is not an attractive proposition. Care needs to be taken to avoid this. %----------------------------------------------------------------------- \section{Locating Datasets in Memory} %----------------------------------------------------------------------- \CiteDate{20180721} We can see that \Rvariable{ds} and \Rvariable{weather} reference the same dataset in memory using \Sexpr{Rfunction(location)} and \Sexpr{Rfunction(changes)}. <>= location(weather) location(ds) changes(weather, ds) @ This gets rather technical (or geeky), but the strings of digits and characters within the angle brackets are actual memory addresses---that is, they are pointers to a direct location in our computer's memory. The \texttt{0x} at the beginning of each identifies that a hexadecimal scheme is used, thus we see digits 0 to 9 and then the letters a, b, c, d, e, and f being used. That is, 16 digits. The thing to note is that the addresses recorded for \Rvariable{weather} and \Rvariable{ds}, including the addresses where we find the actual variables (columns) within each dataset, are identical. This is confirmed by the call to \Sexpr{Rfunction(changes)}. %----------------------------------------------------------------------- \section{Changing Datasets in Memory} %----------------------------------------------------------------------- \CiteDate{20180721} Let's make a change to the weather dataset by simply changing a single cell, changing the value of \Variable{MinTemp} (the third variable) for the first observation to \texttt{5}. <>= weather[1,3] <- 5 @ Notice the divergence of the two datasets. They still share a lot in common, and hence only one copy of that data, but where they diverge, they now use different memory locations. <>= location(weather) location(ds) @ Using \Sexpr{Rfunction(changes)} makes clear the changes. <>= changes(weather, ds) @ That's an interesting aside, but we now get back to our actual data analysis and processing. %----------------------------------------------------------------------- \section{Reviewing Variable Names} %----------------------------------------------------------------------- The names of the variables within the dataset as supplied to us may not be in any particular form and may use different conventions. For example, they may use a mix of upper and lower case letters (\texttt{TempToday9AM}) or be very long (\texttt{Temperature\_Recorded\_Today\_9am}) or use sequential numbers to identify each variable (\texttt{V004} or \texttt{V004\_rainToday}) or use codes (\texttt{XVn34\_rain}) or any number of other conventions. Often we prefer to simplify the variable names to ease our processing and thinking and to enforce a standard and consistent naming convention for ourselves. We use \Sexpr{Rfunction(names)} to list the names of the variables within a dataset. <>= # Review the variables to consider normalising their names. names(ds) @ Notice that the names here use a scheme whereby the initial letter is capitalised and each word within the variable name is also capitalised. That's a reasonable naming scheme and is preferred by some. %----------------------------------------------------------------------- \section{Normalizing Variable Names} %----------------------------------------------------------------------- \CiteDate{20180721} A convenient convention that I personally prefer is to map all variable names to lowercase. \R{} is case sensitive so that doing this will result in different variable names as far as \R{} is concerned. Such (so called) normalisation is useful when different upper/lower case conventions are intermixed inconsistently in names like \Variable{Incm\_tax\_PyBl}. Remembering how to capitalize when interactively exploring the data with thousands of such variables can be quite a cognitive load for us. Yet we often see such variable names arising in practise especially when we import data from databases which are often case insensitive. We can use \Sexpr{Rfunction(normVarNames)} to make a reasonable attempt of converting variables from a dataset into a preferred standard form. The actual form follows a style that is presented in Appendix~\ref{cha:style}. The example below shows the transformation into a normalised form. We make extensive use of the function \Sexpr{Rfunction(names)} to work with the variable names. <>= # Normalise the variable names. names(ds) %<>% normVarNames() %T>% print() @ Notice the use of the assignment pipe here as introduced in Chapter~\ref{cha:intror}\index{assignment pipe|see{\%<>\%}}. We will recall that the \Sexpr{apipe.fn.nsp} operator pipes the left-hand data to the function on the right-hand side and then returns the result to the left-hand side overwriting the original contents of the memory referred to on the left-hand side. %----------------------------------------------------------------------- \section{Effect on Data Storage} %----------------------------------------------------------------------- \CiteDate{20180721} When the names of the variables within a dataset are changed \R{} does not make a complete new copy of the dataset. Instead, the actual data in the column remains in tack whilst the variable itself (\Rvariable{ds}) references a new memory location where the new variable names get noted. The underlying data within the table is unaffected. <>= location(weather) location(ds) @ %----------------------------------------------------------------------- \section{Special Case Variable Name Transformations} %----------------------------------------------------------------------- \CiteDate{20180721} When reviewing the variables of a dataset we often notice other changes that could be made to the variable names. This might be to simplify the variables or to clarify the meaning of the variable. The string processing functions provided by \Sexpr{Rpackage(stringr)} come in handy for such processing. In the following example we remove the prefix of the variable names where we identify that the prefix consists of all characters up to the first underscore. This is useful where a dataset has prefixed each variable by a sequential number or by some other code and we have no real use of such a prefix in our processing. <>= names(ds) %<>% str_replace("^[^_]*_", "") @ \noindent This will take a variable name like \verb|ab123_tax_payable| and convert it to \verb|tax_payable|. <>= str_replace("ab123_tax_payable", "^[^_]*_", "") @ The odd looking characters in the argument to \Sexpr{Rfunction(str_replace)} are a \ConceptF{regular expression}. Regular expressions are a very powerful concept and can get quite complex. The reader is referred to the many resources on-line that cover regular expressions. The regular expression is a pattern used to match some part of the variable name. The pattern begins with \verb|^| which anchors the match to the beginning of the variable name. This can be followed by zero or more characters (\verb|*|) that do not match the underscore (\verb|[^_]|)---the \verb|*| specifies that the preceding pattern can be repeated zero or more times. The preceding pattern here is actually a list of characters included between square brackets. Since this list begins with \verb|^| the listed characters are excluded from the matching. That is, the pattern preceding the \verb|*| will match any character that is not an underscore. The third component of the match is then an actual underscore. Combined this regular expression matches any sequence (including an empty sequence) of characters (except for an underscore) that is at the beginning of the variable name and followed by an underscore. The next argument to \Sexpr{Rfunction(str_replace)} is the replacement string. In this case we are replacing the matched pattern with an empty string. The example here is simply one example of very many possible transformations we become used to in cleaning our datasets. The aim in transforming the variable names is to make then easier to use and to understand, both for ourselves and for others. %----------------------------------------------------------------------- \section{Data Review} %----------------------------------------------------------------------- \CiteDate{20180721} Having ingested the dataset and an initial review, normalising the variable names, we are now ready to explore more. In particular, what do the data within the dataset look like. We again gain \Sexpr{Rfunction(glimpse)} into the dataset: <>= # Review the dataset. glimpse(ds) @ Observe the variety of data types here, ranging from \Rclass{Date} (\texttt{date)}, through \Rclass{character} (\texttt{chr}) and \Rclass{numeric} (\texttt{dbl}). The data mostly looks as expected though it is odd that \Variable{evaporation} and \Variable{sunshine} are identified as character. Probably because they seem to be all missing, at least in the first 10 or so observations. We begin question other aspects of the data too. For example, is \Variable{date} an ongoing sequence of days as it appears to be here? Does \Variable{location} have values other than \texttt{Albury}? What is the distribution of the different variables? These are all questions we will start asking ourselves in the context of ``living and breathing'' our data. Our aim should be to gleam all we can about the data that we are dealing with. Data science is very much about understanding, not blindly processing. The excitement is in the discovery of patterns in the data and the narrative the data is seeking to tell. %----------------------------------------------------------------------- \section{Dataset Head and Tail} %----------------------------------------------------------------------- \CiteDate{20180721} Datasets can be very large, with many observations (millions) and many variables (thousands). We can't be expected to browse through all of the observations and variables. Instead we might review the contents of the dataset using \Sexpr{Rfunction(head)} and \Sexpr{Rfunction(tail)} to consider the top six (by default) and the bottom six observations. <>= # Review the first few observations. head(ds) %>% print.data.frame() @ <>= # Review the last few observations. tail(ds) %>% print.data.frame() @ All the time we are building a picture of the data we are looking at. It is beginning to confirm that \Variable{location} has multiple values whilst \Variable{date} does appear to be a sequence for each location. %----------------------------------------------------------------------- \section{Random Observations} %----------------------------------------------------------------------- \CiteDate{20180721} It is also useful to review some random observations from the dataset to provide a little more insight. Here we use \Sexpr{Rfunction(sample_n)} to randomly select six rows from the dataset. <>= # Review a random sample of observations. sample_n(ds, size=6) %>% print.data.frame() @ %----------------------------------------------------------------------- \section{Characters} %----------------------------------------------------------------------- \CiteDate{20180723} On ingesting the dataset into \R{} we observe the variables identified (automatically) as having \Rclass{character} \Sexpr{Rfunction(class)}. The expected values for such variables are strings of characters. We often call such variables \Concept{categoric} variables. Within \R{} these are usually represented as a data type called \Rclass{factor} and handled specially by many of the modelling algorithms. We can observe some meta data for each of the character variables. Let's first identify the character variables. <>= # Identify the character variables by index. ds %>% sapply(is.character) %>% which() %T>% print() -> chari # Identify the chracter variables by name. ds %>% names() %>% '['(chari) %T>% print() -> charc @ We will review each one of these in more detail so as to understand how we make use of them in our analyses. In particular we consider which of the variables might be handled as factors. Where a \Rclass{character} variable takes on a limited number of possible values we might convert the variable from \Rclass{character} into \Rclass{factor} (categoric\index{categoric|see{factor}}) so as to take advantage of special handling of factors in \R{}. In fact, we think of a \Rclass{factor} as a variable that can only take on a specific number of known distinct values which we call the \ConceptF{levels} of the \Rclass{factor}. %----------------------------------------------------------------------- \section{Factors} %----------------------------------------------------------------------- \CiteDate{20180908} For datasets that we load into \R{} we will not always have examples of all possible levels of a factor. Consequently it is not always possible to automatically list all of the levels automatically. Be default the tidyverse ingests these variables as \Rclass{character} so that we can take specific action to convert them to \Rclass{factor} as required. We first review the number of unique \textit{levels} for each of the factors. <>= # Observe the unique levels. ds[charc] %>% sapply(unique) @ If we decide to convert all of these variables from character into factor, then we can do so using \Sexpr{Rfunction(factor)}. <>= # Convert all chracter variables to be factors. ds[charc] %<>% map(factor) @ We don't actually do so here as we will consider each character variable in turn to decide how to handle it, especially that we might observe that evaporation and sunshine appear to be numeric. %----------------------------------------------------------------------- \section{Location} %----------------------------------------------------------------------- \CiteDate{20180723} From our review of the data so far we start to make some observations about the character variables. The first is \Variable{location}. We note that several locations were reported in the above exploration of the dataset. We can confirm the number of locations by counting the number of \Sexpr{Rfunction(unique)} values the variable has in the original dataset. <>= # How many locations are represented in the dataset. ds$location %>% unique() %>% length() @ We may not know in general what other locations we will come across in related datasets and we already have quite a collection of \Sexpr{ds$location%>%unique()%>%length()} %$ locations. We will retain this variable as a character data type. Here is a list of locations and their frequencies in the dataset. <>= ds$location %>% table() @ %----------------------------------------------------------------------- \section{Evaporation and Sunshine} %----------------------------------------------------------------------- <>= c("evaporation", "sunshine") %>% glue("\\verb|{var}|", var=.) %>% glue_collapse(", ") -> cvars @ \CiteDate{20180723} The next two character variables are: \Sexpr{cvars}. It does seem odd that these would be character, expecting bpth to be numeric values. If we look at the top of the dataset we see they have missing values. <>= # Note the character remaining variables to be dealt with. head(ds$evaporation) head(ds$sunshine) # Review other random values. sample(ds$evaporation, 8) sample(ds$sunshine, 8) @ The heuristic used to determine the data type when ingesting data only looks at a subset of all the data before it determines the data type. In this case the early observations are all missing and so default to character which is general enough to capture all potential values. We need to convert the variables to numeric. <>= # Identify the vairables to process. cvars <- c("evaporation", "sunshine") # Check the current class of the variables. ds[cvars] %>% sapply(class) # Convert to numeric. ds[cvars] %<>% sapply(as.numeric) # Review some random values. sample(ds$evaporation, 10) sample(ds$sunshine, 10) @ %----------------------------------------------------------------------- \section{Wind Directions} %----------------------------------------------------------------------- <>= ds %>% select(contains("_dir")) %>% names() %>% glue("\\verb|{var}|", var=.) %>% glue_collapse(", ") -> wgvars @ \CiteDate{20180723} The three wind direction variables (\Sexpr{wgvars}) are also identified as \Rclass{character}. We review the distribution of values here with \Sexpr{Rfunction(select)} identifying any variable that \Sexpr{Rfunction(contains)} the string \verb|_dir| and then build a \Sexpr{Rfunction(table)} over those variables. <>= # Review the distribution of observations across levels. ds %>% select(contains("_dir")) %>% sapply(table) @ Observe all 16 compass directions are represented and it would make sense to convert this into a factor. Notice that the directions are in alphabetic order and conversion to factor will retain that. Instead we can construct an ordered factor to capture the compass order (from \texttt{N}, \texttt{NNE}, to \texttt{NW} and \texttt{NNW}). We note the ordering of the directions here. <>= # Levels of wind direction are ordered compas directions. compass <- c("N", "NNE", "NE", "ENE", "E", "ESE", "SE", "SSE", "S", "SSW", "SW", "WSW", "W", "WNW", "NW", "NNW") @ %----------------------------------------------------------------------- \section{Ordered Factor} %----------------------------------------------------------------------- \CiteDate{20180723} Given our knowledge that compass directions have an obvious order, we convert the direction variables into an ordered factor. We do so using \Roption{ordered=}\texttt{TRUE} with \Sexpr{Rfunction(factor)}. <>= # Note the names of the wind direction variables. ds %>% select(contains("_dir")) %>% names() %T>% print() -> vnames # Convert these variables from character to factor. ds[vnames] %<>% lapply(factor, levels=compass, ordered=TRUE) %>% data.frame() %>% tbl_df() # Confirm they are now factors. ds[vnames] %>% sapply(class) @ We can again obtain a distribution of the variables to confirm that all we have changed is the data type. <>= # Verify the distribution has not changed. ds %>% select(contains("_dir")) %>% sapply(table) @ %----------------------------------------------------------------------- \section{Rain} %----------------------------------------------------------------------- <>= ds %>% select(starts_with("rain_")) %>% names() %>% glue("\\verb|{var}|", var=.) %>% glue_collapse(", ") -> ravars @ \CiteDate{20180723} The two remaining character variables are: \Sexpr{ravars}. Their distributions are generated by \Sexpr{Rfunction(select)}ing from the dataset those variables that start with \verb|rain_| and then build a \Sexpr{Rfunction(table)} over those variables. We use \Sexpr{Rfunction(sapply)} to apply \Sexpr{Rfunction(table)} to the selected columns to count the frequency of the occurrence of each value of a variable within the dataset. <>= # Review the distribution of observations across levels. ds %>% select(starts_with("rain_")) %>% sapply(table) @ Noting that \verb|No| and \verb|Yes| are the only values these two variables will take it makes sense to convert them both to factors. We will keep the ordering as alphabetic and so a simple call to \Sexpr{Rfunction(factor)} will to convert from character to factor. <>= # Note the names of the rain variables. ds %>% select(starts_with("rain_")) %>% names() -> vnames # Confirm these are currently character variables. ds[vnames] %>% sapply(class) # Convert these variables from character to factor. ds[vnames] %<>% lapply(factor) %>% data.frame() %>% tbl_df() # Confirm they are now factors. ds[vnames] %>% sapply(class) @ %----------------------------------------------------------------------- \section{Numeric} %----------------------------------------------------------------------- \CiteDate{20180723} Summaries of numeric data are provided using \Sexpr{Rfunction(summary)}. In the following we identify the numeric variables and summarise each. In doing so, as a data scientist, we want to again observe any oddities and to explain them. <>= ds %>% sapply(is.numeric) %>% which() %>% names %T>% print() -> numi ds[numi] %>% summary() @ Reviewing this information we can make some obvious observations. Temperatures, for example, appears to be in degrees Celsius rather than Fahrenheit. Rainfall looks like millimetres. There are some quite skewed distributions with min and median small but large max values. As data scientists we will further explore the distributions as in Chapter~\ref{cha:explore}. %----------------------------------------------------------------------- \section{Logical} %----------------------------------------------------------------------- \CiteDate{20180723} Above we converted \Variable{rain\_today} and \Variable{rain\_tomorrow} to factors. They have just two values as we confirm here, in addition to a small number of missing values (\texttt{NA}). <>= ds %>% select(rain_today, rain_tomorrow) %>% summary() @ As binary valued factors, and particularly as the values suggest, they are both candidates for being considered as logical variables (sometimes called Boolean). They can be treated as FALSE/TRUE instead of No/Yes and so supported directly by \R{} as class \Rclass{logical}. Different functions will then treat them as appropriate but not all functions do anything special. If this suits our purposes then the following can be used to perform the conversion to logical. \begin{knitrout} \definecolor{shadecolor}{rgb}{0.906, 0.906, 0.906}\color{fgcolor}\begin{kframe} \begin{alltt} \hlstd{ds} \hlopt{%<>%} \hlkwd{mutate}\hlstd{(}\hlkwc{rain_today} \hlstd{= rain_today} \hlopt{==} \hlstr{"Yes"}\hlstd{,} \hlkwc{rain_tomorrow} \hlstd{= rain_tomorrow} \hlopt{==} \hlstr{"Yes"}\hlstd{)} \end{alltt} \end{kframe} \end{knitrout} Best to now check that the distribution itself has not changed. \begin{knitrout} \definecolor{shadecolor}{rgb}{0.906, 0.906, 0.906}\color{fgcolor}\begin{kframe} \begin{alltt} \hlstd{ds} \hlopt{%>%} \hlkwd{select}\hlstd{(rain_today, rain_tomorrow)} \hlopt{%>%} \hlkwd{summary}\hlstd{()} \end{alltt} \begin{verbatim} ## rain_today rain_tomorrow ## Mode :logical Mode :logical ## FALSE:110319 FALSE:110316 ## TRUE :31880 TRUE :31877 ## NA's :3261 NA's :3267 \end{verbatim} \end{kframe} \end{knitrout} Observe that the TRUE (\texttt{Yes}) values are much less frequent than the FALSE (\texttt{No}) values, and we also note the missing values. The majority of days not having rain can be cross checked with the rainfall variable. In the previous summary of its distribution we note that rainfall has a median of zero, consistent with fewer days of actual rain. As data scientists we perform various cross checks on the hunt for oddities in the data. As data scientists we will also want to understand why there are missing values. Is it simply some rare failures to capture the observation, or for example is there a particular location not recording rainfall? We would explore that now before moving on. For our purposes going forward we will retain these two variables as factors. One reason for doing so is that we will illustrate missing value imputation using \Sexpr{Rfunction(na.roughfix)} and this functoin does not handle logical data but keeping \Variable{rain\_tomorrow} as character will allow missing value imputation. Of course we could skip this variable for the imputation. %----------------------------------------------------------------------- \section{Variable Roles} %----------------------------------------------------------------------- \CiteDate{20180723} Now that we have a basic idea of the size and shape and contents of the dataset and have performed some basic data type identification and conversion we are in a position to identify the roles played by the variables within the dataset. First we will record the list of available variables so that we might reference them below. <>= # Note the available variables. vars <- names(ds) %T>% print() @ By this stage of the project we will usually have identified a business problem that is the focus of attention. In our case we will assume it is to build a predictive analytics model to predict the chance of it raining tomorrow given the observation of today's weather. In this case the variable \Variable{rain\_tomorrow} is the \textit{target variable}\index{target variable}. Given today's observations of the weather this is what we want to predict. The dataset we have is then a \textit{training dataset}\index{training dataset} of historic observations. The task is to identify any patterns among the other observed variables that suggest that it rains the following day. <>= # Note the target variable. target <- "rain_tomorrow" # Place the target variable at the beginning of the vars. vars <- c(target, vars) %>% unique() %T>% print() @ We have taken the opportunity here to move the target variable to be the first in the vector of variables recorded in \Rvariable{vars}. This is common practice where the first variable in a dataset is the target (dependent variable) and the remainder are the variables (the independent variables) that will be used to build a model to predict that target. %----------------------------------------------------------------------- \section{Risk Variable} %----------------------------------------------------------------------- \CiteDate{20180723} With some knowledge of the data we observe \Variable{risk\_mm} captures the amount of rain recorded tomorrow. We refer to this as a \textit{risk variable}\index{risk variable}, being a measure of the impact or risk of the target we are predicting (rain tomorrow). The risk is an output variable and should not be used as an input to the modelling---it is not an independent variable. In other circumstances it might actually be treated as the target variable. <>= # Note the risk variable - measures the severity of the outcome. risk <- "risk_mm" @ For this risk variable note that we expect it to have a value of 0 for all observations when the target variable has the value \texttt{No}. <>= # Review the distribution of the risk variable for non-targets. ds %>% filter(rain_tomorrow == "No") %>% select(risk_mm) %>% summary() @ Interestingly, even a little rain (defined as 1mm or less) is regarded as no rain. That is useful to keep in mind and is a discovery of the data that we might not have expected. As data scientists we should be expecting to find the unexpected. A similar analysis for the target observations is more in line with expectations. <>= # Review the distribution of the risk variable for targets. ds %>% filter(rain_tomorrow == "Yes") %>% select(risk_mm) %>% summary() @ %----------------------------------------------------------------------- \section{ID Variables} %----------------------------------------------------------------------- \CiteDate{20180723} From our observations so far we note that the variable (\Variable{date}) acts as an identifier as does the variable (\Variable{location}). Given a \Variable{date} and a \Variable{location} we have an observation of the remaining variables. Thus we note that these two variables are so-called identifiers. Identifiers would not usually be used as independent variables for building predictive analytics models. <>= # Note any identifiers. id <- c("date", "location") @ We might get a sense of how this works with the following which will list a random sample of locations and how long the observations for that location have been collected. <>= ds[id] %>% group_by(location) %>% count() %>% rename(days=n) %>% mutate(years=round(days/365)) %>% as.data.frame() %>% sample_n(10) @ The data for each location ranges in length from 4 years up to 9 years, though most have 8 years of data. <>= ds[id] %>% group_by(location) %>% count() %>% rename(days=n) %>% mutate(years=round(days/365)) %>% ungroup() %>% select(years) %>% summary() @ %----------------------------------------------------------------------- \section{Ignore IDs and Outputs} %----------------------------------------------------------------------- \CiteDate{20180723} The identifiers and any risk variable (which is an output variable) should be ignored in any predictive modelling. Always watch out for treating output variables as inputs to modelling---this is a surprisingly common trap for beginners. We will build a vector of the names of the variables to ignore. Above we have already recorded the \Variable{id} variables and (optionally) the \Variable{risk}. Here we join them together into a new vector using \Sexpr{Rfunction(union)} which performs a set union operation---that is, it joins the two arguments together and removes any repeated variables. <>= # Initialise ignored variables: identifiers and risk. ignore <- union(id, risk) %T>% print() @ We might also check for any variable that has a unique value for every observation. These are often identifiers and if so they are candidates for ignoring. We select the \Variable{vars} from the dataset and pipe through to \Sexpr{Rfunction(sapply)} for any variables having only unique values. In our case there are no further candidate identifiers. as indicated by the empty result, \texttt{character(0)}. <>= # Heuristic for candidate indentifiers to possibly ignore. ds[vars] %>% sapply(function(x) x %>% unique() %>% length()) %>% equals(nrow(ds)) %>% which() %>% names() %T>% print() -> ids # Add them to the variables to be ignored for modelling. ignore <- union(ignore, ids) %T>% print() @ %----------------------------------------------------------------------- \section{Ignore Missing} %----------------------------------------------------------------------- \CiteDate{20180723} We next remove any variable where all of the values are missing. There are none like this in the weather dataset but in general for other datasets with thousands of variables there may be some. Here we first count the number of missing values for each variable and then list the names of those variables that have no values. <>= # Identify variables with only missing values. ds[vars] %>% sapply(function(x) x %>% is.na %>% sum) %>% equals(nrow(ds)) %>% which() %>% names() %T>% print() -> missing # Add them to the variables to be ignored for modelling. ignore <- union(ignore, missing) %T>% print() @ It is also useful to identify those variables which are very sparse---that have mostly missing values. We can decide on a threshold of the proportion missing above which to ignore the variable as not likely to add much value to our analysis. For example, we may want to ignore variables with more than 70\% of the values missing: <>= # Identify a threshold above which proportion missing is fatal. missing.threshold <- 0.7 # Identify variables that are mostly missing. ds[vars] %>% sapply(function(x) x %>% is.na() %>% sum()) %>% '>'(missing.threshold*nrow(ds)) %>% which() %>% names() %T>% print() -> mostly # Add them to the variables to be ignored for modelling. ignore <- union(ignore, mostly) %T>% print() @ %----------------------------------------------------------------------- \section{Ignore Excessive Level Variables} %----------------------------------------------------------------------- \CiteDate{20180723} Another issue we traditionally come across in our datasets are those factors with very many levels. This is more common when we read data as factors rather than as character, and so this step depends on where the data has come from. Nonetheless We might want to check for and ignore such variables. <>= # Identify a threshold above which we have too many levels. levels.threshold <- 20 # Identify variables that have too many levels. ds[vars] %>% sapply(is.factor) %>% which() %>% names() %>% sapply(function(x) ds %>% extract2(x) %>% levels() %>% length()) %>% '>='(levels.threshold) %>% which() %>% names() %T>% print() -> too.many # Add them to the variables to be ignored for modelling. ignore <- union(ignore, too.many) %T>% print() @ %----------------------------------------------------------------------- \section{Ignore Constants} %----------------------------------------------------------------------- \CiteDate{20180723} We also ignore variables with constant values as they add no extra information to the analysis. <>= # Identify variables that have a single value. ds[vars] %>% sapply(function(x) all(x == x[1L])) %>% which() %>% names() %T>% print() -> constants # Add them to the variables to be ignored for modelling. ignore <- union(ignore, constants) %T>% print() @ %----------------------------------------------------------------------- \section{Correlated Variables for Numerics} %----------------------------------------------------------------------- \CiteDate{20180726} It is often useful to identify highly correlated variables. Such variables will often record the same information but in different ways and often arise when we combine data from different sources. We identify the numeric variables of a dataset by \Sexpr{Rfunction(sapply)}ing the function \Sexpr{Rfunction(is.numeric)} to find \Sexpr{Rfunction(which)} are numeric. Their integer column positions are stored into the variable \Variable{numi}. <>= extract <- magrittr::extract @ <>= # Note which variables are numeric. vars %>% setdiff(ignore) %>% extract(ds, .) %>% sapply(is.numeric) %>% which() %>% names() %T>% print() -> numc @ %----------------------------------------------------------------------- \section{Calculating Correlations} %----------------------------------------------------------------------- \CiteDate{20180726} The correlation is calculated by \Sexpr{Rfunction(select)}ing the numeric columns from the dataset and passing that through to \Sexpr{Rfunction(cor)}. This matrix of pairwise correlations is based on only the complete observations so that observations with missing values are ignored. We set the upper triangle of the correlation matrix to \texttt{NA}'s as they are a mirror of the values in the lower triangle and thus redundant. We also set \Roption{diag=TRUE} to set the diagonals as \texttt{NA} since they will always be perfect correlations. The processing continues by making all values positive using \Sexpr{Rfunction(abs)}. With conversion to \Sexpr{Rfunction(data.frame)} then to \Sexpr{Rfunction(tbl_df)} the dataset column names need to be reset appropriately using \Sexpr{Rfunction(set_colnames)}. We \Sexpr{Rfunction(mutate)} the dataset with a new column using \Sexpr{Rfunction(mutate)}, reshape the dataset using \Sexpr{Rfunction(gather)} from \Sexpr{Rpackage(tidyr)} and then omit missing correlations using \Sexpr{Rfunction(na.omit)}. Finally the rows are \Sexpr{Rfunction(arrange)}'d with the highest absolute correlations appearing first. <>= # For the numeric variables generate a table of correlations ds[numc] %>% cor(use="complete.obs") %>% ifelse(upper.tri(., diag=TRUE), NA, .) %>% abs %>% data.frame %>% tbl_df %>% set_colnames(numc) %>% mutate(var1=numc) %>% gather(var2, cor, -var1) %>% na.omit %>% arrange(-abs(cor)) %T>% print() -> mc @ %----------------------------------------------------------------------- \section{Dealing with Correlations} %----------------------------------------------------------------------- \CiteDate{20180726} From the final result we can identify pairs of variables where we might want to keep one but not the other variable because they are highly correlated. We will select them manually since it is a judgement call. Normally we might limit the removals to those correlations that are 0.90 or more. In our case here the three pairs of highly correlated variables make intuitive sense. <>= # Note the correlated variables that are redundant. correlated <- c("temp_3pm", "pressure_3pm", "temp_9am") # Add them to the variables to be ignored for modelling. ignore <- union(ignore, correlated) %T>% print() @ %----------------------------------------------------------------------- \section{Removing Ignored Variables} %----------------------------------------------------------------------- \CiteDate{20180726} Once we have identified all of the variables to ignore we remove them from our list of variables to use. <>= # Check the number of variables currently. length(vars) # Remove the variables to ignore. vars <- setdiff(vars, ignore) # Confirm they are now ignored. length(vars) @ %----------------------------------------------------------------------- \section{Feature Selection} %----------------------------------------------------------------------- \CiteDate{20180726} The \Sexpr{Rpackage(FSelector)} package provides functions to identify subsets of variables that might be more effective for modelling. We can use this (and other packages) to assist us in reducing the variables that will be useful in our modelling. As we find useful functionality we will add them to our standard template so that for our next dataset we have the functionality readily available. We first use \Sexpr{Rfunction(cfs)} to identify a good subset of variables using correlation and entropy. We then list the variable importance using \Sexpr{Rfunction(information.gain)} to advise a useful subset of variables. Note that the \Sexpr{Rfunction('%s+%',show.paren=FALSE)} operator is a convenience to concatenate strings together to produce a formula that indicates we will model the target variable using all of the other variables of the dataset. % Do we need options(scipen=10) here - Tony does? <>= # Construct the formulation of the modelling we plan to do. form <- formula(target %s+% " ~ .") %T>% print() # Use correlation search to identify key variables. cfs(form, ds[vars]) # Use information gain to identify variable importance. information.gain(form, ds[vars]) @ The two measures are consistent in this case in that the variables identified by \Sexpr{Rfunction(cfs)} are the more important variables identified by \Sexpr{Rfunction(information.gain)}. %----------------------------------------------------------------------- \section{Missing Targets} %----------------------------------------------------------------------- \CiteDate{20180726} Sometimes there may be further operations to perform on the dataset prior to modelling. A common task is to deal with missing values. Here we remove observations with a missing target. As with any missing data we should also analyse whether there is any pattern to the missing targets. This may be indicative of a systemic data issue rather than simply randomly missing values. <>= # Check the dimensions to start with. dim(ds) # Identify observations with a missing target. missing.target <- ds %>% extract2(target) %>% is.na() # Check how many are found. sum(missing.target) # Remove observations with a missing target. ds %<>% filter(!missing.target) # Confirm the filter delivered the expected dataset. dim(ds) @ %----------------------------------------------------------------------- \section{Missing Values} %----------------------------------------------------------------------- \CiteDate{20180726} Missing values for the variables are an issue for some but not all algorithms. For example \Sexpr{Rfunction(randomForest)} omits observations with missing values by default whilst \Sexpr{Rfunction(rpart)} has a particularly well developed approach to dealing with missing values. We may want to impute missing values in the data (though it is not always wise to do so). Here we do this using \Sexpr{Rfunction(na.roughfix)} from \Sexpr{Rpackage(randomForest)}. This function provides, as the name implies, a rather basic algorithm for imputing missing values. Because of this we will demonstrate the process but then restore the original dataset---we will not want this imputation to be included in our actual dataset. <>= # Backup the dataset so we can restore it as required. ods <- ds @ <>= # Count the number of missing values. ds[vars] %>% is.na() %>% sum() # Impute missing values. ds[vars] %<>% na.roughfix() # Confirm that no missing values remain. ds[vars] %>% is.na() %>% sum() @ As foreshadowed we now restore the dataset with its original contents. <>= # Restore the original dataset. ds <- ods @ %----------------------------------------------------------------------- \section{Omitting Observations} %----------------------------------------------------------------------- \CiteDate{20180726} An alternative is to remove observations that have missing values. Here \Sexpr{Rfunction(na.omit)} identifies the rows to omit based on the \Variable{vars} to be included for modelling. The list of rows to omit is stored as the \Rvariable{na.action} attribute of the returned object. We then remove these observations from the dataset. Notice we keep a copy of the original dataset and then restore it. <>= # Backup the dataset so we can restore it as required. ods <- ds # Initialise the list of observations to be removed. omit <- NULL @ <>= # Review the current dataset. ds[vars] %>% nrow() ds[vars] %>% is.na() %>% sum() # Identify any observations with missing values. mo <- attr(na.omit(ds[vars]), "na.action") # Record the observations to omit. omit <- union(omit, mo) # If there are observations to omit then remove them. if (length(omit)) ds <- ds[-omit,] # Confirm the observations have been removed. ds[vars] %>% nrow() ds[vars] %>% is.na() %>% sum() @ <>= # Restore the original dataset. ds <- ods @ %----------------------------------------------------------------------- \section{Normalise Factors} %----------------------------------------------------------------------- \CiteDate{20180726} Some variables will have levels with spaces, and mixture of cases, etc. We may like to normalise the levels for each of the categoric variables. For very large datasets this can take some time and so we may want to be selective. <>= # Note which variables are categoric. ds %>% sapply(is.factor) %>% which() -> catc # Normalise the levels of all categoric variables. for (v in catc) levels(ds[[v]]) %<>% normVarNames() @ %----------------------------------------------------------------------- \section{Target as a Factor} %----------------------------------------------------------------------- \CiteDate{20180726} We often build classification models. For such models we want to ensure the target is categoric. Often it is 0/1 and hence is loaded as numeric. We could tell our model algorithm of choice to explicitly do classification or else set the target using \Sexpr{Rfunction(as.factor)} in the formula. Nonetheless it is generally cleaner to do this here and note that this code has no effect if the target is already categoric. <>= # Ensure the target is categoric. ds[[target]] %<>% as.factor() # Confirm the distribution. ds[target] %>% table() @ We can visualise the distribution of the target variable using \Sexpr{Rpackage(ggplot2)}. The dataset is piped to \Sexpr{Rfunction(ggplot)} whereby the target is associated through \Sexpr{Rfunction(aes_string)} (the aesthetics) with the x-axis of the plot. To this we add a graphics layer using \Sexpr{Rfunction(geom_bar)} to produce the bar chart, with bars having \Roption{width=} \texttt{0.2} and a \Roption{fill=} color of \texttt{"grey"}. The resulting plot can be seen in Figure~\ref{fig:data:plot_target_distribution}. <>= ds %>% ggplot(aes_string(x=target)) + geom_bar(width=0.2, fill="grey") + theme(text=element_text(size=14)) @ %----------------------------------------------------------------------- \section{Identify Variable Types} %----------------------------------------------------------------------- \CiteDate{20180726} Metadata is data about the data. We now record data about our dataset that we use later in further processing and analysing our data. In one sense the metadata is simply a convenient store. We identify the variables that will be used to build analytic models that provide different kinds of insight into our data. Above we identified the variable roles such as the target, a risk variable and the ignored variables. From an analytic modelling perspective we identify variables that are the model inputs. We record then both as a vector of characters (the variable names) and a vector of integers (the variable indicies). <>= inputs <- setdiff(vars, target) %T>% print() @ The integer indices are determined from the \Sexpr{Rfunction(names)} of the variables in the original dataset. Note the use of \Roption{USE.NAMES=} from \Sexpr{Rfunction(sapply)} to turn off the inclusion of names in the resulting vector to keep the result as a simple vector. <>= inputi <- sapply(inputs, function(x) which(x == names(ds)), USE.NAMES=FALSE) inputi @ For convenience we record the number of observations: <>= nobs <- nrow(ds) %T>% print() @ Here we simply report on the dimensions of various data subsets primarily to confirm the dataset appear as we expect: <>= dim(ds) dim(ds[vars]) dim(ds[inputs]) @ %----------------------------------------------------------------------- \section{Identify Numeric and Categoric Variables} %----------------------------------------------------------------------- \CiteDate{20180726} Identifying numeric and categoric variables may be useful for example for cluster analysis algorithms that only deal with numeric variables. Here we identify them by name (a character string) and by index. When using the index we have to assume the variables remain in the same order within the dataset and all variables are present, otherwise the indicies will get out of sync. <>= # Identify the numeric variables by index. ds %>% sapply(is.numeric) %>% which() %>% intersect(inputi) %T>% print() -> numi # Identify the numeric variables by name. ds %>% names() %>% extract(numi) %T>% print() -> numc # Identify the categoric variables by index and then name. ds %>% sapply(is.factor) %>% which() %>% intersect(inputi) %T>% print() -> cati ds %>% names() %>% extract(cati) %T>% print() -> catc @ %----------------------------------------------------------------------- \section{Save the Dataset} %----------------------------------------------------------------------- For large datasets we may want to save it to a binary RData file once we have wrangled it into the right shape and collected the metadata. Loading a binary dataset is generally quicker than loading a CSV file---a CSV file with 2 million observations and 800 variables can take 30~minutes to \Sexpr{Rfunction(read.csv)}, 5~minutes to \Sexpr{Rfunction(save)}, and 30~seconds to \Sexpr{Rfunction(load)}. <>= osave <- save save <- function(...) {invisible()} @ <>= # Timestamp for the dataset. dsdate <- "_" %s+% format(Sys.Date(), "%y%m%d") %T>% print() # Filename for the saved dataset dsrdata <- dsname %s+% dsdate %s+% ".RData" %T>% print() # Save relevant R objects to binary RData file. save(ds, dsname, dspath, dsdate, nobs, vars, target, risk, id, ignore, omit, inputi, inputs, numi, numc, cati, catc, file=dsrdata) @ <>= save <- osave @ <>= # Do this so we know what to load into ModelsO.Rnw dsdate <- "_" %s+% "160720" dsrdata <- dsname %s+% dsdate %s+% ".RData" save(ds, dsname, dspath, dsdate, nobs, vars, target, risk, id, ignore, omit, inputi, inputs, numi, numc, cati, catc, file=dsrdata) @ Notice that in addition to the dataset (\Variable{ds}) we also store the collection of \ConceptF{metadata}. This begins with items such as the name of the dataset, the source file path, the date we obtained the dataset, the number of observations, the variables of interest, the target variable, the name of the risk variable (if any), the identifiers, the variables to ignore and observations to omit. We continue with the indicies of the input variables and their names, the indicies of the numeric variables and their names, and the indicies of the categoric variables and their names. Each time we wish to use the dataset we can now simply \Sexpr{Rfunction(load)} it into \R{}. The value that is invisibly returned by \Sexpr{Rfunction(load)} is a vector naming the R objects loaded from the binary RData file. <>= load(dsrdata) %>% print() @ We place the call to \Sexpr{Rfunction(load)} within a call to \Sexpr{Rfunction('(',show.paren=FALSE,show.package=FALSE)} (i.e., we have surrounded the call with round brackets) to ensure the result of the function call is printed. A call to \Sexpr{Rfunction(load)} returns its result invisibly since we are primarily interested in its side-effect. The side-effect is to read to \R{} binary data from disk and to make it available within our current \R{} session. %----------------------------------------------------------------------- \section{A Template for Data Preparation} %----------------------------------------------------------------------- Through this chapter we have built a template for data preparation. An actual knitr template based on this chapter for data preparation is available as \url{http://HandsOnDataScience.com/scripts/data.Rnw}. An automatically derived version including just the \R{} code is also available as \url{http://HandsOnDataScience.com/scripts/data.R}. Notice that we would not necessarily perform all of the steps, such as normalising the variable names, imputing missing values, omitting observations with missing values, and so on. Instead we pick and choose as is appropriate to our situation and specific datasets. Also, some data specific transformations are not included in the template and there may be other transforms we need to perform that we have not covered here. As we discover new tools to support the data scientist we can add them into our own templates. %----------------------------------------------------------------------- \section{Command Summary} %----------------------------------------------------------------------- This chapter has introduced, demonstrated and described the following \R{} packages, functions, commands, operators, and datasets: \\[2mm] \begin{outdent} \textbf{\Sexpr{Rfunction(get, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(get, PkgOp=TRUE)}.} Returns the named dataset. \textbf{\Sexpr{Rfunction(glimpse, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(glimpse, PkgOp=TRUE)}.} Summarise a dataset. \textbf{\Sexpr{Rfunction(head, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(head, PkgOp=TRUE)}.} Display the first few rows of a dataset. \textbf{\Sexpr{Rfunction(names, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(names, PkgOp=TRUE)}.} Column names of a dataset. \textbf{\Sexpr{Rfunction(normVarNames, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(normVarNames, PkgOp=TRUE)}.} Normalize variable names. \textbf{\Sexpr{Rfunction(ncol, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(ncol, PkgOp=TRUE)}.} Number of columns in a dataset. \textbf{\Sexpr{Rfunction(nrow, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(nrow, PkgOp=TRUE)}.} Number of rows in a dataset. \textbf{\Sexpr{Rfunction(read_csv, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(read_csv, PkgOp=TRUE)}.} Load data from a CSV file. \textbf{\Sexpr{Rfunction(sample_n, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(sample_n, PkgOp=TRUE)}.} Random sample of n rows. \textbf{\Sexpr{Rfunction(sapply, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(sapply, PkgOp=TRUE)}.} Apply a function to columns of a dataset. \textbf{\Sexpr{Rfunction(str_replace, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(str_replace, PkgOp=TRUE)}.} Replace a string with another. \textbf{\Sexpr{Rfunction(system.file, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(system.file, PkgOp=TRUE)}.} Locate a system file. \textbf{\Sexpr{Rfunction(tail, DefnOp=TRUE)}} \textit{Function from \Sexpr{Rfunction(tail, PkgOp=TRUE)}.} Display the last few rows of a dataset. \end{outdent} %----------------------------------------------------------------------- \section{Exercises} %----------------------------------------------------------------------- \begin{Exercise}[title={Exploring the Weather}] We have worked with the Australian \Dataset{weatherAUS} dataset throughout this chapter. For this exercise we will explore the dataset further. For each exercise, extend the knitr document with the analyses performed. \begin{enumerate} \item Create a data preparation script beginning with the template available from \url{http://HandsOnDataScience.com/scripts/data.Rnw} and replicating the data processing performed in this chapter. \item Investigate the \Sexpr{Rfunction(group_by)} and \Sexpr{Rfunction(summarise)} functions, combined through a pipeline using \Sexpr{pipe.fn.nsp} to identify regions with considerable variance in their weather observations. The use of \Sexpr{Rfunction(var)} might be a good starting point. \end{enumerate} \end{Exercise} %% \begin{Exercise}[title={Using the Data Preparation Template}] %% We have worked with the Australian \Dataset{weatherAUS} dataset %% throughout this chapter. For this exercise we will replace %% \texttt{weatherAUS.csv} with \url{http://xyz.csv} and work through %% the data template available as %% \url{http://HandsOnDataScience.com/scripts/data.Rnw}. %% \begin{enumerate} %% \item Create a data preparation script beginning with the template %% available from %% \url{http://HandsOnDataScience.com/scripts/data.Rnw} and tune it %% for the \Dataset{xyz} dataset. %% \end{enumerate} %% \end{Exercise} %% > ds <- read_csv("http://rattle.togaware.com/weatherAUS.csv") %% Warning message: %% 127435 problems parsing 'http://rattle.togaware.com/weatherAUS.csv'. See problems(...) for more details. %% > problems(ds) %% Source: local data frame [127,435 x 4] %% row col expected actual %% 1 4414 6 T/F/TRUE/FALSE 12 %% 2 4414 7 T/F/TRUE/FALSE 12.3 %% 3 4415 6 T/F/TRUE/FALSE 14.8 %% 4 4415 7 T/F/TRUE/FALSE 13 %% 5 4416 6 T/F/TRUE/FALSE 12.6 %% 6 4416 7 T/F/TRUE/FALSE 13.3 %% 7 4417 6 T/F/TRUE/FALSE 10.8 %% 8 4417 7 T/F/TRUE/FALSE 10.6 %% 9 4418 6 T/F/TRUE/FALSE 11.4 %% 10 4418 7 T/F/TRUE/FALSE 12.2 %% .. ... ... ... ... %% > ds %% Source: local data frame [105,379 x 24] %% Date Location MinTemp MaxTemp Rainfall Evaporation Sunshine %% 1 2008-12-01 Albury 13.4 22.9 0.6 NA NA %% 2 2008-12-02 Albury 7.4 25.1 0.0 NA NA %% 3 2008-12-03 Albury 12.9 25.7 0.0 NA NA %% 4 2008-12-04 Albury 9.2 28.0 0.0 NA NA %% 5 2008-12-05 Albury 17.5 32.3 1.0 NA NA %% 6 2008-12-06 Albury 14.6 29.7 0.2 NA NA %% 7 2008-12-07 Albury 14.3 25.0 0.0 NA NA %% 8 2008-12-08 Albury 7.7 26.7 0.0 NA NA %% 9 2008-12-09 Albury 9.7 31.9 0.0 NA NA %% 10 2008-12-10 Albury 13.1 30.1 1.4 NA NA %% .. ... ... ... ... ... ... ... %% Variables not shown: WindGustDir (chr), WindGustSpeed (int), WindDir9am (chr), %% WindDir3pm (chr), WindSpeed9am (int), WindSpeed3pm (int), Humidity9am (int), %% Humidity3pm (int), Pressure9am (dbl), Pressure3pm (dbl), Cloud9am (int), %% Cloud3pm (int), Temp9am (dbl), Temp3pm (dbl), RainToday (chr), RISK_MM (dbl), %% RainTomorrow (chr) \begin{Exercise}[title={Understanding Ferries}] % See c02_exercise.R % See http://seananderson.ca/2014/09/13/dplyr-intro.html % http://esapubs.org/archive/ecol/E090/184/ %% \noindent A dataset of mammals is available as %% \url{http://HandsOnDataScience.com/data/mammals.txt}\footnote{The %% original source of the mammals dataset is %% \url{http://esapubs.org/archive/ecol/E090/184/PanTHERIA_1-0_WR05_Aug2008.txt}. %% The dataset and its associated paper %% \citep{jones.etal:2009:ecology} are available under a Creative %% Commons Attribution (CC BY 4.0) license.} We will use this %% dataset to exercise our data template. % ds2 <- read_tsv("http://HandsOnDataScience.com/data/mammals.txt") % ds3 <- read_csv("http://data.gov.au/dataset/7be6e3ee-043a-4c47-a6eb-a97702419ccd/resource/c6b211ad-3aa2-4f53-8427-01b52a6433a7/download/mobileblackspotdatabase.csv") \noindent A dataset of ferry crossings on Sydney Harbour is available as \url{http://HandsOnDataScience.com/data/ferry.csv}. The original source of the Ferry dataset is \url{http://www.bts.nsw.gov.au/Statistics/Ferry/default.aspx?FolderID=224}. The dataset is available under a Creative Commons Attribution (CC BY 3.0 AU) license. We will use this dataset to exercise our data template. <>= ds <- read_csv("http://HandsOnDataScience.com/data/ferry.csv") # Some possible questions from Tony # is there a difference between years, and seasons, or when they travel , # even questions like, do they travel later in winter than in summer, etc # plus we can factor in weather, and location, etc so do more people travel one way, than return, etc # do people in the eastern suburbs travel more than people in the western suburbs, # also some ideas about the boats , cause they are different types of ferrysw # on the greenwhich run, they change ferries after the first run, etc @ \begin{enumerate} \item Create a data preparation script beginning with the template available from \url{http://HandsOnDataScience/scripts/data.R}. \item Change the sample source dataset within the template to download the ferry dataset into \R{}. \item Rename the variables to become normalized variable names. \item Create two new variables from \Variable{sub\_route}, called \Variable{origin} and \Variable{destination}. \item Convert dates. \item Convert appropriate variables into factors. \item Work through the template to explore and prepare the dataset. \item Develop some visualisations. \end{enumerate} \end{Exercise} %----------------------------------------------------------------------- \section{Further Reading} %----------------------------------------------------------------------- \rattlereading \begin{TODO} We list below further resources that augment the material we have presented in this chapter. \begin{itemize} \item \todo[inline]{Under development: list any other resources of use that extend beyond what we have presented in this chapter.} \end{itemize} \end{TODO} %----------------------------------------------------------------------- <>= @ %----------------------------------------------------------------------- \end{document}