\documentclass[a4paper]{article} \usepackage{mycourse} \newcommand{\Title}{Programming with Data: Introducing \R{}} <>= Module <- sub(".Rnw", "", current_input()) cat(paste0("\\newcommand{\\Module}{", Module, "}")) @ \begin{document} % Getting familiar with R, RStudio and Data Science % % Preliminary Program: % % * 1 hour to download and install R and RStudio % * 2 hours working through the material in this chapter % * 2 hours reading through the R introduction % * 2 hours doing the exercises. <>= # These are dependencies that would otherwise be attached as required. library(dplyr) library(utils) library(stringr) library(tidyr) library(ggplot2) library(rattle) @ <>= @ \input{crossref.tex} \maketitle\thispagestyle{empty} \label{cha:intror} Data scientists write programs to ingest, fuse, clean, wrangle, visualise, analyse, and model data. Programming over data is a core task for the data scientist. We will primarily use \R{} as our programming language \citep{R-base} and assume basic familiarity of \R{} as may be gained from the many resources available on the Intranet, particularly from \url{https://cran.r-project.org/manuals.html}. Programmers of data develop sentences or \Concept{code}. Code instructs a computer to perform specific tasks. A collection of sentences written in a language is what we might call a \Concept{program}. Through \Concept{programming by example} and \Concept{learning by immersion} we will share programs to deliver insights and outcomes from our data. \R{} is a large and complex ecosystem for the practice of data science. There is much freely available information on the Internet from which we can continually learn and borrow useful code segments that illustrate almost any task we might think of. We introduce here the basics for getting started with \R{}, libraries and packages which extend the language, and the concepts of functions, commands, and operators. We also introduce the powerful concept of pipes as a foundation for building sophisticated data processing pipelines in \R{}. <>= @ %----------------------------------------------------------------------- \section{Tooling For \R{} Programming} %----------------------------------------------------------------------- We will first need to have available to us the computer software that interprets our programs that we write as sentences in the \R{} language. Usually we will install this software, called the \R{} Interpreter, on our own computer and instructions for doing so are readily available on the Internet. The \fref{https://www.r-project.org}{R Project} is a good place to start. Most GNU/Linux distributions freely provide \R{} packages from their application stores and a commercial offering is available from Microsoft who also provide direct access within SQL Server and through Cortana Analytics on the Azure cloud. Now is a good time to install \R{} on your own computer if that is required. The open source and freely available \fref{https://www.rstudio.com}{\RStudio}\index{RStudio} software is recommended as a modern % \Sexpr{Wikipedia("integrated development environment")} (IDE)\index{IDE} for writing \R{} programs. It can be installed on the common desktop or server computers running GNU/Linux, Microsoft/Windows, or Apple/MacOS. A browser version of \RStudio{} also allows a desktop to access \R{} running on a back-end cloud server. The server version of \RStudio{} runs on a remote computer (e.g., a server in the cloud) with the graphical interface presented within a web browser on our own desktop. The interface is much the same as the desktop version but all of the commands are run on the server rather than on our own desktop. Such a setup is useful when we require a powerful server on which to analyse very large datasets. We can then control the analyses from our own desktop with the results sent from the server back to our desktop whilst all the computation is performed on the server itself which may be a considerably more powerful computer than our usually less capable personal computers. %----------------------------------------------------------------------- \section{Introducing RStudio} %----------------------------------------------------------------------- The \RStudio{} interface includes an \R{} \texttt{Console} on the left through which we directly communicate with the \R{} software. The window pane on the top right provides access to the \texttt{Environment} within which \R{} is operating. This includes data and datasets that are defined as we interact with \R{}. Initially it is empty. A \texttt{History} of all the commands we have asked \R{} to run is available on another tab within the top right pane. The bottom right pane provides direct access to an extensive collection of \texttt{Help}. On another tab within this same pane we can access the \texttt{Files} on our hard disk. Other tabs provide access to \texttt{Plots}, \texttt{Packages}, and a \texttt{Viewer} to view documents that we might generate. In the \texttt{Help} tab displayed in the bottom right pane of Figure~\ref{intro:fig:rstudio_startup} we can see numerous links to \R{} documentation. The \textsf{Manual} titled \href{https://cran.r-project.org/doc/manuals/R-intro.pdf}{\textit{An Introduction to \R{}}} within the \texttt{Help} tab is a good place to start if you are unfamiliar with \R{}. There are also links to learning more about \RStudio{} and these are recommended if you are new to \RStudio{}. % The screenshots from RStudio use a window with the following dimensions. % % $ wmctrl -lG | grep RStudio % 0x02800006 0 1682 198 1022 647 N/A RStudio % % We can place the window in a fixed location and fixed size for % repeatable screen capture using: % % $ wmctrl -r RStudio -e 0,1682,198,1022,647 % % Another useful command is: % % $ xwininfo % % xwininfo: Please select the window about which you % would like information by clicking the % mouse in that window. % % xwininfo: Window id: 0x2800006 "RStudio" % % Absolute upper-left X: 1681 % Absolute upper-left Y: 170 % Relative upper-left X: 1 % Relative upper-left Y: 28 % Width: 1022 % Height: 647 % Depth: 24 % Visual: 0x154 % Visual Class: TrueColor % Border width: 0 % Class: InputOutput % Colormap: 0x2800001 (not installed) % Bit Gravity State: ForgetGravity % Window Gravity State: NorthWestGravity % Backing Store State: NotUseful % Save Under State: no % Map State: IsViewable % Override Redirect State: no % Corners: +1681+170 -1+170 -1-463 +1681-463 % -geometry 1022x647-0+142 \begin{figure}[h] \centering \includegraphics[width=0.97\textwidth]{rstudio_startup} \caption[\RStudio{}: Initial layout.]{The initial layout of the \RStudio{} window panes showing the \texttt{Console} pane on the left with the \texttt{Environment} and \texttt{Help} panes on the right.} \label{intro:fig:rstudio_startup} \end{figure} \section{Editing Code} Often we will be interacting with \R{} by writing code and sending that code to the \R{} Interpreter so that it can be run (locally or remotely). It is always good practice to store this code into a file so that we have a record of what we have done and are able to replicate the work at a later time. Such a file is called an \R{} Script. We can create a new \R{} Script by clicking on the first icon on the \RStudio{} toolbar and choosing the first item in the resulting menu as in Figure~\ref{intro:fig:rstudio_startup_editor}. A keyboard shortcut is also available to do this: \texttt{Ctrl+Shift+N} (hold the \texttt{Ctrl} and the \texttt{Shift} keys down and press the \texttt{N} key). A sophisticated file editor is presented as the top left pane within \RStudio. The tab will initially be named \texttt{Untitled1} until we actually save the script to a file on the disk. When we do so we will be asked to provide a name for the file. The editor is where we write our \R{} code and compose the programs that instruct the computer to perform particular tasks. The editor provides numerous features that are expected in a modern program editor. These include syntax colouring, automatic indentation to improve layout, automatic command completion, interactive command documentation, and the ability to send specific commands to the \R{} Console to have them run by the \R{} Interpreter. <>= system(paste("cd graphics;" , "convert rstudio_startup_editor.png" , "-fill none -stroke red -strokewidth 2" , '-draw "roundrectangle 4,55 38,76 5,5"' , "rstudio_startup_editor_annotate.png")) # system("display graphics/rstudio_startup_editor_annotate.png") @ \begin{figure}[h] \centering \includegraphics[width=0.97\textwidth]{rstudio_startup_editor_annotate} \caption[\RStudio{}: Ready to program in \R{}.]{Ready to edit \R{} scripts in \RStudio{} using the \texttt{Editor} pane on the top left created when we choose to add a new \R{} Script from the toolbar menu displayed by clicking the icon highlighted in red.} \label{intro:fig:rstudio_startup_editor} \end{figure} \section{Executing Code} To send \R{} commands to the \R{} Console we can use the appropriate toolbar buttons. One line or a highlighted region can be sent using the \texttt{Run} button. Having opened a new \R{} script file we can enter the commands below. The four commands make up a program which begins with an \Sexpr{Rfunction(install.packages,Defn=TRUE,add.seen=FALSE)} to ensure we have installed two requisite software packages for \R{}. The second and third commands use \Sexpr{Rfunction(library,Defn=TRUE,add.seen=FALSE)} to load those packages into the current \R{} session. The fourth command produces a plot of the \Sexpr{Rdataset(weatherAUS,Defn=TRUE)} dataset. This dataset contains observations of weather related variables across almost 50 weather stations in Australia over multiple years. <>= install.packages(c("ggplot2", "rattle")) library(ggplot2) library(rattle) qplot(data=weatherAUS, x=MinTemp, y=MaxTemp) @ \noindent \Sexpr{Rfunction(qplot,Defn=TRUE)} allows us to quickly code a plot. Using the \Sexpr{Rdataset(weatherAUS,Defn=TRUE)} dataset we plot the minimum daily temperature (\Variable{MinTemp}) on the x-axis against the maximum daily temperature (\Variable{MaxTemp}) on the y-axis. The resulting plot is a \Sexpr{Wikipedia("scatter plot")} as we see the plot in the lower right pane. <>= system(paste("cd graphics;" , "convert rstudio_weatherAUS_scatterplot.png" , "-fill none -stroke red -strokewidth 2" , '-draw "roundrectangle 370,112 425,132 5,5"' #, '-draw "roundrectangle 178,112 230,132 5,5"' #, '-draw "roundrectangle 528,112 580,132 5,5"', , "rstudio_weatherAUS_scatterplot_annotate.png")) # system("display graphics/rstudio_weatherAUS_scatterplot_annotate.png", wait=FALSE) @ \begin{figure}[h] \centering \includegraphics[width=\textwidth]{rstudio_weatherAUS_scatterplot_annotate} \caption[\RStudio{}: Running the \R{} program.]{Running \R{} commands in \RStudio{}. The \R{} programming code is written into a file using the editor in the top left pane. With the cursor on the line containing the code we click the \texttt{Run} button to pass the code on to the \R{} \texttt{Console} to have it run by the \R{} Interpreter to produce the plot we see in the bottom right pane.} \label{fig:intror:rstudio_weather_scatterplot} \end{figure} \section{RStudio Review} Figure~\ref{fig:intror:rstudio_weather_scatterplot} shows the \RStudio{} editor as it appears after we have typed the above commands into the \R{} \texttt{Script} file in the top left pane. We have sent the commands to the \R{} \texttt{Console} to have it run by \R{}. We have done this by ensuring the cursor within the \R{} \texttt{Script} editor is on the same line as the command to be run and then clicking the \texttt{Run} button. We will notice that the command is sent to the \R{} \texttt{Console} in the bottom left pane and the cursor advances to the next line within the \R{} \texttt{Script} editor. After each command is run any text output by the command is displayed in the \R{} \texttt{Console} whilst graphic output is displayed in the \texttt{Plots} tab of the bottom right pane. This is now our first program in \R{}. We can now provide our first observations of the data. It is not too hard to see from the plot that there appears to be quite a strong relationship between the minimum temperature and the maximum temperature: with higher values of the minimum temperature recorded on any particular day we see higher values of the maximum temperature. There is also a clear lower boundary that might suggest, as logic would dictate, that the maximum temperature can not be less than the minimum temperature. If we were to observe data points below this line then we would begin to explore issues with the quality of the data. As data scientists we have begun our observation and understanding of the data, taking our first steps toward immersing ourselves in and thereby beginning to understand the data. %----------------------------------------------------------------------- \section{Packages and Libraries} %----------------------------------------------------------------------- <>= options(repos="https://cran.rstudio.com") pkgs <- packageStatus() npkgs <- format((nrow(pkgs$avail) %/% 1000) * 1000, big.mark=",") @ %$ The power of the \R{} ecosystem comes from the ability to extend the language by its nature as \Sexpr{Wikipedia("open source software")}. Anyone is able to contribute to the \R{} ecosystem and by following stringent guidelines they can have their contributions included in the comprehensive \R{} archive network, abbreviated as CRAN\footnote{CRAN: \url{https://cran.r-project.org}.}. Such contributions are collected by an author into what is called a \Concept{package}. Over the decades many researchers and developers have contributed very many \Concept{package}s to CRAN. A \ConceptF{package} is how \R{} collects together \Concept{command}s for a specific collection of tasks. A \ConceptF{command} is a verb in the computer language used to tell the computer to do something. There are over \Sexpr{npkgs} packages available for \R{} from almost as many different authors. Hence there are very many verbs available to build our sentences to command \R{} appropriately. With so many packages there is bound to be a package or two covering essentially any kind of processing we could imagine though we will also find packages offering the same or similar commands (verbs) perhaps even with very different meanings. Beginning with Chapter~\ref{cha:knitr} we will list at the beginning of each chapter the \R{} \Concept{package}s that are required for us to be able to replicate the examples presented in that chapter. Packages are installed from the Internet (from the securely managed \fref{https://cran.r-project.org}{CRAN} package repository) into a local \Concept{library} on our own computer. A \ConceptF{library} is a folder on our computer's storage which contains sub-folders corresponding to each of the installed packages. To install a package from the Internet we can use the command \Sexpr{Rfunction(install.packages,show.package=FALSE)} and provide to it as an \Concept{argument} the name of the package to install. The package name is provided as a \ConceptF{string} of \ConceptF{characters} within quotes and supplied as the \Roption{pkgs=} \ConceptF{argument} to the command as in the following code. Here we choose to install a package called \texttt{dplyr}---a very useful package for data manipulations. <>= # Install a package from a CRAN repository. install.packages(pkgs="dplyr") @ Once a package is installed we can access the commands provided by that package by prefixing the command name with the package name as in \Sexpr{Rfunction(qplot,pkg="ggplot2")}. This is to say that \Sexpr{Rfunction(qplot,pkg="ggplot2",show.package=FALSE)} is provided by the \Sexpr{Rpackage(ggplot2)} package. \section{A Glimpse of the Dataset} Another example of a useful command that we will find ourselves using often is the \Sexpr{Rfunction(glimpse,pkg="dplyr",show.package=FALSE)} command from the \Sexpr{Rpackage(dplyr)} package. This command can be accessed in the \R{} console as \Sexpr{Rfunction(glimpse,pkg="dplyr")} once the \Sexpr{Rpackage(dplyr)} package has been installed. This particular command accepts an \Concept{argument} \Roption{x=} which names the dataset we wish to glimpse. In the following \R{} example we use the \Sexpr{Rdataset(weatherAUS,show.package=FALSE)} dataset from the \Sexpr{Rpackage(rattle)} package. <>= # Review the dataset. dplyr::glimpse(x=rattle::weatherAUS) @ We can see here the command that was run and the output from running that command. As a convention used in this book the output from running \R{} commands is prefixed with ``\verb|## |''. The ``\verb|#|'' introduces a comment in an \R{} script file and tells \R{} to ignore everything that follows on that line. We use the ``\verb|## |'' convention throughout the book to clearly identify output produced by \R{}. When we run these commands ourselves in \R{} this prefix is not displayed. Long lines of output are also truncated for our presentation here. The \verb|...| at the end of the lines and the \verb|....| at the end of the output indicate that the output has been truncated for the sake of keeping our example output to a minimum. \section{Attaching a Package} We can \ConceptF{attach} a package to our \textit{current} \R{} session from our local \Concept{library} for added convenience. This will make the command available during this specific \R{} session without the requirement to specify the package name each time we use the command. Attaching a package tells the \R{} software to look within that package (and then to look within any other attached packages) when it needs to find out what the command should do (the \ConceptF{definition} of the command). We can see which packages are currently attached using \Sexpr{Rfunction(search)}. The order in which they are listed here corresponds to the order in which \R{} searches for the definition of a command. \begin{knitrout} \definecolor{shadecolor}{rgb}{1, 1, 1}\color{fgcolor}\begin{kframe} \begin{alltt} \hlstd{base}\hlopt{::}\hlkwd{search}\hlstd{()} \end{alltt} \begin{verbatim} ## [1] ".GlobalEnv" "package:stats" ## [3] "package:graphics" "package:grDevices" ## [5] "package:utils" "package:datasets" ## [7] "package:methods" "Autoloads" ## [9] "package:base" \end{verbatim} \end{kframe} \end{knitrout} Notice that a collection of \Concept{package}s are installed by default among a couple of other special objects (\Rvariable{.GlobalEnv} and \Rvariable{Autoloads}). A package is attached using the \Sexpr{Rfunction(library)} command. The \Sexpr{Rfunction(library)} command takes an argument to identify the \Roption{package=} we wish to \Concept{attach}. <>= # Load packages from the local library into the R session. library(package=dplyr) library(package=rattle) @ Running these two commands will affect the search path by placing these packages early within the path. \begin{knitrout} \definecolor{shadecolor}{rgb}{1,1,1}\color{fgcolor}\begin{kframe} \begin{alltt} \hlstd{base}\hlopt{::}\hlkwd{search}\hlstd{()} \end{alltt} \begin{verbatim} ## [1] ".GlobalEnv" "package:rattle" ## [3] "package:dplyr" "package:stats" ## [5] "package:graphics" "package:grDevices" ## [7] "package:utils" "package:datasets" ## [9] "package:methods" "Autoloads" ## [11] "package:base" \end{verbatim} \end{kframe} \end{knitrout} \section{Simplifying Commands} By attaching the \Sexpr{Rpackage(dplyr)} package we can drop the package name prefix for any commands from the package. Similarly by attaching \Sexpr{Rpackage(rattle)} we can drop the package name prefix from the name of the dataset. Our previous \Sexpr{Rfunction(glimpse)} command can be simplified. <>= # Review the dataset. glimpse(x=weatherAUS) @ We can actually simplify this a little more. Often for a command we don't have to explicitly name all of the arguments. In the following example we drop the \Roption{package=} and the \Roption{x=} arguments as the commands themselves know what to expect implicitly. <>= # Load packages from the local library into the R session. library(dplyr) library(rattle) # Review the dataset. glimpse(weatherAUS) @ % "start up" is a verb, "start-up" is an adjective and "startup" is a noun. A number of packages are automatically attached when \R{} starts. The first \Sexpr{Rfunction(search)} command above returned a vector of packages and since we had yet to attach any further packages those listed are the ones automatically attached. One of those is the \Sexpr{Rpackage(base)} package which provides the \Sexpr{Rfunction(library)} command. \section{Working with the Library} To summarise then, when we interact with \R{} we can usually drop the package prefix for those commands that can be found in one (or more) of the attached packages. Throughout the running text in this book we retain the package prefix to clarify where each command comes from. However, within the code we tend to drop the package name prefix. A prefix can still be useful in larger programs to ensure we are using the correct command and to communicate to the human reader where the command comes from. Some packages do implement commands with the same names as commands defined differently and found in other packages and the prefix notation is then useful to specify which command we are referring to. Each of the following chapters will begin with a list of \Concept{packages} to \Concept{attach} from the \Concept{library} into the \R{} session. Below is an example of attaching five common packages for our work. Attaching the listed packages will allow the examples presented within the chapter to be replicated. In the code below take note of the use of the hash (\verb||) to introduce a comment which is ignored by \R{}---\R{} will not attempt to understand the comments as commands. Comments are there to assist the human reader in understanding our programs. <>= # Load packages required for this script. library(rattle) # The weatherAUS dataset and normVarNames(). library(readr) # Efficient reading of CSV data. library(dplyr) # Data wrangling and glimpse(). library(ggplot2) # Visualise data. library(magrittr) # Pipes %>%, %<>%, %T>%, %$%. @ In starting up an \R{} session (for example, by starting up \RStudio{}) we can enter the above \Sexpr{Rfunction(library,show.package=FALSE)} commands into an \R{} script file created using the \texttt{New \R{} Script File}\index{R script file} menu option in \RStudio{} and then ask \RStudio{} to \texttt{Run} the commands. \RStudio{} will send each command to the \R{} \texttt{Console} which sends the command on to the \R{} software. It is the \R{} software that then runs the commands. If \R{} responds that the package is not available then the package will need to be installed, which we can do from \RStudio's \texttt{Tools} menu or by directly using \Sexpr{Rfunction(install.packages)} as we saw above. %----------------------------------------------------------------------- \section{Functions} \label{sec:functions} %----------------------------------------------------------------------- \Concept{commands} in \R{} instruct the \R{} software to perform particular actions. Formally commands are actually \Concept{functions} and we generally use this term to convey the mathematical concept of a function. A function simply takes a set of inputs and produces an output. \R{} provides a wealth of \Concept{function}s, and we will generally think of each one as either a \Concept{function}, a \Concept{command} or an \Concept{operator}. All \R{} \Concept{functions} take arguments and return values. When we run a function (as distinct from what we will identify shortly as a \Concept{command}) we are interested in the value that it returns. Below is a simple example where we \Sexpr{Rfunction(sum)} two numbers.\footnote{It is a good idea to replicate all of the examples presented here in \R{}. Simply open an \R{} script file to edit and type the text \texttt{sum(1, 2)} using the editor in \RStudio{}. Then instruct \RStudio{} to \texttt{Run} the command in the \R{} \texttt{Console}.} <>= # Add two numbers. sum(1, 2) @ As previously we precede the output from the function by the double hash (\verb|##|). We will however not see the double hash in the \R{} \texttt{Console} when we run the functions ourselves. Instead, the output we see will begin with the \verb|[1]| which indicates that the returned value is a \Concept{vector} which starts counting from index~1. A \ConceptF{vector} is a collection of items which we can access through a sequential index---in this case the vector has only a single item. We can store the result value from running the function (the value being a vector of length 1 containing just the item \texttt{3}) into a \Concept{variable}. A \ConceptF{variable} is a name that we can use to refer to some location in the computer's memory where we store data while our programs are running. To store data in the computer's memory so that we can refer to that data by the specific variable name we use the \ConceptF{assignment} \Concept{operator} \Sexpr{Rfunction('<-',show.paren=FALSE)}. <>= # Add two numbers and assign the result into a variable. v <- sum(1, 2) @ \section{Accessing Stored Values} We can access the value stored in the variable (or actually stored in the computer's memory that the variable name refers to) simply by requesting through the \R{} \texttt{Console} that \R{} ``run'' the command \Variable{v}. We do so below with a line containing just \verb|v|. In fact the \R{} software actually runs the \Sexpr{Rfunction(print)} function to display the contents of the computer's memory that \Variable{v} refers to---this is a convenience that the \R{} software provides for us. <>= # Print the value stored in a variable. v print(v) @ By indexing the variable with \verb|[1]| we can ask for the first item referred to by \Variable{v}. Noting that the variable \Variable{v} is a vector with only a single item when we try to index a second item we get an \verb|NA| (meaning not available or a missing value). <>= # Access a particular value from a vector of data. v[1] v[2] @ %----------------------------------------------------------------------- \section{Commands} \label{sec:commands} %----------------------------------------------------------------------- For \ConceptF{command}s (rather than \Concept{function}s) we are generally more interested in the actions or side-effects that are performed by the command rather than the value returned by the command. For example, the \Sexpr{Rfunction(library)} command is run to attach a package from the library (the action) and consequently modifies the search path that \R{} uses to find commands (the side-effect). <>= # Load package from the local library into the R session. library(rattle) @ A \Concept{command} is also a function and does in fact return a value. In the above example we do not see any value printed. Some functions in \R{} are implemented to return values invisibly. This is the case for \Sexpr{Rfunction(library)}. We can ask \R{} to print the value when the returned result is invisible using the function \Sexpr{Rfunction(print)}. <>= # Demonstrate that library() returns a value invisibly. print(library(rattle)) @ We see that the value returned by \Sexpr{Rfunction(library)} is a vector of character strings. Each character string names a package that has been attached during this session of \R{}. Notice that the vector begins with item \verb|[1]| and then item \verb|[5]| continues on the second line, and so on. We can save the resulting vector into a variable and then index the variable to identify packages at specific locations within the vector. <>= # Load a package and save the value returned. l <- library(rattle) # Review one of the returned values. l[7] @ %----------------------------------------------------------------------- \section{Operators} \label{sec:operators} %----------------------------------------------------------------------- The third type of function is the \Concept{operator}. We use \ConceptF{operator}s in the flow of an expression. Technically they are called \Concept{infix} operators. In the following example we use the \Sexpr{Rfunction('+',show.paren=FALSE)} infix operator. <>= # Add two numbers using an infix operator. 1 + 2 @ Internally \R{} actually converts the operator into a functional prefix form to be run in the same way as any other function. <>= # Add two numbers using the equivalent functional form. `+`(1, 2) @ \begin{WebOnly} Note that R also supports using quotes and double quotes instead of the backtick we used in the code above. The backtick is preferred as per page 142 of \cite{wickham:2014:advanced_r}. \end{WebOnly} The key thing to remember is that all commands and operators are functions and we instruct \R{} to take action by running functions. %----------------------------------------------------------------------- \section{Pipes} %----------------------------------------------------------------------- A function (Section \ref{sec:functions}) performs some action on some input and returns some output. We can think of functions as the verbs of the language---the action words of our sentences. As learn many new functions useful for the data scientist we will construct longer sentences that string together a sequence of verbs to deliver the results. This is a powerful programming concept, combining dedicated and well designed and implemented functions, each function focused on achieving a specific outcome. To combine single focus functions into more complex operations we can use \Concept{pipe}s. Pipes are a powerful concept for combining functions. The concept will be familiar to most who have used Unix and Linux. The idea is to pass the output of one function on to another function as its input. Each function does one task and aims to do that task very well, very accurately, and very simply from a user's point of view. We can then pipe together many such specialist functions to deliver very complex and quite sophisticated data transformations in an easily accessible manner. Pipes were introduced in \R{} through the \Sexpr{Rpackage(magrittr)} package. To illustrate the concept of pipes we will again use the \Sexpr{Rdataset(weatherAUS)} dataset. We review the basic contents of the \Sexpr{Rdataset(weatherAUS)} dataset by printing it. <>= # Review the dataset of weather observations. weatherAUS @ We might be interested in the distribution of specific numeric variables from the dataset. For that we will \Sexpr{Rfunction(select,pkg="dplyr")} a few numeric variables by \textit{piping} the dataset into \Sexpr{Rfunction(select,pkg="dplyr")}. <>= # Select variables from the dataset. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) @ Notice how \Variable{weatherAUS} by itself will list the whole of the dataset. Piping the whole dataset to the \Sexpr{Rfunction(select,pkg="dplyr")} function using the \Concept{pipe} operator \Sexpr{pipe.fn.nsp} tells \R{} to send the \Sexpr{Rdataset(weatherAUS)} dataset on the left to the \Sexpr{Rfunction(select,pkg="dplyr")} function on the right. We provided \Sexpr{Rfunction(select,pkg="dplyr")} with a list of the variables we wish to select. The end result returned as the output of the pipeline is a subset of the original dataset containing just the named columns. %----------------------------------------------------------------------- \section{Multiple Pipes} %----------------------------------------------------------------------- Suppose we now want to produce a \Sexpr{Rfunction(summary)} of these numeric variables. We will \Concept{pipe} the dataset produced by \Sexpr{Rfunction(select,pkg="dplyr")} into \Sexpr{Rfunction(summary)}. <>= # Select variables from the dataset and summarise the result. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% summary() @ Perhaps we would like to summarise only those observations where there is more than a little rain on the day of the observation. To do so we will \Sexpr{Rfunction(filter)} the observations. <>= # Select specific variables and observations from the dataset. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall >= 1) @ We can see that this sequence of functions operating on the original \Sexpr{Rdataset(weatherAUS)} dataset returns a subset of that dataset where all observations have some rain. %----------------------------------------------------------------------- \section{Backward Assignment} %----------------------------------------------------------------------- We saw earlier in this chapter the assignment operator \Sexpr{Rfunction('<-',show.paren=FALSE)} which is used to save a result into the computer's memory and give it a name that we can refer to later. We can use this operator to save the result of the sequence of operations from the pipe. <>= # Select variables/observations and save the result. rainy.days <- weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall >= 1) @ An alternative that makes sense within the pipe paradigm is to use the forward assignment operator \Sexpr{Rfunction('->',show.paren=FALSE)} to save the resulting data into a variable in a logically consistent way. <>= # Demonstrate use of the forward assignment operator. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall >= 1) -> rainy.days @ In this example we see that using the forward assignment operator tends to hide the actual variable into which the assignment is made. The important side effect of the series of commands, the assignment to \Variable{rainy.days}, could easily be missed. Compare that to the prior code. Instead, when using hte forward assignment, be sure to un-indent the final variable name <>= # Best to unindent the final assigned variable for clarity. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall >= 1) -> rainy.days @ \section{Comparing Distributions} Continuing with our pipeline example, we might want a \Sexpr{Rfunction(summary)} of the dataset. <>= # Summarise subset of variables for observations with rainfall. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall >= 1) %>% summary() @ It could be useful to contrast this with a \Sexpr{Rfunction(summary)} of those observations where there was virtually no rain. <>= # Summarise observations with little or no rainfall. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall < 1) %>% summary() @ Any number of functions can be included in a \ConceptF{pipeline} to achieve the results we desire. In the following chapters we will see many examples and some will string together ten or more functions. Each step along the way is of itself generally easily understandable. The power is in what we can achieve by stringing together many simple steps to produce something more complex. \section{Pipeline Syntactic Sugar} For the technically minded we note that what is actually happening here is that the syntax, or how we write the sentences, is changed by \R{} from the normal embedded functional approach in order to increase the ease with which we can read the code. This is an important goal as we need to always keep in mind that we write our code for others (and ourselves later on) to read. Below we see a pipeline version of a series of commands operating on the dataset and not how it is mapped by \R{} into the functional construct. For many of us it will take quite a bit of effort to parse this traditional functional form of the expression so as to understand what it is doing. The pipeline alternative provides a clearer narrative <>= # Summarise observations with little or no rainfall. weatherAUS %>% select(MinTemp, MaxTemp, Rainfall, Sunshine) %>% filter(Rainfall < 1) %>% summary() @ <>= # Functional form equivalent to the pipeline above. summary(filter(select(weatherAUS, MinTemp, MaxTemp, Rainfall, Sunshine), Rainfall < 1)) @ Anything that improves the readability of our code is useful. Computers are quite capable of doing the hard work of transforming a simpler sentence into this much more complex looking sentence. \section{Assignment Pipe} The are several variations of the pipe operator available. A particularly handy operator implemented by \Sexpr{Rpackage(magrittr)} is the assignment pipe\index{assignment pipe|see{\%<>\%}} \Sexpr{apipe.fn.nsp}. This operator should be the left most pipe of any sequence of pipes. In addition to piping the dataset on the left onto the function on the right the result coming out of the right-hand pipeline is piped back to the original dataset overwriting its original contents in memory with the results from the pipeline. A simple example is to replace a dataset with the same dataset after removing some observations (rows) and variables (columns). In the example below we \Sexpr{Rfunction(filter)} and \Sexpr{Rfunction(select,pkg="dplyr")} the dataset to reduce it to just those observations and variables of interest. The result is piped backwards to the original dataset and thus overwrites the original data (which may or may not be a good thing). We do this on a temporary copy of the dataset and use the \Sexpr{Rfunction(dim)} function to report on the dimensions (rows and columns) of the resulting datasets. <>= # Copy the dataset into the variable ds. ds <- weatherAUS # Report on the dimensions of the dataset. dim(ds) # Demonstrate an assignment pipeline. ds %<>% filter(Rainfall==0) %>% select(MinTemp, MaxTemp, Sunshine) # Confirm that the dataset has changed. dim(ds) @ Once again this is so-called \textit{syntactic sugar}. The commands are effectively translated by the computer into the following code. <>= # Functional form equivalent to the pipeline above. ds <- select(filter(weatherAUS, Rainfall==0), MinTemp, MaxTemp, Sunshine) @ \section{Tee Pipe} Another useful operation is the tee-pipe \Sexpr{tpipe.fn.nsp} which causes the command that follows to be run as a side-pipe whilst piping the same data into that command and also into the next following command. The output from the first command is ignored. \begin{WebOnly} Note that a new operator \Sexpr{Rfunction('%T>%',show.paren=FALSE)}% \index{tee pipe|see{\%T>\%}} from \Sexpr{Rpackage(magrittr)} has been introduced here to process the transformed dataset in two ways---once with \Sexpr{Rfunction(glimpse)} and then with \Sexpr{Rfunction(summary)}. The so-called \textit{tee} pipe will take the flow of the dataset coming out of the previous junction which is the \Sexpr{Rfunction(select)} and pipe it in two directions, like a T pipe. The first flow proceeds to the junction where \Sexpr{Rfunction(glimpse)} is run. The second flow proceeds directly to the following junction, without going via the \Sexpr{Rfunction(glimpse)}, where \Sexpr{Rfunction(summary)} is then run. Any further pipes would flow the output of the \Sexpr{Rfunction(summary)} command onward. One use case is to generate a summary of an intermediate dataset whilst continuing to process the dataset. <>= weatherAUS %>% select(Rainfall, MinTemp, MaxTemp, Sunshine) %>% filter(Rainfall==0) %T>% head() %>% select(MinTemp, MaxTemp, Sunshine) %>% summary() @ \end{WebOnly} A common use case is to \Sexpr{Rfunction(print)} the result of some data processing steps whilst continuing on to assign the dataset itself to a variable. We will often see the following example. <>= # Demonstrate usage of a tee-pipe. no.rain <- weatherAUS %>% filter(Rainfall==0) %T>% print() @ \section{Tee Pipe Alternatives} We could simply \Sexpr{Rfunction(print)} the value of the variable \Variable{no.rain} after the assignment or else enclose the assignment within round brackets but there is some elegance in including it explicitly as part of the single pipeline. <>= # Equivalent alternatives to the above tee-pipe example. ( weatherAUS %>% filter(Rainfall==0) -> no.rain ) @ <>= print(no.rain) @ \begin{WebOnly} \section{Tee Pipe Use Case: Load all CSV Files} Example of usage of the tee operator. Load all \verb|.csv.gz| files in the data folder into a single data frame and include a message for each file loaded to monitor progress. <>= fpath <- "data" files <- dir(fpath, "*.csv.gz") ds <- data.frame() for (i in seq_len(length(files))) { ds <- fpath %>% file.path(files[i]) %T>% cat("\n") %>% readr::read_csv() %>% rbind(ds, .) } @ This can also be useful when combined with an on the fly function which is introduced with curly braces. We can use the global assignment operator to define variables that can be used later on in the pipeline. This example is a little contrived but illustrates its use. The on the fly function calculates the order of wind gust directions based on the maximum temperature for any day within each group defined by the wind direction. This is then used to order the wind directions and saving that order to a global variable \Variable{lvls}. That variable is then used to mutate the original data frame (note the use of the tee pipe) to reorder the levels of the \Variable{WindGustDir} variable. This is typically done within a pipeline that feeds into a plot where we want to reorder the levels so that there is some meaning to the order of the bars in a bar plot, for example. <>= # List the levels of a factor. levels(weatherAUS$WindGustDir) # weatherAUS %>% filter(Rainfall>0) %T>% { lvls <<- select(., WindGustDir, MaxTemp) %>% group_by(WindGustDir) %>% summarise(MaxMaxTemp=max(MaxTemp)) %>% arrange(MaxMaxTemp) %>% pull(WindGustDir) } %>% mutate(WindGustDir=factor(WindGustDir, levels=lvls)) %$% levels(WindGustDir) @ \section{Exposition Pipe} We also see here yet another useful pipe operation \verb|%$%|. This is the exposition pipe which is also useful for commands that do not take a dataset as their argument but instead some columns from the dataset. It evaluates the following function within the context of the dataset passed to it, so that the variables of the dataset become available without the need to quote them <>= weatherAUS %>% filter(Rainfall==0) %$% cor(MinTemp, MaxTemp) @ We might otherwise use with. <>= weatherAUS %>% filter(Rainfall==0) %>% with(cor(MinTemp, MaxTemp)) @ \end{WebOnly} \section{Pipe and Plot} A common scenario for pipeline processing is to prepare data for plotting. Indeed, plotting itself has a pipeline type concept where we build a plot by adding layers to it. Below the \Sexpr{Rdataset(weatherAUS)} dataset is \Sexpr{Rfunction(filter)}ed for observations from four Australian cities. We \Sexpr{Rfunction(filter)} observations that have missing values for the variable \Variable{Temp3pm} using an embedded pipeline. The embedded pipeline pipes the \Variable{Temp3pm} data through the \Sexpr{Rfunction(is.na)} function which tests if the value is missing. These results are then piped to \Sexpr{Rfunction(not)} which inverts the true/false values so that we include those that are not missing. A plot is generated using \Sexpr{Rfunction(ggplot)} into which we pipe the processed dataset. We add a geometric layer using \Sexpr{Rfunction(geom_density)} which consists of a density plot with transparency specified through the \Roption{alpha=} argument. We also add a title and label the axes using \Sexpr{Rfunction(labs)}. <>= cities <- c("Canberra", "Darwin", "Melbourne", "Sydney") weatherAUS %>% filter(Location %in% cities) %>% filter(Temp3pm %>% is.na() %>% not()) %>% ggplot(aes(x=Temp3pm, colour=Location, fill=Location)) + geom_density(alpha=0.55) + labs(title = "Density Distributions of the 3pm Temperature", x = "Temperature Recorded at 3pm", y = "Density") @ <>= @ We now observe and tell a story from the plot. Our narrative will begin with the observation that Darwin has quite a different and warmer pattern of temperatures at 3pm than Canberra, Melbourne and Sydney. \section{Sophisticated Pipeline} % Add a section to load this dataset and explain it. <>= avgdis <- read_csv("data/SA4_AvgDis_PoR_PoW.csv") @ <>= avgdis %>% filter(GCCSA_CODE_2011=="1GSYD") %>% select(O_SA4name, O_Avg_KM, D_Avg_KM) %>% mutate(O_SA4name=str_replace(O_SA4name, "^Sydney - ", "")) %$% { nms <<- O_SA4name[order(O_Avg_KM)] ; . } %>% mutate(O_SA4name=factor(O_SA4name, levels=nms)) %>% gather("OD", "Distance", -O_SA4name) %>% arrange(O_SA4name) %$% { lbls <<- data.frame(x = sort(c(seq(0.75, 14.75), seq(1.2, 15.2))+0.02), y = (Distance-0.75), lbl = round(Distance,1)) ; . } %>% ggplot(aes(x=O_SA4name, y=Distance)) + geom_bar(stat="identity", position="dodge", aes(fill=OD)) + geom_text(data=lbls, aes(x=x, y=y, label=lbl), colour="white", size=3, hjust=0.5) + theme(axis.text.x=element_text(angle=45, hjust=1, size=10)) + theme(legend.position=c(.15, .9)) + scale_fill_manual(values=c(rgb(246, 160, 26, max=255), rgb( 0, 101, 164, max=255)), labels=c("By SA4 of residence", "By SA4 of work")) + theme(legend.title=element_text(colour="transparent"), legend.background=element_rect(fill="transparent"), legend.direction="horizontal") + labs(x="Sydney", y="Average Distance (KM)") @ %\begin{figure}[t] % \centering % \includegraphics{sydney_distance_travel} % \caption{Figure generated from a pipline of commands.} % \label{fig:intror:sydney_distance_travel} %\end{figure} %----------------------------------------------------------------------- \section{Pipeline Identity Operator} \index{identity} %----------------------------------------------------------------------- A handy trick when building a pipeline is to use what is effectively the identity operator. An identity operator simply takes the data being communicated through the pipeline, and without changing it passes it on to the next operator in the pipeline. An effective identity operator is constructed with the syntax \verb|{.}|. In \R{} terms this is a compound statement containing just the period whereby the period represents the data. Effectively this is an operator that passes the data through without processing it---an identity operator. Why is this useful? Whilst we are building our pipeline, one line at a time, we will be wanting to put a pipe at the end of each line, but of cause we can not do so if there is no following operator. Also, whilst debugging a pipeline, we may want to execute only a part of it, and so the identity operator is handy there too. As a typical scenario we might be in the process of building a pipeline as here and find that including the \Sexpr{pipe.fn.nsp} at the end of the line of the \Sexpr{Rfunction(select)} operation: <>= weatherAUS %>% select(Rainfall, MinTemp, MaxTemp, Sunshine) %>% {.} @ We then add the next operation into the pipeline without having to modify any of the code already present: <>= weatherAUS %>% select(Rainfall, MinTemp, MaxTemp, Sunshine) %>% summary() %>% {.} @ And so on. Whilst it appears quite a minor convenience, over time as we build more pipelines, this becomes quite a handy trick. %----------------------------------------------------------------------- \section{Getting Help} %----------------------------------------------------------------------- A key skill of any programmer, including those programming over data, is the ability to identify how to access the full power of our tools. The breadth and depth of the capabilities of \R{} means that there is much to learn around both the basics of \R{} programming and the multitude of packages that support the data scientist. \R{} provides extensive in-built documentation with many introductory manuals and resources accessible through the \RStudio{} \texttt{Help} tab. These are a good adjunct to our very brief introduction here to \R{}. Further we can use \RStudio's search facility for documentation on any action and we will find manual pages that provide an understanding of the purpose, arguments, and return value of \Concept{function}s, \Concept{command}s, and \Concept{operator}s. We can also ask for help using the \Sexpr{Rfunction("?", show.paren=FALSE)} operator as in: <>= # Ask for documentation on using the library command. ? library @ The \Sexpr{Rpackage(utils,cite=FALSE)} package which provides this operator is another package that is attached by default into \R{}. Thus we can drop its prefix as we did here when we run the command in the \R{} Console. \RStudio{} provides a search box which can be used to find specific topics within the vast collection of \R{} documentation. Now would be a good time to check the documentation for the \Sexpr{Rfunction(library)} command. Beginning with this chapter we will also list at the conclusion of each chapter the functions, commands, and operators introduced within the chapter together with a brief synopsis of the purpose. The following section reviews all of the functions introduced so far. %----------------------------------------------------------------------- \section{Command Summary} %----------------------------------------------------------------------- This chapter has introduced, demonstrated and described the following \R{} packages, functions, commands, operators, and datasets: \\[2mm] \begin{TODO} \todo[inline]{Review each command description to assure correct and sufficient.} \end{TODO} \begin{outdent} \textbf{\Sexpr{Rfunction('<-', DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('<-', PkgOp=TRUE)}.} The base \R{} assignment operator. The expression on the right hand side is evaluated and the result is stored into the memory location identified on the left hand side. The left hand side is usually the name of a variable and has associated with it memory where it stores its values. \textbf{\Sexpr{Rfunction('->', DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('->', PkgOp=TRUE)}.} The forward assignment operator. The expression on the left hand side is evaluated and the result is stored into the memory location identified on the right hand side. This assignment is not commonly used in \R{} however it is a natural for use in a pipeline expression where the pipeline processes a dataset and we then store the resulting dataset into a variable. \textbf{\Sexpr{Rfunction('?', DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('?', PkgOp=TRUE)}.} Display the documentation provided for a command. The documentation will be displayed in the lower right pane of the \RStudio{} window and often includes links to other documentation. \textbf{\Sexpr{Rfunction('%<>%', DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('%<>%', PkgOp=TRUE)}.} The assignment pipe will pipe a dataset through to a function and the result is piped backwards to overwrite the original dataset. \textbf{\Sexpr{Rfunction('%>%',pkg="magrittr",DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('%>%',pkg="magrittr",PkgOp=TRUE)}.} Pipe a dataset through a series of functions. Each function takes the dataset as its input, processes the dataset in some way, and then returns a processed dataset which is piped onto the next function, if there is another pipe. \textbf{\Sexpr{Rfunction('%T>%',pkg="magrittr",DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('%T>%',pkg="magrittr",PkgOp=TRUE)}.} The Tee-Pipe operator pipes a dataset through to the next function and also on to the following function. The results of the immediately following function are ignored and the pipeline then continues with the function following the immediately following function. \textbf{\Sexpr{Rfunction('+', DefnOp=TRUE)}} \textit{operator from \Sexpr{Rfunction('+', PkgOp=TRUE)}.} Determine the sum of two arguments. The arguments are usually numbers but we have also seen an example where we add layers to a plot using this operator. This is an infix binary operator in that it is used between the two arguments that are to be added. \textbf{\Sexpr{Rfunction(dim, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(dim, Pkg=TRUE)}.} Returns a count of the number of rows and the number of columns in a dataset---the dimensions. \textbf{\Sexpr{Rfunction(filter, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(filter, Pkg=TRUE)}.} Choose only those observations from the input dataset that satisfy a specified criteria. The output dataset consists of all of the same variables as the input dataset but a subset of the observations from the input dataset. \textbf{\Sexpr{Rfunction(geom_density, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(geom_density, Pkg=TRUE)}.} Add a density plot to a ggplot. A density plot shows the distribution of the different values of the variable over the observations in the dataset. \textbf{\Sexpr{Rfunction(ggplot, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(ggplot, Pkg=TRUE)}.} Initiate a new plot and return an object that represents that (currently empty) plot. In order to be able to display the plot we need to add layers. We have illustrated the layering in this chapter where we added density geometries, a title and labels. The \texttt{gg} here refers to the \textit{grammar of graphics} which is implemented by \Sexpr{Rpackage(ggplot2)}. \textbf{\Sexpr{Rfunction(glimpse, Defn=TRUE)}} \textit{command from \Sexpr{Rfunction(glimpse, Pkg=TRUE)}.} Summarise a dataset. This is a convenient command for quickly obtaining a view of a potentially very large dataset. \textbf{\Sexpr{Rfunction(install.packages, Defn=TRUE)}} \textit{command from \Sexpr{Rfunction(install.packages, Pkg=TRUE)}.} Install a package into a library. Often the package will be downloaded from the Internet though it can also be a locally available file that was separately downloaded. The file will be an archive and this command unpacks the archive and stores the result in a folder that is referred to as a library. \textbf{\Sexpr{Rfunction(is.na, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(is.na, Pkg=TRUE)}.} Test if a value (or for that matter a vector) is ``not available'' (missing). Returns TRUE or FALSE appropriately or a vector of TRUE/FALSE if the argument is a vector. \textbf{\Sexpr{Rfunction(labs, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(labs, Pkg=TRUE)}.} Change the text of the labels used for the x or y axis of a plot built using \Sexpr{Rfunction(ggplot)} . \textbf{\Sexpr{Rfunction(library, Defn=TRUE)}} \textit{command from \Sexpr{Rfunction(library, Pkg=TRUE)}.} Attach a package from an \R{} library. The name of a package is provided as the argument to this command. That package is searched for within the available libraries on our local computer. When it is found the functions, commands, operators and datasets are made available to this \R{} session. \textbf{\Sexpr{Rfunction(not, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(not, Pkg=TRUE)}.} Invert the meaning of TRUE and FALSE, so that TRUE becomes FALSE and FALSE becomes TRUE. This is useful in a pipe as an alternative to the \Sexpr{Rfunction('!',add.seen=FALSE,show.paren=FALSE)} operator. \textbf{\Sexpr{Rfunction(print, Defn=TRUE)}} \textit{command from \Sexpr{Rfunction(print, Pkg=TRUE)}.} Print a representation of the memory referred to by the named object supplied as the argument. \textbf{\Sexpr{Rfunction(qplot, Defn=TRUE)}} \textit{command from \Sexpr{Rfunction(qplot, Pkg=TRUE)}.} Quickly code a plot of the dataset. The generated plot is displayed as a side-effect of the command. \textbf{\Sexpr{Rfunction(search, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(search, Pkg=TRUE)}.} List the currently attached packages. This can be useful when we wish to see which packages have been attached to the current \R{} session. The order in which the packages are listed is also the order in which \R{} searches for a function. When the same function name is defined in two packages the first that is found is the one that will be used. \textbf{\Sexpr{Rfunction(select, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(select, Pkg=TRUE)}.} Choose specified variables from a dataset. This function returns a smaller dataset than the input dataset consisting of all of the same observations as the input dataset but only of the selected variables. \textbf{\Sexpr{Rfunction(sum, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(sum, Pkg=TRUE)}.} Determine the sum of two arguments (usually numbers). This is the traditional functional form of the \Sexpr{Rfunction('+',show.paren=FALSE,show.package=FALSE)} operator. \textbf{\Sexpr{Rfunction(summary, Defn=TRUE)}} \textit{function from \Sexpr{Rfunction(summary, Pkg=TRUE)}.} From the input dataset this function generates a simple statistical summary of each variable. \begin{WebOnly} \textbf{\Sexpr{Rdataset(weather, Defn=TRUE)}} \textit{dataset from \Sexpr{Rdataset(weather, Pkg=TRUE)}.} The weather dataset contains daily observations of numerous weather variables from a weather station at the Canberra airport in Australia. It is a small dataset consisting of just \Sexpr{nrow(weather)} observations of \Sexpr{ncol(weather)} variables. It is used here to illustrate complex processing when size is not relevant for demonstrating the processing. \end{WebOnly} \textbf{\Sexpr{Rdataset(weatherAUS, Defn=TRUE)}} \textit{dataset from \Sexpr{Rdataset(weatherAUS, Pkg=TRUE)}.} The weather dataset contains daily observations of numerous weather variables from nearly 50 weather stations across Australia. This is a moderately sized dataset consisting of \Sexpr{scales::comma(nrow(weatherAUS))} observations of \Sexpr{scales::comma(ncol(weatherAUS))} variables. \end{outdent} %----------------------------------------------------------------------- \section{Exercises} %----------------------------------------------------------------------- \begin{TODO} \todo[inline]{For each exercise produce worked answers as an external resource (not included in the book).} \end{TODO} \begin{Exercise}[title={Exploring \RStudio.}] \begin{enumerate} \item Install \R{} and \RStudio{}. \item Start up \RStudio{} and explore the menus, tool bars, and panes and tabs. \item Explore the various options for learning about \R{}. \end{enumerate} \end{Exercise} \begin{Exercise}[title={Getting help in \RStudio.}] \begin{enumerate} \item Attach the \Sexpr{Rpackage(rattle)} package in \RStudio. \item Obtain a list of the functions available from the \Sexpr{Rpackage(rattle)} package. \item Review the documentation for \Sexpr{Rfunction(normVarNames)} and then apply the function to the \Sexpr{Rfunction(names)} of the \Sexpr{Rdataset(weatherAUS)} dataset from \Sexpr{Rpackage(rattle)}. \end{enumerate} \end{Exercise} \begin{Exercise}[title={Interacting with \RStudio.}] \begin{enumerate} \item Create a new \R{} script file and add a comment at the top with your name and today's data. Then save the file to disk. \item By interacting with the \R{} script file, attach the \Sexpr{Rpackage(rattle)} package. \item Plot the distribution of \Variable{WindSpeed9am} against \Variable{Humidity9am} from the \Sexpr{Rdataset(weatherAUS)} dataset. \item Filter the dataset by different locations and then review and compare the plots. \item View and write a short report on the \Sexpr{Rdataset(weatherAUS)} dataset. The report should describe the dataset, including a summary of the number of variables and observations. Describe each variable and summarise its distribution. Include some interesting plots. \end{enumerate} \end{Exercise} \begin{TODO} \begin{Exercise}[title={Functions, commands and Operators.}] \todo[inline]{Include an exercise relating to functions, operators and commands. Cover return values and perhaps include TRUE/FALSE. Use the '-' operator both infix and as a function call.} \end{Exercise} \end{TODO} \begin{Exercise}[title={Using pipes.}] \begin{enumerate} \item Load the \Sexpr{Rdataset(weatherAUS)} dataset into \R{}. \item Explore the rainfall for a variety of locations, generating different plots similar to the two examples provided in this chapter. \item Write a short report on what you discovery including the \R{} code you develop (using pipelines) and include sample plots that support your narrative. \end{enumerate} \end{Exercise} %----------------------------------------------------------------------- \section{Further Reading}\label{sec:intror:further} %----------------------------------------------------------------------- \rattlereading \begin{TODO} \todo[inline]{List other resources of use that extend beyond what we have presented in this chapter.} \end{TODO} We list below other resources that augment the material we have presented in this chapter. \begin{itemize} \item The manual titled \textit{An Introduction to \R{}} is a good place to learn some of the basics of \R. Follow the link provided within \RStudio{}. \item \url{http://www.rstudio.org/} is the home of \RStudio{}. The \RStudio{} software can be downloaded from here as well as finding documentation and discussion forums. For a guide to using \RStudio{} visit \url{https://support.rstudio.com}. \item Documentation within \RStudio{} through the Help tab in the bottom right pane provides entry points to many of the standard resources for getting started with \R{} and \RStudio, as well as more advanced references. \item \fref{https://stackoverflow.com}{Stack Overflow} is a great question and answer resource. There we find crowd sourced answers with crowd sourced ratings. It is always a good first port of call for any questions about using \R{}. \end{itemize} %----------------------------------------------------------------------- <>= @ %----------------------------------------------------------------------- \end{document}