Understanding the details of how a computer works is not necessary for programming. However, understanding the basics of how your computer's memory works helps reveal why data typing is such an important topic early in computer programming training. Also, understanding of how the lowest level machine language programs are executed by the processor's logic circuitry helps reveal the differences in efficiency between the customized machine language programs from compiled languages versus the higher level scripting language programs that are ultimately executed on the processor by a precompiled interpreter program (3:56 min).

With a basic understanding of how the memory works, we can review the fundamentals of how the processor runs a program and in particular an individual instruction of machine language (5:09 min).

The lowest level human readable language that can be used to generate the machine language instructions that run directly on the processor is called assembly language. Assembly language is specific to the instruction set for a given processor and is essentially just a one-to-one translation from the series of bits representing a given instruction and a narrative word or phrase for that instruction. Assembly code is rarely directly generated by humans except for educational exercises. If customized machine language is needed, it is generally created from a compiled programming language like C or Fortran. For automating workflows where memory use or processor efficiency is less critical or can be handled by external programs, programming in scripting languages that are executed by precompiled interpreters like R or python engines are popular. The following video summarized where compiled or scripting languages may be more appropriate and describes why R is used for the activities here (5:48 min).

Slides from videos

The remainder of this module assumes that you have R and RStudio installed. You will need to have R installed first, so the RStudio installation can find the current R installation and automatically create the necessary links to it. The links to the R installation used by RStudio can easily be changed in RStudio's Global Options under the Tools menu.

Here are the direct links for installing R and RStudio, in the order in which they should be installed:

• The Comprehensive R Archive Network (CRAN) for downloading and installing R (among many other R packages).

• Open source RStudio Desktop for downloading and installing RStudio (should install R first).

This module also assumes that plentiful public resources are available for helping with the installation of these software packages on your operating system of choice, and does not provide specific narrative or videos on this topic. 

Depending on the default configuration of the web browser, this HTML code will render to the following formatted text.

With this strategy in mind, lets work through building the basic structure of a pipeline for our sensitivity analysis project and creating an RStudio project file in the protocol directory that will help us stay organized when working on the R code performing the workflow. Consistent practice of using a project file to open RStudio for work on a given pipeline will eventually facilitate the use of relative paths in code that will enhance the portability and reproducibility of the data product (i.e. the ability to move the entire folder structure to another computer and it will still work). Let's start by creating a root directory for a data product and add the pipeline folders to the root (4:18 min).

Once the pipeline folders are in place, we can create an RStudio project file in the protocol folder that will make it easier to bring up a unique RStudio session for work on the R code that generates the data product. This is the first step toward working with the data product in a way that will ultimately make it more portable and reproducible (4:55 min).

Before we start programming, this is a good time to review the more commonly used tools in the RStudio integrated development environment in more detail, including thinking about the different parts of your computer's software stack with which you are interacting when using those tools (18:22 min).

Slides from videos

Retention of material while learning how to program computers is aided by active learning exercies that have a meaningful goal for what we want the computer to do. For this exercise, we are going to learn the programming skills necessary to create an analytical data product that provides a sensitivity analysis of a commonly used mathematical model. We are going to write a program that performs the calculations of the model with several different parameter values, then creates a visualization that allows us to assess how altering that parameter value influenced the behavior of the model. This is an extremely common exercise in scientific analysis when we are trying to determine whether a given mathematical model will be appropriate for describing patterns in our data. Let's review the theoretical origins of the exponential decay model, which is prolifically used across scientific disciplines to describe many different source-limited behaviors in nature. In this case we'll think about it in the context of first-order kinetics from the rate theory of chemistry (6:49 min).

To get R to do calculations with this model that will help us understand how it predictions concentration will change over time, we need to have a way to tell R to create a series of times at which we want to know the concentration. Therefore, the first step to writing R code is to understand how to assign values to variables in the global environment, and how to query or manipulate the variables contained within the global environment (7:53 min).

The most fundamental and irreducible data type in R is the atomic vector. Even when we assign a single value to a variable, R is still creating at atomic vector to hold that value; but that vector is composed of a single element containing the assigned value. Let's briefly review a few of the ways to create vectors with more than one element, with careful attention to the data type being assumed by R in allocating space in RAM for the value (12:04 min).

Now that you are familiar with creating numeric vectors, we can start a script that will execute a sensitivity analysis of exponential decay in the context of first-order kinetics (14:19 min).

Now that we have a vector of times, we need to do some math with it to calculate concentrations. R needs to have special rules for doing mathematical operations with variables representing numeric vectors because each vector in the equation may have a different number of elements. The following video shows examples of how R will recycle the values in shorter vectors to match the number of elements in the longer vector when doing calculations (6:00 min).

Understanding the recycling of values of shorter vectors in R calculations is critical to understanding how the exponential decay equation results in a vector of values the same as the length of the time vector (9:54 min).

When a compound data structure you are using has more than one element, the ability to extract specific elements from those data structures for calculations or other data processing is often necessary. R has a wide variety of different ways you can index a vector to create a new vector with only specific elements of the original (11:59 min).

Let's work through a simple example of the application of vector indexing by using it to filter the data we consider in the exponential decay sensitivity analysis (5:41 min).

Note that NA values are generally ignored by R's graphing functions. Try plotting the results of the above code to see how the graph is now truncated to the fist 99% of reactant consumption.

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

Slides from videos

Once you have a grasp on the fundamental data types in a computer programming language, you quickly get to a point where you need some way to group some of these simple data structures together into a more complex data type. In this case, we have perhaps reached a point where we want to save the results of our analysis to a file in the incremental folder for later processing, and the ability to group variables together into a single object would be convenient. The list data type in R is an abstract version of a vector (i.e. non-atomic) allowing you to keep track of a vector of references to other R objects (6:35 min).

Each reference in a list can point to an object of a different data type, which is why lists are commonly used to build more complex data structures from a combination of simpler data structures (3:36 min).

Just like any other vector, the individual elements of a list can be assigned a name that can later be used as an alternative to the numerical index for accessing individual elements of the list. Naming the elements of a list is a more common practice than naming the elements of an atomic vector because using names rather than integers for the elements of a list helps with a more intuitive way to remember the data type referenced by each element (8:03 min).

Note that following along with the next step depends on having a project configured similar to the completion of the exercise in the previous topic.

Now that we know the nature of lists and how to create them, lets use a list to collect and save the results of our sensitivity analysis. First we need to create the list object that we want to save for later use (7:12 min).

Once a list is created, we can save it to a binary file that can be easily reloaded back into R in a separate script. Reading data from or writing data to other files in the pipeline folders is where the value of having a consistent working directory based on using a project file to open RStudio becomes more clear. This step therefore provides an example of the use of relative file system paths for writing a data file to the incremental pipeline folder. In particular, you will need to understand the "../" notation used for specifying the parent of the current working directory at the beginning of the relative path "../3_incremental/results.RData" used to designate the data file to be saved to permanent memory (6:59 min).

Strategies like using relative paths from designated working directories are critical for the code included in a given data product to run correctly regardless of where the data product is located on a file system. Any file input or output code that uses full paths to files will inherently break if the data product is moved to a different computer or even to a different place in the file system on the same computer. If those paths are sprinkled throughout the code for a complex analysis, the task of correcting all those paths to get the code to run again after it is moved can be quite tedious, time consuming, and error prone. I HIGHLY recommend not doing that to your future self, and using relative paths is one of the most widely accepted practices for avoiding it.

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

Slides from videos

The results of a version of our analysis have been written to a list stored in a binary RData file in the incremental folder. If we are running a new script for the purpose of generating a product with the visualization of the analysis, we will need to know how to read those data back into R (6:29 min).

Now that we have the results data, let's review how to make a plot, but this time lets choose a graphics device that will allow us to create the plot as a vector graphics file that is part of the data product (8:15 min).

Let's start learning how to take more control over how the graph looks. The first step is to understand how to define the coordinate system you will use for putting symbols at particular locations on the canvas, and the first step defining the coordinate system is to change the graphics parameters that define where the four edges of the rectangular graphing area are located relative to the four edges of the canvas (7:47 min).

Note that I am using the Sumatra PDF viewer in these videos, which allows me to leave the PDF file open while I am changing the graph with the code. The Sumatra PDF viewer will also automatically update the image when the source PDF file is rewritten. This will not work with all PDF viewers. If you have the PDF file open in Adobe's viewer, for example, the R code will generate an error if trying to write to the PDF file while it is open in Adobe. If your code breaks for this reason, or for any reason that causes it to break before the dev.off() function is called, you may have to call dev.off() manually in the console to get back to where R can write to the PDF file again. Alternatively, you can call the graphics.off() function to completely reset the graphics engine.

After you establish the location of the graphing area, the coordinate system is established by defining the coordinate values associated with the extremes of the graphing area. The x limits define the bounds for the left and right edges of the graphing area and the y limits define the bounds for the bottom and top edges of the graphing area (8:59 min).

Once we have axes, we can add data to the figure and then define axes labels to give the data meaning (12:18 min).

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

Note that a graph just like this can be created using different arguments to the plot() function used earlier to first demonstrate the graphing capacity of R. However, if you learn how to break a call to plot() down into the component functions it is using to create the graph, then you will be able to be much more creative and take much more control over the rendering of the figure. We are not yet taking advantage of features beyond what plot() can do when using the appropriate arguments, but this exercise is designed to train you to understand the lower level graphing functions that are eventually necessary to take advantage of highly granular control over figures.

Creating the same plot by aggregating several function calls above into a single call to plot() would look like the following. While this code is indeed more compact, I find it a bit harder to read in terms of understanding what is actually happening when the plot() function is called. Also, the following version has no computational benefits other than the code taking a little bit less space to store. With either version, the same functions are ultimately being called to generate the graph.

Because multi-element data structures like atomic vectors or lists are a common component of nearly all programming languages, designing iterative algorithms for dealing with each element of these data structures is commonly an early topic in computer programming training. Let's alter our first-order exponential decay example to repeat the analysis for several different values of the initial concentration as a relatively simple example of designing iterative algorithms with a program control structure called a for loop (15:20 min).

We have updated our analysis to include the results of three different initial concentrations, but our script for generating a graph of the results is not expecting this new heirarchical data structure. We need to use more loops and iterative algorithms to alter the product.R script to be able to graph the results for the three different initial concentrations on the same axes (20:23 min).

Since we have a graph with three different symbols on it, a legend specifying the meaning of those symbols would be nice (23:39 min).

The sensitivity analysis now gives us an idea of how changing the initial concentration affects the exponential decay curve, but what about the rate coefficient? Let's expand the sensitivity analysis to include a full factorial combination of variation in the initial concentration and the first-order rate coefficient. This means we need to learn about nested loops and we need to add another to the hierarchy of the data structure for our results (9:29 min).

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

Now we need to update the code for generating the graphs for the more complex sensitivity analysis (25:48 min).

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

Let's put some final touches on our sensitivity analysis data product. First, our data product does not include plots of all the data our sensitivity analysis has generated. Let's add to our iterative algorithm to generate plots of the filtered data as well as the unfiltered data we are already plotting (16:45 min).

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

If we want to fully automate the workflow for this sensitivity analysis, we may want to be able to reconfigure the analysis without changing the code. That means we need to read the configuration of the analysis from input files rather than hardcoding the configuration in our analysis script. This gives us some additional files we will want to put in the the input folder of our pipeline to define the configuration of the sensitivity analysis (21:39 min).

After the videos above, you should have an R script in the protocol of the pipeline that looks something like the following. Note that I have added more thorough comments than those typed in the video.

We now have a workflow for generating the sensitivity analysis that may be considered fully automated. We can configure the sensitivity analysis from the text files in the input folder of the pipeline and rerun the analysis without having to change any code. To convince you of this, I will change the configuration and rerun the analysis script directly from R using the command line. You do not need to know how to do this to be functional at using R for your data analysis. Out of convenience, I seldom run R programs without running them from R Studio. This is just a demonstration of what it means to have a fully automated workflow that can be executed without any of the tools in RStudio (4:09 min).

A parallel data product developed with VS Code and python

A general tutorial on R data structures

A general tutorial on base R graphics