Deciding how to add metadata to a data product first requires thinking about the content you intend to add and how you are going to organize that content. Before diving into more technical detail, perhaps the best place to start is to make sure you understand the overall goals of including metadata in the first place. One way to summarize these goals is to define the FAIR principles for data stewardship (3:58 min).

For a more detailed perspective on the appropriate content of metadata, you can consider the questions you want your metadata to be able to answer proactively (4:51 min).

With these FAIR and interrogative principles in mind, what is the pathway to a more specific strategy for organizing the metadata for a given data product? A definitive answer to that question is not possible without more information about the use cases for the data, but abstractions that are transferable across most curatorial metadata applications have been defined, and specific schema that are appropriate for a wide variety of environmental science applications are available (5:19 min).

Slides from videos 

Using EDI's EML requirements as a model, we can start building metadata for the photosynthetically active radiation data product we started in the previous module. The centerpiece of metadata for a data product is often found in a human readable file with the name "readme" located in the same folder with the data product. This file name allows for some Alice in Wonderland style intuition to prompt a data user to open the readme file to find more contextual information about the product (11:18 min).

Adding hierarchical numbering to the section headings makes it easier to see the organization of the outline for a document. Here is a simple way to automate the generation of section numbers using the features of Rmarkdown. This automation means that we don't have to worry about manually renumbering sections if we add a new section in the middle or if we decide to change the order of sections. In other words, it takes a little time to write the code now, but potentially saves us quite a bit of time making manual changes later (9:40 min).

After the videos above, you should have an Rmarkdown file for generating a readme metadata file that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

We have a start on an Rmarkdown file that will render a readme file, but the source of the metadata in this report might come from definition earlier in the workflow. For example, we might want to couple the definition of the metadata describing the data table more directly to the code that is generating the data table. Keeping this code together helps ensure the metadata stays synchronized with any changes to the data through the development of the data product (22:40 min).

Perhaps the most important metadata for a tabular data product is the description of each of the columns in the data tables included. Let's define the column attribute metadata by constructing a data frame that can be saved as a machine readable table in the data product as well as included in the human readable report (32:22 min).

Now that we have tightly coupled the generation of some of the metadata with the generation of the data, we can propagate that information through to the data product readme file in human readable form. Note that we will have copies of the same information in more than one location, which has the tendency to create synchronization problems if we are trying to maintain multiple copies of the same information manually. However, we have automated the workflow such that just one authoritative version of the metadata is copied to all those locations when we regenerate the data product. Automated documentation generation helps eliminate concerns with synchronization issues when the same information needs to be reflected in more than one location (15:57 min).

After the videos above, you should have an updated Rmarkdown file for data processing that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

Perhaps the next most important metadata for reuse of the data product is more information about when and where the data were collected. Our processing workflow already includes some of this information, so maybe we should define space and time coverage metadata in the processing script and propagate it to generation of the readme file (22:47 min).

After the videos above, you should have an updated Rmarkdown file for data processing that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

Our readme file is growing and we have quite a few elements of metadata yet to add. Rmarkdown's render function allows us to add some dynamic HTML to provide an interactive table of contents allowing quick navigation of the sections of the readme file. This feature is ultimately accomplished by pandoc adding Javascript code to the the HTML file (5:34 min).

After the video above, you should have an updated R Script for rendering the readme file that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

The next metadata category that is a high priority for allowing reuse of a data product describes how the data were acquired and processed into the product. EML uses the intuitive term methods to refer to this category (19:42 min).

After the video above, you should have an updated Rmarkdown file for generating a readme file that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

While we have covered the detailed metadata that should make our data product more interoperatble and reusable, we are still missing some elements that are geared for improving findability. The next video discusses adding a title, abstract, keywords, and purpose statement that would help a potential user quickly locate the data product and then quickly decide if it is appropriate for their use (12:51 min).

After the video above, you should have an updated Rmarkdown file for generating a readme file that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

The last major category of metadata to cover is to provide information on who was responsible for collecting the data and generating the data product. First, we need to list the personnel responsible and describe their role using the CRediT taxonomy, https://credit.niso.org/ (12:48 min).

Any project that provided resources for generation of a data product should also be acknowledged (2:42 min).

After the videos above, you should have an updated Rmarkdown file for generating a readme file that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

Even if you don't care about if and how people reuse your data product, you should still put a license on the code and data to protect yourself legally. The two main protections you want are for liability if the code or data are misused and to allow you or anyone else to continue to use the code even if someone else includes it in a product they are selling (2:42 min).

After making the videos above, I realized that the location of the description of the readme.html file as a data entity was not at the appropriate level in the outline. Let's correct that issue (1:36 min).

Finally, let's add some technical information regarding our R session to the metadata, in case it is helpful to someone who is having trouble trying to rerun the code (6:06 min).

After the videos above, you should have an updated Rmarkdown file for data processing that looks something like the following. Note that I have added more thorough comments than those typed in the video. 

The following provides a brief description of the contents for each of the sections of a primary data product readme file introduced above. The formatting of the embedded preview may not be clear if viewed on a small screen. 

Link to a full page version of the template.

Download your own copy of the encapsulated HTML file here.

The following code is the Rmarkdown used to generate the readme template above. 

Complete pipeline

A general tutorial on markdown

A general tutorial on R data structures

A parallel data product developed with VS Code, Quarto, and python