Home‎ > ‎

NetCDF Information Sources

NetCDF and Related Information Sources

Compiled by Ben Domenico
Last Edited: 2 April 2012

The NetCDF data model has many facets that can be arranged in several layers. As the netCDF become an integral part of the international standards for data systems, those facets and layers are described in many different ways so as to fit into each context.

Library of Congress Digital Preservation Standards Description

Among the best sources of concise descriptions of and pointers to additional information about the netCDF are those in the Library of Congress Digital Preservation web site.

  • NetCDF-3, Network Common Data Form, version 3
  • NetCDF-4, Network Common Data Form, Version 4
  • NetCDF-4C, Network Common Data Form, Version 4, Classic Model

NetCDF materials from the most recent Unidata netCDF training workshop

  NetCDF Online Workshop materials


The Unidata NetCDF Data Model Documentation, the syntactic layer

From the Unidata netCDF documentation which is found at:

http://www.unidata.ucar.edu/software/netcdf/

What Is netCDF?

NetCDF (network Common Data Form) is a set of interfaces for array-oriented data access and a freely-distributed collection of data access libraries for C, Fortran, C++, Java, and other languages. The netCDF libraries support a machine-independent format for representing scientific data. Together, the interfaces, libraries, and format support the creation, access, and sharing of scientific data.

NetCDF data is:
  • Self-Describing. A netCDF file includes information about the data it contains.
  • Portable. A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers.
  • Scalable. A small subset of a large dataset may be accessed efficiently.
  • Appendable. Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure.
  • Sharable. One writer and multiple readers may simultaneously access the same netCDF file.
  • Archivable. Access to all earlier forms of netCDF data will be supported by current and future versions of the software.
The netCDF software was developed by Glenn Davis, Russ Rew, Ed Hartnett, John Caron, Steve Emmerson, and Harvey Davies at the Unidata Program Center in Boulder, Colorado, with contributions from many other netCDF users.

The NetCDF Classic Data Model

The classic netCDF data model consists of variables, dimensions, and attributes.

This way of thinking about data was introduced with the very first netCDF release, and is still the core of all netCDF files.

nc-classic-uml.png
NetCDF Classic Data Model

In version 4.0, the netCDF data model has been expanded. See The Enhanced Data Model.

Variables

N-dimensional arrays of data. Variables in netCDF files can be one of six types (char, byte, short, int, float, double).

Dimensions

describe the axes of the data arrays. A dimension has a name and a length. An unlimited dimension has a length that can be expanded at any time, as more data are written to it. NetCDF files can contain at most one unlimited dimension.

Attributes

annotate variables or files with small notes or supplementary metadata. Attributes are always scalar values or 1D arrays, which can be associated with either a variable or the file as a whole. Although there is no enforced limit, the user is expected to keep attributes small.

Meteorological Example

NetCDF can be used to store many kinds of data, but it was originally developed for the Earth science community.

NetCDF views the world of scientific data in the same way that an atmospheric scientist might: as sets of related arrays. There are various physical quantities (such as pressure and temperature) located at points at a particular latitude, longitude, vertical level, and time.

A scientist might also like to store supporting information, such as the units, or some information about how the data were produced.

The axis information (latitude, longitude, level, and time) would be stored as netCDF dimensions. Dimensions have a length and a name.

The physical quantities (pressure, temperature) would be stored as netCDF variables. Variables are N-dimensional arrays of data, with a name and an associated set of netCDF dimensions.

It is also customary to add one variable for each dimension, to hold the values along that axis. These variables are called “coordinate variables.” The latitude coordinate variable would be a one-dimensional variable (with latitude as its dimension), and it would hold the latitude values at each point along the axis.

The additional bits of metadata would be stored as netCDF attributes.

Attributes are always single values or one-dimensional arrays. (This works out well for a string, which is a one-dimensional array of ASCII characters.)

The Enhanced Data Model in NetCDF-4/HDF-5 Files

The Enhanced Data Model in NetCDF-4/HDF5 Files

Files created with the netCDF-4 format have access to an enhanced data model, which includes named groups. Groups, like directories in a Unix file system, are hierarchically organized, to arbitrary depth. They can be used to organize large numbers of variables.

nc4-model.png

Each group acts as an entire netCDF dataset in the classic model. That is, each group may have attributes, dimensions, and variables, as well as other groups.
The default group is the root group, which allows the classic netCDF data model to fit neatly into the new model.

Dimensions are scoped such that they can be seen in all descendant groups. That is, dimensions can be shared between variables in different groups, if they are defined in a parent group.

In netCDF-4 files, the user may also define a type. For example a compound type may hold information from an array of C structures, or a variable length type allows the user to read and write arrays of variable length values.

Variables, groups, and types share a namespace. Within the same group, variables, groups, and types must have unique names. (That is, a type and variable may not have the same name within the same group, and similarly for sub-groups of that group.)

Groups and user-defined types are only available in files created in the netCDF-4/HDF5 format. They are not available for classic or 64-bit offset format files.

The Climate and Forecast Conventions: A Layer for Semantics:

From the CF conventions document:

http://cf-pcmdi.llnl.gov/documents/cf-conventions

This document describes the CF conventions for climate and forecast metadata designed to promote the processing and sharing of files created with the netCDF Application Programmer Interface [NetCDF]. The conventions define metadata that provide a definitive description of what the data in each variable represents, and of the spatial and temporal properties of the data. This enables users of data from different sources to decide which quantities are comparable, and facilitates building applications with powerful extraction, regridding, and display capabilities.

The CF conventions generalize and extend the COARDS conventions [COARDS]. The extensions include metadata that provides a precise definition of each variable via specification of a standard name, describes the vertical locations corresponding to dimensionless vertical coordinate values, and provides the spatial coordinates of non-rectilinear gridded data. Since climate and forecast data are often not simply representative of points in space/time, other extensions provide for the description of coordinate intervals, multidimensional cells and climatological time coordinates, and indicate how a data value is representative of an interval or cell. This standard also relaxes the COARDS constraints on dimension order and specifies methods for reducing the size of datasets.

Some key areas where CF conventions enable semantic information in netCDF files are:

coordinates

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coordinate-types

A variable's spatiotemporal dimensions are used to locate data values in time and space. This is accomplished by associating these dimensions with the relevant set of latitude, longitude, vertical, and time coordinates. This section presents two methods for making that association: the use of coordinate variables, and the use of auxiliary coordinate variables.

a controlled vocabulary for standard names

http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/19/cf-standard-name-table.html

data sampling categories

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#coordinate-system

and

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#discrete-sampling-geometries


Relationship to Other Data Models, e.g., ISO Coverages

S. Nativi, J. Caron, B. Domenico, L. Bigagli, “Unidata's Common Data Model Mapping to the ISO 19123 Data Model”, Earth Science Informatics, Volume 1, Number 2, September, 2008, pp. 58-78, ISSN 1865-0473, Springer Berlin / Heidelberg, 2008

It's available online in pdf at:


and html


Additional references in the literature:

S. Nativi, B. Domenico, J. Caron, L. Bigagli, “Extending THREDDS middleware to serve OGC community”, Advances in Geosciences, Journal of the European Geosciences Union. Page(s) 57-62. SRef-ID: 1680-7359/adgeo/2006-8-57. June 2006

 

S. Nativi, J. Caron, E. Davis and B. Domenico, “Design and implementation of netCDF Markup Language (NcML) and Its GML-based extension (NcML-GML)”, Computers & Geosciences Journal, Volume 31, Issue 9, November 2005, Pages 1104-1118, Elsevier Publication.


NetCDF in the vernacular of the NASA Earth Science Data Systems (ESDS) Standards Process Group (SPG)

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc

NetCDF Classic

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc/esds-rfc-011

NetCDF Climate and Forecast Conventions

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc/esds-rfc-021

NetCDF-4/HDF-5 File Format

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc/esds-rfc-022

HDF 5

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc/esds-rfc-007

DAP 2.0

http://earthdata.nasa.gov/our-community/esdswg/standards-process-spg/rfc/esds-rfc-004

NetCDF in the OpenGeospatial Consortium (OGC) arena

The OpenGeospatial Consortium has adopted netCDF with CF (Climate and Forecast) Conventions as an international binary encoding standard.   The OGC material regarding CF-netCDF can be found at:

http://www.opengeospatial.org/standards/netcdf

Related data models for scientific data

The THREDDS (THematic Real-time Environmental Distributed Data Services) web services for netCDF

http://www.unidata.ucar.edu/software/tds/

Comments