Summary
Summary
HDF5 has long been a standard for sharing scientific data across High Performance Computing (HPC) platforms. From the HDF website "HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5." Other data models, like NetCDF from UCAR, are built on top of HDF5 and are heavily used in oceanography, meteorology, and other earth science domains. Given this, many applications rely on having a usable working version of HDF5 that they can link into their applications.
The Portland Group Compilers are gaining popularity with the growing interest in accelerating HPC applications with Graphics Processing Units (GPUs). PGI compilers ship completely with the CUDA toolkit and a Fortran compiler (pgfortran) that support OpenACC and CUDA-Fortran, languages for writing code that operates on GPUs. This is a plus for Fortran programmers, who have a large presence in Department of Energy labs and in earth science departments around the world. Other options for GPU acceleration include FortranCL, CUDA (C/C++), OpenCL, and HIP. It is fair to say that the lack of activity on the main FortranCL repository for the last 6 years suggests the project has long been forgotten; this makes it unattractive for developers to latch on to this as a solution. The other languages are supported in C/C++ syntax and require Fortran programmers to develop C/C++ in addition to an ISO_C_BINDING layer to call their C-kernels from Fortran.
Building HDF5 with parallel and Fortran support with PGI compilers is not as straight-forward as building with other compilers, like GCC. I came across this issue while setting up clusters for hackathons, and defining build instructions for SELF-Fluids ( personal software project ). Through this experience, I have discovered that there are hints at the solution on the HDF Forums; in fact, this is where I found that a GNU C-preprocessor needs to be used in place of PGI's. This did enable a build of HDF5 with serial support, but the OpenMPI build that ships with PGI compilers cannot be used for parallel support, and instead OpenMPI was built from scratch.
This document provides details necessary for compiling HDF5 from source with the PGI compilers. HDF5 is built using an autotools build system. Template configure incantations are provided with the install notes along with expected output at the configure stage of the build. Ultimately, this was a roughly 16 hour exploration into this build issue that ultimately led to its resolution.
Install Notes
Install Notes
These notes cover how to install serial and parallel implementations of HDF5-1.10.2 with PGI-18.4.
Summary
Summary
In short, to build HDF5 with the PGI compilers ( v18.4 )
The C-preprocessor is replaced with the GCC preprocessor
At a minimum, the -fPIC flag must be added for C, C++, and Fortran compilers. Here, we have added “-fPIC -m64 -tp=px”, per Carl Ponder’s script that can be found on this forum.
When building in parallel, the precompiled OpenMPI libraries shipped with PGI community edition compilers cannot be used. To work around this, we’ve opted to compile OpenMPI (v3.1.1) from scratch.
Below, I’ll provide the arguments that need to be passed at the configure stage of the serial and parallel builds. For each build, I’ll provide the summary output that was obtained at the end of the configure stage.
Prerequisites
Prerequisites
HDF5 depends on zlib, and parallel builds depend on some flavor of MPI.
Serial Build
Serial Build
$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=pgcc CXX=pgc++ FC=pgfortran ./configure --with-zlib=<path/to/zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --prefix=<path/to/hdf5>
$ make
$ make install
Below shows the output of the configure stage when invoking a similar incantation to that shown above.
SUMMARY OF THE HDF5 CONFIGURATION
=================================
Compiling Options:
------------------
Build Mode: production
Debugging Symbols: no
Asserts: no
Profiling: no
Optimization Level: high
Linking Options:
----------------
Libraries: static, shared
Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib
Extra libraries: -lpthread -lz -ldl -lm
Archiver: ar
AR_FLAGS: cr
Ranlib: ranlib
Languages:
----------
C: yes
C Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgcc ( pgcc 18.4-0 64-bit target on x86-64 Linux -tp px )
CPPFLAGS:
H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L -DNDEBUG -UH5_DEBUG_API
AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include
C Flags: -fPIC -m64 -tp=px
H5 C Flags: -c99 -Minform=inform -fast -s
AM C Flags:
Shared C Library: yes
Static C Library: yes
Fortran: yes
Fortran Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgf90 ( pgf90 18.4-0 64-bit target on x86-64 Linux -tp px )
Fortran Flags: -fPIC -m64 -tp=px
H5 Fortran Flags: -fast -Mnoframe -s
AM Fortran Flags:
Shared Fortran Library: yes
Static Fortran Library: yes
C++: yes
C++ Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgc++
C++ Flags: -fPIC -m64 -tp=px
H5 C++ Flags:
AM C++ Flags:
Shared C++ Library: yes
Static C++ Library: yes
Java: no
Features:
---------
Parallel HDF5: no
High-level library: yes
Threadsafety: yes
Default API mapping: v110
With deprecated public symbols: yes
I/O filters (external): deflate(zlib)
MPE: no
Direct VFD: no
dmalloc: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Memory allocation sanity checks: no
Metadata trace file: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no
Parallel Build
Parallel Build
For the parallel build, you’ll first want to build OpenMPI with PGI. To do so, with CUDA enabled, you can do something similar to the following
Build MPI with CUDA support
Build MPI with CUDA support
$ CPP=cpp ./configure --with-cuda=<path-to-pgi-install>/linux86-64/2018/cuda/9.0 --prefix=<path-to-mpi-install>
$ make
$ make install
Note that PGI-18.4 ships with a CUDA-9.0 toolkit and you can use that installation to provide CUDA libraries.
The config summary is
$ Open MPI configuration:
-----------------------
Version: 3.1.1
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi, use mpi_f08
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: (none)
Miscellaneous
-----------------------
CUDA support: yes
Transports
-----------------------
Cisco usNIC: no
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): no
Intel SCIF: no
Intel TrueScale (PSM): no
Mellanox MXM: no
Open UCX: no
OpenFabrics Libfabric: no
OpenFabrics Verbs: no
Portals4: no
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: no
TCP: yes
Resource Managers
-----------------------
Cray Alps: no
Grid Engine: no
LSF: no
Moab: no
Slurm: yes
ssh/rsh: yes
Torque: no
Build HDF5 with parallel support
Build HDF5 with parallel support
Make sure that the previous build of MPI is in your path. Additionally, be sure to fill in the path to your zlib installation (--with-zlib) and where you want to install HDF5 (--prefix)
$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=mpicc CXX=mpic++ FC=mpif90 ./configure --with-zlib=<path-to-zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --enable-parallel --prefix=<path-to-hdf5-install>
$ make
$ make install
Below is the configuration summary for our build.
SUMMARY OF THE HDF5 CONFIGURATION
=================================
Compiling Options:
------------------
Build Mode: production
Debugging Symbols: no
Asserts: no
Profiling: no
Optimization Level: high
Linking Options:
----------------
Libraries: static, shared
Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib
Extra libraries: -lpthread -lz -ldl -lm
Archiver: ar
AR_FLAGS: cr
Ranlib: ranlib
Languages:
----------
C: yes
C Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpicc
CPPFLAGS:
H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L -DNDEBUG -UH5_DEBUG_API
AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include
C Flags: -fPIC -m64 -tp=px
H5 C Flags: -c99 -Minform=inform -fast -s
AM C Flags:
Shared C Library: yes
Static C Library: yes
Fortran: yes
Fortran Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpif90
Fortran Flags: -fPIC -m64 -tp=px
H5 Fortran Flags:
AM Fortran Flags:
Shared Fortran Library: yes
Static Fortran Library: yes
C++: yes
C++ Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpic++
C++ Flags: -fPIC -m64 -tp=px
H5 C++ Flags:
AM C++ Flags: -DOLD_HEADER_FILENAME -DHDF_NO_NAMESPACE -DNO_STATIC_CAST
Shared C++ Library: yes
Static C++ Library: yes
Java: no
Features:
---------
Parallel HDF5: yes
High-level library: yes
Threadsafety: yes
Default API mapping: v110
With deprecated public symbols: yes
I/O filters (external): deflate(zlib)
MPE:
Direct VFD: no
dmalloc: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Memory allocation sanity checks: no
Metadata trace file: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no
Page updated
Report abuse