NCDFUTIL: Marine, earth, and atmospheric modeling tools on Henry2 at NC State

Contact: Lisa L. Lowe, lllowe@ncsu.edu

Marine, earth, and atmospheric modeling tools

cmaq-libs

The cmaq-libs modules contain all the libraries/dependencies necessary to compile and run CMAQ (Community Multiscale Air Quality Modeling System). They do not contain CMAQ itself.

The following are instructions for running the CMAQ 5.3.2 Benchmark using the default module. It is a modification of this tutorial.



Instructions


Clone the CMAQ repo, and checkout 5.3.2 (that is what I tested this with):

cd /share/$GROUP/$USER

git clone -b main https://github.com/USEPA/CMAQ.git CMAQ_REPO

cd CMAQ_REPO

git checkout CMAQv5.3.2_12Oct2020



Set the CMAQ environment:

module load /usr/local/usrapps/ncdfutil/modulefiles/cmaq-libs/intel2018.4


Copy the Henry2 specific configure script:

cp /usr/local/usrapps/ncdfutil/cmaq-libs/intel2018.4/config_cmaq.csh .


The script has been modified from the original as such -

  • Sets 'compiler' to 'intel', as to make the command line argument unnecessary

  • Gets library paths as set from the CMAQ-libs module (e.g. "H_IOAPI", etc.)

  • Compile argument "-xHost" has been removed so jobs may be run on nodes without AVX2 instructions

  • Extra_lib "-shared_intel" has been added.


Run configure:

./config_cmaq.csh


Go to the scripts directory and build the model:

cd CCTM/scripts/

./bldit_cctm.csh intel


Copy the modified run script for the Benchmark. Places that were modified have "Henry2".

cp /usr/local/usrapps/ncdfutil/cmaq-libs/intel2018.4/run_cctm_Bench.csh .


The modifications were:

  • NPCOL and NPROW are left undefined in the run script so that they can be set in the submission script. That way, the run script does not need to be modified when changing the decomposition.

  • The mpirun command has been modified for our LSF (i.e. to not use -np).

  • The Benchmark data has been put in a space for databases, /gpfs_partners/databases/CMAQ, so the run script sets INPDIR to point there.


Copy the sample submit script and submit:

cp /usr/local/usrapps/ncdfutil/cmaq-libs/intel2018.4/submit_Bench.csh .

bsub < submit_Bench.csh


The optional argument of ptile ensures you get a node with at least 16 cores and the job is not spanned over more than one node. My run took about 15 minutes with the sample batch script. Output from the run script is written to cctm.log.