DCMIP-2025 - Workflow

Running on izumi (when casper is down). Currently, this is only for tests 1 (all dycores) and 2 (only SE and FV3)

Logon to moffatt first using the following command:
ssh -X moffatt.cgd.ucar.edu (on Windows machines)
ssh -Y moffatt.cgd.ucar.edu (on Macs)
and authenticate with your CIT password and the DUO App
NOTE: You may need to add USER@moffatt.cgd.ucar.edu if your NCAR $USER differs from your local machine.
Then logon to izumi:

ssh -X izumi.cgd.ucar.edu (on Windows machines)
ssh -Y izumi.cgd.ucar.edu (on Macs)
and authenticate with your CIT password and the DUO App.

setenv PATH ${PATH}:/project/dcmip/scripts
Load the required python module with: module load lang/python/3.11.5 (Do this each time you log on)
Clone the case. The commands are given explicitly for each case ($NEW_CASE_NAME is already specified), so you don't need to change anything.

Test case 1 (gravity wave breaking) with SE ne30:
- - /project/amp02/dcmip2025/CAM_6_4_082_21052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_gw_break_se_ne30 --clone /project/amp02/dcmip2025/CAM_6_4_082_21052025/cases/dcmip2025_gw_break_se_ne30 --cime-output-root /scratch/cluster/$USER
Test Case 1 (gravity wave breaking) with FV3 C96:
- - /project/amp02/dcmip2025/CAM_6_4_082_21052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_gw_break_fv3_C96 --clone /project/amp02/dcmip2025/CAM_6_4_082_21052025/cases/dcmip2025_gw_break_fv3_C96 --cime-output-root /scratch/cluster/$USER
Test Case 1 (gravity wave breaking) with mpasa120:
- - /project/amp02/dcmip2025/CAM_6_4_082_21052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_gw_break_mpasa120 --clone /project/amp02/dcmip2025/CAM_6_4_082_21052025/cases/dcmip2025_gw_break_mpasa120 --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with se ne30:
- - /project/amp02/dcmip2025/CAM_6_4_082_21052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_se_ne30 --clone /project/amp02/dcmip2025/CAM_6_4_082_21052025/cases/dcmip2025_horiz_mount_flow_se_ne30 --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with se ne60:
- - /project/amp02/dcmip2025/CAM_6_4_082_21052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_se_ne60 --clone /project/amp02/dcmip2025/CAM_6_4_082_21052025/cases/dcmip2025_horiz_mount_flow_se_ne60 --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with fv3 C96 and no Coriolis force
- - /project/amp02/dcmip2025/CAM_6_4_083_22052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_fv3_C96_no_rot --clone /project/amp02/dcmip2025/CAM_6_4_083_22052025/cases/dcmip2025_horiz_mount_flow_fv3_C96_no_rot --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with fv3 C192 and no Coriolis force
- - /project/amp02/dcmip2025/CAM_6_4_083_22052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_fv3_C192_no_rot --clone /project/amp02/dcmip2025/CAM_6_4_083_22052025/cases/dcmip2025_horiz_mount_flow_fv3_C192_no_rot --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with fv3 C96 and with the Coriolis force
- - /project/amp02/dcmip2025/CAM_6_4_084_23052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_fv3_C96_with_rot --clone /project/amp02/dcmip2025/CAM_6_4_084_23052025/cases/dcmip2025_horiz_mount_flow_fv3_C96_with_rot --cime-output-root /scratch/cluster/$USER
Test Case 2 (gap flow and vortex shedding) with fv3 C192 and with the Coriolis force
- - /project/amp02/dcmip2025/CAM_6_4_084_23052025/cime/scripts/create_clone --case /home/$USER/dcmip2025_horiz_mount_flow_fv3_C192_with_rot --clone /project/amp02/dcmip2025/CAM_6_4_084_23052025/cases/dcmip2025_horiz_mount_flow_fv3_C192_with_rot --cime-output-root /scratch/cluster/$USER

In /home/$USER/$NEW_CASE_NAME/ run ./case.setup
Change xml settings using ./xmlchange or namelist settings in user_nl_cam if necessary (for higher vertical resolution, run time, etc.)
To compile the model, run qcmd -- ./case.build
Submit the job: ./case.submit
The output data will be written to your work directory: cd /scratch/cluster/$USER/$NEW_CASE_NAME where you can find a directory with the name of the configuration. Move into this directory and its 'run' subdirectory.
If you intend to conduct more simulations using the same case, make sure to rename your output, or it will be overwritten. For example, an SE ne30 mountain test case simulation will produce an output file like dcmip2025_horiz_mount_flow_se_ne30.cam.h0i.0001-01-01-00000.nc. If this data is describing the gap flow test case with the Coriolis force included, you might rename the file (using the Unix 'mv' command, i.e. mv [old_file_name] [new_file_name]) to: dcmip2025_horiz_mount_flow_se_ne30.cam.h0i.0001-01-01-00000_gap_cor.nc
Make any modifications to the test case (step 7). If you want to make xmlchanges, run ./case.build --clean-all before re-building the model. If making namelist modifications (in the user_nl_cam file) you only need to run ./case.build.

Making initial condition changes in izumi:

Copy across the initial condition file into your SourceMods:

Test case 1 (gravity wave breaking):
- - cp /project/amp02/dcmip2025/CAM_6_4_082_21052025/src/dynamics/tests/initial_conditions/ic_gw_break.F90 /home/$USER/$NEW_CASE_NAME/SourceMods/src.cam/ic_gw_break.F90
Test Case 2 (gap flow and vortex shedding) with SE:
- - cp /project/amp02/dcmip2025/CAM_6_4_082_21052025/src/dynamics/tests/initial_conditions/ic_horiz_mount_flows.F90 /home/$USER/$NEW_CASE_NAME/SourceMods/src.cam/ic_horiz_mount_flows.F90
Test Case 2 (gap flow and vortex shedding) with fv3 and no Coriolis force
- - cp /project/amp02/dcmip2025/CAM_6_4_083_22052025/src/dynamics/tests/initial_conditions/ic_horiz_mount_flows.F90 /home/$USER/$NEW_CASE_NAME/SourceMods/src.cam/ic_horiz_mount_flows.F90
Test Case 2 (gap flow and vortex shedding) with fv3 and with the Coriolis force
- - cp /project/amp02/dcmip2025/CAM_6_4_084_23052025/src/dynamics/tests/initial_conditions/ic_horiz_mount_flows.F90 /home/$USER/$NEW_CASE_NAME/SourceMods/src.cam/ic_horiz_mount_flows.F90

Make the desired modifications to the initial condition .F90 file.

Re-build the case and resubmit.

Regridding with izumi:

Create a new regrid script:

touch regrid.sh

Copy across the commands in the regrid_izumi.sh file in the github repository: https://github.com/ta440/DCMIP2025/blob/main/other_scripts/regrid_izumi.sh

Run the regridding file:

bash regrid.sh [native_grid (mpasa120, C96, C192)] [output_grid (1x1 or 0.5x0.5)] [output_file_to_regrid]

Important: remove the .nc extension when writing the output file name.

Visualizing unstructured NetCDF data with ncvis

There is a cobbled-together version of ncvis on Izumi

~zarzycki/ncvis/ncvis /home/aherring/cesm_inputfiles/topo/ne20pg3_nc3000_Co060_Fi001_PF_nullRR_Nsw042_20180606.nc

You need to login via proxyjumps
ssh -Y -J your_login_name@moffatt.cgd.ucar.edu your_login_name@izumi.cgd.ucar.edu

Downloading individual files from Izumi to your local machine:

Method 1:

scp -J USER@moffatt.cgd.ucar.edu USER@izumi.cgd.ucar.edu:FULL_PATH_TO_FILE.nc MY_LOCAL_FOLDER

Example:

scp -J zarzycki@moffatt.cgd.ucar.edu zarzycki@izumi.cgd.ucar.edu:/home/aherring/cesm_inputfiles/topo/ne20pg3_nc3000_Co060_Fi001_PF_nullRR_Nsw042_20180606.nc ~/Desktop

You will need to authenticate using your CIT password twice since it "hops" the scp command from moffat to izumi

Method 2 (potentially faster):

The machine goldhill has access to the same project/ directory as izumi. You can scp from goldhill to your local machine to directly transfer files without hopping through moffat. You'll have to move your output to the project/dcmip/$USER directory first.

scp USER@goldhill.cgd.ucar.edu:project/dcmip/$USER/FULL_PATH_TO_FILE_WITHIN_PROJECT.nc MY_LOCAL_FOLDER

Example:

scp nandrosk@goldhill.cgd.ucar.edu:project/dcmip/nandrosk/test.txt ~/Desktop

Note: If you're trying to download someone elses data you need to first ensure it has the right permissions. You can run

chmod 666 project/dcmip/$USER/FULL_PATH_TO_FILE_WITHIN_PROJECT.nc

Cloning: How to clone and run a test case

Logon to Casper using the following command:
ssh -X casper-login1.hpc.ucar.edu (on Windows machines)
ssh -Y casper-login1.hpc.ucar.edu (on Macs)
and authenticate with your CIT password and the DUO App
Create sub-directory "dcmip25" in your work directory: mkdir /glade/work/$USER/dcmip25/
Clone the case (all one command; find SRC_DIR, CASE_DIR, and CASE_NAME in flowchart below): $SRC_DIR/cime/scripts/create_clone --case /glade/u/home/$USER/$NEW_CASE_NAME --clone $CASE_DIR$CASE_NAME --cime-output-root /glade/work/$USER/dcmip25

You are free to choose $NEW_CASE_NAME which is the name of the cloned case.

In /glade/u/home/$USER/$NEW_CASE_NAME/ run ./case.setup
Change xml settings using ./xmlchange or namelist settings in user_nl_cam if necessary (for higher vertical resolution, run time, etc.)
For the SE-squall line test case only: run git config --global --add safe.directory /glade/u/home/nandrosk/StormSPEED_CAM6_076.01_05212025/libraries/kokkos

**This command should only need to be run once**

To compile the model, run qcmd -A UMIC0107 -q workshop -l walltime=01:00:00 -- ./case.build
Open env_mach_specific.xml with an editor, search for the variable TMPDIR and make sure it's set to your work directory /glade/work/$USER/

Note: Any time you run ./case.setup --reset, make sure to check TMPDIR as it might get reset to the default scratch directory.

Submit the job: ./case.submit
The output data will be written to your work directory: cd /glade/work/$USER where you find a directory with the name of the configuration. Change into this directory and its 'run' subdirectory.
If you intend to conduct more simulations using the same case, make sure to rename your output, or it will be overwritten. For example, an SE ne30 mountain test case simulation will produce an output file like dcmip2025_horiz_mount_flow_se_ne30.cam.h0i.0001-01-01-00000.nc. If this data is describing the gap flow test case with the Coriolis force included, you might rename the file (using the Unix 'mv' command) to: dcmip2025_horiz_mount_flow_se_ne30.cam.h0i.0001-01-01-00000_gap_cor.nc
If you need to re-build the model, run ./case.build --clean-all then go back to 5). Re-building is necessary whenever you change namelist or xml settings.

DCMIP Case Flowchart

Remapping: Moving the data to a lon-lat grid (FV3, MPAS)

Load the necessary module: module load nco
Take a look at the DCMIP2025 github repository:
https://github.com/ta440/DCMIP2025/tree/main/other_scripts
Locate the remapping algorithm in the DCMIP2025 repository: DCMIP2025/other_scripts/regrid.sh
In your home directory, create a Bash regridding script:

touch regrid.sh

Open your new remapping script using the editor of your choice. For example,

vim regrid.sh

Copy the contents of the regrid.sh script on the Github into your new remapping script. If you are working with MPAS, instead look at regrid_mpas.sh; otherwise, the interpolation for omega, w, theta will not be correct.
Alternatively, clone the DCMIP2025 Github repository in your home directory:

git clone https://github.com/ta440/DCMIP2025.git

and access the remapping script from ~/DCMIP2025/other_scripts/regrid.sh. If working with MPAS, use regrid_mpas_2.sh

Identify the native grid of your output from the following: mpasa120, mpasa60, C96, C192
These grids correspond to the following lat-lon grids (in degrees), respectively: 1x1, 0.5x0.5, 1x1, 0.5x0.5
Run the remapping algorithm in your run directory (don't include .nc in filename):

bash ~/regrid.sh [native_grid] [latlon_grid] [filename]

where native_grid = {mpasa120, mpasa60, C96, C192} , latlon_grid = {1x1, 0.5x0.5}

Example for C96 gap flow test case (in one line):

Go to the location that contains your output data on the /glade/work/$USER/XXX/run directory where XXX stands for the configuration.
In the example below this would be 'cd /glade/work/$USER/1x1 dcmip2025_horiz_mount_flow_fv3_C96_no_rot/run'

bash ~/regrid.sh C96 1x1 dcmip2025_horiz_mount_flow_fv3_C96_no_rot.cam.h0i.0001-01-01-00000_gap

--This produces a regridded file in the same directory:

dcmip2025_horiz_mount_flow_fv3_C96_no_rot.cam.h0i.0001-01-01-00000_gap.regrid.1x1.nc

Visualization

Once your model has successfully run and your data regridded to a lat-lon grid, it's time to visualize the data. For quick methods, command line tools like ncview or ncvis can show your output at a glance. For more sophisticated plots and customizations, use JupyterHub to open a Jupyter Notebook environment to make plots in Python. The NPL 2025a kernel on Casper includes many of the packages you may need (numpy, scipy, matplotlib, xarray, uxarray, metpy, etc.). Several example Jupyter Notebook have been made to make simple plots of model output for each test case at the DCMIP 2025 Github repository.

To see more tools on visualization and how to access the DCMIP Jupyter Notebook templates on Casper see Editors & Visualization .

If you're interested in visualizing output on the native, unstructured grids without remapping to a lat-lon grid, ask your mentors. Tools like ncvis on the command line and the uxarray Python library can handle data on unstructured grids.

An example running ncvis on casper: ~zarzycki/ncvis gw_break.cam.h0i.0001-01-01-00000.nc

Modifying the Case

Below is a quick overview of how to make common changes to the case. For more details to the CAM workflow see CAM Configuration Notes and for modifications specific to the test case, see their respective pages.

XML Changes:

Within your case directory you can query XML variables with ./xmlquery and change XML variables with ./xmlchange

Useful XML variables:

JOB_WALLCLOCK_TIME: Time allocated for the submission run. The wallclock limit is 12 hours, but for these ideal test cases wallclock times longer than 2 hours will likely not be necessary.

Ex: Change wallclock time to 1 hour: ./xmlchange JOB_WALLCLOCK_TIME=01:00:00

ATM_NCPL: The number of physics time steps within an Earth day. The physics time step is calculated as t_phys = 86400 s / ATM_NCPL.

CAM_CONFIG_OPTS: This setting is important for specifying ideal physics schemes (like Kessler or Held Suarez), using analytic initial conditions, and setting the number of vertical levels. These settings will already be set in the case, but if you want to make any changes to the number of vertical levels, the change must be reflected here as well.

Ex: For the squall line test case which uses Kessler microphysics and a 40 level vertical grid CAM_CONFIG_OPTS is set to:

CAM_CONFIG_OPTS="--phys kessler --analytic_ic --nlev=40"

Ex: for the gravity wave test case:

CAM_CONFIG_OPTS="--phys held_suarez --analytic_ic --nlev=88"

STOP_N: The length of the simulation in units determined by STOP_OPTION.

STOP_OPTION: The unit used by STOP_N to control the length of the simulation. For these test cases, we use nsteps so that STOP_N is the total number of physics time steps for the model run.

Ex: Suppose the physics time step is 30 seconds and STOP_OPTION = nsteps. Then STOP_N = 720 means the simulation will run for 360 time steps and corresponds to 180 minutes or 3 hours.

user_nl_cam Changes:

To make changes to model output or dycore-specific variables (time-stepping, diffusion, vertical grid, etc.), we edit the user_nl_cam file located in the case directory. You can see several default options used for the test cases at the DCMIP-2025 Github repository. For more details see the specific test cases or ask the mentors.
The CAM documentation for SE and FV3 is here: https://docs.cesm.ucar.edu/models/cesm2/settings/current/cam_nml.html
For MPAS consult the MPAS User's Guide https://mpas-dev.github.io/atmosphere/atmosphere_download.html
and the material from the MPAS tutorials: https://www2.mmm.ucar.edu/projects/mpas/tutorial/Howard2024/agenda.html

Initial Conditions Changes:

Modifying the test case initial conditions will require editing the ic_<test_case_name>.F90 file. This is currently in the source code in SRC_DIR that you cloned (see table from earlier) in SRC_DIR/src/dynamics/tests/initial_conditions/ic_<test_case_name>.F90, with test_case_name = gw_break for case 1, horiz_mount_flows for case 2, and squall for case 3.

To edit the initial condition, you need to clone the initial condition .F90 file into your <CASE_DIRECTORY>/SourceMods/src.cam/ directory. The goal of the SourceMods directory is to allow for changes in the CAM code without changing the original source code itself. Any changes of the variables in these SourceMods files will require a rebuild of the model. The workflow goes like:

Copy the initial condition script into your SourceMods:

cp SRC_DIR/src/dynamics/tests/initial_conditions/ic_<test_case_name>.F90 <CASE_DIRECTORY>/SourceMods/src.cam/

Edit the ic_<test_case_name>.F90 file in <CASE_DIRECTORY>/SourceMods/src.cam/ with your desired modifications.
Make any other XML changes you desire. Some XML changes like to CAM_CONFIG_OPTS require a rebuild, while others like ATM_NCPL or JOB_WALLCLOCK_TIME do not. CAM will typically give you a message if a rebuild is required for the changes to come into effect after you make an XML change.
Clean the case build.

./case.build --clean-all

Rebuild the case. This step may require as much as 10 minutes in some cases.

qcmd -A UMIC0107 -q workshop -l walltime=01:00:00 -- ./case.build

This command simply submits a job to Casper using the UMIC0107 project number that runs ./case.build. This avoids running the case build on Casper login nodes.

Make any desired changes to the namelist variables in user_nl_cam with your text editor of choice.
Preview your namelist variables and your run. It's good practice to double check your JOB_WALLCLOCK_TIME and STOP_N, especially if you expect your changes to require more computation time.

./preview_namelists

./preview_run

./xmlquery JOB_WALLCLOCK_TIME,STOP_N

Submit the case!

./case.submit