We use git for code management and version control. More information on git is available here. To get a copy of Jetstream on the HPC, you will clone a bare repository from the group’s central repository on the local system. On the HPC system, first tell git about yourself.
>> git config --global user.name "Your Name Comes Here"
>> git config --global user.email you@yourdomain.example.com
>> git config --global pull.rebase false
Then, get a copy of the code with,
>> cd
>> git clone -o central <your-oddjob-username>@oddjob.utias.utoronto.ca:/nfs/carv/d1/people/comp-aero/jetstream.git jetstream
You should now have a directory Jetstream which contains the source code.
There are two ways to compile Jetstream. The first uses a script included in the Jetstream directory called make_jetstream. This script compiles all of the third-party software packages Jetstream uses (except PETSc, as will be discussed later). This only needs to be used when first installing Jetstream, or after a major code overhaul for the group. During regular code development, you can simply issue a make command. Doing so only compiles the modified portions of Jetstream, and none of the third-party packages. Doing a make clean is sometimes required; this recompiles all of the Jetstream source. In either case, you may want to modify the file Makefile which contains compiler options. In the Makefile, make sure that the variable sys is set to the name of the system you are currently on, i.e. sys?=niagara for Niagara. The second thing which you will change from time to time is the version number, VER. This version number gets appended to the library names and the executable, so you can have different versions compiled simultaneously. There must be no spaces before or after the VER specification, e.g. 'VER= 111 ' is illegal. There are other flags in the Makefile, but these are more case-specific so will not be discussed here.
So, to compile Jetstream for the first time, do
>> cd
>> cd jetstream
>> ./make_jetstream
If successful, this should create the executable jetstream_x86_64 in your ~/bin directory with a current time stamp. Also, make sure that Jetstream has access to all the required libraries. Do
>> ldd ~/bin/jetstream_x86_64
and make sure that there are no lines saying "not found". If any libraries are not found seek help from the code administrator.
When compiling subsequent times (such as during development), do
>> cd
>> cd jetstream
>> make
If you are running optimizations for all but the simplest cases, you will want to use the third party package PETSc which is used to parallelize the mesh movement code. The PETSc installation is HPC-specific, and is described for all current HPC systems below.
Get a copy of PETSc by
>> cd
>> cp -r /home/z/zingg/treist/petsc-3.8.4/ .
To configure, do
>> cd
>> cd petsc-3.8.4
>> ./configure --with-debugging=0 --with-blas-lapack-dir=$MKLROOT
If that is successful, compile with
>> make PETSC_DIR=/home/z/zingg/<username>/petsc-3.8.4/ PETSC_ARCH=arch-linux2-c-opt all
Note: if you use 2019 compilers (NiaEnv/2019b), you will likely see a message at the end of the configure command that tells you to use PETSC_ARCH=arch-linux2-c-opt in the make command. If you use 2022 compilers (NiaEnv/2022a) however, you will likely instead see PETSC_ARCH=arch-linux-c-opt . If this is the case, you should use this value in the above make command instead, as well as in the .bashrc file.
Finally, add the following to your .bashrc file (make sure to source it after you do)
export PETSC_DIR=/gpfs/fs1/home/z/zingg/<username>/petsc-3.8.4
export PETSC_ARCH=arch-linux2-c-opt
PETSc should now be compiled and ready to use. To use PETSc with Jetstream, you must make Jetstream with petsc=on in the Jetstream Makefile. Do
>> cd
>> cd jetstream
(edit the Makefile to set petsc=on)
>> cd movegrid
>> make clean
>> cd ..
>> make
To run Jetstream with PETSc, set GRID%PETSC = .true. in the input.param file.
Future-Proofing: Although the above steps work just fine to get PETSc, here is a way to install it directly from source. Note we checkout to the same version as above. In the future if we wish to update the version of PETSc, these are the steps one would use.
>> git clone -b release https://gitlab.com/petsc/petsc.git petsc
>> cd petsc
>> git checkout 995a3e3744c
>> ./configure --with-blaslapack-dir=$MKLROOT --with-batch --with-debugging=0 --download-sowing-cc=icc --download-sowing-cxx=icpc
>> make PETSC_DIR=/home/z/zingg/<username>/petsc/ PETSC_ARCH=arch-linux-c-opt all