Environment variables and run script

There are a number of python scripts that have been written for use with NWCSAF-GEO data. This python code can be downloaded from the NWCSAF website. However, before we start doing that we will modify the setup of our computer so avoid conflicts between anaconda and the NWCSAF-GEO installation we have already created. When installing NWCSAF-GEO a number of software dependencies were included in the form of the COTS directory. This approach means that everything that is needed to run NWCSAF-GEO is supplied with its source code. However, it can cause some problems when we have to install new software such as Anaconda. To prevent this from occurring we will have to remove some of the lines relevant to NWCSAF from our .bashrc file and create some scripts that allow us to run NWCSAF-GEO without these environment variables being defined.

First navigate to your home directory and open your .bashrc file.

cd

vim .bashrc

Now remove the following lines from your .bashrc file.

export BUFR_TABLES=$SAFNWC/COTS/bufrtables/

export CC=gcc

export FC=gfortran

export CFLAGS="-O3 <openmp>"

export FFLAGS="-O3 <openmp>"

export ARCH=linux

export HDF5_PLUGIN_PATH=$SAFNWC/COTS/hdf5/lib/plugin/

export PATH=$SAFNWC/bin:$SAFNWC/COTS/bin:$PATH

export LD_LIBRARY_PATH=$SAFNWC/COTS/lib:$LD_LIBRARY_PATH

. .nwcgeo

Navigate to the scripts directory that we created before and create a script called SAFNWCTM.sh. In the text file add the following lines of code, save and exit.

#!/bin/bash

args=$*

export BUFR_TABLES=$SAFNWC/COTS/bufrtables/

export CC=gcc

export FC=gfortran

export CFLAGS="-O3 <openmp>"

export FFLAGS="-O3 <openmp>"

export ARCH=linux

export HDF5_PLUGIN_PATH=$SAFNWC/COTS/hdf5/lib/plugin/

export PATH=$SAFNWC/bin:$SAFNWC/COTS/bin:$PATH

export LD_LIBRARY_PATH=$SAFNWC/COTS/lib:$LD_LIBRARY_PATH

/bin/bash $SAFNWC/bin/SAFNWCTM ${args}

When called this script will read in all the arguments to the script (there are some useful options for running NWCSAF-GEO) defines the environment variables that we just removed from our .bashrc file and runs the SAFNWCTM command with the appropriate arguments. Now in order to be able to run NWCSAF-GEO as we have been doing, we must now add an alias to our .bashrc file so that when we use the command SAFNWCTM, our new script is called instead.

vi ~/.bashrc

Your .bashrc file should look something like this now

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

        . /etc/bashrc

fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:

# export SYSTEMD_PAGER=

# User specific aliases and functions

export TZ=UTC

export SAFNWC=/full/path/to/home/directory/safnwc

alias SAFNWCTM="/bin/bash ${SAFNWC}/scripts/SAFNWCTM.sh"

Installing Anaconda

To be able to use the python scripts provided by NWCSAF we will be installing Anaconda (a package manager, environment manager and python distribution). This will allow us to set up an environment that will allow us to simply run the scripts supplied by NWCSAF.

First of all visit the Anaconda website (https://www.anaconda.com/)

At the top right of the page click on the download link. Scroll down the page and select the Anaconda installer for Linux (either version will work, the python code from NWCSAF is in python 2 but I have always used the python 3 version of Anaconda in the past due to other code that I work with). This tutorial will use the python 3.7 distribution of the Anaconda installer. Click the download link and wait.

When the download is complete move the installer script into your home directory, change the permissions to make it executable and run the installer script.

mv Anaconda3-2019.10-Linux-x86_64.sh ~/.

cd ~/.

chmod +x Anaconda3-2019.10-Linux-x86_64.sh

./Anaconda3-2019.10-Linux-x86_64.sh

This will launch the Anaconda installer, you will be asked to review the License Agreement, confirm the location of the install (in your home directory) and initialize Anaconda.

When the installation is complete you have to source your .bashrc before Anaconda will work.

source ~/.bashrc

Creating an NWCSAF python environment

The next step will be to create a Conda environment that will allow us to include all the dependencies in the python code. A guide to managing Conda environments can be found here (https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).

We will use the environment we have already set up that we know works. This can be replicated by downloading the NWCPY environment yml file (here).

All you have to do to create your new conda environment is use the following command and all the packages included in the environment yaml file will automatically be downloaded and installed.

conda env create -f NWCPY_environment.yml

Now you should have a conda environment called nwcsaf capable of plotting all the different types of NWCSAF-GEO output.

Finally you should also make sure that you have imagemagick installed on your machine to prevent convert errors from the python scripts

To do this you will require root access (sudo)

sudo yum install php-pear php-devel gcc

sudo yum install ImageMagick ImageMagick-devel ImageMagick-perl

This will not only mean that you will not get error messages when running your python code but will also mean that you will be able to simply visualise the .gif files created using the display command.

Setting up the NWCPY scripts

Go to the NWCSAF web-page and sign in (http://www.nwcsaf.org/). Hover over the Software drop down and select Extra Tools from the menu. Here you will see a link to download the NWCPY tool. Download the latest tarred and zipped NWCPY file.

Our aim here is not to teach you how to use python. However if you already know some python then feel free to open the scripts and investigate the way in which these scripts access the data and generate imagery.

The scripts are currently in a .tgz file. Move the tar file to your safnwc directory and and untar it there.

mv NWC-CDOP3-GEO-AEMET-SW-CODE-NWCPY_v2.0.tgz $SAFNWC

cd $SAFNWC

tar -xvf NWC-CDOP3-GEO-AEMET-SW-CODE-NWCPY_v2.0.tgz

A new directory should have been created named NWCPY. In here there are a number of python scripts for plotting different NWCSAF-GEO products. To see what products are available from NWCSAF-GEO you can check on line here (http://www.nwcsaf.org/web/guest/nwc/geo-geostationary-near-real-time-v2018).

To make sure that the scripts work as expected there are two things we need to do before we start.

1. Set the NWCPY environment variable in the .bashrc file to the NWCPY directory we just created.

Open your .bashrc file vim ~/.bashrc and add the following line.

export NWCPY_PATH=$SAFNWC/NWCPY

Once you have updated this either source the bashrc file in your current terminal or close your current terminal and open a new terminal for the environment variable to be set.

2. Activate the conda environment for NWCSAF that has the correct python version and required libraries to run the plotting scripts.

conda activate nwcsaf/

This conda environment has its python version set to python 2.7 and has the additions of matplotlib, NetCDF4, Basemap and h5py. To exit this environment we can use the command conda deactivate

Now we can start produce some plots based on the files we have generated "operationally".