Raw data management

The aim of this next section is to try to replicate the arrival of data into the NWCSAF-GEO software if it were being run operationally. As it is possible that you do not yet have a satellite dish receiving EUMETCast data this section will use archive data that has been prepared for you. This data can be downloaded using this link (here). The running of NWCSAF-GEO for this data will take approximately 6 hours (as this is the time period being covered), in line with simulating the use of NWCSAF-GEO operationally.

To replicate the arrival of data (both NWP and satellite data) we first need to move the test data into your safnwc directory and unzip it.

mv SWIFT_testdata_compressed.zip $SAFNWC

cd $SAFNWC

unzip SWIFT_testdata_compressed.zip

If we look in the new Operational_test_data directory that has been created we can see two additional directories

GFS_20180802  MSG_20180802

These contain GFS and Meteosat data for 2nd August 2018 1800 UTC to 2345 UTC (approximately 6 hours) and will be the source of the data being delivered to the NWCSAF-GEO software.

The next step is to create the scripts to manage the "arrival" of this data. These scripts will move the data files at appropriate times. Most of the required NWP data will be available to the NWCSAF software on opening, with a single initiation slot being added approximately 4 hours into the simulation. The MSG data will arrive every 15 minutes as would be the case during real life operation.

First, navigate back to your safnwc directory and create a new directory to contain the scripts that we will be writing.

cd $SAFNWC

mkdir scripts

In here we will be keep the scripts for both replicating the arrival of data and the general file management and automation needed for actual operational running of NWCSAF-GEO.

NOTICE: DO NOT RUN THESE SCRIPTS STRAIGHT AFTER CREATING THEM. COMPLETE THIS WHOLE SECTION AND ONLY RUN RAW DATA MANAGEMENT SCRIPTS WHEN ALL PREPARATIONS ARE COMPLETE AND YOU ARE READY TO RUN NWCSAF-GEO.

Moving the NWP data

Using vim create a new text file called GFS_move.sh

cd $SAFNWC/scripts

vim GFS_move.sh

in this file add the following lines (to create a bash script)

#!/bin/bash

src_dir=$SAFNWC/Operational_test_data/GFS_20180802

dest_dir=$SAFNWC/import/NWP_data

status_file=$SAFNWC/scripts/GFS_status.txt

while true;

do

   if [ ! -f ${status_file} ];

   then

      touch ${status_file}

      mv ${src_dir}/S_NWC_NWP_2018-08-02T00:00:00Z_*.grib ${dest_dir}/.

      mv ${src_dir}/S_NWC_NWP_2018-08-02T06:00:00Z_*.grib ${dest_dir}/.

      mv ${src_dir}/S_NWC_NWP_2018-08-02T12:00:00Z_*.grib ${dest_dir}/.

      sleep 4h

   else

      mv ${src_dir}/S_NWC_NWP_2018-08-02T18:00:00Z_*.grib ${dest_dir}/.

      rm ${status_file}

      break

   fi

done

This script first defines the source directory and the destination directories, these reference the environment variable SAFNWC and so will automatically reference the directories in your safnwc directory.

Next a status file is defined (but not yet created). This is simply an empty text file, its presence indicates what stage the script has reached and will affect the behaviour of the while loop and if statement below.

Inside the loop an if statement checks for the existence of our status file, if it is not present then it is created and the a number of GFS NWP files are moved into the import/NWP_data directory ready for NWCSAF-GEO to map them on opening. The script is then told to sleep for 4 hours. After 4 hours this script will enter another iteration of the while loop. This time on checking for the existence of our status file, it will be found and the final set of NWP data will be moved into the import/NWP_data directory.

This delay will replicate the arrival of new NWP forecast data as would occur in the operational running of NWCSAF-GEO.

Before this (and any other script can be run) it has to be made executable. To do this use the chmod command as follows.

chmod +x GFS_move.sh

Remember, do not run this script yet!

Moving MSG data

Now, before we begin to create the script to drip feed the MSG files into the NWCSAF-GEO system we should consider how these files arrive via the satellite antenna. The majority of the MSG files are compressed and as such cannot simply be supplied to the software. Therefore we should create a directory in which to store our compressed files so that we can decompress any new files that arrive (using xRITDecompress). Navigate to your import directory and create our compressed satellite data directory

cd $SAFNWC/import

mkdir compressed_Sat_data

now navigate back to your scripts directory and create a new bash script called MSG_move.sh

cd $SAFNWC/scripts

vim MSG_move.sh

in this file add the following lines of code

#!/bin/bash

src_dir=$SAFNWC/Operational_test_data/MSG_20180802

dest_dir=$SAFNWC/import/compressed_Sat_data

tim=201808021800

while true;

do

   if [ ! ${tim} == 201808030000 ];

   then

      mv ${src_dir}/*${tim}* ${dest_dir}

      tim=$( date --date="${tim:0:4}-${tim:4:2}-${tim:6:2} ${tim:8:2}${tim:10:2} + 15 minutes" +'%Y%m%d%H%M' )

      sleep 15m

   else

      break

   fi

done

This script first defines the source directory and the destination directories, these reference the environment variable SAFNWC and so will automatically reference the directories in your safnwc directory. Next a time variable (tim) is created and set to a value of 201808021800 (the first time slot of the tutorial data).

Inside the loop, an if statement checks whether the variable tim is equal to 201808030000 (the time slot after the last in the tutorial data). If this is not the case then all the files with tim as a timestamp are moved into the destination directory, the timestamp (tim) is updated to be 15 minutes later than its current value and the script sleeps for 15 minutes. The loop will continue to increment the timestamp by 15 minutes and move files until it matches 201808030000 at which point the script will finish.

The effect of this is that the destination directory will receive a new time slot every 15 minutes in the same way as if data were being received via a satellite antenna.

Remember to make this script executable before you try to run it

chmod +x MSG_move.sh

Remember, do not run this script until you are ready to run NWCSFA-GEO!

The two scripts discussed above should not be considered necessary for the use of NWCSAF -GEO operationally but instead allow us to use archive data to replicate its conditions. Please now complete the following sections to be ready to run NWCSAF-GEO "operationally".