Embedded Files
GSICS Software: My Project At ISRO
GSICS Software: My Project At ISRO
This blog details about my work as a Software Development Intern at the prestigious Indian Space Research Organization, Government of India on 'Developing GSICS software for satellite calibration'
What is GSICS?
What is GSICS?
1. Introduction to GSICS
1. Introduction to GSICS
GSICS is a system designed to ensure that the data collected from different satellites is accurate and consistent, initiated by WMO.
Satellites observing the same area may produce varying measurements due to differences in instruments, sensors, and data formats, such as recording different temperatures for the same location.
GSICS helps to calibrate these measurements, allowing scientists and researchers to trust that the data they’re using from various sources is comparable.
2. How Does GSICS Work?
2. How Does GSICS Work?
Data Collection: GSICS collects data from various satellites. This includes measurements from different sensors and instruments.
Calibration Techniques: GSICS uses several techniques to calibrate these sensors, like Inter-Calibration: Comparing measurements from different satellites to see how they differ and adjusting them to match.
Collaboration: GSICS involves collaboration among various space agencies, like NASA, ISRO, ESA and others. They share data and methods to improve calibration processes.
Data Distribution: Once calibration is complete, GSICS makes this data available to researchers and meteorologists so they can use the improved data in their work.
In summary, GSICS is all about making sure that satellite measurements are accurate and consistent across different satellites and sensors.
GSICS Backend Architecture
GSICS Backend Architecture
ISRO satellite to be calibrated: INSAT 3D/3R/3S (lower resolution GEO satellite)
Satellite to be calibrated against: MetOp (higher resolution LEO satellite)
- Preparation: [My Code]
config.ini: Configuration file for storing configuration settings in key-value pairs
Reading config.ini: Read using configparser
ParamInfo.py: Stores configuration parameters of config.ini for easy access and management.
MapInfo.py: Defines class to store metadata and properties related to mapping and satellite data.
- Downloading INSAT and MetOp Data: [My Code]
INSAT data - downloaded from ISRO Data Center MOSDAC; file format: HDF5 (.h5); transfer mode: SFTP.
MetOp data - downloaded from NOAA CLass website; file format: Native Binary (.nt); transfer mode: FTP.
Automated download processes via Python scripts.
- Creating files list: [My Code]
Made 2 lists in Python - one for INSAT files (INSAT_list) and other for IASI files (MetOp_list).
- The loop: [My Code]
Loop through every ISAT file and find the files satisfying Δt condition and call them valid_files.
Δt condition: MetOp files which were received ± 30 minutes from INSAT file.
If number of valid_files > 0: exit loop, else go to next INSAT file.
- Extracting files: [My Code]
INSAT .h5 files: Extracted using h5py, pandas and NumPy (developed by ISRO).
MetOp .nt files: Extracted via specialized Python script (developed by European Space Agency).
Multiprocessing: From multiprocessing library, I used Process to utilize multiple processors for extracting data from IASI files and Lock to synchronize access to shared resources.
- Collocation of MetOp data over INSAT data: [Senior Scientist's Code]
Collocation of the data was done by creating KD_Tree for INSAT and MetOp satellite points.
The collocation produces MetOp_over_INSAT.png image, in which the grey part is INSAT image, and the red lines is the path traced by MetOp satellite.
7. Analysis: [My Code]
7. Analysis: [My Code]
From the collocation data, I filtered out only the useful points using NumPy data handling techniques.
For the remaining data, I created 2 Pandas DataFrame, one for each satellite data.
Using NumPy, I calculated slope, offset, bias and root_mean_square_deviation from the 2 Pandas DataFrame.
Plotted the calculated parameters using matplotlib.pyplot.
8. Correction: [Senior Scientist's Code]
8. Correction: [Senior Scientist's Code]
Once the bias, offset etc are known, the final step is to correct the data.
Data correction is done using advanced mathematical and Physics calculations by Senior ISRO Scientists.
Ater bias correction plot looks like this:
Tech Stack
Tech Stack
1. Operating System:
Linux
2. Programming Language:
Most of the software is written in form of Python modules, except for some bash scripts.
3. Python Libraries:
Data Handling-
NumPy: For advanced numerical calculations, particularly during Analysis.
pandas: To create DataFrames for managing and analyzing satellite data.
h5py: For reading INSAT data from HDF5 (.h5) files
glob: To create lists of files (e.g., INSAT_list, MetOp_list) by matching patterns like date-time, satellite version, etc.
System & File Managment-
sys: To interact with the operating system, and handle command-line arguments
os: To manage files, directories, environment variables, and system processes
shutil: To create temporary config files (config_temp.ini) for each user instance
time: Controlling execution time during communication of Frontend & Backend
datetime : For handling various date-time operations
Parallel Processing-
multiprocessing: To utilize multiple processors for parallel-processing
Visualization-
matplotlib.pyplot: Used for generating and visualizing Analysis plots (Fig. )
matplotlib.colors: For managing color formats, ensuring proper visualizations in plots and images.
PIL: For handling and manipulating image outputs (such as collocation results, graphs, etc.)
scipy.spascipy.spampl_toolkits.basemap: To create advanced plots using basemap
Scientific Computing-
scipy.spatial: To generate KD-Tree structures, used in collocation of MetOp data over INSAT data
External Command Execution-
subprocess: To run external commands directly within the Python environment
Dynamic Importing-
17. imp: For dynamically importing modules as needed during runtime
Google Sites
Report abuse