This page is intended to help people run the SEO image pipeline from their personal computer to get a first-hand experience of its execution and results. The pipeline is based on the DarePype package described at darepype.readthedocs.io, the SEO pipeline is available on github at yerkesobservatory/pipeline.
The simplest way to learn about the pipeline is by using the SEO_PipeColab notebook. It's on the google drive under
SEO_UChicago > PipeLine > SEO_PipeColab.ipynb
Use the latest version of this document (the filename has V4 or so in it for the version number). This and other colab notebooks are in the google drive folder at
This folder also contains other useful notebooks to run individual pipe steps.
Python (3.3 or later) must be installed as well as following packages - darepype, numpy, astropy, logging and configobj. To run some of the Stonedge Steps you will also need the ccdproc, drizzle, lmfit, requests, the PIL (or PILow), sep, astroquery and pandas packages.
All source code and config files can be found on GitHub at https://github.com/yerkesobservatory/pipeline. To run the pipeline following folders and files are required from the Git repo:
The pipeline code is under the folder source. You need to download the drp and stonesteps subfolders to get the parent classes and pipesteps.
A valid configuration file, i.e. pipeconf_stonedge_remote.txt. You might have to make a copy of that file and adjust it for your computer.
Download Astrometrica (optional) - steps given under Image Pipeline page. This software package is used to get WCS added files.
If using the command line, set the python path to include the folder that contains the drp and stonesteps subfolders. Command to use is - export PYTHONPATH=/Path/to/source/folder
If using an iPython shell, commands to use are - import sys and then sys.path.append('/Path/to/source/folder')
You can run the pipeline from the system shell or from the python command line.
To run the pipeline form the system command line, run the pipeline.py program. Use the following command:
python pipeline.py your_pipeconf.txt your_input_files.fits
There are several optional parameters, use
python pipeline.py -h
for a full list. For example to run all raw files in a folder you run
python source/drp/pipeline.py pipeconf_stonedge_remote.txt Examples/M81/*.fits
You can also run a single pipestep using the following command:
python stepfile.py -c your_pipeconf.txt your_input_files.fits
Again use
python stepfile.py -h
for a full list of parameters. You can run a single file or multiple files through a pipestep.
You can run the pipeline from a python shell using the following commands:
# Import the pipeline objectfrom drp.pipeline import PipeLine # Create a pipe object and set correct configuration pipe = PipeLine(config = '/path/file/name/of/pipeconf_stonedge.txt') # Run pipeline result = pipe(['data1.raw.fits', 'data2.raw.fits']) # Save the result result.save('output_filename.fits')
If you want to see the pipeline log messages, set up logging:
# import the logging libraryimport logging # Configure logging to print messages of level INFO logging.basicConfig(level = logging.INFO)
if you want to see more detailed messages use logging.DEBUG instead of logging.INFO.
You can also run individual steps from the python shell.
#Import pipedata and reduction step objectsfrom drp.pipedata import PipeData from drp.stepmystep import StepMyStep # Make a pipedata object and load data into it inputdata = PipeData(config = 'path/file/name/of/pipeconf_stonedge.txt') inputdata.load('path/file/name/of/input/file.fits') # Make a pipestep object mystep = StepMyStep() # Run the step on the data outputdata = mystep(inputdata) # Save the data outputdata.save('path/filw/name/of/output/file.fits')
Use pythondoc if you have further questions.
Can not find stonesteps module:
Make sure you set the path correctly (see Setting Paths section above).
December 2018: Rewritten by Marc Berthoud
August 2017: Page created by Atreyo Pal (copy of Pipeline internship 2017 summary attached below)