Image Pipeline

Table of Contents:

Goal: Automatic creation of RGB images on the STARS server for incoming Stone Edge images.

Status: We currently have a working pipeline which reduces data every day at 11am CDT. The raw and reduced data can be accessed on the Stars server under stars.uchicago.edu/fitsview20. Data from the queue can be accessed under stars.uchicago.edu/astroclass. The pipeline usually works but if there's time there are a few things which could be improved (see task list below). To install and run the pipeline yourself see Pipeline User Manual.

Task List / Ideas

• • StepRGB:

    • Update stepsubgroup to be able to run bin1/bin2 images separately as well as images for each filter.

    • RGB Error FIX: Line 138 there is a certain case which generates IndexError b/c there are not enough files in the list.

    • Update steprgb such that selection criteria for R, G and B frame can be specified in the pipeline configuration.

    • For StepRGB add a parameter which specifies an optional folder where the jpeg is copied to.

    • Use improved algorithm (for example algorithm used by Trilogy) for image scaling

  • Make simple image shift to align all images for WCS.

  • Install tifffile library on server so .tif images can be created (better for Photoshop and such)

  • Add to stepastrometry: for images with too few stars, find a way to use other filter images to get accurate WCS.

  • Idea: Automatically post reduced RGB images on slack.

  • Add response to the \tostars command on izanma to give link where data is.

  • Web Viewer: For image list, add thumbnail of JPEG images if available. Also search images.

  • StepFluxCalSex: Exclude sources which have bad flags.

  • StepFluxCalSex: Find a way to calibrate images in unlisted bands (clear, Halpha)

  • Coadd Integration: Test stonesteps.StepCoadd to make mosaics and stack images. Integrate the step with the full pipeline. Image merging for multiple images of same bin, filter and exposure time.

Further Ideas:

• • Size Matching Pipestep: Right now, images with different dimensions won't be applied to each other (the step will be masked out). Implement a pipestep to resolve this issue (current ideas: take bin 2 images, and ‘cut’ each pixel into four, creating a bin 1 image, or take the average of 4 adjacent pixels in a bin 1 and average them in order to take an image from bin 1 to bin 2).

  • Retroactively updating stars database: After a new master flat is made, run the pipeline again for the observations taken closer to the new flat than the second-most recent master flat.

  • Include intermediate products to assess whether data reduction is working: Create hdu files containing bias, darks, and flats that can be assessed after a complete pipeline run

  • Use Sextractor Background for good Flatfielding: StepFluxCalSex uses sextractor which (as a byproduct) generates a background image (which is currently deleted by StepFluxCalSex). We could apply this background image as an additional background image.

  • Improved organization of Itzamna Reduced data: Before the data gets reduced it should be organized into separate folders, one folder per object and per observation. We might want to do that in PipeExecuteAutoDay.py.

  • StepRGB Improvements: There are a couple of small but significant problems with StepRGB. Firstly, the section that determines how many files in each band are being provided (and thus, which ones to use when stacking), is based on a series of if/elif/else statements intended to cover edge cases that do not actually cover all the edge cases. For example, in the case where no files with "iband", "rband", "gband" or similar are found, the files are all placed into a list called "other" - which is then never actually used. The logic of the if/elif statements is such that an error will be thrown earlier than necessary. Secondly, if the images are of different dimensions, StepRGB will not function, and will be unable to stack the images. This is particularly a problem when using files that have had drizzle applied, which will cover slightly different areas of the sky. Suggested improvements would be rewriting the section determining files with better logic, and implementing a way to somehow center and stack images that have different dimensions.

Notes / Usage

• • Click here to visit the page on the Stars Server .

  • File location on the Stars Server: /data/scripts/DataReduction

  • The files are on GITHUB under github.com/yerkesobservatory/pipeline.

  • How to execute the pipeline on the STARS server on SEO data: Simply type
    python /data/scripts/DataReduction/PipeExecuteAutoDay.py /path/to/desired/image/date/folder/
    there must be a space between the two entries. For example the folder can be
    /data/images/StoneEdge/0.5meter/2017/2017-04-01

  • Data for an entire year can be reduced with the command:
    python PipeExecuteAutoYear.py /data/images/StoneEdge/0.5meter/2016

  • Raw and reduced files: raw fits images are located in various folders under images/StoneEdge/0.5meter/year/date/etc. The reduced files are created in the same folder as the raw images. For the queue the raw and reduced files are under images/astroclass/user/observation

Pipeline Process

The pipeline runs all images through several steps. The input of the pipeline are RAW files from the telescope, the final output is a jpeg preview image. The following pipe steps are used:

• • StepAddKeys adds certain fits keywords to the header based on information in the filename.

  • StepBiasDarkFlat determines the most appropriate master bias, dark and flat files based on binning, date and, where appropriate, exposure time and filter. If no precise matche can be found, the closest file is used. These master files have already been corrected in the process which generates the master files. The bias is subtracted from the image, then the dark is subtracted. Finally the image is flat corrected using the normalized flat. For these operations the functions from the ccdproc library are used.

    • For HDR data which has a low and high gain image, StepHdr should be used instead of StepBiasDarkFlat. StepHdr uses different master flat and dark files, it also uses PFIT files.

  • StepHotpix identifies hot pixels and replaces them with an average of adjacent pixels. The step generates a blurred image using a median filter with a 2x2 mask. Each pixel more than 10 * stdev(rawimage - blurredimage) is replaced by the blurred value.

  • StepSrcExtPy uses the python SEP source extraction library to extract point sources in the image. The point sources are stored in the table in the output image.

  • StepAstrometry performs astrometrical correction by running the astrometry.net program. This program identifies bright stars in the image and compares their location with a full sky database to find the correct location of the image in the sky. If the data has been reduced with StepSrcExpPy the table of point sources is used for astrometry. There are two versions of StepAstrometry:

    • StepAstrometryLocal: This version of the step requires an installation of the astrometry.net software on the computer running the pipeline. See Pipeline Development page for information on installing astrometry. The advantage of using this version is that you can always run the software but it can be a challenge to install it.

    • StepAstrometryWeb: This version of the step uses the online astrometry.net tool to which it uploads the image (or the source table) to get the astometry. The advantage of this step is that you don't need to install the astrometry software but sometime the astrometry.net server is in high demand and may not process new requests.

  • StepFluxCalSex calibrates the flux of the image to janskys. This step determines the image brightness of bright stars in the image by using the SExtractor program. The step retrieves data from the Guide Star Catalog II using the StSci WebServices interface. The catalog brightness values are compared to the values from the image to fit the calibration factor which converts the image flux to Jy. This step generates a FCALplot.png plot that shows brightness data and the fit.

    • StepFluxCal: Does the same as StepFluxCalSex but it relies on the source extraction by StepSrcExtPy instead of using SExtractor.

  • StepRGB combines three images to make a color jpeg. The step makes an educated guess on which images to use. If less than 3 images are available, some image(s) are used for more than one color. It also adds text information to the image.

The data is saved after StepHotpix (HPX.fits), StepAstrometry (WCS.fits) and StepFluxCalSex (FCAL.fits). To see which settings are used for each step, look at the configuration file, pipeconf_stonedge_auto.txt, on github. For further details on the algorithms look at the pipeline code directly under github.com/yerkesobservatory/pipeline.

Master Bias / Dark / Flats

The master bias, dark and flatfiles are generated by the pipemodes mode_masterbias, mode_masterdark and mode_masterflat (see pipeconf_SEO.txt at github for more details). Each of these pipemodes uses StepDataGroup to split one nights calibration files by XBIN/EXPTIME/FILTER into groups of similar files which are then reduced with StepMasterBias, StepMasterDark and StepMasterFlat to make master calibration files.

Example Images

Development

Sample files and commands to use for development:

• • /data/images/StoneEdge/0.5meter/2016/2016-07-09/m57/m57_g-band_10sec_bin2_2016jul09_lindsaymberkhout_num2_seo.fits

  • /data/images/StoneEdge/0.5meter/2016/2016-07-09/m57/m57_i-band_10sec_bin2_2016jul09_lindsaymberkhout_num4_seo.fits

  • /data/images/StoneEdge/0.5meter/2016/2016-07-09/m57/m57_r-band_10sec_bin2_2016jul09_lindsaymberkhout_num3_seo.fits

  • /data/images/StoneEdge/0.5meter/2016/2016-07-09/m57/dark_m57_i-band_10sec_bin2_2016jul09_lindsaymberkhout_num5_seo.fits

Full command to run the pipeline with these files (must be in the /data/scripts/DataReduction folder).

export PYTHONPATH=/data/scripts/DataReduction/source

To run one pipe step: type "python", the name of the step file, specify the configuration file, list the file(s) you want to reduce.

python source/stonesteps/stepastrometrica.py --loglevel DEBUG --config pipeconf_stonedge_auto.txt

Examples/m57/m57_i-band_10sec_bin2_2016jul09_lindsaymberkhout_num4_seo.fits

To run the entire pipeline:

python source/drp/pipeline.py --loglevel DEBUG --pipemode stoneedge pipeconf_stonedge_auto.txt

Examples/m57/m57_i-band_10sec_bin2_2016jul09_lindsaymberkhout_num4_seo.fits

Examples/m57/m57_r-band_10sec_bin2_2016jul09_lindsaymberkhout_num3_seo.fits

Examples/m57/m57_g-band_10sec_bin2_2016jul09_lindsaymberkhout_num2_seo.fits

To test PipeExecuteAutoDay:

python PipeExecuteAutoDay.py /data/images/StoneEdge/0.5meter/2016/2016-10-13

Workarounds:

1. message=could not broadcast input array from shape (1024,1024) into shape (682,682), or (2048,2048) into shape (1024,1024) - this happens when different size images are in the same observation, a future version of the pipeline will fix this problem.

2. LoadHead: no such file or head read error, Unable to open Astrometrica output file = tmpe3t3iW.fits /data/images/StoneEdge/0.5meter/2017/2017-03-26/m51/tmpQ9bMbP.new. Pipeline for object = m51 returned Error. This happens when astrometrica can't reduce a file.

Progress / Log

A more detailed log can be found in the commits list on github.

• • 2020 Apr: Updated pipeline to run on darepype.

  • 2019 Jan: Added StepFluxCalSex to make fit plots and store data tables in the file. -- Marc

  • 2018 August: Master Bias / Dark / Flat implemented: The pipeline now makes masters files for each night. -- MMerz

  • 2017 August: Pipeline installation procedure and steps how to run it for one or more raw image on your own computer has been documented on new subpage Pipeline User Manual -- Atreyo

  • 2017 August: Created pipestep "BiasDarkFlat" which automatically calibrates WCS added FITS images using the day's bias, dark and flat frames so that the resulting JPG image is cleared of background noise and 'bad' pixels ---Atreyo

  • 2017 July: Updated the pipeline configuration file by adding a 'biasdarkflat' section and removing the separate sections for darks and flats. Also daterange is set to 0.5 days (before and after day of input file) ---Atreyo

  • 2017 July: Updated parent program loadaux to optimize searching for aux files (bias, dark and flat files) --Atreyo

  • 2017 July: Created script ExecuteAutoYear which can reduce raw data from StoneEdge for the entire year. 2017 successfully tested.

  • 2017 July: Improved automatic execution on Stars Server for incoming StoneEdge images at 11:11am--Atreyo

  • 2017 June: Atreyo Pal updated stepRGB to select images: It contains the image selection code in PipeExecuteAutoDay.py. Advantage now is that all images can be reduced and only the three relevant images are forwarded to StepMakeRGB

  • 2017 April: Joe Polk added a step to add missing keywords to the FITS header.

  • 2017 March: Fixed stepAstrometrica for variable format. Checked with DS9 that .WCS.fits images are indeed correctly stacked -- Marc

  • 2016 Fall: Pipeline moved to github, implemented stepastrometrica which generates .WCS.fits images -- Marc

  • 2015 July: Pipeline is automatically executed for all images at the end of every day. -- Ben

  • 2015 January: Pipeline execution has been updated to use the correct order of i/r/g-band filters if they exist (pipeline is optimized for use with i/r/g-bands since those seem to be the most common choice). Other combinations of desired filters can easily be added. -- NS

  • 2015 January: Pipeline now has two automatic versions: PipeExecuteAutoDay.py and PipeExecuteAutoYear.py. Day version goes through every object folder found on a specified day and reduces any compatible files found in it. Year version reduces images for every object from every day of the specified year. The Day version is the one that should be used normally. Execution instruction can be found in previous section. -- Neil

  • 2014 August: Edited code for manual operation so the pipeline can accept 1-3 files for processing (one makes grey-scale, two makes strange colors); otherwise an error message is printed -Neil

  • 2014 August: Added code that uses raw inputs for scaling values. Also made a copy of the execution code that runs the pipeline without raw input (automatically uses default values).

  • 2014 July: Installed the DataReduction folder on the Stars server for manual execution - Neil

  • 2014 July: Tested the pipeline (using an executable python file that requires only one command line) using various StoneEdge images - Neil

  • 2014 July: Took HAWC pipeline structure and wrote first version of StepHotpix and StepRGB - Neil