Flux extraction, aka 'reduction' performs two tasks: (1) it traces and identifies the fibers and orders in a 2D master flat and (2) performs both regular (aka 'box') and optimum extraction to produce 1D extracted spectra for 2D input spectra.
Tracing and identifying fibers and orders is done on a (preferably background subtracted) 2D master flat. This step needs to be done only once per flat and the results can be applied to all subsequent flux extraction steps for other data. The routine allows to specify which fibers are illuminated by flat light to minimize wrong order/fiber identification.
Box extraction is the simple summation of all spatial pixels in a given fiber/order combination. The trace of the 'box' is taken from the master flat field. Box extraction is performed for all types of input spectra, including darks (D). Optimal extraction is using the flat as a model for the spatial profile and applies the common optimal extraction algorithm with outlier rejection. This method is computationally expensive but yields good cosmic ray rejection and optimal SNR. Due to the steep edges of the spatial profile of MAROON-X's rectangular fibers, rejection is very sensitive to small (sub-pixel) mismatches in the position/illumination of the spatial profile. The optimal extraction algorithm has been adjusted to accommodate this feature and has parameters to tweak the result. Optimal extraction is per default only applied to fibers illuminated with flat (F) and science (O) input.
The procedure takes input either from (1) a small GUI, from (2) the command line, or (3) via an API call. The API call is just a wrapper for the command line arguments and was introduced to support external batch scripts for multi-threaded processing (file concurrent), see discussion below.
Camera arm specific config file.
2D (fits) or 1D (hdf) master flat field.
Raw (fits) science spectrum.
Optional: 2D (fits) master dark for the science spectrum.
Reduced 1D spectra in specified subfolder with same filename as the input data but of type .hdf.
Log file with ending .log
Depending on the setting of the debug parameter, a number of plots are displayed (see below).
GUI:
From the base directory (maroonx_reduce) call: PYTHONPATH=${PWD} python reduce/extraction.py
and fill in the fields in the GUI. GUI data can be saved (in extraction.ini) and are automatically retrieved when extraction.py is called without commend line parameters.
Command line:
with the following parameters:
Input:
-ds DARK_SCIENCE, --dark_science DARK_SCIENCE Filename or file pattern of dark frame(s). Filename is relative to path. If asterix is found,
it is interpreted as a pattern and a master_frame will be calculated.
-df DARK_FLAT, --dark_flat DARK_FLAT Filename or file pattern of dark frame(s). Filename is relative to path. If asterix is found,
it is interpreted as a pattern and a master_frame will be calculated.
-f FLAT, --flat FLAT Filename of flat frame. Filename is relative to path.
-s SCIENCE, --science SCIENCE Filename of science frame. Filename is relative to path.
Misc:
-dbg {0,1,2,3,4}, --debug_level {0,1,2,3,4} Debug level: {0,1,2,3,4} 0 no plots will show up. 1...4 - more important plots will show up.
Output:
-o OUTPUT_FILE, --output_file OUTPUT_FILE Path of HDF output file. Can be absolute filename, relative filename (to path) or just a directory.
If path is a directory, the output file is named like the science input file (with ending .hdf).
-over OVERWRITE, --overwrite OVERWRITE Overwrite output file if existing.
Configuration:
-flat_fibers FLAT_FIBERS, --flat_fibers FLAT_FIBERS Flat fibers to extract. If not specified all fibers expected by config are extracted.
Can either be specified as list of ints (e.g. '1 2 5') or as a range (e.g. '2-3').
-fibers FIBERS, --fibers FIBERS Input fibers to extract. If not specified all fibers found in the flat are extracted.
Can either be specified as list of ints (e.g. '1 2 5') or as a range (e.g. '2-3').
-oe OPT_EXTRACTION, --opt_extraction OPT_EXTRACTION Input fibers to perform optimum extraction for. If not specified only fibers with containing flat
or object spectrum are optimally extracted.
Can either be specified as list of ints (e.g. '1 2 5') or as a range (e.g. '2-3').
-sd SKIP_DARK, --skip_dark SKIP_DARK Input fibers for which the dark frame will NOT be subtracted. Useful for dark frames containing
simultaneous etalon (DDDDE).
Can either be specified as list of ints (e.g. '1 2 5') or as a range (e.g. '2-3').
-orders ORDERS, --orders ORDERS Echelle orders to extract. If not specified all orders found are extracted.
Can either be specified as list of ints (e.g. '90 91 92') or as a range (e.g. '90-92').
-c CONFIG, --config CONFIG Path to config file. See documentation for structure of config file.
-p PENALTY, --penalty PENALTY Multiplicative factor on the profile-mismatch-correction algorithm. Set to 0 to disable this feature.
-sc SIGMACLIP, --sigmaclip SIGMACLIP Sigma clipping threshold for optimal extraction.
-b BACKGROUND, --background BACKGROUND Background variance estimate in the absence of a dark.
API:
The following command line parameters are exposed as a function call execution_withparameters():-s, -c, -f, -dbg, -over, -ds -o
Example: PYTHONPATH=${PWD} python reduce/extraction.py -s '/data2/MaroonX_spectra_reduced/Maroonx_masterframes/202011xx/flats/20201119T18_masterflat_backgroundsubtracted_FFFFF_r_0000.fits' -f '/data2/MaroonX_spectra_reduced/Maroonx_masterframes/202011xx/flats/20201119T18_masterflat_backgroundsubtracted_FFFFF_r_0000.fits' -c '/data/MaroonX_spectra_reduced/Maroonx_configfiles/202005xx/config_r.hdf' -o '/data2/MaroonX_spectra_reduced/Maroonx_masterframes/202011xx/flats/' -dbg 0 -over True
Command line example call to reduce the 2D master flat to produce a 1D flat as input for subsequent calls where the fiber/order tracing will be skipped.
From the base directory (maroonx_reduce) call: PYTHONPATH=${PWD} python reduce/batch_reduce_SOOOE.py
with the following parameters:
Input:
-dd DATA_DIRECTORY, --data_directory DATA_DIRECTORY Directory for raw files. Default = '/data2/MaroonX_spectra/'.
-d DATE, --date DATE UTC date of observation, e.g. '20200901'.
-t EXPTIME, --exptime EXPTIME Exposure time in sec, e.g. '300'.
-nd ND_SETTING, --nd_setting ND_SETTING ND Filter wheel setting of dark frame in deg
-dkd DARK_DATE, --dark_date DARK_DATE UTC date of dark file, e.g. '20200901'. Default: --date
-dkt DARK_EXPTIME, --dark_exptime DARK_EXPTIME Exposure time of dark in sec, e.g. '300'. Default: --exptime
-fr RED_FLAT, --red_flat RED_FLAT Reduced master flat for red camera (hdf). Default: /data2/MaroonX_spectra_reduced/
Maroonx_masterframes/202011xx/flats/
20201119T18_masterflat_backgroundsubtracted_FFFFF_r_0000.hdf
-fb BLUE_FLAT, --blue_flat BLUE_FLAT Reduced master flat for blue camera (hdf). Default: /data2/MaroonX_spectra_reduced/
Maroonx_masterframes/202011xx/flats/
20201119T18_masterflat_backgroundsubtracted_FFFFF_b_0002.hdf
-cr RED_CONFIGFILE, --red_configfile RED_CONFIGFILE Config file for red camera. Default: /data/MaroonX_spectra_reduced
/Maroonx_configfiles/202005xx/config_r.hdf
-cb BLUE_CONFIGFILE, --blue_configfile BLUE_CONFIGFILE Config file for blue camera. Default: /data/MaroonX_spectra_reduced
/Maroonx_configfiles/202005xx/config_b.hdf
-c CAMERA_ARM, --camera_arm CAMERA_ARM Camera arm, e.g. 'b' or 'r'.
Output:
-od OUTPUT_DIRECTORY, --output_directory OUTPUT_DIRECTORY Directory for reduced files. Default = '/data2/MaroonX_spectra_reduced'.
Example 2: PYTHONPATH=${PWD} python reduce/batch_reduce_SOOOE.py -c 'r' -dd '/data2/MaroonX_spectra/' -d '20201120' -t 300
Example call for the multi-threaded collects and reduces all red SOOOE frames with 300 sec exposure time and saves results in default output directory (/data2/MaroonX_spectra_reduced). Assumes DDDDE master dark frames for the same date and same exposure time in '/data2/MaroonX_spectra_reduced/Maroonx_masterframes/20201120/darks/'
Please see extraction.py for in-depth documentation.
The batch routine only multi-processes files with the same exposure time taken in the same night. If multiple nights are specified, it used the same dark for all. This should be changed. As a first step, files with the same exposure time and ND filter setting but from different nights need to be grouped with the closest dark in time, which is not always taken with the same UT date. As a second step, the pre-sorting/grouping should also work for different exposure times and ND filter settings so that the script doesn't need to be called for each unique exposure, making use of the 40-96 cores available on the number crunchers.
We need a check that background subtraction with a DDDDE frame is matching the etalon line strength reasonably well. For example the median for all pixels that have counts over say 2000 in the raw spectrum of fiber 5 should be output and logged for each order for the raw-DDDDE frame. Raise a warning if over or under a certain threshold (see next point).
QC: Procedure should raise a flag if it fails or if certain thresholds in the statistics are exceeded (e.g., large number of rejected pixels and/or columns, flux mismatch between dark and science in fiber 5 and/or mismatch in ND filter setting).
Some statistics are output only on the command line. These could be either written into the fits header or into a log file.
Optimal extraction should also be performed for Iodine spectra ('I' in type descriptor). Fixed by Andreas Seifahrt.