Example of reduction sequence

Step 1: Create Master Flatfield

1. Combine individual FDDDF and DFFFD frames into master frames to remove outliers (cosmics) and improve SNR.:
   maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/recipes/make_master_flats.py -dd /data8/MaroonX_spectra/20220523

2. Extract spectra from the master FDDDF and DFFFD frames (used for location and masking of orders/fibers for subsequent background subtraction). Use GUI version:
   PYTHONPATH=${PWD} python reduce/extraction.py

3. Copy hdf frames from ~repos/2022xxxx/ to masterframe directory (/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202205xx/flats/)

4. Fit background and combine frames to FFFFF master frames:
   maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/recipes/backgroundfit.py -dd /data8/MaroonX_spectra_reduced/Maroonx_masterframes/202205xx/flats/

5. Extract spectra from master FFFFF frames Use GUI version:
   PYTHONPATH=${PWD} python reduce/extraction.py

Step 2: Create Master Darks

Master darks are easiest created on a per-night basis. The code automatically recognizes all exposure times and ND filter settings and produces all master darks for a given night.

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/recipes/make_master_darks.py -dd '/data8/MaroonX_spectra/' -d '20220427'

Step 3: Extract flux from science frames

Flux extraction requires manual specification of a matching master dark. Hence flux extraction is per arm, per night, and per exposure time.

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/batch_extraction.py -c 'r' -fr '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/flats/20220323T23_masterflat_backgroundsubtracted_FFFFF_r_0001.hdf' -dd '/data8/MaroonX_spectra/' -od '/data8/MaroonX_spectra_reduced/' -e 0900 -t SOOOE -d 20220413

or, with a specific data for a dark (e.g., if non for the same UT date as the science frames exists):

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/batch_extraction.py -c 'b' -fb '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/flats/20220323T23_masterflat_backgroundsubtracted_FFFFF_b_0007.hdf' -dd '/data8/MaroonX_spectra/' -od '/data8/MaroonX_spectra_reduced/' -e 1800 -t SOOOE -d 20220413 -dkd 20220414

Step 4: Extract flux from etalon frames

Flux extraction in DEEEE frames for drift and wavelength calibration does not require darks and can thus be per arm for a range of dates.

maroonx@mxblue:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/batch_extraction.py -c 'b' -fb '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/flats/20220323T23_masterflat_backgroundsubtracted_FFFFF_b_0007.hdf' -dd '/data8/MaroonX_spectra/' -od '/data8/MaroonX_spectra_reduced/' -e '0022' -t 'DEEEE' -d 2022042* -sd 'all'

Step 5: Etalon line fitting in etalon frames

Etalon line fitting is a very computationally expensive process. It must be done per arm and for a range of dates.

maroonx@mxblue:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/batch_extract_etalon_positions.py -t 'DEEEE' -dd '/data8/MaroonX_spectra_reduced/' -fb '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/flats/20220323T23_masterflat_backgroundsubtracted_FFFFF_b_0007.hdf' -c 'b' -e '????' -d '2022042*'

Step 6: Etalon line fitting in science frames

Etalon line fitting is a very computationally expensive process. It must be done per arm and for a range of dates.

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python reduce/batch_extract_etalon_positions.py -t 'SOOOE' -dd '/data8/MaroonX_spectra_reduced/' -d '2022042*' -fr '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/flats/20220323T23_masterflat_backgroundsubtracted_FFFFF_r_0001.hdf' -c 'r' -e '????' -n 8

Step 7: Fitting a wavelength and drift solution to etalon frames

A wavelength solution, including instrumental drifts, is fitted to the DEEEE frames. This is done per arm and can be done for a range of dates:

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python analyze/recipes/batch_etalon_spline_wls.py -d '2022041[3-9]' -c 'b' -m True -dd /data8/MaroonX_spectra_reduced/ -o 'DEEEE'

Step 8: Transferring the wavelength and drift solution from etalon to science frames

The wavelength and drift solution is transferred from the etalon frames to the science frames using the etalon spectrum in the simcal fiber. This is done per arm and can be done for a range of dates:

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python analyze/recipes/batch_science_spline_wls_dynamic.py -d '2022041[3-9]' -c 'b' -m True -dd /data8/MaroonX_spectra_reduced/ -o 'SOOOE'

Step 9: Combining science fibers into virtual fiber 6

Combining the flux of the three science fibers to create a new, 'virtual' fiber 6. The wavelength and drift solution is transferred from the etalon frames to the science frames using the etalon spectrum in the simcal fiber. This is done for a range of dates:

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python analyze/recipes/combine_science_fibers.py -dd '/data8/MaroonX_spectra_reduced/' -d '2022041[3-9]' --max_clip 20

Step 10: Determine barycentric velocity based on photometer data

Barycentric velocity is determined based on photometer data and simbad information. The photometer data need to be first retrieved from influx. This is done once for a complete run. First set the port forwarding on mxred:
maroonx@mxred:~$ ssh -N -C -L 8086:127.0.0.1:8086 maroonx@128.171.88.21

Then edit the python script analyze/InfluxQuery.py for the desired data range and name of output pickle file. Execute via:
maroonx@mxred:~/Repos/maroonx_reduce$ python analyze/InfluxQuery.py

Finally, call the barycentric correction script. It is done per target and can be done for a range of dates. Target names not recognized by Simbad need to be replaced with a known name or target coordinates:

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python analyze/recipes/batch_barycor.py -dd '/data8/MaroonX_spectra_reduced/' -p '/data8/MaroonX_spectra_reduced/Maroonx_masterframes/202203xx/expmeter/expmeter032022.pkl' -d '20220*' --zp_frd 2.8 --zp_pc 16.0 -n 'TOI 1450' -sn 'LSPM J1907+5905'

Step 11: Producing pandas hdf files for distribution to external PIs

Reformatting the hdf files to simplified pandas store files (still hdf format) for external PIs. This is done for a range of dates and/or exposure times:

maroonx@mxcrunch:~/Repos/maroonx_reduce$ PYTHONPATH=${PWD} python analyze/recipes/batch_spec_to_pandas.py -dd /data8/MaroonX_spectra_reduced/ -d 20220602 -t 340

Note: If blue and red arm exposure times differ (e.g. for transit observations), use the blue arm exposure time.