Pipeline Development

Alignment Test

Our Goal: to see whether DS9 realigns misaligned RGB images.

Test 1

Figure 1.

Screenshot of misaligned JPEG image from Pipeline.

Figure 2.

Screenshot of DS9 RGB image.

For this test we took a misaligned JPEG image of Messier 36 (fig. 1) and downloaded its FITS files:

  • 2018-02-11_GalaxiesLab6Group2_Messier_36_i-band_0.BDF.fits

    • 2018-02-11_GalaxiesLab6Group2_Messier_36_r-band_0.BDF.fits

    • 2018-02-11_GalaxiesLab6Group2_Messier_36_g-band_0.BDF.fits

We then created an RGB image on DS9 with these files (fig. 2).

Test 2

We were curious about the distance relationship (in pixels) between the misaligned JPEG images and the aligned RGB we created using DS9.

Figure 3.

Screenshot of misaligned JPEG image from Pipeline.

Figure 4.

Screenshot of DS9 RGB image.

We measured the distance between the misaligned stars in the JPEG and divided that by the width of the entire image to get .224, thus the images were misaligned by 22.4%. We then measured the distance between the frames of the DS9 RGB image, divided it by the width of one frame, and got the same result. They agreed to an accuracy better than 0.2% this is the same as the one pixel error margin in our measurements.

Observation and Results

By creating an RGB image on DS9 with the FITS files from the misaligned JPEG image, we found that DS9 realigns the three color images. We also know that the distance relationship between the misaligned images in the JPEG and the RGB frames created on DS9 is the same. This is very helpful to astronomers and scientists who look to create RGB images using DS9.

This test was conducted by Jack Adams and Thomas Curry during March of 2018.

Development of Master Bias, Dark, Flat

Things which should be in the output fits header

    • List of all input raw bias, dark, flats

    • Ambient Temperature

StepNIParent

    • How does the step get config?

        • command line (execfiles - make data object with only config).

        • Python comand line:

            • You always have to make a data object with config

            • OR Allow call to have None as data object only for NIParent but then it needs to be set in a different way (maybe setconfig for stepniparent only) but it breaks the idea that config always travel with the data.

        • What about pipeline with stepniparent.

    • How does pipeline.py have to change?

      • It has to allow operation without input files (only if mode is defined and first step is a StepNIParent child)

Goal for the pipeline is having a script with the following commands

python pipeline.py --pipemode=bias bla/pipeconfig.txt

python pipeline.py --pipemode=dark bla/pipeconfig.txt

python pipeline.py --pipemode=flat bla/pipeconfig.txt

Calibration

To install sextractor:

    • Download from www.astromatic.net/software/sextractor

    • Install the FFTW c library at http://www.fftw.org/download.html (or otherwise)

      • Make sure your compiler knows where it is (adapt paths as needed):

      • export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

      • export CFLAGS="-I/usr/local/include"

      • export CFLAGS="-I/usr/local/include -I/Users/berthoud/science/utils/lapack/lapack-3.8.0/CBLAS/include -I/Users/berthoud/science/utils/lapack/lapack-3.8.0/LAPACKE/include"

      • Make sure it's compiled in single precision (use ./configure --enable-float)

    • Need gfortran gcc.gnu.org/wiki/GFortranBinaries for compiling packages

    • Need the BLAS/LAPack libraries. See pheiter.wordpress.com/2012/09/04/howto-installing-lapack-and-blas-on-mac-os for Mac installation.

Instead of sextractor we could also use sep.readthedocs.io/en/v1.0.x/index.html, it may be easier to install.

Astrometry.net

Setup / Installation notes:

    • General information can be found at astrometry.net -- has links for project summary, download instructions, etc.

    • Use in the pipeline: pipeline gives original RA & Dec coordinates found in the image FITS header to Astrometry, which then processes the image to find its exact coordinates (the original coordinates given may not be exact), as well as a list of objects found in the field.

    • Required Python Libraries: go to http://astrometry.net/doc/build.html and look under "Prerequisites" and "Ubuntu or Debian-like systems." These sections list all the packages required to properly install Astrometry.

    • Installation (Linux): Follow initial steps outlined on astrometry.net/use. Once the program folder has been downloaded and unzipped, see http://astrometry.net/doc/build.html for further instructions; scroll down to "Getting/Building" for instructions on installation.

    • Installation (macOS): The astrometry.net documentation for this was last updated in 2012 tested on a vastly outdated software version. More recent instructions may be found here: https://www.ap-i.net/ccdciel/en/documentation/astrometry.net, although these instructions are not perfect either. For index files, it is more constructive to refer to the next step. Astrometry may install into the directory /usr/local/Cellar/astrometry-net.

    • Index Files: See http://astrometry.net/doc/readme.html for details about which index files are needed for download and where/how to install. Index files can also be found on the File Manager under stars/Research/Tools/AstrometryNet or under /data/filemanager/Research/Tools/AstrometryNet it's 32GB in total.

    • Config File: The astrometry.cfg file is used to specify the folder the program should look in for the index files. The correct file to edit is (in Linux) /etc/astrometry.cfg. A sample config file is also uploaded to the File Manager in the same folder as the index files.

    • Using Astrometry to solve image fields: See the README page (linked two lines above) for details on how to solve for different fields sizes, as well as what each resulting file is (under Solving and Output Files, respectively). Keep in mind that the program is easiest to use for solving FITS files, not JPG (but the commands are the same). Using the addition of --no-plots in the command may be necessary.

    • For using the Online API for astrometry.net use the yerkesstar@starsatyerkes.net account (and default password) and the API key is xpolczmnfaxzkihm (thanks Jorge).

Installation on stars:

    • The program is installed on /data/scripts/astrometry

      • The config file is under /data/scripts/astrometry/astrometry.net-0.70/etc???

      • The index files under /data/scripts/astrometry/install/data

    • The executables are under install/bin

    • To run stonedge images, run the program:

    • solve-field --batch --no-plots --overwrite --continue --skip-solved --index-xyls none --axy none --temp-axy --solved none --match none --rdls none --corr none --pnm none --downsample 2 --scale-units arcsecperpix --scale-low 0.9 --scale-high 1.1 [input FITS filename here] --out [enter output filename here]

Test of Different Parameters (compare with Emil's list):

astrocmd = '/data/scripts/astrometry/install/bin/solve-field --batch --no-verify

--crpix-center --no-plots --overwrite --continue --skip-solved --index-xyls none

--axy none --temp-axy --solved none --match none --rdls none --corr none

--pnm none --scale-units arcsecperpix --cpulimit 300 --guess-scale %s --out %s'

Emil's commands:

--sort-column FLUX --no-fits2fits --overwrite --downsample 4 --crpix-center

--scale-low %f --scale-high %f --scale-units arcsecperpix --ra %s --dec %s

--radius %f --cpulimit 5000 --no-plots

Emil's commands that stars doesn't have:

--sort-column FLUX --scale-low %f --scale-high %f

he usually uses scale range 0.5 . . . 2.0

--no-fits2fits (stars version doesn't know this command)

Stars commands that Emil doesn't have:

--no-verify (don't check existing fits header)

--continue (overwirte existing output files)

--skip-solved (don't re-solve - why do we need this?)

--crpix-center (force reference point to be in center for output)

--solved none --match none --rdls none --corr none --pnm none --index-xyls none --axy none

(don't save auxiliary files)

--temp-axy (write augmented xy list to temp file - why do we need this?)

--batch (run once for all files - shouldn't make a difference for us, we only submit one file at time)

--guess-scale %s --out %s (we have that instead of scale-low / scale-high)

List of downsample factors to try: downsample = 2, 4, 6, 8, 10

Things will be updated accordingly.

Question mark (?)