Sky images are acquired at Astronomical Research Institute (ARI) and then transferred via the ARI FTP site to the University of Chicago where they are provided via the Yerkes Website under http://astro.uchicago.edu/yerkes/outreach/observing/iasc/ (username is iasc key is iascsearch). The alternative site is on the Stars server, located at: stars.uchicago.edu/science/iasc.
Set up and maintain the processes necessary to get ARI images efficiently and reliably distributed to the public.
- Transfer and run the program on Stars (Raki) - being set up at stars.uchicago.edu/science/iasc.
- Make small changes as suggested by Patrick (Raki)
- Add a check to maintenance to make sure the FTP tasks were completed successfully.
- From talking to Patrick:
- Need all files in ONE folder for zip files (some groups use 3, others 5 stacks)
- Zip filename should be Asteroidname-MMDD-NN.zip ex: 2012 LR1-0621-24.zip
- Add: Folder date and observation time (date and time) to the page
- List both dates in a certain night, not just date of following day (i.e. 28-29 instead of 29)
- Search for observations with RA/Dec / date / object name
- Automatically add position angle and speed to the object table
- Add a report/log page for problems
- Add YYYYMMDD-ariNN (where NN is telescope size in inch) to zipfilename and to top folder in zipfile.
- Rename all zip files to replace spaces with underscores _
- Transfer new files only to web
- Add the following check in maintenance.py: If there are RAW files in workpath/raw/telescope/today but no zipped files in webpath/grouped/telescope/today → raise an error
A python program at ARI is transferring the files via ftp to the ARI FTP site. The kronos server on the University of Chicago astronomy network then runs the getfiles script to download the images from the ARI FTP site. On this server the move2web/u_groupzip program is unpacking the images and creating one ZIP file with all the FITS images for each asteroid observation. These images are provided on the Yerkes Observatory website under http://astro.uchicago.edu/yerkes/outreach/observing/iasc/ (username is iasc key is iascsearch). To monitor the success, the maintenance program runs once a day and sends an E-mail if there is a problem with any of the components. All programs are described in greater detail under Programs.
The code is under:
/home/berthoud/asteroids/code
The downloaded zipped files are stored in the following folder structure:
/data/home/berthoud/asteroids/raw/NNinch/YYYY_MM_DD/NNNNNNNN.ObjectName.REDUCED.FIT.zip
The repackaged files are stored in the following folder structure:
/data/home/berthoud/asteroids/grouped/NNinch/YYYY_MM_DD/ObjectName.zip
The uploaded files are in:
http://astro.uchicago.edu/yerkes/outreach/observing/iasc/
which is under
/astro/www/groups/yerkes/outreach/observing/iasc on kronos. The files are protected using .htpasswd and .htaccess files. The users are ari and iasc.
Temporary files are in:
/data/home/berthoud/asteroids/temp/unziped/Filename.FIT - unziped raw files
/data/home/berthoud/asteroids/temp/group/ObjectName/groupI/Filename.FIT - files to be grouped
List files are in:
/data/home/berthoud/asteroids/lists/NNinch/YYYY_MM_DD.txt
The list files are tab delineated and the columns are ordered as follows:
Object Name Telescope Folder Date Object RA Object Dec Zipfile Name File Count UTC Date/Time.
Stars Server Folder Structure
Code:
/data/science/iasc/code
Downloaded zipped files:
/data/science/iasc/raw/NNinch/YYYY_MM_DD/NNNNNNNN.ObjectName.REDUCED.FIT.zip
Repackaged files:
/data/science/iasc/grouped/NNinch/YYYY_MM_DD/ObjectName.zip
Uploaded files:
stars.uchicago.edu/science/iasc/
/var/www/html/science/iasc
Temporary files:
/data/science/iasc/temp/unzipped/Filename.FIT
/data/science/iasc/temp/group/ObjectName/group/Filename.FIT
List files:
/data/science/iasc/lists/NNinch/YYYY_MM_DD.txt
Bugs / Problems / Solutions
- Problem: The repackaging of a day does not go well and the next days are corrupted as well. Fix: Remove all files in the temporary grouped and unziped folders → Re-run the u_groupzip command.
- Problem: Sometimes a file has no RA / Dec entry and crashes the u_groupzip. Fix: Remove the .zip file and re-run the program. Most files that generate this problem have filenames like NNNNNNNN.REDUCED.FIT. Resolution: This problem should now be permanently fixed, as all files missing the OBJCTRA keyword are ignored.
- Problem: Due to the large data volume the hard drives for Kronos and the webserver were filling up. Fix: Automatic file removal by the maintenance program have fixed that problem.
Please add an entry below whenever there is a significant change to the program. Newest entries should be first
- Updated to the new sfhartung.com server as a FTP source for the files. Added the 50inch telescope to the program. Marc Berthoud - October 2014
- Followed Patrick's suggestions (with an added component to the zip file names, as suggested by Vivian). Zip file names now show asteroidname-YYYYMMDD-NN.zip. Raki Dawis - July 2012
- Followed Patrik's suggestions. All reduced images are in one folder in the zip file. Raki Dawis - July 2012
- Revamped Grant's maintenance program. Easier to find program errors vs file errors. More structured. More consistent reading. Raki Dawis - July 2012
- Moved IASC file server to Stars Server. Raki Dawis - July 2012
- Fix problem with missing OBJCTRA keyword by ignoring those files. Marc Berthoud - February 2012
- New installation on Kronos. Data has been moved to /data/home/berthoud/asteroids folder. Marc Berthoud - February 2012
- Various problems with new files: - Marc Berthoud - 2011 September
- Files with missing OBJRA and OBJDEC crash u_grupzip.py
- Added Folder Date and UTC observation time to log files and the web page. - Grant Luberda - 2011 January
- Make sure that today's log file ends with **Finished overall if not send an E-mail - Grant Luberda - 2010 December
- Auto_Delete for files older than 2 weeks from the website, older than 60 days from kronos - Grant Luberda - 2010 November
- List information about each observation on the website with links to ZIP files - Grant Luberda - 2010 October
- Add possibility to store information about each observation (a file for each day for each telescope) - Grant Luberda - 2010 October
- 2010-December: Increased program reliability by only uploading log files after all of them have been completed.
- The program used is now move2web rather than movefileset2web.
- 2010-October: Added the creation of files lists. One file for each day for each telescope that contains the list of observed objects.
- 2010-August: Various debugging
- 2010-April: Initial Setup:
- u_groupzip identifies which files belong to the same observation, bundles and zips them
- movefileset2web just copies everything