Timeline

The purpose of this page is to keep track of all work done on the 1.5 m telescope, or any equipment used in association with the 1.5 m. This is not limited to CHIRON, but any work on SIMON, the dome, etc.


2015.06.29

Low count calibrations reported
It could be that something got misaligned when we fixed the decker motor. In this is the case, all position would have low counts. Is this the case, or is only the narrow slit?. 
I just checked and the reported position for the narrow slit is correct. I also moved it, so the motor is doing fine.
(Marco)


2015.06.27

EON calibrations stuck in Narrow_slit mode. 
We opened the box; the motor was indeed very stuck at the end. We were
able to move it manually, and after that started to work.

OK, so calibrations finished well -I see no errors in the slicer motor-. We just closed up everythign in the coude. It would be good if someone can check the images to be sure everything is still aligned -we did not touche the fiber, but we certainly needed to manipulate around, so it would be good to check.
If that's ok, observations should proceed normally tonight -except that the exposure meter computer we setup is not working yet, so exposure meter will need to wait a bit still.
(Marco)


2015.06.23

TelOps: Unfortunately yesterday's tests with the vintage generator were unsuccessful.  It had been running fine for several days without load, but only held up for 2.5 hours under the full load of the mountain before it  overheated and cut out; we are evaluating whether this can be fixed.  We  have also finally been able to track down a rental generator with the correct characteristics but it won't be in place for another 2-3 weeks. In the meantime we can no longer delay the 2000 hour maintenance on our main generator (its now pushing 2500).

We will therefore do this major maintenance on Thursday 25 June. The power will go of at around 9:30, and may be off for as long as eight (8) hours. Please prepare for this outage and let us know if you need any help with power down or power back up after this longer interval.


2015.06.22

TelOps: We will carry out maintenance on the generator again on the morning of Monday June 22. Preparatory work will begin at 9:00am, power will be switched off at 10:00am. Power should resume around 12:00 noon. 

During this shut down we will also be testing a second standby generator resurrected from our vintage equipment collection. If this is successful it will mean that, going forward, we will be able to do maintenance on our primary generator without having to cut the power to everything. Fingers crossed!

Meanwhile we continue to work to restore the commercial power.


2015.06.15

TelOps: The telescope worked without any problems. No errors or variations were observed in the position of the declination axis. 


2015.06.12/13

DEC slew clutch broke around midnight. - TelOps
Quote from Electroid - 14 weeks (process initiated by Victoria)
"We have repaired the system by changing a DEC clutch found in some spare parts. Telescope is aligned and in working order." - Jorge Briones/Cristian Diaz (still waiting for replacement)


2015.06.11

PLANNED Generator Maintenance, preparatory work began at 9:00am, power switched off at 10:00am local time. Power restored around 12:00pm local time.

"Work continues to resolve the problem with the commercial power system but it looks like this may be a lengthy process, during which generator maintenance will need to be carried out every ten days or so."


2015.05.20

PLANNED Generator Maintenance, preparatory work began at 9:00am, power switched off at 10:00am local time. Power restored around 12:00pm local time.

"Work continues to resolve the problem with the commercial power system but it looks like this may be a lengthy process, during which generator maintenance will need to be carried out every ten days or so."


2015.05.11

PLANNED Generator Maintenance, preparatory work began at 9:00am, power switched off at 10:00am local time. Power restored around 12:00pm local time.

"Work continues to resolve the problem with the commercial power system but it looks like this may be a lengthy process, during which generator maintenance will need to be carried out every ten days or so."


2015.05.08

PLANNED Generator Maintenance, preparatory work began at 9:00am, power switched off at 10:00am local time. Power restored around 12:00pm local time.

"Work continues to resolve the problem with the frequency converter with guidance from the original manufacturer."


2015.04.28

PLANNED Generator Maintenance, preparatory work began at 9:00am, power switched off at 10:00am local time. Power restored around 12:00pm local time.

"The transformer was delivered and installed as planned, however, unfortunately it appears that the frequency converter also suffered damage from the lightning strike which was not apparent from tests performed earlier. So we will have to continue running on generator while that is diagnosed and fixed. So there will likely be scheduled maintenance in another 10 days time."


2015.04.24

"The repair of the transformer has been completed and factory testing performed successfully. It is on its way up the mountain today , and will be installed over the weekend. However, there are several tests to be performed once it's installed, and there may still be surprises when we attempt to restore power. We are therefore scheduling another round of generator maintenance for Tuesday morning (2015.04.28). If we still do not have commercial power by then, this will require another complete power cut as for the last cycle. If we do have the commercial power back there should be no interruption at all." -Steve Heathcote


2015.04.17

PLANNED power outage. Maintenance work on generator. Preparatory work at 9:00am, power off at 10:00am local time.


2015.04.06

UNPLANNED power outage. Maintenance work on generator was planned for 150407, but failed unexpectedly on the 6th. Still awaiting reinstallation of step down transformer.  


2015.03.26

Major Electrical Storm: Lightening caused damage to main step down transformer, which was sent to Santiago for repair by the manufacturer. Transformer should be ready by around April 15th. 


2015.03.20

Narrow Resolution dipping below lower limit (150314, 150315, 150317, 150318).

20150301     Narrow     1534.00     136857.33 
20150302     Narrow     1522.00     137106.07 
20150303     Narrow     1569.00     138193.45 
20150304     Narrow     1540.00     136686.82 
20150305     Narrow     1526.00     136259.52
20150306     Narrow     1543.00     135752.63
20150307     Narrow     1537.00     135878.63
20150308     Narrow     1534.00     135879.56
20150309    Narrow     1565.00     136271.35 
20150310     Narrow     1474.00     135689.26 
20150311     Narrow     1517.00     135119.69
20150312     Narrow     1537.00     135700.49
20150313     Narrow     1563.00     136320.32 
20150314     Narrow     1544.00     134706.85 RED
20150315     Narrow     1524.00     134073.78 RED
20150316     Narrow     1505.00     135103.14 
20150317     Narrow     1538.00     134585.13 RED
20150318     Narrow     1537.00     134858.67 RED
20150319     Narrow     1545.00     135470.09 


2015.03.13

Missing reduced file in plan 287 distribution file. Turns out that the fiber decker observation was binned 3 by 1 instead of the standard 4 by 4 and was therefore not reduced via regular pipeline.

1187N Sco 2015OUT07:32:07.3681800.003 1normalfiber287


2015.03.07

Terrible Resolution after initial reduction. 

Decker# LinesResolution
Slicer517.6726794.23
Narrow517.3345335.64
Slit697.5048139.66
Fiber508.0013714.81

Bleeding Thar (1000), junked, rereduced, bary log updated.

Decker# LinesResolution
Slicer1564.0080233.47
Narrow1537.00135878.63
Slit1424.0096022.61
Fiber1038.0027445.98



2015.01.21

BON Calibs failure. From ctimac3:/var/log/BONCalibs.log:/mir7/scripts/bon_countdown.scpt: execution error: Terminal got an error: Application isn’t running. (-600)
/mir7/scripts/bon_countdown.scpt: execution error: Terminal got an error: Application isn’t running. (-600)
-rwxrwxrwx  1 observer  wheel  2871 Jan 21 15:02 BONCalibs.log
Last log in ctimac3:/mir7/logs/BONCalibs is 150119BONCalibs.log


2015.01.20

BON Calibs failure. From ctimac3:/var/log/BONCalibs.log: /mir7/scripts/bon_countdown.scpt: execution error: Terminal got an error: Application isn’t running. (-600)


2015.01.06

BON Calibs failure. From ctimac3:/mir7/logs/BONCalibs/150106_beg.log: ERROR111 writing message.


2014.12.15

TCS Failure/Maintenance:
Rolando Cantarutti
Javier Rojas

The TCS is working after this work.

The reason of the maintenance was a failure of the TCS. It was important to have Rolando because of his expertise as one of the architects of the TCS.
We could reproduce yesterday's failure. However we understand, that we can't reproduce it at will because it is related to processes inside the clock board we dont control.
We experienced last night three different problems;
-The Time&Frequency Processor Board (clock board) bc635  , which captures an IRIGB signal from the GPS and feeds the TCS with UT, was broken , sending garbage.
-Heurikon , the CPU of the VME, had troubles with loading his software because of an out of timing interruption. This interruption comes from the bc635 board before it can get initialized by the Heurikon. It wasn't possible to define if this "feature" is a malfunction, but the board is working.
-The power supply was unstable and working marginal. We changed it today.

-After several tries we changed the bc635 board by a spare board (last one) and it worked fine. Since we also had the problem with the power supply, we suspected first all problems were related to it.
-Checked many old Heurikon spares but none worked. Finally we went back to the original one and it worked after many restarts. There are a couple of Heurikon boards that we eventually can repair if we get the right RAMs (obsolete) but , we can't be sure they work satisfactory after this repairing.
The VME is consuming far more power as it should be. That is something we need to  look at. Best window  in summer.
This extra consumption could also be the reason for some power supply instability, what again can produce an instability in the system.    

Streaking Problems: There was streaking on the chip. First exposure of the night (ThAr: chi141214.1000.fits) was junked. 


2014.11.10

At the suggestion of Marco B., we added a "set shutter close" command to the beginning of the calibration Setup
(ctioe1:/apps/CHIRON/bin/geom_Setup)


2014.11.09

Streaking Problems: There was streaking on the chip again. First exposure of the night (ThAr: chi141109.1000.fits) was junked.
Missing EM: CHIRPS Target HR472, chi141109.1135.fits, No EM information.


2014.11.07

Streaking Problems: There was streaking on the chip again. First exposure of the night (ThAr: chi141107.1000.fits) was junked.
Missing EM:  No EM information, chi141107.1145.fits.


2014.10.10

IOS: The computer hosting the CHIRON Interactive Observing Script has been switched from ctimac2 to ctimac3. Matt skyped with the telescope operators to inform them of the changes and sent this email:

Hi all, 

John Brewer and I completed the transition of MySQL database replication from ctimac2 to ctimac3 yesterday. To access the nightly observing script go to the address ctimac3.ctio.noao.edu within Safari on ctimac3. 

I documented changes made to exoplanets, ctimac2, and ctimac3 on an internal wiki here:

Best regards,
Matt


2014.10.07

Lamps: The relays for the lamp box were replaced. Here was the email from Marco Bonati on the issue:

On Tue, Oct 7, 2014 at 12:10 PM,  <mbonati@ctio.noao.edu> wrote:
Hi,

WE just finished the replacement of the relays (all the work was done by
Victor, one of our electronic technicians). We just tested the
functionality (ThAr and Quartz on/off, darks, etc) and it seems to be
working fine. So there was no need to delay/skip calibrations (I did not
touch the enabling of the calibrations).
Now we just need to wait and see. I hope it makes a difference. We also
replaced a logic chip. THe only one we did not replace was a driver,
because we did not have spares. But those drivers are very robust -we have
never seen them fail-. Let's see.

We found as loose cable that carried 5 volts to an unused switch. However,
this would not explain the problem, as the loosing of the 5V (for example,
if it touched the body of the box) would have caused the lamps to go off,
not on.
Anyway we fixed that.

M.


2014.10.06

Streaking Problems: There was streaking on the chip again. First exposure of the night was junked.
Exposure meter: There were problems with the exposure meter again. Andy S. looked into it and responded with the following:

From: Andrew Szymkowiak
Date: Tue, Oct 7, 2014 at 10:29 AM
Subject: No EM Fits last night

The QC shows FITS info in any of the headers from last night.  The EM was working, as it was terminating exposures.  It looks like all the info is in my logs on the expmeter computer.  Again I'll say, I should finally write the program to read the logs and put the data into existing files...this might be the time to do it (but I'm kind of tied up today (Tues) so it won't be super soon...

Later (after a telecon) I'll try to look and see, in the other logs, if there is any symptoms about why the stuff wasn't being written through the socket stream to the Chiron DB machine...


2014.09.23

Andy S. noticed in the Quality Control pages that the guiding corrections were very high in RA and DEC on the telescope. Additionally, Carlos reported the guiding was not working, and he was guiding by hand. Andy asked the tel-ops people to have a look at the TCS. They repaired it, and the guiding correction plots on the QC page show improvement. Here is Esteban's summary below:

Today the telescope was inspected by mechanics and electronics and in some positions it was detected that the stepper motor doesn't move. Mechanics changed the belt and the motor by spares and they had to balance the telescope as well, we don't know exactly yet why or how it was modified.
Electronics checked the telescope movements and it looks OK, we have to keep on eye on this


2014.08.26

Javier repaired the dome shutter last night, after Carlos noticed it was having trouble opening. Copied below are Carlos' comments from the EON report

Between 19:10 to 19:40, there was a problem with the shutter. It was not opening specifically the left side. Due to that , we called to Javier Rojas who found the failure in the clamps. After that , it was fit properly.


2014.08.22

Marco, Javier, and Humberto worked on the comparison lamp box to see why the ThAr lamp is on at the same time as the quartz lamp. Here is a summary of his work

Today I came to the mountain to take a look at the comparisson lamp box. I
got help from Javier and HUmberto to disasemble it.
Unfortunately we did not find anything obvious. We checked all the parts
and signals. Just in case, wer eplaced two of the digital logic chips,
just in case. We asl left out the signals for the handpaddle (manual lamps
switch) just in case the problem was due to these inputs (accidentally
left on). I doubt this is the problem, but this all we can do by now. If
the problem continues, we can replace the actula solenoids, but this is a
major job since it requieres dismounting several pieces.

I also installed the switch that Matt sent. We left plugged there both
ctioe1 and expmeter60.


2014.08.19

Tel Ops worked on the telescope to do maintenance during the observer vacation from 2014.08.19-2014.08.20. During the maintenance, someone accidental used the manual switch to leave the quartz lamps on. As a result, they burned out. The afternoon calibrations for the quartz and iodine showed no signal because of the burnt out quartz lamp. The ThAr lamp never switched on because the quartz was turned on manually.

Marco communicated with the Tel Ops crew and they exchanged the burnt out lamp with a new one. Matt reviewed the counts for the new lamps calibration frames and saw that they were within 1% of the former lamp, and so no recalibration was needed.


2014.08.18

Problem with ThAr being on at same time as Quartz persisted. Bad frames were replaced.
chi140818.1005-1053.fits were replaced with chi140817.1005-1053.fits
chi140818.1094-1115.fits were replaced with chi140817.1094-1115.fits
chi140818.1172-1220.fits were replaced with chi140816.1179-1227.fits

logstructure and qc page recreated before reduction pipeline was run again


2014.08.17

The problem with the ThAr and Quartz lamps being on at the same time occurred again. This time, for frames 1182-1230, inclusive. They were exchanged with chi140816.1179-1227.fits. These frames were then renamed so their sequence numbers reflected those of the frames they replaced. For example, chi140816.1179.fits -> chi140817.1182.fits. After this, logmaker and chi_quality were run, and then the reduction code was run again.


2014.08.05

The reduction code crashed when running 140802. This was due to the ThAr and Quartz lamp being on at the same time. There is logic to prevent this situation from happening, and all logs show that CHIRON "thought" the ThAr lamp was on. Marco is going to investigate potential hardware failures. In the meantime, Imran used flats from 140801 to reduce data from 140802. This problem also occurred on 140804. 

2014.08.04

We continue to have troubles guiding with the 1.5 m. Andrei wrote to Esteban to see if he can look into the poor guiding performance some more. Here is a copy of his message:

 Dear Esteban,

   Our colleagues at SMARTS are concerned about guiding of the 1.5-m in 
declination. They noted that the telescope's response changes during the 
night, and suspect a backlash or balance problem that started in mid-July.

1. The backlash in DEC is eliminated by a special motor. Can you, 
please, check that this motor is on and operates normally? In the past, 
we had a problem when the backlash motor was turned off for a long 
period of time.

2. Is the telescope balanced? The instrument (SIMON) is asymmetric and 
requires 4 balance adjustments (two in each axis). In other words, the 
DEC balance should be reached with the telescope near-horizontal and 
near-zenith. Has this been checked?

Here is Esteban's response

Andrei

the DEC backlash motor is properly working and the balance was adjusted 
by mechanics, in fact SIMON is not being used, so for some time it is 
not being LN2 refilled and it affect the balance anyway.
They should check it again tonight

2014.07.31

We continue to have problems with the exposure meter not working correctly. After including many addition print statements to track down the error, Andy and Marco found that some messages being sent from ctioe1 to expmeter60 are not being received by expmeter60 showing what are most likely network issues. Matt sent down a spare fast gigabit ethernet switch to hook ctioe1, expmeter60, ctimac2, and ctioxb up to. Hopefully this will alleviate the network problems. 


2014.06.16

Marco, Humberto, and Javier Rojas restored communication of the temperature and pressure sensors for CHIRON to the software which communicates to it. A summary of their work is given by Marco in an email thread, copied below

(June 15)
For some reason yesterday the software could not talk to the iodine cell
omega controller. The controller was working, though, so I refered not
power-cycling the controller as this would cause the disturbance on the
iodine cell temperature.
Today I'll ask the electronics to dot he power-cycling. The issue is most
likey simply that the serial port on the controller is taken/hanged.

(June 16)
Today Javier and Humberto power cycled the omega controller, and restarted
the software that talks to it. After that everything is working fine, so
it was indeed the serial port hanged/taken on the controller. All is OK
now.


2014.06.06

Calibration frames showed very low counts for CHIRON. Marco and Andrei looked at the instrument, and noticed an electronics problem with the instrument that seemed to result from faulty wiring. He also made changes to the temperature inside CHIRON to keep it warmer in the winter months

Andrei's email is copied below:

"As Marco wrote, the CHIRON electronics on telescope was without power (as well as the guide camera). I did nothing, just on/off the box. Still no power. The it appeared "by itself", apparently after touching the cables. I suspect poor contact in the 110V power. David Rojas revised today.

Normally such situations will be evident by the loss of guiding signal from the camera. But not evident during calibrations. Some missing calibrations may find explanation there.

Temperature control: The CHIRON was 1deg lower than nominal, instrument heating power at 100%. The room was below nominal (at 21.5), and the room heater was "ON". The air temp. in the coude room was +10.5C.

I did two things:

1. Double the room heater power by switching on the 2nd element. We should swap from one to two elements regularly between summer/winter. Full power in summer over-heats the room.

2. Increase the enclosure setpoint from 22.0 to 22.5. With this, there will be less power needed for the instrument heater. At this moment (14:20) ctioe1 is offline (network problem), so I can't check the result of this action."


2014.02.05.
Humberto & Marco worked on the CHIRON dewar. His summary (send 2014.02.06) is below:

Yesterday Rodrigo went up to Tololo to perform the dewar works. The dewar was warmed up during the day yesterday,and it reached room temperature around 10:30 p.m. After that Rodrigo and Humberto proceeded to change the molecular sieve. The dewar was then left pumping overnight, and this morning, at around 9 a.m. The cooldown process started. At around noon the detector reached nominal temperature (-128 C). The pump was taken out of the dewar, and the pressure stabilized at around 1.5E-6
I just took some biases and images look normal. I'm measuring a read noise of around 7.7 e-, which seem a bit high, but I compared that with a bias taken the last observation night and it seems around the same. This may require some investigation, but by now the system is back and ready to be used normally.
Special thanks to Humberto and Rodrigo who stayed in the mountain until very late last nigh who worked to have the system ready for today.


2014.01.28:
A. Tokovinin repaired the CHIRON shutter. Find his report below:

The problem of CHIRON shutter was mechanical. The rubber pad that fixed its open position has moved, and that caused the shutter to stay open.
Such event was foreseen during the initial installation, the rubber was fixed by a droplet of glue. However, on October 4 2013, when we adjusted the shutter together with Peter, I moved the pad slightly and forgot to put the glue. The pad probably drifted slowly and eventually caused the problem, at first intermittent then permanent.

Now the rubber pad is replaced by a metallic flexible leaf, so this should not occur again. I tested ThAr exposures as short as 50ms, seem normal (test* in 140128/). I don't think we need to replace or modify the driver. If it "half-worked" even with such a mechanical fault, we have sufficient reliability margin.


2014.01.24:
continuing problems with the shutter. Quartz, Iodine and ThAr images were over exposed. In this case, calibration frames were noticed to be over exposed even in the BON calibration files. The QC page caught science exposures which had over exposed quadrants.


2014.01.23:
Over exposed ThAr images were noticed for data on January 4th. The shutter to Chiron was not closing, and images were being over exposed, and readout while the shutter was open. many thar calibration frames were lost (i.e. junked) because of this. Images after the BON calibration frames were noticed to be problematic. Peter worked on alternating the electronics to the shutter on 2014.01.23.


2014.01.09:
A. Tokovinin and Roberto Tighe worked on realigning the telescope. It was for 3 hours.( 21:30 to 00:30)


2014.01.06:

A. Tokovinin and prepared telescope to be worked on for realignment. Additionally, protective foam was installed around the CHIRON FEM fiber to protect it from becoming misaligned in the future.
Lastly, a an was installed to blow air below CHIRON to provide a more stable environment for the dewar (temperature fluctuations were previously noted).
more information from an email thread, message from A. Tokovinin sent 2014.01006 entitled "alignment of 1.5m (plan for Jan 9?) and CHIRON"


2013.12.27:
After it was noticed that the throughput of CHIRON dropped to lower levels than ever before, Andrei checked the fiber alignment. It was discovered to be misaligned and he fixed it. See email from A. Tokovinin with the subject "Checks on CHIRON" from 2013.12.27 for more details.


Comments