Burnt Plastic From Above

September 15th, 2010Early on at my first company out of college we worked in a smaller building. At that point we were still in start up mode so we didn't have lots of extra space. The engineering lab was carved up into a general open area with several test benches and 480V/240V 25A disconnects and a separate four test bay, higher-power area. The four test bays were arranged around one central pole that had the 400A 480V feed. That area reminded me of those quads of baseball diamonds you see in parks when you fly over a city. One day I was banging away out in the lab in one of the bays on some embedded C code. Back in those days we didn't do a lot of formal planning so a lot of the designs went directly from the engineers' heads into a working prototype, with little to no process or documentation. On that day I was in the mode where I finally had a very good idea of what everybody wanted, had thought it through in my head for a few days, and was ready to get the prototype working.The walls separating the four bays were only four feet tall so you could always see who was working in the other bays and keep an eye one each other. Safety- wise that was a good thing but having the "half-walls" seemed to multiply the normal lab distractions. As I was really into what I was working on that day my patience for sidebars was pretty low and every time somebody stopped out I got invited into random shoot-the-shit conversations.By early afternoon I was in "just leave me alone" mode and actively ignoring anybody who did not address me by my first name twice. I could hear one of the production techs behind and to my right working on re-wiring and testing that was at that time the largest inverter we had ever developed. It was two megawatts and it was huge. We staged it adjacent to the four test bays because the inverter section and switchgear would not fit in one test bay. We could not run full power on that unit in the building but we were still doing the initial development using the capacity we had available. We had a large tan colored auto transformer between the unit and the source to bump up the voltage slightly for testing. That auto transformer was in the test bay to my left and the big unit was staged to my right about 15 feet away. The switchgear for the unit was designed to have the power come in through the top so we built a wire-way that suspended the power cables in the air between the main disconnect and the unit. My development machine happened to be parked under that wire way that day.The tech working behind me was rewiring the switchgear attached to the big inverter, I think, to reverse the phase order of the source. I am not completely sure what he was doing but something went wrong and a phase to ground short occurred. I remember hearing the ominous 60Hz hum but I was so focused on what I was doing I kept typing away at my keyboard. Our lab was all developmental power electronics so squealing inductors and humming transformers were part of everyday life. When I heard the hum of the tan transformer I assumed they were doing overload testing so I thought it was no big deal.About five seconds after the transformer starting humming the situation escalated and demanded my attention. A ground wire in the over head cable bundle above my development machine came alive. The cable bundle was supported by four temporary support towers that were about eight feet tall. The span of wire I happened to be sitting under was about six feet in length. When the short occurred the first indication that something was horribly wrong was the ground wire started drooping. The heat produced by the short heated the copper wire, which expanded its length way more than I thought possible. The low point of the wire span dropped about three feet in as many seconds. It was about two feet above my keyboard at that point.

Right after the wire came down the real excitement started. The plastic insulation on the wire burst into smoke. It was terrifyingly beautiful because the entire span of the wire burst into smoke the exact same time. I would have expected a weak spot in the wire to start smoking first, then spread along the wire in random fashion but what happened could not have been more symmetrical if you planned it that way. Within the time it takes to snap your fingers the overhead wire went from dropping but still intact to completely in flames along the entire span. Within the next second, bits of carbonized wire insulation were raining down on my keyboard.

During this whole incident I was dumbstruck at my computer looking up. You would think that some sort of self preservation instinct would have kicked in and told me to move away from the melting, smoking wire but that wasn't the case for me that day. It was like those B grade horror movies where you sit there watching some poor schmuck about to be carved up my a knife wielding maniac and you sit there thinking "why don't you just run away?" or "don't open that door." I sat there looking at myself in the same way screaming "MOVE!" inside but nothing happened.

Luckily other people in the lab had it together. One of the other techs (Tong, great tech and really smart guy) hit the emergency off on the main breaker. Another engineer (Bob, always reliable) grabbed a fire extinguisher and came running. Bob must have been rather excited also because we never did find the pin from that extinguisher after he pulled it off the wall as he charged toward the fire.

After taking an extended lunch to let the smoke clear from the building we figured out what happened. Normally a short circuit should just trip a breaker or blow a fuse within milliseconds, long before any wires overload and start smoking. However in this case due to the auto transformer we had wired in between the source and the unit there was enough impedance in the circuit to keep the upstream breaker from tripping. Along with that the ground wire was not sized large enough to handle the fault current. It was the perfect storm of three mistakes that lead to a serious failure: transformer fusing incorrect, ground wire not large enough, and the short circuit caused by the technician.

For the next year I would occasionally find bits of that burnt wire in my keyboard. It was a good reminder about how quickly things can happen in a power electronics lab.