Machine Gun Active Rectifier
September 20th, 2010
Updated May 22nd, 2012 (after receiving feedback from the technician at the center of the story).
One day I was out in the lab working next to a test cell where some of the other engineers were working on a new active rectifier design. An active rectifier is a power electronics device used to draw power from the utility grid in a coherent fashion to reduce the electrical pollution that gets put back out one the line.
Most non-techie, non-engineer types have probably never heard of an active rectifier, or even a rectifier for that matter. Until recently times active front ends were not very common in most household electronics because historically the increased performance was not justified versus the added cost relative to a passive rectifier. However in commercial and industrial applications they have been somewhat common for the 20 last years. The prototype in the test cell next to me was for a telecom application that had requirements that could me met only with an active topology.
The key point to take away regarding the active rectifier is that its a bit more complex than run of the mill power supplies. Unlike a simpler diode bridge based passive rectifier, where there is no active switching going on, in an active rectifier there are a lot more opportunities for stuff to go horribly wrong. This is doubly true during development. The nearby prototype rectifier had been well wrung out previously so I didn't expect anything exciting to happen that day.
It was just after lunch when I noticed a technician working in the test cell with the active rectifier. He making some hardware updates one of the engineers requested and after he was done he removed the EEPROM chip from the control board and plugged in an emulator. An emulator is a development tool which is basically a specialized cable that allows somebody to tether the control board to a PC so new code can be tried out and the internal variables of the processor can be monitored. Engineers use emulators during development then once the code is in a stable state the emulator is disconnected and the code is burned onto an EEPROM chip which is then plugged back into the board so the system can run on its own. When the technician removed the EEPROM then connected the emulator to the board he was basically getting the system ready for the engineer to work on new software development.
After the technician had the unit ready for development the engineer stopped out, made a few changes to the code, then downloaded it to the processor via the emulator. Shortly thereafter he was called away for some other issue (or maybe he was just going for a soda, I can't remember). On his way out of the lab he asked the technician to give the new code a try and said he would be back shortly. The technician let me know he was about to apply power then he hit the run button.
This particular topology of active rectifier had diodes in anti-parallel configuration with the power switches. The diodes provide a path to maintain the current flowing in the rectifier's input filter inductors when the power switches are cycled on and off during normal operation. Without these diodes the current in the inductors would create a destructive inductive voltage spike whenever the switches were turned off (this mechanism is how the voltage for spark plugs is generated).
During normal operation the power switches are modulated to boost the voltage to a set level. However the anti-parallel diodes act like a passive rectifier even when the power switches are not switching. This means as soon as the rectifier is energized the DC link capacitors see well over half their rated voltage with the unit just idling.
I knew something was up as soon as I heard a loud 60Hz hum sound coincidentally with the technician enabling the rectifier controls. Loud 60Hz hums are not at all uncommon in a power electronics lab but I knew that particular unit should not be drawing any significant current when it was first turned on because it really wasn't loaded yet. The technician and I had a few seconds to look up and over at the unit before the first explosion roared through the lab.The bang was loud enough to ring my ears and I was about ten feet away at the time. The technician was half way between me and the unit. The first bang visibly startled him and he started backing away. Then the rest of the capacitors started popping like a string of over sized M80's and as he turned to get away from the unit his feet got tangled up and the technician hit the ground. I still have the vision of him with his hands over his ears rolling away from the unit as smoke and sparks were pouring out of it. After about ten or so rapid fire explosions in the span of about four seconds the upstream fuses finally blew which cleared the fault.At first I was a considerably alarmed. I wasn't sure exactly why the technician hit the ground. I assumed the worse and that he had somehow gotten shocked or hung up on the unit. Once the fuse blew and he had a few seconds to recover he got back up from the ground under his own power. He was a bit disoriented but other then temporary hearing loss he was fine.What happened was the when the engineer had updated the code he had accidentally set the controls to regulate the DC bus voltage to zero. As mentioned above as soon as power was applied to the unit the inherent diodes of the power switches create several hundred volts on the DC link caps. The engineers code was basically violating the physics of the rectifier.
When the unit tried to run the internal control signs ended up in state they were never designed for and the switching commands of the controls ending up pumping up the bus voltage well above the rating of the DC link capacitors. As this was a prototype unit not all the protective faults were working correctly so the unit keep running in its out of control state.
The cap bank was made up of a dozen or so individual capacitors in a series parallel configuration. The weakest capacitor in the bank when first, followed by the next weakest, followed by the next, and so on resulting in the rapid fire, machine gun like explosions.
The technician spent the next day repairing the unit. All he ever got out of the engineer regarding the errant version of the control code was a muffled "oops." About a year after that incident the technician decided to change careers and go into IT.