This artifact is a relic from Seymour Cray's attempt to build a multiprocessor Gallium Arsenide based supercomputer in the mid 1980's.
The project was started by Cray Research (who built the famous Cray-1), and was transferred to the Cray Computer Corporation before ultimately failing in the mid 1990's.
This little object may not look all that special, but is actually a most extraordinary work of creativity and electronic art. It reminds me of the Sutton Hoo Gold artifacts; they exist, but nobody really knows how they were made. So if you are curious as to how Mr. Gray could make a four cubic inch computer module containing over a thousand L.S.I. Gallium Arsenide chips, well over a hundred thousand connections, and used a kilowatt of power, please read on.
Included in these pages are images of my own example, a discussion of the truly remarkable technology used in its manufacture, and various transcribed texts from the original patent documentation.
The smallest squares are the outlines of the individual Gallium Arsenide integrated circuit chips (16 to 24 per PCB).
The bigger squares are the logic PCBs themselves. These are 1-inch square rectangles containing some 2,500 plated through holes, and have 10 (?) electrical layers.
The logic PCBs are mounted in 2 separate layers either side of a central sandwich of power and signal routing PCBs These boards are 4 inches square, and also serve to hold the whole module
together.
The white rectangles at the top of the module are 40 pin multiway connectors used to connect the module to the computer's wiring 'cortex'. Each module can have up to 48 of these connectors,
giving a total of 1,920 contacts. The gold-plated rectangles at the base are high current power connectors that provide the module with various DC voltages. (VP: 3.2V, VN:-2V)
It is popularly accepted by the people who knew him or know of his achievements, that Seymour Cray was quite simply a genius. Considered to be the father of the supercomputer, Mr. Cray was a quiet and conservative man who did not seek publicity, but preferred to work alone with just a lined writing pad and pencil. With these, the most simple of tools, he designed some of the World's most complex machines. He did this mostly from his home in Chippewa Falls Minnesota, while teams of engineers back at the factory checked his workings and converted them into the World's fastest 'number crunchers'. One only has to read a little about this very clever man to begin to appreciate his modest and down to earth approach to his life's work, and to become very
impressed.
But he is gone now; killed by a freak road accident, and all that is left of his unfinished work are a few artifacts in museums and private collections. There are however a number of web sites devoted to this man and his machines, and so my intention here is not to provide a life story of Seymour Cray or to describe in detail his various supercomputers, but to give to the reader as much information I have been able to obtain about the modules used in the Cray-3 machine, a sample of which I happen to have (just a module, not the computer sadly).
My interest in Cray's remarkable work started from simple web browsing, where by chance I stumbled across articles and images of the Cray-1 supercomputer. My curiosity was aroused as I had obviously almost completely missed what seemed to have been a most creative period in computer design, a golden age perhaps. The fact that Cray's machines looked so unusual and interesting immediately got my brain into gear to try and understand why they looked the way they did. Was this just a vacuous styling exercise or was there something much deeper at work here?
Like many a youngster back in the 1960's I was excited by the prospect of those mighty new 'electronic brains' and later in secondary school I took a course in computing (in 1967). And it is with a little antiquarian pride that I can admit to writing simple programs onto arcane items such as 80 column punch cards and 8 track paper tape during those still pioneering days of great expectation. I was though rather more interested in the engineering aspects of computers, less so in programming them, and for various reasons eventually lost interest completely.
An ambivalence towards computers has remained with me since those early days, and was often to be re-enforced during various traumatic 'computerizations' of my workplace during the 1980's. The mainframes nearly destroyed the company and the arrival years later of the PC seemed only to underline my disappointment with the rather pathetic achievements made by the computer industry so far. The new systems were better weren't they? (you will say yes...) And weren't we supposed to be having intelligent conversations with the things by now? Were IS the HAL 9000?
But wait, in the late 1970's this Seymour Cray fellow in America had actually been doing some very interesting things indeed...
So I spent time on the web, and was pleased to acquire from the Atomic Museum in Albuquerque a Cray-1 module (no not from the first machine gents, but thanks anyway), and also kept half an eye open on Ebay. Recently Tony Cole has been auctioning some more of his stock. What? A chance to own a Cray-3 module? An artifact from the great man's last completed supercomputer? Well why not...
So here we have the object itself; a little rather booring flat brown thing of squares and tiny connectors. To an untutored eye pretty much of a non-event, but with just a little reasearch it starts to take on the aspect of a relic from a lost super civilization...
So what makes it so very special? Well...
As is usual with Mr Cray's designs, each of his computers was a single minded state-of-the-art tour-de-force, and a highly elegant trapeze act between packaging and technology. This, the third iteration of his highly uncompromising philosophy was an attempt to provide to his established customers some 'more of the same', but with better memory management and much faster (8,000 MIPS) performance. He unfortunately did not win his Cray -1-2-3 'hat trick', because the end product of the lengthy and increasingly desperate Cray-3 project ultimately failed to find even one paying customer. But despite this minor drawback (!), the beast itself was indeed a wonderful
technological achievement; a machine of truly breathtaking circuit and packaging audaciousness. Mind you the Cray-4 was supposed to have been even better!
A few Cray-3 numbers:
Approximate number of logic printed circuit boards in each machine: more than 12 Thousand.
Approximate number of integrated circuits: more than 200 Thousand.
Approximate number of logic cells: more than 50 Million.
Approximate number of electrical interconnections: more than 18 Million.
Approximate current consumption: more than 22 Thousand Amps.
Approximate length of logic connecting wire: about 30 Miles.
Approximate volume of the supercomputer itself: less than a quarter of a Cubic Foot.
Total number of prototype mainframes produced: 7
At a cost of: 2 to 3 Hundred Million dollars?
Total number sold: none!
Consider from the above just two quantities; a machine of a quarter of a cubic foot in volume (think of a six inch high fish tank covering your desk) consuming a continuous twenty two
thousand amps of electricity. My rough calculations may be slightly in error, but even so...
I am not much interested in (or can understand) the 'computer' or software aspects of the machine, it was the fastest of its time (well at least when conceived) etc, but what excites me about this computer is the sheer bravado of Mr Cray's hardware. Here is where we can find some truly science fiction engineering. As an asside, it has always struck me how computer software has always been rather feeble in comparison to the emerging hardware. We do so badly need some
fundamental creativity in this area (when will we ever rid ourselves of Microsoft?).
But while the Cray-3 was a stunning achievement in technology, it was also a stunning commercial failure. For Cray Computer Corporation times and fashion changed rather too
rapidly; the cold war ended, the evil little PC got faster and Cray's machine took much longer to develop than was expected. A fairly standard story perhaps, but as they say; 'they don't make them like they used to'. Actually, I am caused to ponder if anyone at all could make such a thing as the Cray-3 computer again, as (naturally) many of those unique circumstances of its birth no longer apply now, and might never again.
And so perhaps this little, otherwise un-inspiring Cray-3 fragment might indeed be worth some serious contemplation.
In contrast to his previous designs, with this his third machine, Cray embraced the very latest and the most difficult of semiconductor technologies to provide a four times increase in processing speed. Back in the 1980's Gallium Arsenide semiconductors were to be found in the most critical
of high frequency applications, and were widely held to be a worthwhile new direction in the quest for faster electronics. Trouble was, anything much more complex than a single diode or transistor in this material was very difficult to make economically, and while slower silicon based semiconductors could with ease be produced in ever larger circuit densities, no one was producing Gallium Arsenide circuits in even medium densities, let alone in the large quantities that Cray needed.
Cray used this technology to 'expand the envelope', and his new 'Cray 3' machine was to be a remarkable combination of large scale integration Galluim Arsenide chips, the highest integrated circuit packaging density ever achieved, and a liquid cooling system to stop the whole lot from catching fire. The main guts of the new supercomputer was to be a series of logic module 'clusters' with a combined volume of just 405 cubic inches (depending on processor quantity required). These were to be immersed in shallow octagonal 'fish tanks' filled with 3M's inert 'artificial blood', that were quite compact enough to easily sit on an ordinary office desk. Though these particular fish tanks would consume some twenty thousand amps of electricity and dissipate some one hundred and thirty thousand Watts of power. So, as with other Cray supercomputers,
they would be rather like icebergs. As what you didn't see were the complex liquid cooling system and the epic power supplies, some of which was built into the decorative pedestal of the machine, but most taking up large amounts of space in rooms out of sight. Oh, and that cooling fluid had to be pumped through the fish tank at sufficient velocity to cause concern about it actually damaging the circuitry.
Seymour's logic wiring was also legendary, in the Cray-1 he used between 60 to 100 miles of hand wired twisted pair cable to connect all the 1,600 modules together. In that machine the
wiring was deep inside the processor, but in the tiny Cray-3 the wiring actually covered the processor rather like a tangled mass of wool. This compact wiring 'cortex' (having some 7,000
connectors and over 300,000 contacts) would be arranged to surround the half dozen or so module clusters of the computer itself. And while no signal wire was to be longer than 1 foot, he
still must have used approximately 30 miles of the stuff. Apparently these wiring mats were known to wave gently in the strong current of the cooling fluid, flowing through the modules at around 10 inches per second, 'like kelp in the ocean'.
The Cray-3 module itself (of which there were between 120 and 200 in a machine, according to the number of processors specified) has a unique structure, that may well prove to be the highest integrated circuit packing density ever achieved. Each 4 cubic inch module held up to 1024 (or possibly a few more) Gallium Arsenide logic chips, each of which contained between 200 and 300 logic gates. The logic is implemented with D-MESFET transistors and Schottky diodes, 'producing performance levels of less than 150 pico-second per gate propagation delay'. There are varying combinations of three types of logic cell used in each chip, in addition to a balanced
clock amplifier cell. The exact quantity and layout of the logic cells within each of the integrated circuits, are as individually required to make up each complete computer module. Each chip holds an equivalent of some 4,000 transistors and dissipates 0.8 of a Watt. The un-encapsulated 'nude' chips are mounted directly to the multi-layer PCB by a process of inverting and plugging them directly into a pre-drilled matrix of 52 plated through holes. The chip's gold lead-out wires
being arranged to point vertically straight up from the substrate's surface, and the inverted chip is simply pushed into matching plated through holes in the PCB. This is done by placing a 'caul' plate on each side of the board, and bringing them together. As the slightly long chip leadout wires meet the plate at the bottom of the through hole, they become crushed or crumpled slightly within the hole as the chip is seated home. And thus both make a good electrical connection and hold the chip firmly in place. The lead-out wires have truncated conical ends to help with the alignment. Each chip has in fact two rows of lead-out wires placed around its periphery (52 wires usually, with power input connection wires placed at the corners). These 'nude' chips are usually mounted sixteen to a one inch square module PCB, and the inert low viscosity cooling fluid easily flows around them. (Some PCBs actually hold more than 16 chips, as 2 smaller dies are sometimes fitted within a single IC space.) The printed circuit boards are mounted in a 4 by 4
matrix (rather like tiles) on to a central 4 inch square surface that contains power and signal routing multi-layer PCBs. Two layers of logic PCBs are placed each side of the central
distribution boards, and the whole 'strata' of a Cray-3 module is as follows:
Layer:1)Side A transparent outer insulating overlay.
2)Side A outer logic 4 X 4 PCB tile matrix.
3)Side A outer logic PCB cooling channels and flow blockers.
4)Side A inner logic 4 X 4 PCB tile matrix.
5)Side A inner logic PCB cooling channels and flow blockers.
6)Side A multi-layer logic signal interconnection board.
7)Central, multi-layer power distribution printed circuit board
8)Side B multi-layer logic signal interconnection board.
9)Side B inner logic PCB cooling channels and flow blockers.
10)Side B inner logic 4 X 4 PCB tile matrix.
11)Side B outer logic PCB cooling channels and flow blockers.
12)Side B outer logic 4 X 4 PCB tile matrix.
13)Side B transparent outer insulating overlay.
Remarkably, all this adds up to 64 individual logic PCBs, 1024+ integrated circuits, and some 69 electrical layers.
To connect (and hold) all this lot together, Mr Cray used some 11,000 tiny rope-like gold-alloy multistranded 'Z jumper' links. There were of two types of these; power jumpers and signal jumpers. Because each PCB would have differing connection layouts, it was given a standard pre-drilled matrix of 2,500 plated through holes, which would be filled with a 'Z jumper' or not as required. At each corner of the PCB there were 4 slightly larger holes which received the thicker power distribution 'Z jumpers'. There are rather a lot of precision (by Laser?) drilled microscopic holes in a Cray-3 module, adding up all the various layers, my estimate is somewhere around 350,000 holes, all of which need to be lined up with reasonable accuracy. But there again, there are about 120,000 connections in a Cray-3 module, most of which are replaceable (if you had the robots).
These Z jumpers are themselves a stunning feat of miniature fabrication. Essentially one needs to imagine a 6mm long twisted gold-plated beryllium wire rope of 7 strands about the same thickness as a strand of cotton (the whole rope -not the strands). At 6 regularly spaced intervals along the length of this 1/4 inch long rope are small bulges that Mr. Cray called 'bird cages'. These areas being where the rope had been slightly un-twisted to make a resilient contact area. These stranded ropes were laser welded at each end, and the 'bird cage' bulges formed by untwisting the wires in a pair of collets. These wire ropes and bird cages would of course be slightly 'springy' and therefore be able to make reliable electrical contacts that also had some tolerance of PCB misalignment. At the front of each jumper a laser was used to melt the end into a rounded bullet shape, and a single straight pull-through wire was welded to this point. The other end was also zapped (to a lesser extent) to fuse the wires and prevent the strands unravelling. Some 11 thousand of these jumpers were inserted in precisely defined and unique positions for each of the 4 by 4 matrix of logic PCBs, and were pulled through the module to make the some
66,000 through connections. The pulling wires were then cut off and the link left in position. Not only did these 'Z links' simply make through connections between all the module board layers, but the friction of the connections also held the layers, and thus the complete module together. One might therefore be able to dismantle and reassemble (for repair) sections of a Cray-3 module by simply pulling out and replacing these Z links. Of course this requires that the 13 layers of drilled boards are all perfectly lined up. In production this was done by assembling each PCB with locating pins passed through the slightly larger corner power distribution holes. Once assembled, the locating pins were then replaced by power distribution Z links (which were thicker than the
signal links and had 5 'bird cages'). The resilient nature of the stranded Z link system allowed nicely for manufacturing tolerances.
Bear in mind that each complete Cray-3 module may have had a unique 11,000 Z link connection scheme, and many of the 1,000 ICs within each of those modules may also have had a unique mixture of the 300 standard logic cells.
So, in an object four inches square and a quarter of an inch thick Cray managed to drill three hundred and fifty thousand precision holes, fit over one thousand and twenty four highly special integrated circuits into sixty four boards, and make one hundred and twenty thousand connections with two hundred and forty feet of stranded wire rope. And of course connected it all up in the correct way and er, test it. (Apparently the modules were too 'powerful to test by conventional methods of the time, and could only be checked by substitution in a known working computer. To save the time of having to drain and refill the 'fish tank' with coolant for board testing, the dry machine would be pulsed on for a few milliseconds. Just long enough to make the various checks and before things started to catch fire...)
Perhaps even more miraculous was the sheer intelligence needed in the machine's production process to be able to plan and keep track of what went where. A thousand plus ICs on 64 boards, the mix of which was possibly unique to each module, and with an individual matrix of 11,000 connecting links. And then 200 plus of these modules in varying configurations (4 to 8 processors) plugged together by hand (?) with 6,000+ multiway connectors and 30 miles of wire... How do you make a wiring loom with 300,000 connections and 30 miles of wire that fits into a fraction of a cubic foot? How do you just draw the circuit of such a beast? Of course it must have all been done by CAD, but the production infrastructure to build a Cray-3 must have been quite something. I am not sure what is more impressive, the computer and it's technology, or the ability to build such a device in the first place.
Quite mind boggling.
As Mr Spock once might have said to Captain Kirk, 'It's electronics Jim, but not as we know it'...
-----------------
Cray Patents
United States Patent 4,965,863
Cray October 23, 1990
Gallium arsenide depletion mode MESFIT logic cell
Abstract
A gallium arsenide logic design system is described for designing custom or semi-custom LSI integrated circuits using standard cells from a cell library. D-MESFET transistors and Schottky diodes are used for implementing the cell types in gallium arsenide to produce performance levels of less than 150 pico-second per gate propagation delay. Each integrated circuit die is built from a cell library containing three standard cells. The limitation on the number of standard cells used for logic design allows for fast and efficient turnaround time between logic design and fabrication. A minumum number of masks are required for implementing the custom integrated circuit due to the efficient design of the cell types. The
placement and interconnect of the cells on the die are also performed in an efficient manner due to the predefined allowable locations for cell placement and the predefined allowable route channels for the interconnect. A clock amplifier cell is described for the cell library which differentially phase corrects a two-phase clock signal to ensure that the two clock lines are perfectly out of phase at all times. The combination of this strictly controlled two-phase clock with the gallium arsenide cell designs allows digital logic implementations at an LSI level to operate at 1-GHz clock frequencies.
Inventors: Cray; Seymour R. (Chippewa Falls, WI)
Assignee: Cray Computer Corporation (Colorado Springs, CO)
Appl. No.: 104758
Filed: October 2, 1987
Description
FIELD OF THE INVENTION
This invention pertains to the field of electronic digital computers, and specifically to improved high speed electronic circuitry for computers.
BACKGROUND OF THE PRIOR ART
A considerable amount of effort has gone into the development of gallium arsenide as an alternative semiconductor material for high-speed digital computers, because of its known advantage in gate switching speed over silicon-based integrated circuits. However, despite this switching speed
advantage, gallium arsenide has not achieved widespread use. Although there are many reasons for the less than rapid acceptance of gallium arsenide in the semiconductor industry, one reason of particular importance in the field of large high-speed digital computers, or supercomputers, is the lack of effective systems for implementing custom or semi-custom gallium arsenide-based integrated circuits which are needed because of custom design requirements for various sections within the architecture of the computer system. Off-the-shelf small-scale integration of gallium arsenide is available in the prior art and can be used for implementing custom designs of supercomputers. However, the relatively small number of logic gates per package limits the overall performance of the computer due to propagation delays between packages. Hence, any speed improvement in gate switching speed over silicon is at least partially lost in the interpackage propagation delays. In the meantime, silicon-based integrated circuit technology has continued to evolve and has provided increased performance through improvements in packaging density and large-scale integration techniques. Attempts to translate the circuit designs and integration techniques which have been successful in silicon into the field of gallium arsenide has not been successful in application to the design of large computing systems.
Therefore, there exists a need in the high-speed computer industry for a simple and effective design system for implementing custom or semi-custom gallium arsenide-based integrated circuits. Due to the unique characteristcs of gallium arsenide semiconducting materials and the differences between these materials and silicon, unique constraints and requirements are placed on a gallium arsenide-based logic
system heretofore unknown in the prior art. Such a logic design system should allow the fabrication of integrated circuits with packaging densities of equivalent gates per chip at a much higher level than currently available in the prior art. Such a logic design system should also be simple and easy to
maintain by having a minimum number of cell types to choose from, keeping the number of fabrication steps and cell masks required for implementation at a minimum.
SUMMARY OF THE INVENTION
The present invention overcomes these and other limitations of the prior art by describing a design system for implementing high speed gallium arsenide integrated circuits in a custom or semicustom fashion for use in high speed digital computers. Performance levels of less than 200 picaseconds per gate propagation delay are achieved using a custom gallium arsenide integrated circuit logic cell approach to building each custom integrated circuit. Each integrated circuit die is built from a cell library which results in a package (die) containing 40 to 80 multi-level logic cells equivalent to 200 to 300 gallium arsenide logic gates. A standard approach to implementing the integrated circuits results in efficient and fast turnaround to allow logic designers to test the design actually implemented in the integrated circuits. The gallium arsenide integrated circuits implemented are based on a low voltage D-MESFET technology and are fabricated using a minimum number of fabrication steps and only four cell types.
Accordingly, the present invention describes number of gallium arsenide cell types for performing logic functions which can be used as building blocks for constructing larger custom digital logic designs. By using a minimum number of simple cell types to construct larger logical decisions, a minimum amount of
mask types and fabrication steps are necessary for constructing a gallium arsenide custom integrated circuit.
According to one aspect of the logic design system of the present invention, a gallium arsenide depletion mode MESFET logic cell is described which includes a diode logic section for performing a logical function connected to a voltage level shifter section for providing signal conditioning of the resulting logical decision, connected for driving an amplifier section having load driving transistors for providing the necessary power and voltage excursion to drive other logic cells.
According to another aspect of the logic design system of the present invention, a gallium arsenide integrated circuit is described having a regular, ordered, and predefined placement configuration having the power supply busses being attached to the cells along predefined paths and having the intercell
interconnections be confined to other predefined paths. A plurality of the logic cells are placed in predefined locations for connection to the power supply busses and the interconnection paths. First and second level metal interconnect is used to connect ones of the cells with other ones of the cells to form a
custom logic function on the integrated circuit.
According to yet another aspect of the logic design system of the present invention, a method of designing gallium arsenide custom LSI integrated circuits is described having the steps of selecting, from a limited number of cell types, the appropriate cells required to implement a logic function. The cells are then placed at selected predefined locations within a regular, ordered configuration. After placement, metallization layers are included to interconnect the cells to perform a custom design logic function.
According to yet another aspect of the logic design system of the present invention, a method for performing a logic function in gallium arsenide is described comprising the steps of performing a logical function using diode logic, level shifting the resulting logical result and buffering the result to produce a higher drive power in the form of a voltage swing for use by other cells.
According to yet another aspect of the logic design system of the present invention, a differential clock amplifier cell is described comprising two identical clock cells connected to perform a differential function on an incoming two-phase split clock signal. Each phase of the clock signal is sent to one of the clock amplifier cells and the clock amplifier cells compare the signals to perform phase correction. The first stage of the clock amplifier is differentially coupled with the opposite cell.
I discovered on the US Patent Office's excellent web site a number of patents that apply to the Cray-3 module. To me they make fascinating reading, however my web browser (thanks Microsoft) won't let me view the large TIFF image files, so for my own 'document collection' I removed many of the text
references to the applicable drawings. While I was doing this I also decided to amend some of the quaint legal language used, also removing some of the repetition to make things more readable. What remains though is essentially what was on the original patents and should be not taken as any attempt on my
behalf to bypass any legalities. This information is presented here for completeness and to give the reader a far better insight into Cray's achievements than I am able. I believe that this information can be regarded as being in the public domain, and I hopefully have just made it a little more accessible to the
casual reader. Naturally I recommend a visit to the US Patent Office site itself to view the patents 'un-edited'.
I discovered on the US Patent Office's excellent web site a number of patents that apply to the Cray-3 module. To me
they make fascinating reading, however my web browser (thanks Microsoft) won't let me view the large TIFF image
files, so for my own 'document collection' I removed many of the text references to the applicable drawings. While I
was doing this I also decided to amend some of the quaint legal language used, also removing some of the repetition
to make things more readable. What remains though is essentially what was on the original patents and should be not
taken as any attempt on my behalf to bypass any legalities. This information is presented here for completeness and to
give the reader a far better insight into Cray's achievements than I am able. I believe that this information can be
regarded as being in the public domain, and I hopefully have just made it a little more accessible to the casual reader.
Naturally I recommend a visit to the US Patent Office site itself to view the patents 'un-edited'.
--------------
United States Patent 5,014,419
Cray , et al. May 14, 1991
Twisted wire jumper electrical interconnector and method of making
Abstract
A method and apparatus for interconnecting electronic circuit boards through the use of twisted wire jumpers which are formed from multifilament wire and which have enlarged bird cages formed along the pins. The pins are drawn through a stack of circuit boards to position the cages in contact with interconnection apertures located in the printed circuit boards. The frictional engagement of the cages in the apertures provides both electrical inter connection of, and mechanical coupling between the printed circuit boards.
Inventors:
Cray; Seymour R. (Chippewa Falls, WI); Krajewski; Nicholas J. (Chippewa Falls, WI)
Assignee:
Cray Computer Corporation (Colorado Springs, CO)
Appl. No.: 347507
Filed: May 4, 1989
Description
TECHNICAL FIELD OF THE INVENTION
The present invention relates to the field of electrical circuit connectors, and more specifically to both an apparatus and a method for interconnecting stacks of printed circuit boards.
BACKGROUND OF THE INVENTION
Integrated circuits are typically fabricated on wafers which are then cut up to form individual integrated circuits. These individual circuits are packaged within hermetically sealed ceramic or plastic packages. The signal and power lines from the integrated circuit are brought out to the pins of the package by means of leads attached to bonding pads on the integrated circuit chips. The chips are then used to form larger circuits by interconnecting the integrated circuit packages by means of printed circuit boards. These circuit boards may contain several layers of electrical interconnect. Typically the integrated circuit packages are soldered to the circuit board. The soldering process forms an electrical and mechanical connection between the integrated circuit package and the circuit board.
To form still larger circuits called modules, circuit boards may be arranged and interconnected in a variety of ways. One popular high density interconnect scheme is to stack the circuit boards in a sandwiched relationship and electrically interconnect the circuit boards with jumpers passed through the stack along the Z axis. This packing scheme achieves a relatively high packing density limited by heat dissipation and connector spacing requirements.
The aforementioned technique of forming larger circuits by using individually packaged integrated circuits mounted on
circuit boards limits packing density. The actual integrated circuit chips themselves are typically smaller than one-
tenth of a square inch, and only cover only 10-20 percent of the board area. Due to the low density achieved through the use of individually packaged integrated circuit chips and traditional interconnection technology, it is difficult to
increase the operating speed of the system. Additionally, the inter-board spacing of stacked circuit boards is limited by
the height of the integrated circuit packages and the inter-board connects. This limits packing density in the z direction
as well. Configurations which limit packing density limit the interboard signal speed due to the long propagation delays associated with the long interconnect lines.
Another problem presented by traditional configurations relates to the ease with which modules can be disassembled.
Forms of construction which involve soldering and staking of the board assemblies typically result in modules which
cannot be disassembled or repaired.
The present invention provides a new apparatus and method for high-density interconnects of circuit boards which
overcomes these disadvantages of the prior art.
SUMMARY OF THE INVENTION
The present invention provides for the interconnection of sandwiched circuit boards through the use of twisted wire
jumper connectors installed in interconnection apertures of circuit boards.
The circuit boards disclosed for use with this invention have the integrated circuit chips attached directly to the printed
circuit without the traditional ceramic or plastic packaging. The circuit boards themselves are manufactured with
plated-through holes, having hole patterns substantially matching the bonding pad patterns of the integrated circuit
chips.
The integrated circuit chips are manufactured with flying leads which are positioned facing the circuit board. The flying leads are inserted through the plated holes so that the flying leads protrude from the circuit board. Caul plates are then positioned on the outer sides of this sandwich and pressed together so that the sticky or soft gold of the flying leads is compressed within the plated holes, causing the soft gold to deform against the surface of the plated holes and thereby forming a strong electrical and mechanical bond. The caul plates are then removed and the integrated circuit package remains firmly attached to the circuit board. This results in improved packing density of integrated circuit chips on circuit boards.
Two or more stacked circuit boards are interconnected using electrically conductive twisted wire jumper connectors or jumpers inserted into the plated-through holes of the stacked circuit boards. The twisted wire jumper connectors are made from multifilament wire and have enlarged portions called bird cages, formed along their length. These bird cages bow out to a large outer radius, which is larger than the inner radius of the plated-through holes of the printed circuit boards. The twisted wire jumper connectors are used as inter board jumpers for the transmission of power or logic signals. The jumpers are preferably drawn through the stacked circuit boards through the use of a leader. The wiping action of the insertion creates a low impedance electrical connection between the circuit boards. The twisted wire jumper connector is made slightly longer than the stack height of the module so that a portion of the twisted wire jumper connector protrudes through one or both sides of the sandwich of circuit boards forming a stub. This stub may then be used to assist in the removal of the twisted wire jumpers to facilitate module repair.
----------
United States Patent 5,260,850
Sherwood , et al.
November 9, 1993
Logic module assembly for confining and directing the flow of cooling fluid
Abstract
An improved logic module comprises integrated circuit chips located in channels. The channels convey a flow of a cooling fluid or a combination of cooling fluids, bringing the cooling fluid into contact with the integrated circuit chips. Flow blockers and power blades form boundaries of the channels and substantially exclude the undesirable entry or exit of cooling fluids at the boundaries. A support frame holds the logic modules of a cluster of logic modules in a generally parallel relationship. Sealing plates between logic modules and an assembly gasket substantially prevent flow short circuiting. The power blades mate sealably with a horizontal buss to conduct electric power to the module.
Inventors: Sherwood; Gregory J. (Colorado Springs, CO); Quaderer; Chris M. (Colorado Springs, CO)
Assignee: Cray Computer Corporation (Colorado Springs, CO)
Appl. No.: 813966
Filed: December 24, 1991
Description
BACKGROUND OF THE INVENTION
This invention relates to a new and improved assembly of electronic components in a module such as a logic module used in a computer. More particularly this invention relates to a module and an assembly of modules having a configuration adapted for confining and directing a flow of cooling fluid through the module and into contact with the electronic components, which is particularly useful in a cooling system utilizing a flow of forced turbulent gas-liquid cooling fluid in a computer system.
In electronic computers substantially all the electrical energy consumed by the computer is ultimately converted to heat. This heat must be removed at a rate equal to the rate at which electrical energy is converted into heat, otherwise the components of the computer will be destroyed by the accumulated heat. Furthermore, the cooling effect must be distributed appropriately to maintain all of the components of the computer within appropriate operating temperatures. Achieving adequate heat removal is of significant concern in all computer systems, but is of particular concern in high speed, high capacity digital computers, referred to hereinafter as
supercomputers, which operate with relatively high heat generating densities, for example in the range of 275
watts per cubic inch.
Virtually all computers are cooled by heat transfer to a liquid fluid or a gas fluid flowing through or within the computer. To attain effective cooling it is important that the cooling fluid be effectively delivered to, and removed from, the components to be cooled.
An advantageous cooling technique for supercomputers, which combines a cooling gas and a cooling liquid in a turbulent flow to achieve improved cooling is described in detail in the aforementioned U.S. patent application Ser. No. 07/666,362, now U.S. Pat. No. 5,131,233. In the cooling system of this invention a plurality of the logic modules are installed in a sealed upper chamber of a computer housing. A lower chamber of the computer housing contains a power supply for the computer. A partition with an opening separates the upper chamber from the lower chamber. The opening is located directly below the logic modules. The logic modules include
integrated circuits (ICs) attached to circuit boards and located along channels formed in the modules. The cooling liquid is sprayed into the upper chamber and a flow of gas pressurizes the upper chamber. A pressure differential between the upper and lower chambers combines the cooling gas and the sprayed droplets of liquid into a turbulent flow which travels through channels in the logic modules and through the opening in the partition to the lower chamber. Both the latent heat of vaporization and the sensible heat gain of the cooling fluids
effectively remove heat from the ICs located along the channels. An enhanced cooling effect of the logic modules results. The enhanced cooling effect is particularly important because ICs in the logic modules are capable of generating higher heat densities and are more susceptible to damage from increased temperature than the power supply and other components of the computer.
A flow of cooling liquid or gas or both which passes from the upper chamber to the lower chamber without flowing through the channels of the logic modules is referred to as flow short circuiting.
To achieve the best cooling effect, it is desirable to direct substantially all of the turbulent flow of cooling gas and liquid through the channels of the logic modules. Cooling fluid which does not flow through the channels does not contact the heat generating components of the logic modules and does not contribute to cooling. Furthermore, gas and liquid which enters the channels at points other than the entrance or escapes from the channels without contacting the components in the channels can result in disturbing the proportions of the
mixture of the gas and liquid to cause uneven and possibly inadequate cooling. In areas which experience excessive gas flow relative to the liquid flow, localized dry spots can develop where inadequate cooling may take place. In areas which experience excessive liquid flow relative to gas flow the advantages of turbulence in the cooling fluid are reduced leaving the primary cooling to be by less desirable immersion effects.
It is against this background that the present invention has evolved, to obtain even further significant improvements and advancements in the field of cooling supercomputers, general purpose computers, electronic components and other high density heat generating configurations.
SUMMARY OF THE INVENTION
One of the significant aspects of the present invention is obtaining a relatively uniform distribution of cooling gas and cooling liquid in a turbulent flow through the logic modules of a computer, to thereby reduce or eliminate localized dry spots. In accordance with this aspect of the invention, each logic module incorporates at least one new and improved power blade which mates with an improved power buss to provide a
substantial seal against the entry of dry cooling gas into the channel and to reduce flow short circuiting while delivering high electric currents to the modules. Each power blade incorporates an integral leaf spring and a securing fastener to center the blade and provide an effective seal for the flow of cooling fluid through the channels of the module. Also, in accordance with this aspect of the invention, flow blockers are located between components of the module and cooperate with the power blades to direct the flow of cooling fluid through the channels and evenly over the components of the module. Flow short circuits between the module and the power buss are avoided and the entry of dry cooling gas into the flowing mixture of cooling fluids within the modules is avoided. The arrangement of the power blade and power buss and the flow blockers more effectively distributes the cooling fluid mixture throughout the channels of the logic modules.
Another significant aspect of the invention relates to an installation of the logic modules in the upper chamber of a computer system. An effective seal is formed to reduce or eliminate flow short circuiting between the upper and lower chambers. In accordance with this aspect of the invention a plurality of logic modules are assembled into a logic module cluster. Sealing plates are provided between adjacent modules and between modules and adjacent end caps of a supporting frame to prevent cooling fluid from flowing between and
around modules in the cluster. A gasket seals the logic module cluster to the perimeter of the opening in the partition between the upper chamber and the lower chamber to prevent the flow of cooling fluid from entering the lower chamber without passing through the channels of the logic modules. All or a majority of the cooling liquid and gas pass through the channels of the logic modules, providing effective cooling of integrated circuits on the modules.
The power requirements for these modules is pretty scary; I estimate that each one of these power blades has to carry about 40 Amps...
The above enlarged edge shot gives some idea of the various layers of circuitry in these modules.
In this edge close up the bright horizontal bars are actually a few of the 11,000 or so Z links. The vertical brown strip is the central power distribution board.
Like everything else in a Cray module, these 44 pin miniature connecters are unique and patented. The prototype versions of these modules used miniature wires between these connectors and the module itself (each of the connectors is connected to one of the 4 logic board layers). The production modules have printed
wiring conductors between connector and module.
The connector 'pole' consists of a tiny Gold-plated bifurcated resilient pin welded (?) to the printed circuit conductor.
Special tooling was used to separate and mate these multiway connectors safely.
Apparently replacement Cray-3 modules had their own rather nice laptop sized antistatic transport cases complete with a tiny padlock!.
Only logical when one considers the likely cost to replace one of these items...
This is an enlargement of one of the 64 individual logic boards that make up a Cray-3 module.
The board is covered with 2,500 plated through holes, and in this picture you can just see the positions that are filled with the twisted wire Z links. (Each board may of course have a different combination of these.) The outer surface of each logic board has an insulating translucent overlay (as here) which is also pre drilled with the Z link hole matrix. The board is one inch square.
In the above enlargement one may clearly see the stranded end 'tails' of the Z links and the ends of the IC leads crushed into their plated through-holes. (In this image you are looking through the transparent outer insulaion overlay.)
Alan Kilian pulled his Cray-3 memory board apart (many thanks for all the interesting pictures), here we can see some of those tiny Z-link 'bird cages'. I suppose they are about a millimeter or so high.