ICE System
The ICE FPGA motherboard and backplanes are designed to meet the requirements of our radio astronomy correlator and mm-wave TES bolometer readout system. It is unique in its IO bandwidth, processing capabilities, and its clocking scheme.
A user manual, describing the motherboard, is kept at http://icecore-docs.readthedocs.org .
The ICE System capabilities have been described in a published JAI article here: https://arxiv.org/abs/1608.06262
The blue Motherboard (Figure 1 - ICE Motherboard with CHIME data acquisition boards) has a high end ARM CPU and a cutting edge Xilinx K7 FPGA to perform the real-time data processing and high speed communications. In the image you see two red CHIME data acquisition boards mounted onto the motherboard. Together the system shown will sample 16 antenna signals with an 8 bit resolution at 800 mega samples per second and fast fourier transform the incoming data.
16 ICE motherboards can be plugged into a backplane (Figure 2 - Topside view of ICE backplane). This backplane boasts a 10gbit/s full mesh (each slot can communicate directly with any other slot). This corresponds to a total of 4.8 Terabits/second. This rate of data transfer is comparable with some of the fiber links crossing from the USA to Europe, see Transatlantic communications cable.
The backplane has 2 additional powered slots for miscellaneous hardware and provides multiple clock distribution to each slot.
The image on the right, Figure 3 , shows a backplane mounted onto a subrack with one ICE board plugged in. Additionally, 8 double height QSFP cages can be seen. A fully populated crate can steam a total of 640 Gbit/second through these cages.
The ICE system is extremely flexible and compatible with alternative mezzanine boards, including those purchased off the shelf. In the image below, see figure 4, you can see the digital frequency multiplexing mezzanine board (DfMux mezzanine) designed for use in an ICE system.
The extremely low noise digital to analog conversion and analog to digital conversion circuitry allows a single DfMux mezzanine to monitor 64 cryogenically cooled bolometers on each of its 4 Frequency Multiplexed combs (a total of 256 bolometers). The board supports channel frequencies up to 10Mhz. With further research and development, it is expected that number of channels will vastly extend, with no hardware modifications required. To ensure peak performance, the onboard switching power supplies are synchronised with the main system clock. A unique identification code stored in onboard memory enables the ICE system to self-discover connected hardware. Experiments that will use this mezzanine shortly include Polarbear 2 (located in Chile) and SPT3G (located at the South pole) .
Figure 1 - ICE Motherboard with CHIME data acquisition boards
Figure 2 - Topside view of ICE backplane
Figure 3
Figure 4 - DfMux Data Acquisition and frequency comb generator board
Summary of ICE Motherboard Specifications
Figure 5 - ICE motherboard block diagram
FPGA: Xilinx Kintex 7 XC7K420T
Two high pin count (HPC) FPGA Mezzanine Connectors (FMC)
High speed serial connections:
2x QSFP+ ports (rear of board, direct connection to FPGA) each supporting 40 GbE (8 GTX lanes total)
1x SPF+ port or PCIe connection to ARM (1 GTX lane)
15 GTX lanes (each supporting up to 12.5 Gbps) to the backplane for full mesh interconnect
4 GTX lanes (each supporting up to 12.5 Gbps) go to separate QSFP+ connectors on the backplane, which are used to interconnect multiple crates.
ARM Processor: Texas Instruments AM3871
two GbE ports
DDR3 SDRAM, 2 x 256 Mb x 16
connection to the FPGA via one PCIe lane (optional, controlled by switch), JTAG, and parallel SPI.
SD card
two USB ports
one SATA port
Dimensions: 9U tall, 160mm deep.
Power: all on-board power, including FMC slots, is derived from a single input 12V-20V. All buck regulators are switched synchronously with the system clock.
Clocking: all clocks on the board are synthesized from a single 10 MHz clock that can be provided through the backplane, through an SMA connector, or from an on-board crystal oscillator. The clocking system requirements are derived from the rigid requirements of a GHz radio telescope interferometer. Synchronization signals can be provided as well, and fan out to the FMCs.
Housekeeping: all housekeeping is accessible through an I2C matrix by both the FPGA and ARM processor. Current and voltage monitoring for each regulator, as well as several temperatures across the board, are provided.