Using ROACH DRAM

Requirements: Tutorial 1, 2 and 3.

Introduction

In this tutorials we learn the basics of how to use the ROACH DRAMs to store data and read them afterwards. The first thing to know is that in the lab we have only been able to successfully use the ROACH1's DRAM, so that is what are we using. For the implementation details of the DRAM in ROACH you can check the CASPER page, in this tutorial we'll be focusing in showcasing some DRAM example models.

NOTE from the future: We were able to get some functionality from ROACH2 DRAM, but there is no tutorial for it. If you need help please contact MWL people.

Writing individual values into the DRAM

The most simple thing you can do with the DRAM is write individuals values in specific directions of the memory. Download and study the following model (.slx, .bof):

The dram block creates the interface between the FPGA and the physical DRAM. For a full description of every input, output and parameter of the block see the CASPER page. The most important inputs are address, the data_in and the cmd_valid. When the cmd_valid input is high the data from the data_in input is written int the provided address. Two important quirks that you have to be aware when using the DRAM block are:

1. The datawidth of the data_in input is 144 bits, however in order to be written correctly, two groups of 8-bit constant zero valued must be present in the input data as is demonstrated in the model. This means that the actual payload is 128 bits per address. In this example this data is provided by four independent 32-bits registers.
2. In order write to data correctly, the address input must be held for at least two clock cycles, and the input data will be written in two consecutive addresses of the DRAM. So if you held the address 0x00 for two clock cycles, you'll be writing data into physical addresses 0x00 and 0x01, then if you change the address to 0x01, you'll write in 0x02 and 0x03, etc. Since in this simple example you'll be setting the address manually you'll unavoidably will end with duplicated data.

You should keep the DRAM block parameters as the default. Notice that you can change the 'Datapath clock rate'. Different values for this clock rate can change the performance of the DRAM, and a poor choice can corrupt the data written in the DRAM. The rule of thumb is to always keep the DRAM datapath clock rate higher to the FPGA clock in order to never overflow the DRAM FIFO.

Now make a python script to write data into the DRAM using the registers. You can read the data from the DRAM using the corr's function read_dram(<>, <>), where the first parameter is the amount of data to read in bytes, and the second parameter is the offset in bytes from where to start to read. Remember that corr reads roach data in string of bytes so you'll have to reinterpret it using something like Numpy's frombuffer(). If you are stuck you can try the example script from here. Check that your DRAM is working correctly an continue to the next section.

Writing counter data into the DRAM

DRAMs are more useful when you want to write a large burst of data at once. In this section we'll do just that using the output of a counter. Download the following model (.slx, .bof):

One freeze_counter counter is used to generate the addresses for the DRAM, while the other for the input data. The freeze_counter block can be found in CASPER's Misc sublibrary, they are essentially normal counters that 'freeze' in zero after they reach their final value. The counters are controlled with the 'enable' and 'reset' registers.

Notice that the problem of having to hold the address input for two cycles is solved by removing the less significant bit from the output of the freeze_cntr block, that way we don't incur in any type of data loss or duplication.

In theory, the ROACH DRAM can hold 1GB of data. Since the input data is of 128 bits (16 bytes), you should have 1GB / 16 bytes = 2³⁰/2⁴ = 2²⁶ addresses to store data. For some reason we have only been able to use 2²⁴ addresses, which is 256MB worth of data. When we tried to use higher addresses the data gets wrapped around into the lower addresses locations. More study is needed to understand this phenomenon.

Anyways, create a script to fill the DRAM with the counter data. You can do this by simply setting the 'enable' register to 1 and then reading the DRAM data to check that is working correctly. Or you can be lazy and you can use this script directly (I won't judge you :). Notice that in that script we don't read the whole relevant memory in a single read_dram() command. That is because reading such large amount of data produces a timeout in the ROACH-PC connection. Instead we read smaller chunks of data. The exact amount of data you can read in a single read_dram() depends on your connection speed, so test which is the proper size for your case.

Writing spectral data into the DRAM in order to create an spectrogram

Now let's make a more practical usage of the DRAM: save instantaneous data from an spectrometer. With instantaneous data I mean data directly from the power blocks, after the FFTs and with no accumulation. Without the DRAM there is now way to get this data because there is not a fast enough connection from the FPGA to the PC hard disk.

Download the following model (.slx, .bof):

A few things you should notice from the model:

1. We use a freeze counter with the less significant bit removed just as we did in the previous section to avoid data loss.
2. The four parallel data stream are combined into one in order to properly input the data into the DRAM.
3. The writing in the DRAM is controlled by the cnt_rst register.
4. Aside of the DRAM, the model also implements a standard bram spectrometer for debugging purposes.

Other than that the model is pretty straight forward. Now you can write an script to read the spectral data from the DRAM and then make a colored frequency vs time vs power plot, usually called spectrogram plots. A script for plotting with this model can be found here. Finally here is an examples of spectrograms (closeup) captured with the model:

Spectrogram for a fixed frequency tone.

Spectrogram for a frequency sweep, the input signal can seen to change frequency bin twice. Keep in mind that the ROACH process data so fast that is difficult to see variation in the spectrogram plot from standard RF sources.