Implementation

The goal of Stage 1 is to turn the input SPAD burst into a single intermediate image that keeps as much useful signal as possible while reducing noise and motion blur. A naive sum is a natural starting point because summing more frames increases the photon count. However, in dynamic scenes, the camera and objects move across the burst, so directly summing many frames can smear object boundaries and produce significant blur.

The naive-sum baselines exposed the main trade-off. NaiveSum16 was a useful candidate for a Stage 2 input because it preserved sharper structure and had relatively little motion blur, but it discarded most of the available photons. NaiveSum256 recovered much more scene information and had the best quantitative performance among the naive-sum baselines reported for the challenge, so we used it as the main Stage 1 comparison point. MCA was designed to use the larger 256-frame window while reducing the blur caused by summing those frames without alignment.

As an initial experiment, we tried aligning individual frames using translation only, with the last frame as the reference. In practice, this performed poorly because individual binary SPAD frames contain very little visible signal. The estimated shifts were unstable and often reflected photon noise rather than actual camera motion.

To make alignment more reliable, we moved to a blockwise translation method. Instead of estimating motion from single frames, we grouped nearby frames into small non-overlapping blocks and summed each block before alignment. In our experiments, we used 8-frame blocks within a 256-frame window. These block sums provided a higher-signal representation for estimating motion, while still keeping the motion within each block relatively small. We then estimated a translation from each block to the final reference block and applied that same translation to all frames in the block before aggregation.

A convolutional LSTM walks through all 1,024 frames one at a time, maintaining a spatial hidden state that accumulates photon statistics, edges, and coarse motion.​

A small decoder head maps the final hidden state back to an RGB image, supervised with an L1 loss against ground truth.​ The model learns how to integrate photons over time instead of using one fixed averaging rule.​

Single-photon bursts are naturally ordered in time: each of the 1,024 frames is a noisy binary observation of the scene. Recurrent neural networks (RNNs) are a standard way to summarise such sequences into a fixed-size state that depends on the entire history, without committing to a fixed hand-crafted fusion rule (e.g. a simple temporal average). 

For this work we use a convolutional LSTM (ConvLSTM) as the temporal backbone. At each time step the cell receives a downsampled RGB-like frame (binary values in [0, 1] after unpacking and pooling) and updates a spatial hidden state and cell state over the image grid. Convolutional recurrence preserves spatial structure. Gates and updates use convolutions over H × W, so local neighbourhoods evolve coherently across time. The full burst is processed in chunks (e.g. 64 frames) to keep memory manageable. The hidden and cell states are carried across chunks so the representation stays consistent over all 1,024 steps. 

After the last frame, the final hidden state is treated as a high-dimensional feature map encoding a learned summary of the burst (photon statistics, edges, coarse motion, etc.). A small convolutional decoder head maps this map to three channels in [0, 1], yielding an image at the RNN resolution. That image is supervised with an ℓ1 loss to the ground truth RGB (downsampled to match during training), so the recurrent trunk and decoder are trained end-to-end for reconstruction. At evaluation, the decoder output is upsampled to 800×800 to match the challenge resolution and to compare against baselines and metrics (PSNR, MS-SSIM, LPIPS). 

Compared to a naive sum baseline (averaging binary frames), the ConvLSTM can in principle learn how to integrate photons over time rather than applying a single global average. That matters when the burst contains motion or varying overlap between frames. 

Gradient checkpointing is used over chunk boundaries to limit GPU memory while keeping the effective sequence length at 1,024 steps. The resulting model serves as Stage 1 in a two-stage pipeline. It produces a structured intermediate image (or feature map) for a later generative refinement stage. 

The diffusion stage takes the MCA output and produces a perceptually enhanced reconstruction. MCA does an excellent job at recovering structure and reducing photon noise, but the averaging it relies on inevitably smooths over high-frequency texture. Our goal with diffusion is to commit to plausible texture rather than averaging across it. Generative image priors are good choice for this task.

We use a Palette-style conditional latent diffusion model built on top of Stable Diffusion 2.1 with zero-terminal SNR and v-prediction. The MCA output is encoded into a 4-channel latent representation by gQIR's quanta-aligned VAE, a recently published encoder trained specifically on sparse single-photon data. This latent is concatenated channel-wise with a noisy latent and passed into the diffusion UNet. The UNet's input convolution was expanded from 4 to 8 channels (zero-initialized on the new half so training begins from the pretrained behavior), and 20 DDIM steps of reverse diffusion produce the final latent. The standard SD 2.1 VAE then decodes the result back to an 800×800 RGB image. Regarding text-conditioning, null embedding is passed throughout, since the MCA output already carries all the information the model needs.

A key design decision was using a dual-VAE setup. Namely, gQIR's quanta-aligned VAE for encoding the noisy MCA input, and the standard SD VAE for decoding the final output. The two image distributions are different enough that a single encoder trained on natural images alone could not reliably interpret the sparse-photon data.

Another important decision was how to adapt the pretrained UNet. We initially tried LoRA on the attention layers, which is the standard parameter-efficient fine-tuning approach, but the resulting model hallucinated completely. We resolved this by freezing only the encoder and unfreezing the entire decoder and middle block, giving roughly 612M trainable parameters out of roughly 886M total. This approach succeeded where LoRA had failed.

Training was performed on 1,665 paired MCA-output / ground-truth images from the competition dataset, with 185 held out for validation. We used distributed training across 4 GPUs with mixed precision and gradient checkpointing, optimizing v-prediction MSE loss in latent space. At inference time, 20 DDIM steps was found empirically to outperform both shorter and longer schedules (sweeps from 10 to 1000 steps showed that additional denoising past 20 steps degrades all three evaluation metrics simultaneously, likely because extra steps reintroduce hallucinated detail).