Single Photon Challenge

Background & Motivation

What is a Single-Photon Camera? 

Single-photon avalanche diodes (SPADs) are a new class of imaging sensors that operate at the physical limits of light. Unlike conventional cameras that measure intensity, SPADs detect individual photons, making them extremely sensitive, high-speed cameras (up to hundreds of thousands of frames per second). This enables imaging in conditions where traditional cameras fail: extreme low light, high-speed motion, or both.

Why Does This Matter?

Reliable imaging under low light and fast motion is not just a technical curiosity, it’s a real-world necessity. 

The Problem

Each image captured by a SPAD camera is extremely noisy and binary-valued, dominated by photon shot noise. Thus, our central research question is: given a burst of such binary frames, how can we recover a clean, high-fidelity image?

The challenge is not just denoising, it’s recovering signal under motion. A SPAD camera produces a burst of thousands of frames over time. In a dynamic scene, the content of each frame shifts, meaning pixels no longer correspond across time. This creates a fundamental trade-off: Aggregating more frames improves signal quality. Aggregating more frames also introduces motion blur. 

Naively summing frames leads to ghosting and smeared structures, while using fewer frames preserves motion but amplifies noise. Balancing this trade-off is the core difficulty of the problem.

Single Photon Challenge

The Single Photon Challenge, organized by the WISION Lab at UW-Madison, formalizes this problem. 

Task: Reconstruct a single RGB image (800×800×3) from 1,024 noisy binary SPAD frames (each 800×800×3) captured over a dynamic scene. 

Challenge: Since any pixel in frame 1 may correspond to a different part of the scene by frame 1,024, naively summing all frames smears moving objects into ghost trails. But discarding frames to avoid blur means discarding photons, and with so few photons per frame, every discarded frame hurts signal quality. This trade-off between temporal resolution and photon shot noise is the central challenge of the problem.

Dataset: The reconstruction dataset (~425G training set + ~42G test set, ~133G and ~13G compressed respectively) consists of photoncube/image pairs from 50 unique simulated scenes, plus another 5 scenes for the test set (for which ground truths are not made public). Each photoncube consists of 1024 bitplanes, and the associated ground truth reconstruction corresponds to the last bitplane. More details about the dataset: https://singlephotonchallenge.com/download

Evaluation: Submissions are evaluated on three complementary metrics. 

• Peak Signal-to-Noise Ratio (PSNR) measures pixel-level fidelity in dB but correlates poorly with perceived quality. Higher is better.

• Multi-scale structural similarity (MS-SSIM) assesses structural similarity, luminance, and contrast across multiple scales and better reflects human perception. Higher is better.

• Learned Perceptual Image Patch Similarity (LPIPS) compares deep feature activations from a pretrained network and serves as the closest proxy to perceptual image quality. Lower is better.

Baseline: Taking the sum of the last 256 frames produces the following results: PSNR: 17.29 dB, MS-SSIM: 0.516, LPIPS: 0.779. These numbers reflect poor perceptual and structural quality, highlighting how far naive approaches are from solving the problem. 

Our Approach

We approached the Single Photon Challenge as a two-stage reconstruction problem: first aggregate the 1,024 noisy binary frames into a single clean intermediate image, then refine that intermediate using a learned generative prior. The two stages address different sides of the problem. Stage 1 is fundamentally about motion compensation across the burst and maintaining as much information as possible, while Stage 2 is about denoising and recovering perceptual detail that aggregation alone tends to smooth away.

For Stage 1, we explored several techniques. The simplest baseline is a naive sum across the last 16 frames, which provides a useful reference point but can still suffer from motion blur when there is camera or scene movement. We chose NaiveSum16 because it gave the best trade-off in our experiments: enough photon information to recover visible structure, while keeping blur low enough for Stage 2 refinement to remain effective. To move beyond this baseline, our goal was to aggregate information from more frames while compensating for motion and reducing blur. We then implemented blockwise translation alignment, followed by a fully adaptive motion-compensated aggregation (MCA) method that combines per-block scaling, translation, and local refinement. We also experimented with a learned alternative based on ConvLSTM, which processes the burst as a temporal sequence rather than aligning frames explicitly.

For Stage 2, we built a Palette-style conditional latent diffusion model on top of Stable Diffusion 2.1, using a quanta-aligned VAE to encode the aggregated intermediate into a latent representation that the diffusion model can condition on. We also explored a GAN-based alternative for comparison.

Our best-performing pipeline combines adaptive MCA with the diffusion refinement stage. The MCA output establishes a structurally faithful but somewhat smooth reconstruction; the diffusion stage then commits to plausible high-frequency texture, substantially improving perceptual quality (LPIPS) at the cost of only minor changes in pixel-level error (PSNR). However, it is important to note that the ConvLSTM method achieves the best PSNR, while falling behind in the other metrics. 

Submitted Project Assets