Get me Far side!

(A Flow matching model for Novel View Synthesis)

Github

Huggingface

Get Me Far Side is a deep learning project focused on novel view synthesis — the task of predicting how a 3D object looks from a previously unseen camera angle, given only a single reference image. Starting from the Zero-1-to-3 methodology as a conceptual foundation, the entire pipeline was built and trained from scratch: no pretrained image generation backbone, no borrowed weights. The goal was to deeply understand every component of view-conditioned image generation by constructing it ground-up, and to modernize the training objective by replacing the conventional diffusion loss with rectified flow matching.

Dataset

The project uses Zero-1-to-3's Objaverse renderings (views_release) — a large-scale synthetic dataset of approximately 800,000 3D objects, each rendered from 12 camera viewpoints against a clean white background. For this project, a carefully curated subset of 20,000 objects was used, totaling roughly 40 GB of rendered images and corresponding camera matrices. To make training tractable at scale, a one-time preprocessing pipeline encoded every view through a frozen VAE and CLIP image encoder, caching the resulting latents, embeddings, and all precomputed relative camera poses to disk. This eliminated all image decoding overhead from the training loop, keeping the GPU fully utilized throughout.

Model Architecture & Training

Get Me Far Side is built on a 152M-parameter Diffusion Transformer (DiT) trained entirely from random initialization. The model takes as input a noisy target-view latent channel-concatenated with the reference-view latent — allowing the network to directly copy texture and appearance — and is conditioned on a semantic-plus-pose token formed by projecting the concatenation of a CLIP image embedding and a 4-vector relative camera encoding [Δθ, sin Δaz, cos Δaz, Δr]. This conditioning drives AdaLN-Zero modulation across all 14 transformer blocks, while the patch embedder, positional embeddings, and unpatchify head all start from random initialization. Only the VAE and CLIP encoders are kept frozen throughout training, serving purely as fixed feature extractors.

For training, the project adopts a rectified flow matching objective in place of the conventional DDPM noise-prediction loss. For each pair of views, a straight interpolation path is constructed between Gaussian noise and the target latent, and the model is trained to regress the constant velocity field along that path with an MSE loss — a simpler, schedule-free formulation that produces straighter generation trajectories and cleaner ODE integration at inference. The model was trained for 40,000 optimizer steps with an effective batch size of 128 via gradient accumulation, a warmup-then-constant learning rate schedule, and an exponential moving average of weights used exclusively for inference. Conditioning was randomly dropped 10% of the time to enable classifier-free guidance at inference, and model quality was tracked every 1,000 steps via PSNR on a held-out set of 64 objects never seen during training.

Evaluation

Evaluating a novel view synthesis model requires looking past the training objective itself. My model is trained with a flow-matching loss — a per-pixel MSE between the predicted and target velocity fields in latent space — but this loss plateaus early and correlates weakly with visual quality, so I rely on it only as a training-health signal rather than a measure of success. For actual evaluation, I compare generated views against ground-truth renders using PSNR, which measures raw pixel-level reconstruction accuracy, and SSIM, which captures perceived structural similarity and is more forgiving of the small spatial shifts that generative models produce; together they give a more complete picture of fidelity than either alone. To ground these numbers, I benchmarked my 152M-parameter diffusion transformer head-to-head against the pretrained 860M-parameter Zero123 model on 100 image and camera angle pairs, using the same sampling budget (50 steps) and guidance scale for both, with warmed-up, GPU-synchronized wall-clock timing. My model generates a novel view roughly 2× faster (~1.4s vs ~2.7s per image) while achieving comparable fidelity — a slightly higher SSIM (0.835 vs 0.823) at a modest 1.1 dB PSNR cost — demonstrating that a compact, from-scratch architecture can approach the quality of a much larger pretrained model at a fraction of the parameter count and inference cost.

NVS_2.mp4

Page updated

Google Sites

Report abuse