Training-Free Text-Guided Image Editing

Training-Free Text-Guided Image Editing with Visual Autoregressive Models

Introduction

Text-guided image editing asks for something deceptively hard: change what the instruction names, and leave everything else exactly as it was. Methods built on diffusion and rectified flow keep running into the same trade-off between the two. To edit a real photograph they must first invert it back into the model's latent trajectory, and that inversion is never exact; the error it introduces shows up as structural drift. Cross-modal feature entanglement then spreads the edit outward, changing parts of the image nobody asked about.

Framework

This work sidesteps both problems by changing the substrate. Instead of continuous noise removal, it builds on Visual Autoregressive Modeling (VAR), which generates an image as discrete tokens, scale by scale. Because there is no trajectory to invert, tokens can simply be replaced during sampling without disturbing the generation process at all. The pipeline is training-free and inversion-free.

The guiding idea is understanding before editing. Rather than calling an external segmentation tool or tuning thresholds per object category, the model works out for itself which regions the instruction refers to. At each generation scale it reads the cross-modal attention map between text and visual tokens and derives two masks: a high-attention focus mask marking the edit target, and a low-attention preserve mask marking what must survive. Only the targeted tokens are rewritten.

For real images, the source is first encoded into discrete per-scale bitwise tokens by a BSQ-VAE. Background substitution then happens directly at the token level, preserving source content faithfully with no inversion and no continuous feature injection.

Results

On the PIE-Bench benchmark, the method improves SSIM by 7.5% and LPIPS by 21.5% over PnP Inversion, the strongest diffusion-based method, while also scoring higher on both whole-image and edited-region CLIP alignment. Against AREdit, another VAR-based method using the same 2B backbone, it wins four of six shared metrics, with clear margins on perceptual fidelity and text alignment.

Ablations isolate where the gain comes from. Replacing attention-derived masks with uniform scale-level sharing costs 55% SSIM and 67.5% LPIPS: spatially precise, per-token guidance is far more effective than sharing whole scales indiscriminately. The result is a simpler and faster editing pipeline that preserves source fidelity better than recent training-free baselines while remaining competitive on edit quality.

Page updated

Google Sites

Report abuse