Back to Devoured - May 05, 2026

DEVOURED

End-to-End Tokenizer Training for Autoregressive Images

AI computer-vision arxiv.org

Researchers achieve state-of-the-art image generation results by training the image tokenizer and generator jointly instead of separately.

What: A new training pipeline for autoregressive image generation that jointly optimizes the visual tokenizer (which compresses images into compact representations) and the generation model end-to-end, rather than training them as two separate stages as traditional approaches do.

Why it matters: Training tokenizers separately means they only optimize for reconstruction quality without receiving feedback on how well they support generation. Joint training allows the tokenizer to learn representations specifically tuned for generation performance, leading to better results.

Takeaway: The paper, accepted to ICML 2026 as a Spotlight presentation, demonstrates a state-of-the-art FID score of 1.48 on ImageNet 256x256 generation without guidance.

Deep dive

Autoregressive image models compress images into latent representations using visual tokenizers before generation, but traditional two-stage pipelines train tokenizers and generators separately
This work proposes end-to-end joint optimization of both reconstruction quality and generation performance, enabling direct supervision signals from generation results to flow back to the tokenizer
The approach leverages vision foundation models to improve 1D tokenizers specifically for autoregressive modeling tasks
Achieved state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation benchmark
The joint training contrasts with prior work where tokenizers were optimized purely for reconstruction without considering downstream generation performance
Accepted to ICML 2026 as a Spotlight presentation, indicating significant contribution to the field
The method addresses a key limitation where separately-trained tokenizers may learn representations that reconstruct well but don't support optimal generation

Decoder

Autoregressive: A modeling approach that generates data sequentially, predicting each element based on previously generated elements
Tokenizer: A component that converts images into discrete tokens or compact representations processable by generative models
FID (Fréchet Inception Distance): A metric for evaluating image generation quality by comparing distributions of generated versus real images (lower is better)
ImageNet: A large-scale image dataset commonly used as a benchmark for computer vision tasks
Vision foundation models: Large pre-trained models like CLIP or DINOv2 that learn general visual representations from massive datasets
1D tokenizer: A tokenizer that converts 2D images into a one-dimensional sequence of tokens for sequential processing

Original article

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Authors: Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li, Linjie Yang, Yisong Yue, Qiushan Guo

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.

Explore all 52 articles in today's digest