Back to Devoured - July 03, 2026
DEVOURED
Residual Context Diffusion Language Models

Residual Context Diffusion Language Models

AI llmresearch Apple Machine Learning Research
Residual Context Diffusion (RCD) prevents computational waste in diffusion language models by recycling discarded token data as contextual residuals.
What: RCD is a module that injects discarded token information back into the denoising process of block-wise diffusion language models. The research, published by a team including UC Berkeley researchers, shows 5–10 point accuracy gains and 4–5x fewer required denoising steps.
Why it matters: This addresses a major inefficiency in diffusion-based LLMs, potentially making them more competitive with standard autoregressive models for inference tasks.
Deep dive
  • Diffusion LLMs typically discard lower-confidence tokens during block-wise decoding.
  • RCD captures these discarded representations to improve future denoising steps.
  • Uses a decoupled two-stage training pipeline to manage backpropagation memory overhead.
  • Validated on long Chain-of-Thought (SDAR) and short instruction-following (LLaDA) tasks.
  • Achieves significant accuracy improvements with minimal extra compute.
Decoder
  • Diffusion Large Language Model (dLLM): An LLM that generates text through iterative denoising, allowing for parallel token generation instead of strictly sequential (autoregressive) generation.
  • Chain-of-Thought (CoT): A prompting technique where the model explicitly generates reasoning steps before providing a final answer.
Original article

Residual Context Diffusion Language Models

Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a “remasking” mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ∼1 billion tokens. RCD consistently improves frontier dLLMs by 5–10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4–5x fewer denoising steps at equivalent accuracy levels.

Explore all 51 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi