Back to Devoured - November 12, 2026
DEVOURED
LLMs can learn some multi-step reasoning on filler tokens instead of visible CoT

LLMs can learn some multi-step reasoning on filler tokens instead of visible CoT

AI LessWrong
Researchers demonstrated that LLMs can perform multi-step reasoning using hidden internal states via 'filler tokens' without needing to output coherent chain-of-thought text.
What: By training Qwen3.6 and gpt-oss-20b with LoRAs, researchers successfully taught models to solve multi-step tasks using filler dot sequences, bypassing the need for standard reasoning outputs.
Why it matters: This challenges the assumption that reasoning is strictly tied to visible chain-of-thought, implying that hidden reasoning could eventually threaten the monitorability of AI systems.
Deep dive
  • Filler-token reasoning: model uses extra tokens to reason internally without emitting relevant words
  • Demonstrated feasibility on tasks including Mod Sum, GL(3,2), and ProntoQA
  • Used LoRA fine-tuning with a curriculum of increasing task length
  • Theory shows some tasks require deeper algorithmic circuits, setting empirical walls for model performance
  • Challenges standard interpretability assumptions that equate 'thinking' with visible 'chain-of-thought'
Decoder
  • Filler tokens: Tokens emitted by a model (like '.' or '_') that carry no explicit semantic meaning but provide the model with computation steps.
  • Steganographic reasoning: A hypothesis where models perform reasoning internally while emitting innocuous-looking text to hide the process.
  • TC⁰: A complexity class representing problems solvable by circuits of constant depth; conceptually, parallelizable tasks.
Original article

Research update of a project done as part of MARS 5 and the Meridian Visiting Researcher Programme.

Filler-token reasoning is the ability to use extra tokens (such as a chain of thought) to reason about a task without using the words emitted at those token positions. This is possible because attention can move information between token positions without routing it through the words the LLM emits. This can give a model more parallel, but not more serial, computation.

The canonical demonstration of this is the "Let's Think Dot by Dot" paper (Pfau, Merrill & Bowman, 2024). There, the authors train models to solve completely parallelizable (TC⁰) tasks with chains of thought made only of "." tokens.

Our contributions:

  • We change Pfau et al.'s training method so that models can learn multi-step reasoning on filler tokens.
  • We demonstrate this on two models and three out of seven multi-step reasoning tasks.
  • We provide a theoretical analysis of how many layers a transformer needs to solve each of our multi-step reasoning tasks, depending on the length of the task.
  • We measure empirically how many reasoning steps we can train Qwen3.6-35B-A3B and gpt-oss-20b to do on filler tokens for each task.

Overall, this project has informed which side tasks we use to study steganographic reasoning. We also hope this writeup shows that filler-token reasoning is not confined to fully parallelizable tasks.

Motivation

Using the same token over and over guarantees that the filler tokens carry no information about the reasoning. But you could also imagine the same hidden computation happening while the LLM writes text that has nothing to do with its reasoning.

Our main research is on steganographic reasoning (the model uses its words for reasoning, but hides the reasoning in innocuous-looking text): when can models learn to reason steganographically through simple multi-step tasks? When we see a model write some unrelated text and then correctly answer a multi-step question it could not answer immediately, we have to be careful not to attribute to steganographic reasoning what could simply be filler-token reasoning.

The dot-by-dot paper looked at tasks that are in TC⁰, i.e. in theory parallelizable (3SUM, 2SUM-Transform), and showed that models can learn to do them on filler tokens. The authors argue that filler tokens cannot help beyond TC⁰. They found that when they train on serial reasoning (where later reasoning steps depend on earlier reasoning steps) they could not teach their models to do so on filler tokens.

This did not line up with our experience: filler-token reasoning kept cropping up as a failure mode in our experiments. In this small project, we set out to extend Pfau et al.'s setting to multi-step reasoning on filler tokens.

Methods

Training filler-token reasoning

Pfau et al. trained their models to output dots after the problem statement, followed by the answer, and mixed this data with open-reasoning samples.

We instead train models with dots as the input and the open reasoning as the output: at each dot position, the target is one reasoning step. This means the models we train are no longer next-token predictors. We have to prefill their CoT with dots, since they would be reasoning openly otherwise.

Another difference: they train their models from scratch, while we train LoRAs on top of existing LLMs.

We use two training setups here:

  • Curriculum: we start at a number of task steps the model can definitely learn and train it on that distribution. Once answer accuracy reaches 0.9, we increase the length, bit by bit. We can then see how far the curriculum gets before it stops working.
  • Direct: we train the model at full length from the start. By default this is 100 steps, but shorter for tasks where we don't have enough data, or where the prompt grows superlinearly with the number of reasoning steps. Here we can see up to which position the model still predicts the reasoning steps correctly.

Models

We train Qwen3.6-35B-A3B and gpt-oss-20b with rank-32 LoRAs.

Tasks

We use a selection of multi-step reasoning tasks for which prompts, correct step-by-step reasoning and answers can easily be generated synthetically.

Mod sum: compute the sum of a list of numbers modulo n.

A5: track a ball under five cups through a sequence of shuffles. The task is a game of Thimblerig: There are 5 cups, and in the beginning a ball is under the cup at position 0. Each 'step' is one of these operations: Rotate (every cup moves from position p to p+1 mod 5) or Double-Swap (We swap the cups at positions 0,1 as well as 3,4). These two moves generate the group A5. The answer is the final position of the ball.

GL(3,2): track the on/off pattern of three lamps through a sequence of moves. The task is a scenario where there are three lamps in a ring: A, B and C. Only lamp A is on at the start. Then we do a series of moves: Shift (every lamp passes its state to the next) or Couple (if A is on, B toggles). These two moves generate the group GL(3,2), of order 168.

Modular arithmetic: evaluate a chain of +, −, × operations modulo 8.

ProntoQA: follow rules of the form "every A is a B" to decide whether an entity belongs to a category. This is an adjusted version of Saparov & He with problems of arbitrary lengths and no shortcut strategies that let models easily perform better than chance with no serial reasoning.

HornQA: derive new categories with rules of the form "A and B ⇒ C", where a rule may need a fact derived arbitrarily far back. A variant of our ProntoQA setup based on Horn-clause reasoning, which is P-complete.

Function composition: push a value through a sequence of lookup tables. A start value in 0–7, followed by a sequence of lookup tables of 8 entries each (permutations of 0–7). Each table maps a value v to its entry at index v. The answer is the value after applying all tables in order.

Since our setup comes from steganographic reasoning, where our vocabulary to express the reasoning steps is limited, all numbers in the reasoning that have no fixed maximum are encoded in binary.

Results

Models can learn multi-step reasoning on filler tokens

To test whether models can learn multi-step reasoning on filler tokens, we train three model/task combinations (mod sum and GL(3,2) on Qwen3.6-35B-A3B, mod sum on gpt-oss-20b) in three settings: our method, Pfau et al.'s method, and an answer-only baseline that outputs the final answer directly, with no extra tokens.

We can see that on all of these tasks, our method lets us train the models to reason on filler tokens.

Of course, this only works up to some task length: the model has a fixed number of layers, and these tasks need more successive computation the longer they get.

Theoretical limits to learning multi-step reasoning on filler tokens

For every task, the shallowest algorithm we found gets deeper as the task gets longer, so we found no way to fully parallelize any of them. And for every task except HornQA the depth grows much more slowly than the number of steps, which is why the curves stay below the model's depth far beyond 40 or 24 steps.

We see that different tasks scale very differently. In particular, the modular sum task scales very gently, i.e. it is easy to do on filler tokens.

Empirical limits to learning multi-step reasoning on filler tokens

Again, tasks differ a lot in how well they can be learned. And again there are outliers: HornQA, which theory predicted to be hard to learn on filler tokens, fails after very few steps, and the popular modular sum is learned very easily.

But theoretical and empirical limits don't line up in general: some tasks are theoretically easy but empirically hard (modular arithmetic and function composition hit a wall at 1–3 steps). The pattern seems to be that models have a hard time learning tasks on filler tokens when large packets of information need to be transported across token positions.

Conclusion

Multi-step reasoning can also be done on filler tokens. Recent models such as Astra have shown an increased ability to answer reasoning questions without reasoning about them. If that trend continues, it could endanger CoT monitorability even without the specters of looped architectures or steganographic reasoning coming true.

Our experiments serve as a demonstration, and shed some light on which tasks can be learned on filler tokens, and how easily. Any pressure not to verbalize certain multi-step reasoning could, with enough training, lead to similar results.

We also learned a lot about constructing tasks for studying steganographic reasoning. We recommend that others researching steganographic or encoded reasoning put the tasks they use through a test like this.

Explore all 61 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi