DEVOURED
No Space Like J-Space
Anthropic researchers have identified a 'J-space' within language models where conscious reasoning occurs, mapped via a technique called the Jacobian Lens.
What: The Jacobian Lens interpretability technique computes the average causal effect of residual stream changes to trace how specific concepts are associated with different model layers. This allows researchers to pinpoint where a model performs conscious reasoning.
Why it matters: This research provides a concrete method for 'opening the black box' of LLMs, moving beyond superficial observation to understanding the internal mechanics of how models perform multistep reasoning.
Deep dive
- The J-space represents the area of conscious access where models perform reasoning.
- The Jacobian Lens tracks the causal impact of state changes on final output.
- It enables researchers to see which concepts are processed at specific layers.
- This represents a shift from empirical observation to mechanistic interpretability.
Decoder
- Jacobian Lens: An interpretability tool that calculates the Jacobian matrix of a model's output with respect to its internal states, highlighting which internal representations causally drive specific model behaviors.
- Residual Stream: The sequence of hidden states in a Transformer model that persists across layers, acting as a shared memory for information being refined during processing.
Original article
There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here. I encourage reading of the whole original...
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi