DEVOURED
A global workspace in language models
Anthropic researchers identified a 'J-space' in Claude's neural architecture, a silent, emergent workspace that functions as a hub for deliberate reasoning.
What: Researchers at Anthropic found that Claude possesses a collection of internal neural patterns, dubbed 'J-space' using a Jacobian lens technique, which act as a global workspace for complex, multi-step tasks. These patterns operate independently of token output, allowing the model to perform internal reasoning, monitor its own potential for bias, and manage hidden goals without expressing them explicitly in text.
Why it matters: This research provides a mechanism to peek into the 'black box' of neural networks, suggesting that higher-order cognitive functions in models can emerge spontaneously as a result of training for efficiency.
Takeaway: Developers auditing model safety can use the J-lens method to inspect internal activations for evidence of prompt injection or deceptive behavior before the model generates a response.
Deep dive
- The J-space is a small, specialized collection of internal neural patterns that emerges during training without explicit programming.
- It allows the model to reason silently, modulate its internal state, and perform tasks like multi-step math without outputting the reasoning steps.
- 'J-lens' technique uses a Jacobian-based approach to map internal activations to tokens in the model's vocabulary.
- J-space acts as a 'global workspace' where data is broadcasted, allowing for flexible information reuse across different tasks.
- This internal workspace remains distinct from the 'scratchpad' or 'chain-of-thought' outputs.
- It is possible to influence model behavior by editing the J-space directly to swap or inject concepts.
- The J-space can be used to detect adversarial behavior, such as hidden attempts at prompt injection or intentional data manipulation.
- Post-training on 'reflection' tasks can cause the model to adopt more honest behavior as these values become embedded in the J-space.
Decoder
- Jacobian: A matrix of all first-order partial derivatives of a vector-valued function, used here to map how changes in neural activations impact output probability.
- Mixture-of-Experts (MoE): A neural network architecture where only a subset of the model's parameters are active for any given input, improving performance while saving compute.
- Ablation: The process of removing or disabling parts of a neural network to observe the impact on performance and identify the function of specific components.
Original article
Full article content is not available for inline reading.
Explore all 42 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi