Back to Devoured - November 11, 2026
DEVOURED
Neurons as an Interpretable Basis

Neurons as an Interpretable Basis

AI Goodfire
Goodfire researchers argue that individual transformer neurons are more interpretable than previously thought, identifying them as sparse, predictable, and functional units of computation.
What: Goodfire researchers Curt Tigges, Matthew Kowal, Thomas McGrath, Owen Lewis, and Nick Cammarata analyzed neurons in the Qwen3.6-35B-A3B model. They found that up to 92.2% of weak neuron activations can be removed without significant performance loss, and that neurons perform specific computational functions grouped into coherent 'modes' detectable by LLM agents.
Why it matters: This challenges the dominant paradigm that neurons are intractably polysemantic and must be replaced by sparse autoencoders or learned feature dictionaries, suggesting instead that we can study the model's native units directly.
Deep dive
  • Neurons demonstrate functional sparsity, with most causal work performed by activations in the p97/3 range and higher.
  • A neuron pole typically contains a small, finite set of computational 'modes' that can be concisely described in natural language.
  • Researchers identified subcircuits by mapping upstream gate/up-projection inputs and downstream reader neurons.
  • Agentic 'Neuron Scientist' models can successfully simulate neuron behavior, recovering 73% of the damage caused by complete ablation.
  • Polysemanticity is often effectively resolved by downstream readers, which use contextual signals to interpret the neuron's output in the current mode.
  • Standard LLMs can distinguish active modes with high accuracy based on text and circuit analysis.
  • Random direction baselines fail to exhibit the localized, geometric circuit structure observed in actual neuron activations.
  • Sparsified models can be recovered to full benchmark performance using only 100M tokens of fine-tuning, compared to the multi-trillion token original training run.
Decoder
  • Polysemanticity: The state where a single neuron responds to multiple seemingly unrelated semantic concepts.
  • Sparse autoencoder (SAE): A method used to extract interpretable features from a model by projecting activations into a higher-dimensional space where features are assumed to be sparse.
  • SwiGLU: A gated MLP activation function architecture that enables positive and negative activations (bipolarity).
  • RelP (Relevance Patching): An attribution technique that measures the impact of a specific component by patching its output with a baseline and observing the logit change.
Original article

Full article content is not available for inline reading.

Read the original article →

Explore all 54 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi