Back to Devoured - November 16, 2026
DEVOURED
What thousands of conversations look like inside a language model

What thousands of conversations look like inside a language model

Data Operator0x
A visual mapping experiment suggests that LLM token states for different languages often form overlapping geometric regions in vector space.
What: An independent researcher mapped thousands of conversation states inside the Qwen3-1.7B model. Results showed that different languages (English, Chinese, German, Croatian) and code occupy distinct but overlapping 'sheets' within the 2,048-dimensional vector space, indicating that LLMs maintain a shared geometric structure across languages.
Why it matters: Understanding the internal geometry of LLMs helps move beyond the 'black box' metaphor by revealing how models structure linguistic and conceptual knowledge.
Deep dive
  • Experimental Setup: Used 900 QA examples per language; processed via frozen base model Qwen3-1.7B.
  • Findings: Global PCA axes showed Chinese more separated from English than European languages were.
  • Controls: Transliterating Chinese to Pinyin showed that written form influences vector location, but not entirely.
  • Methodology: PCA projections only capture a fraction (~17.6%) of the variance in a 3D view, highlighting the difficulty of visualizing high-dimensional neural states.
Decoder
  • Residual Stream: The central shared workspace in a transformer architecture where vectors are refined by successive layers.
  • PCA (Principal Component Analysis): A technique used to reduce the dimensionality of large datasets while retaining as much variance as possible.
  • Embedding Table: The initial lookup table that maps tokens to numerical vectors.
Original article

Full article content is not available for inline reading.

Read the original article →

Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi