DEVOURED
What thousands of conversations look like inside a language model
A visual mapping experiment suggests that LLM token states for different languages often form overlapping geometric regions in vector space.
What: An independent researcher mapped thousands of conversation states inside the Qwen3-1.7B model. Results showed that different languages (English, Chinese, German, Croatian) and code occupy distinct but overlapping 'sheets' within the 2,048-dimensional vector space, indicating that LLMs maintain a shared geometric structure across languages.
Why it matters: Understanding the internal geometry of LLMs helps move beyond the 'black box' metaphor by revealing how models structure linguistic and conceptual knowledge.
Deep dive
- Experimental Setup: Used 900 QA examples per language; processed via frozen base model Qwen3-1.7B.
- Findings: Global PCA axes showed Chinese more separated from English than European languages were.
- Controls: Transliterating Chinese to Pinyin showed that written form influences vector location, but not entirely.
- Methodology: PCA projections only capture a fraction (~17.6%) of the variance in a 3D view, highlighting the difficulty of visualizing high-dimensional neural states.
Decoder
- Residual Stream: The central shared workspace in a transformer architecture where vectors are refined by successive layers.
- PCA (Principal Component Analysis): A technique used to reduce the dimensionality of large datasets while retaining as much variance as possible.
- Embedding Table: The initial lookup table that maps tokens to numerical vectors.
Original article
Full article content is not available for inline reading.
Explore all 69 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi