DEVOURED
The New American AI Model Designed to be Customized
Thinking Machines released Inkling, a 975B parameter sparse model built from scratch with a unique relative position encoding and effort-based training.
What: Inkling uses 256 experts (6 active) for 41B active parameters. It includes multimodal support without separate encoders and uses a learned 'effort' setting to regulate reasoning time.
Why it matters: This model represents an architectural departure from the RoPE standard, opting for relative position embeddings to improve long-context extrapolation.
Deep dive
- Uses Mixture of Experts (MoE) with a bias-based routing mechanism to prevent expert collapse.
- Features a hybrid attention structure: 55 sliding-window layers and 11 full-attention layers.
- Uses relative position encoding, which outperforms RoPE in long-sequence extrapolation according to the developers.
- Implements local convolutions on keys and values to inject proximity preference without learning.
- Audio/Images are handled by dMel and hMLP modules rather than separate pre-trained vision/audio models.
Decoder
- Mixture of Experts (MoE): An architecture where different sub-networks (experts) are activated for different inputs to increase parameter count without proportional inference costs.
- Rotary Position Embedding (RoPE): The current industry standard for encoding token positions in transformer models.
- Mel Spectrogram: A visual representation of audio frequency data used for audio processing tasks.
- hMLP: Hierarchical Multi-Layer Perceptron, used here for processing independent image patches.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi