Back to Devoured - November 16, 2026
DEVOURED
Embla 1

Embla 1

AI Sevren
Embla 1 is a new reasoning model optimized to run privately and offline on consumer smartphones with sub-2-second latency.
What: The Sevren-built model features a unique architecture including shared K/V caches and sparse attention, scoring 57.4 on combined benchmarks while answering in 1.77 seconds on an iPhone 18 Pro.
Why it matters: This demonstrates a shift toward highly specialized, efficient architectures that trade off generalist performance for extreme local inference speeds on mobile hardware.
Takeaway: Download the 4-bit weights from Hugging Face to experiment with local mobile reasoning.
Deep dive
  • Features 32 layers of 2048 width with 8 routed experts and one shared expert per token.
  • Uses 15 sliding-window layers (128 tokens) and 12 sparse attention layers.
  • Replaces softmax with ReLU squared in full-attention layers for efficiency.
  • Implements 'Engram', a 3B-parameter table hashing the last few tokens for contextual injection.
  • Shared K/V caches allow sparse layers to read from the preceding full-attention layer instead of maintaining their own.
  • Achieved through multi-stage training including QAT (Quantization-Aware Training), GRPO, and MOPD (Multi-Teacher On-Policy Distillation).
Decoder
  • K/V Cache: Memory buffer storing key and value vectors from previous tokens to avoid recomputing attention during inference.
  • QAT (Quantization-Aware Training): Training technique that simulates quantization effects during the forward pass to prepare weights for low-precision formats.
  • GRPO (Group Relative Policy Optimization): A reinforcement learning method that groups model outputs to estimate a baseline for policy improvement without requiring a separate critic model.
Original article

Embla 1: A reasoning model built for phones

Embla 1 runs on a phone, private, offline and free per request. This is how we trained it, and how it compared with other small models in our tests at 4 bits.

At 4 bits Embla 1 averages 57.4 on BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 and answers in 1.77 s end to end on an iPhone 18 Pro. In our measurements, the models that scored higher took at least twice as long on the same phone.

Embla 1 is a reasoning model built to run on a phone, private, offline and free per request. We designed it to be fast and to use few tokens per answer, and trained it on 16 nodes of 8× H100.

Architecture

Embla 1 is built to be fast on a phone in both prefill and decode. It has 32 layers of width 2048, with 8 of 96 routed experts and one shared expert per token. We use 15 sliding-window layers of 128 tokens, 12 sparse attention layers and 5 full attention layers. It is built around the following:

  • Sevren Sparse Attention. Full-attention layers use ReLU² instead of softmax. About 95% of attention weights are exactly zero and decode skips their values.
  • Shared K/V caches. Sparse layers keep no cache of their own. They read from the preceding full-attention layer. K/V bridges build every second-half cache after layer 15 so prefill skips layers 16 to 31.
  • Engram. The last two and three tokens are hashed into a 3B-parameter table. Rows they hit form a second input next to the token embedding for every layer.

Training

The model is pretrained on 1.4T tokens followed by a 20B-token extension to 256K context. For post-training we run SFT on 123B tokens. During the last 2,000 SFT steps we train with quantization-aware training (QAT) and all later stages run at 4 bits. For RL we run 100 steps of on-policy distillation (OPD), 800 + 200 steps of GRPO and 300 steps of multi-teacher on-policy distillation (MOPD). Every stage before GRPO distils from a larger model.

Stage Tokens or steps Teacher or signal
Pretraining 1.4T tokens Qwen3.5-35B-A3B-Base
Long-context extension 20B tokens Qwen3.5-35B-A3B
SFT with QAT 123B tokens Qwen3.5-35B-A3B
OPD 100 steps Qwen3.8-27B and Qwen3.5-35B-A3B
GRPO 800 + 200 steps Verifiers and a judge
MOPD 300 steps GRPO-trained domain specialists

Pretraining

We pretrain at 4K context in two phases: 1T tokens without document masking, then 400B tokens with it. We use Qwen3.8's tokenizer pruned to 65,536 tokens. From the first step the model distils from Qwen3.5-35B-A3B-Base with 0.9 × KL on its top-256 logits plus 0.1 × cross-entropy. Its embeddings and output head start from the teacher's top principal components. Muon trains the weight matrices and AdamW the rest with 4M-token batches and a constant learning rate. We then extend the context to 256K over 20B tokens with Qwen3.5-35B-A3B Instruct as teacher.

Pretraining Phase 1 Phase 2
Web text, filtered and synthetic 50% 43%
Code 19% 19%
STEM, reasoning and QA (synthetic) 10% 10%
Math 9% 8%
PDFs 8% 14%
Multilingual web (10 European languages) 5% 5%

Post-training

SFT. We train on 123B tokens in 32K packed sequences with Qwen3.5-35B-A3B Instruct as teacher on assistant tokens. Reasoning traces in math, code and STEM are capped at 8K tokens. One in ten reasoning prompts gets no reasoning. The last 2,000 steps run QAT in the phone runtime's 4-bit format: round-to-nearest in groups of 64 with a bf16 scale and offset per group.

SFT Share
Chat 25.5%
Math 16%
Tool use 14%
Instruction following 12%
Code 12%
STEM 10%
Multilingual 6%
Long context, safety, other 4.5%

OPD. We run 100 steps with answers up to 8K tokens. Qwen3.8-27B and Qwen3.5-35B-A3B score each answer token by token. Both are pruned to our vocabulary and keep 99.1% of their probability mass.

GRPO. We run 800 steps with 16 answers per prompt and no reference model. Verifiers score math, code, tool calls and instruction following. Another 200 steps add knowledge questions judged by gpt-oss-120b: +1 correct, 0 abstain, −1 wrong. The last 180 of them add a loop penalty of −10−4 per token in a repeated 40-gram of a correct answer up to 0.5. Answers hitting the 16K limit fell from 20.7% to 1.8%.

MOPD. We run 300 steps with GRPO-trained specialists as teachers for four domains: tool use, instruction following, math and abstention.

Results

Score is the mean of BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 with up to 16K tokens per answer and reasoning on. All models run at 4 bits: Embla 1 in the phone runtime's format, the others from their official Q4_K_M GGUF weights. Each model uses its lab's recommended sampling settings. Latency uses 4-bit builds on an iPhone 18 Pro with a 1,024-token prompt and 256 output tokens.

Model Mean tokens per answer vs Embla 1 Score
Embla 1 888 1.0× 57.4
Gemma 4 E4B 1,577 1.8× 60.4
Gemma 4 E2B 1,611 1.8× 52.8
LFM2.5-8B-A1B 3,129 3.5× 58.8
LFM2.5-2.6B 3,557 4.0× 62.7
Granite 4.2 3B 5,633 6.3× 61.0
MiniCPM5-1B 6,461 7.3× 43.1
Qwen3.5-4B 6,619 7.5× 60.0

Embla 1 is clearly behind on math: 76.1 on MATH-500 against 89.6 for LFM2.5-8B-A1B and 90.5 for LFM2.5-2.6B. There is still a lot to improve.

References

  1. Benoit Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", CVPR, 2018.
  2. Rishabh Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes", ICLR, 2024.
  3. Zhihong Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300, 2024.
  4. Xiaomi LLM-Core Team, "MiMo-V2-Flash Technical Report", arXiv:2601.02780, 2026.
  5. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531, 2015.
  6. Zhiqiu Xu et al., "Initializing Models with Larger Ones", ICLR, 2024.
  7. Keller Jordan et al., "Muon: An optimizer for hidden layers in neural networks", blog post, 2024.
  8. OpenAI, "gpt-oss-120b & gpt-oss-20b Model Card", arXiv:2508.10925, 2025.
  9. Edward Yeo et al., "Demystifying Long Chain-of-Thought Reasoning in LLMs", arXiv:2502.03373, 2025.
Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi