DEVOURED
Apple's 20B On-Device Model Uses Just 2GB of Memory
Apple’s Core Advanced model achieves 20B-parameter performance by utilizing flash storage for experts and activating only 1–4 billion parameters per prompt.
What: The model, built in collaboration with Google, uses 'Instruction-Following Pruning' and 2-bit quantization to fit within 2GB of RAM on M3 Max Macs. It relies on a custom routing design that avoids the memory-heavy standard Mixture-of-Experts (MoE) approach.
Why it matters: Apple is proving that for edge devices, memory-efficient routing and clever I/O management are more critical for daily utility than raw parameter count or standard model architecture.
Takeaway: Use Apple’s on-device model for short-context tasks like summarization and classification, but stick to larger open models for general reasoning, math, and code.
Deep dive
- Models stay largely on flash storage, swapping active experts into memory as needed.
- Uses 2-bit weights to minimize memory footprint and energy usage.
- Optimized specifically for the Apple Neural Engine to keep GPU resources free.
- Employs separate safety-guardrail models to screen inputs and outputs independently.
- Lacks the reasoning performance of larger general-purpose models due to its constrained design.
Decoder
- Instruction-Following Pruning (IFP): A method to prune or reduce model parameters specific to certain task-following requirements.
- Mixture-of-Experts (MoE): A neural network architecture where only a subset of the model's 'experts' (sub-networks) are activated for any given token to reduce computation.
Original article
Full article content is not available for inline reading.
Explore all 69 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi