DEVOURED
Optimizing On-Device Inference for Apple Silicon
Apple's new Lily engine optimizes on-device LLM inference by better leveraging unified memory and specialized silicon than existing alternatives.
What: Apple introduced the Lily engine, which outperforms MLX-LM in prefill and decode speeds. It is designed to handle sparse MoE (Mixture of Experts) architectures and Gated DeltaNet layers found in models like Qwen3.6-35B-A3B.
Why it matters: This indicates that specialized inference engines tuned for Apple's specific memory architecture are becoming essential for running large, complex models locally at acceptable performance levels.
Decoder
- MoE (Mixture of Experts): A neural network architecture that activates only a subset of the model's parameters for each input, allowing for large total parameter counts with lower compute costs.
- Prefill and Decode Throughput: Prefill measures how fast the model processes prompt tokens, while decode measures the speed of generating output tokens.
Original article
Apple's Lily engine optimizes on-device LLM inference for Apple silicon. It speeds up processing by leveraging Apple silicon's unified memory and specialized hardware, outperforming MLX-LM in prefill and decode throughput. Qwen3.6-35B-A3B model's unique architecture, including sparse MoE routing and Gated DeltaNet layers, enables advanced engine tuning for efficient model execution on one Mac.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi