DEVOURED
DwarfStar (GitHub Repo)
DwarfStar is a specialized local inference engine designed specifically for running DeepSeek V4 models across consumer hardware.
What: Salvatore Sanfilippo (antirez) released DwarfStar, a highly optimized, non-general inference engine for DeepSeek V4 and GLM 5.2, featuring tensor parallelism and SSD streaming for high-memory models.
Why it matters: By specializing the inference engine for specific model architectures, it achieves performance levels and hardware utilization that general-purpose GGUF runners cannot reach.
Takeaway: If you have high-end consumer hardware (like a 128GB Mac or multi-GPU server), use DwarfStar to run large models locally with specialized quantization.
Deep dive
- Models: Optimized for DeepSeek V4 Flash, PRO, and GLM 5.2.
- Parallelism: Supports pipeline and tensor parallelism over TCP or Thunderbolt.
- SSD Streaming: Allows running models that exceed RAM by streaming routed-expert weights from SSD.
- Tooling: Includes a native coding agent and evaluation suite.
- Hardware: Targeted at Apple Silicon (Metal), NVIDIA (CUDA), and AMD (ROCm).
Decoder
- Tensor Parallelism: A technique that splits a model's matrices across multiple GPUs, allowing a single model to run faster and with less memory per card.
- GGUF: A binary file format for storing models for inference, popularized by the llama.cpp project.
Original article
Full article content is not available for inline reading.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi