DEVOURED
Colibrì (GitHub Repo)
Colibrì is a C-based inference engine that streams massive 744B+ parameter Mixture-of-Experts models from NVMe storage to enable frontier-model performance on consumer hardware.
What: Colibrì is a single-file C inference engine that manages weights across VRAM, RAM, and NVMe as a unified memory hierarchy. It supports nine model families including GLM-5.2 (744B) and Kimi K3 (2.8T) by streaming experts on-demand, allowing systems with limited fast memory to run models that would otherwise not fit.
Why it matters: This challenges the assumption that running frontier models requires massive GPU VRAM clusters, showing instead that intelligent weight-staging and I/O optimization can bring large-scale intelligence to local hardware.
Takeaway: If you have 25GB of RAM and a fast NVMe drive, download a pre-converted gs64 container of GLM-5.2 and run `./coli chat` to test the engine on your local workstation.
Deep dive
- Implements memory multitiering where VRAM, RAM, and NVMe are treated as a single hierarchy.
- Uses an LRU cache and learned pinned hot-store for on-demand expert loading.
- Supports heterogeneous execution across CPU, CUDA, Metal, and Vulkan.
- Includes native MTP and grammar-forced drafting for speculative decoding.
- Provides a cluster mode for distributing inference across multiple worker machines.
- Validates correctness against standard Hugging Face/Transformer implementations.
- Requires specific int4/gs64 weight containers to ensure output quality.
Decoder
- Mixture-of-Experts (MoE): A model architecture where only a subset of parameters (experts) are activated for each token, allowing for high model capacity with lower compute costs per forward pass.
- VRAM: Video Random Access Memory; high-bandwidth memory dedicated to GPU operations.
- NVMe: Non-Volatile Memory Express; a high-performance storage protocol for SSDs.
- NUMA: Non-Uniform Memory Access; a memory design used in multi-processor systems where memory access time depends on the processor's location relative to the memory bank.
- TTFT: Time To First Token; the latency between sending a request and receiving the first part of the generated response.
Original article
Full article content is not available for inline reading.
Explore all 74 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi