DEVOURED
The WASTE inference engine
WASTE is a new open-source inference engine enabling users to run LLMs on hardware with less memory than the model's total weight.
What: Developed by Marco Bambini, the WASTE inference engine is designed for resource-constrained environments. It officially supports the Kimi K3 model, allowing it to run on hardware like a 64 GB unified memory MacBook Pro.
Why it matters: By decoupling model performance from high-end specialized hardware, projects like WASTE lower the barrier to entry for local, private AI deployments and reduce dependency on proprietary cloud infrastructure.
Decoder
- Inference Engine: Software optimized to execute pre-trained machine learning models to generate predictions or content.
- Unified Memory: A memory architecture where the CPU and GPU share the same physical RAM, common in Apple Silicon, which facilitates running large models without discrete video memory.
Original article
WASTE is an open source inference engine designed to run models whose weights are substantially larger than the memory available on the host machine. It is an initial step into a broader effort to make increasingly capable models available on more hardware. The project aims to provide people and organizations greater control over infrastructure costs, data privacy, availability, and deployment. The first fully supported model by WASTE is Kimi K3, which can run on a MacBook Pro with 64 GB of unified memory.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi