DEVOURED
FreeToken: Efficient Edge-Native MoE Serving
FreeToken makes high-end MoE models run on consumer hardware by dynamically remapping computation to match available bandwidth and memory.
What: FreeToken is an edge-native MoE serving system that supports over 20 models, enabling systems like an 8GB laptop GPU to run 35B models and a single workstation GPU to handle 753B parameter models.
Why it matters: It transforms consumer-grade hardware into viable platforms for frontier-scale intelligence, bypassing the need for dedicated datacenter infrastructure.
Deep dive
- Treats personal machines as unified, elastic inference platforms.
- Dynamically maps expert residency and model state to available resources.
- Co-designs the entire serving stack from model loading to CPU-GPU execution.
- Supports agentic state reuse for continuous workload changes.
- Enables 753B parameter models on workstation-class hardware.
Decoder
- MoE (Mixture of Experts): A neural network architecture that activates only a subset of parameters for any given input, improving efficiency.
- Expert residency: The strategy of which part of a model's parameters (experts) are kept in fast vs slow memory.
Original article
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at http://flashml.ai.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi