Miles v0.1: Production-level Post-training

We present Miles v0.1, a full-stack production-ready system for frontier post-training, the successor to our first Miles release. Building upon slime's clean design, Miles optimizes every stage in the RL training loop around a simple principle: verified, clean, and customizable everywhere. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and developers alike. In this blog post, we will walk through Miles end-to-end.

The Miles RL Loop

An RL training job in Miles is a loop over the following stages:

1. Rollout — SGLang engines generate trajectories. In agentic RL, each multi-turn rollout session interacts with its own isolated environment that executes actions and produces the reward.
2. Training — completed trajectory groups are consumed by the trainer (NVIDIA Megatron-LM or FSDP), which computes the RL loss and updates the policy.
3. Weight update — new weights are synchronized back to the rollout fleet with minimal interruption to in-flight rollouts.

In what follows, we will go through each element in the loop to highlight how we make Miles accurate, efficient, reliable, and scalable.

Rollout

Every rollout in Miles is generated by SGLang. Building upon SGLang's native inference efficiency, Miles unlocks the full agentic training workflow: multi-turn sessions, tool execution, sandboxed environments, and token-faithful trajectory capture.

Fast Agentic Rollout by SGLang

Miles provides fast agentic rollout by natively integrating SGLang, which is optimized for long, multi-turn generation. Agentic trajectories vary widely in length, and each new turn reuses most of the previous context. By default, Miles uses the SGLang router, which keeps all turns of a session on the same SGLang engine and DP rank (if DP attention is enabled) to reuse the cached prefix, while assigning new sessions to the least-loaded rank so that a few long trajectories do not overload part of the fleet. The SGLang router also reserves KV-cache capacity for long sessions beforehand. These features ensure a balanced and stable rollout concurrency, keeping cache-hit rate high in agentic training.

Fully Async RL

For long-context, tool-use, and agentic workloads, rollout time is mainly determined by a handful of stragglers. A synchronous schedule exacerbates the straggler issue, because the trainer has to sit idle until the slowest trajectory in the batch returns, and the rollout engines have to wait until the optimizer finishes model update. Miles' fully asynchronous RL eliminates such mutual-blocking by allowing rollout engines to generate persistently: rollout generation stays continuously in flight while the trainer consumes completed groups and updates the model. Neither side is blocked by the other.

Scheduling operates at sample granularity: each completed trajectory immediately frees a slot, keeping generation concurrency stable despite large differences in trajectory length. Completed groups enter a bounded data buffer that decouples rollout throughput from training cadence. This buffer also forms a customizable policy boundary where users decide which samples to keep, retry, discard, or reject as stale without modifying scheduling or execution.

To make evaluation asynchronous, Miles provides three evaluation modes, distinguished by where model weights come from.

• Shared-engine evaluation uses the rollout fleet and temporarily pauses new submissions, making it suitable for small debug sets without extra GPUs.
• Dedicated evaluation uses a separate GPU fleet loaded from checkpoint snapshots, allowing training and rollout to continue uninterrupted.
• External evaluation passes a checkpoint directory to any user-provided evaluator, including non-SGLang services.

Agentic Environments

In agentic RL, much of the work happens in an isolated environment. Training a coding agent, for example, means giving the model its own sandbox for each task: it runs commands, edits files, reads what comes back, and a test suite at the end decides whether the task was solved. The environment holds that state, executes the actions, and runs the verifier that produces the reward. Miles runs many such episodes at once, records each trajectory in a form the trainer can learn from, and carries the verifier's result through as the reward.

Token-In-Token-Out (TITO)

In multi-turn agentic RL, model outputs pass through message parsing, tool execution, and chat-template rendering before entering the next turn. This process can change tokenization, prune historical reasoning, or reserialize tool calls, causing the trainer to see a different token context from the one actually used during rollout.

The TITO session server in Miles preserves the exact token IDs generated by the model. On each new turn, it tokenizes only the newly appended messages and merges them with the existing prefix. This allows the complete trajectory to be assembled into one contiguous training sample, preserving the original rollout log probabilities while loss-masking tokens that are not generated by the model.

Efficient Rollout Routing Replay (R3)

MoE RL is sensitive to tiny numerical differences between rollout and training: a single top-k routing flip changes both the token computation and the expert weights that receive the gradients. To resolve this issue, Miles' Rollout Routing Replay (R3) records SGLang's expert routing results during rollout and replays them during training. Such replay is handled efficiently in SGLang, adding only minimal overhead compared with normal routing.

Training

Trainer is the backbone of RL. In Miles, we ship numerous trainer optimizations that make RL stable, fast, and resource-efficient.

Low-precision Training

Miles supports rollouts in NVFP4, MXFP4, MXFP8, and FP8; end-to-end training recipes for NVFP4, MXFP8, and FP8; and INT4 quantization-aware training (QAT) for most models. Capturing Blackwell's low-precision throughput in RL requires more than swapping GEMM dtypes: quantization on the rollout and training sides must agree, or the mismatch accumulates across weight updates into policy divergence.

Memory Efficiency & Disk Offload

Training a 744B-parameter model asynchronously on 16 nodes poses a significant memory issue. Miles ships with sophisticated memory optimization that large-scale runs depend on, starting with the optimizer: optimizer states can be offloaded to CPU or node-local NVMe and streamed back per bucket during each optimizer step, so they never need to be resident on the GPU at once.

Two Training Backends

Miles supports two training backends behind one interface: NVIDIA Megatron-LM and PyTorch FSDP.

• Megatron-LM is the default, and the backend the model recipes are written against. It splits the model internally across tensor, pipeline, context, expert and expert-tensor parallelism, and supports CPU and NVMe optimizer offload.
• FSDP trains the model's own HuggingFace implementation under PyTorch FSDP2.

Weight Update

In each training step, after the optimizer updates the model weights, the new weights have to be synchronized across all rollout engines. Miles provides two optimized paths for diverse deployment settings. With P2P weight transfer, training ranks re-shard each weight bucket for the target SGLang layout and write only the required shards directly into rollout-rank memory over RDMA. For rollout fleets without direct NCCL or RDMA connectivity, Miles supports disk-delta updates.

Verified Day-0 Model Support

Miles lands new frontier models on the day their weights become public, together with SGLang. Kimi-K3, DeepSeek-V4, Inkling, Qwen3.8 and NVIDIA Nemotron 3 Ultra were all trainable in Miles on release day, because the inference path in SGLang and the RL recipe in Miles are brought up in parallel.

Other Post-Training Recipes

LoRA RL

Miles enables end-to-end LoRA reinforcement learning across training, weight synchronization, and SGLang rollout, for both LLMs and diffusion models. The base model remains frozen and resident, while Miles trains and synchronizes only the adapters, and SGLang applies the latest adapter during rollout.

On-Policy Distillation (OPD)

Miles supports on-policy distillation, which allows a student to learn from teacher guidance on its own rollout distribution. The reverse KL is implemented as a "reward," so users can optionally complement the distillation with conventional GRPO/PPO-style rewards.

Zero-KL Alignment

Miles supports Zero-KL Alignment, minimizing numerical differences between rollout and training engines by aligning attention, GEMM, operator precision, batching behavior, and other execution details, in addition to model weights.

Code Quality Principle

The training driver is intentionally written like pseudocode, while major components, like rollout functions, data sources, losses, and rewards, sit behind small, typed interfaces. The rollout stack is divided into agent, generation, and rollout layers, so an environment or agent framework can replace only the layer it needs while reusing everything else.

Miles-Diffusion

Miles' design philosophy extends beyond LLMs to diffusion models. In Miles-diffusion, sglang-diffusion serves as the high-performance rollout engine and returns the full denoising trajectory with per-step log-probs. An FSDP2 trainer consumes selected SDE steps and optimizes the RL objective under hybrid shard and SP.

Multi-Hardware Support

Miles runs the same training loop on NVIDIA and AMD GPUs. AMD support is native ROCm through HIP and RCCL: Miles runs on MI300X through MI355X with the same SGLang rollout integration and dedicated Docker images.

Example: Training GLM-5.2 on Terminal-Use Tasks with 64 NVIDIA GB300 GPUs

As an end-to-end example of agentic RL training, we demonstrate how we use Miles to train a GLM-5.2 744B-A40B model on terminal-use coding tasks using fully asynchronous RL across 64 NVIDIA GB300 GPUs, with 32 GPUs for rollout and 32 GPUs for training.