DEVOURED
How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference
OpenAI optimizes agentic loops by minimizing network round trips, leveraging persistent WebSockets, and separating prefill from decode phases in inference.
What: OpenAI engineers utilize techniques like stable prompt prefixes, deferred tool discovery, and 'Code Mode'—where agents write small programs to execute multiple tools in parallel—to reduce the cost per task for systems like Codex.
Why it matters: As AI moves from one-off chat responses to iterative agentic loops, optimization must shift from raw compute power to reducing redundant processing and data retransmission.
Deep dive
- Persistent WebSockets: Avoid redundant TCP/TLS handshakes by keeping one connection open across multiple agent calls.
- Stable prompt prefixes: Reuse cached KV state by appending to prompts rather than reconstructing full contexts.
- Deferred tool discovery: Keep tool schemas out of the main prompt, only loading them via lexical search when explicitly needed.
- Code Mode: Emit executable code rather than sequential tool calls to batch operations and keep intermediate data out of the context window.
- Safety/Inference Parallelism: Run safety classifiers in parallel with the prefill phase to mask latency.
- Cache-aware routing: Route requests back to the GPU machine that already holds the conversation state to avoid recomputing the KV cache.
- Speculative Decoding: Use smaller draft models to propose token sequences, verified in parallel by the large model.
Decoder
- KV cache: A key-value store of previously computed attention states; essential for generating long outputs without re-processing the entire history.
- Prefill vs. Decode: Prefill is the compute-heavy phase that processes input tokens to build the KV cache; Decode is the memory-heavy phase that generates new tokens one by one.
- Speculative Decoding: A technique where a small model guesses multiple future tokens and a larger model verifies them simultaneously, significantly reducing latency.
Original article
Full article content is not available for inline reading.
Explore all 59 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi