The Next Frontier of Visual AI Is Code
Visual AI is evolving from generating static pixels to creating editable source code like HTML/CSS and Blender scripts, enabling iterative design and 3D modeling.
Summary
Deep Dive
Decoder
Original Article
Preventing AI Inference Theft at Scale
Vercel revealed how AI inference theft, costing up to $2 per prompt, is a high-margin attack on exposed endpoints, overcoming traditional rate limits and auth walls.
Summary
Deep Dive
Decoder
Original Article
GitHub's plan for Agents
GitHub COO Kyle Daigle discusses how the platform's infrastructure, designed for human speeds, is straining under a 1400% surge in agent-driven code contributions in 2026, leading to uptime issues.
Summary
Deep Dive
Decoder
Original Article
TinyFish Bigset turns text prompts into live datasets
TinyFish launched Bigset, an open-source, multi-agent system that converts text prompts into self-refreshing, structured datasets pulled from the live web using agents powered by Claude Sonnet 4.6 and Qwen3.7-max.
Summary
Deep Dive
Decoder
Original Article
In a surprise launch, China debuts another big rocket designed for reusability
China's state-backed Long March 12B, a new large rocket resembling SpaceX's Falcon 9, completed its first successful test flight with reusability features, indicating an accelerated race for reusable launch vehicles.
Summary
Deep Dive
Decoder
Original Article
The race to field China’s first reusable launch vehicle is far less predictable than a similar competition that played out in the United States a decade ago.
There was never any real question of which company would develop and demonstrate the first reusable orbital-class rocket in the United States. SpaceX landed a Falcon 9 booster for the first time in 2015, and a little more than a year later, it launched it back into space. It took nearly 10 years for anyone else to do the same. Blue Origin celebrated its first orbital-class booster landing last November with the successful recovery of one of its New Glenn boosters, followed by a relaunch of the same rocket in April.
In China, several companies and state-owned enterprises have a realistic shot at landing an orbital-class booster stage this year. For a time, it seemed like China’s new crop of privately funded launch companies might have the advantage in accomplishing the first landing of an orbital-class booster. But Monday’s launch of China’s Long March 12B rocket, backed by the nearly unrestricted resources of the country’s vast state-owned aerospace enterprise, suggests the industry’s legacy players may now have a leg up.
Secrecy reigns
China’s first two attempts to recover heavy boosters failed in December. First, a company named LandSpace, part of China’s recent wave of quasi-commercial launch providers, debuted its Zhuque 3 rocket on December 2. The launch was successful, but the booster crashed near its landing zone downrange from its launch site in the Gobi Desert of northwestern China. Less than three weeks later, a somewhat less powerful rocket named the Long March 12A had a similar result on its first test flight. The Long March 12A is a product of the Shanghai Academy of Spaceflight Technology, part of China’s legacy government-owned space industry.
In early April, another relative newcomer to China’s launch sector launched its new medium-class Tianlong 3 rocket. The 7-year-old firm behind the Tianlong 3, named Space Pioneer, said the rocket failed to reach orbit, an outcome not uncommon for brand-new launch vehicles. Tianlong 3’s first stage booster is designed for recovery and reuse, but a landing attempt will have to wait until a future flight.
The Long March 12B, the largest and most powerful (potentially) reusable rocket China has launched to date, lifted off Monday from a remote launch pad in the Gobi Desert. The 236-foot-tall (72-meter) rocket took off at 4:40 pm Beijing time (08:40 UTC or 4:40 am EDT).
Unusually, Chinese officials appear not to have announced the launch in advance. The Chinese government did not issue any public notices for pilots to avoid the rocket’s flight path, as is customary for space launches around the world. It’s too soon to know if this was a one-off change or the start of a new policy for Chinese launches. Russia’s government, which has historically also released safety notices for its space launches, has begun issuing such warnings to cover extended periods over many days in a bid to conceal when a launch might actually occur.
The existence of the Long March 12B was not a secret. The rocket completed a test firing on its launch pad in China in January, and a launch was expected in the first half of this year. It was developed by China Commercial Rocket Co. Ltd., or CACL, an opaque business venture set up by China’s sprawling state-owned aerospace enterprise. According to Chinese state media reports, engineers designed and developed the Long March 12B in just 21 months. If the claim is true, it would be a remarkably fast timeline to progress from a clean sheet to an orbital flight.
Monday’s launch did not include any attempt to land the first stage booster, but the rocket carried grid fins and landing legs, important hardware elements for future recovery experiments. A statement released by China Aerospace Science and Technology Corporation (CASC), CACL’s parent company, declared the first flight of the Long March 12B a “complete success” in a post-launch statement.
“This launch adds another high-capacity commercial rocket to [China’s] fleet for large-scale Internet constellation networking missions,” CASC said. “No recovery tests were conducted during this mission; however, first-stage recovery tests are scheduled to be carried out at a later, opportune time.”
Satellites for one of these large-scale Internet constellations rode to space aboard the Long March 12B, which released a batch of Qianfan broadband spacecraft into low-Earth orbit. Qianfan is one of China’s two leading mega-constellations, each seeking to replicate for China what SpaceX’s Starlink does in the United States.
Who’s involved?
The Long March rocket family dates back to 1970, when China launched its first satellite into orbit using the Long March 1 vehicle derived from Chinese ballistic missiles. Many iterations have followed. The Long March 2, 3, and 4 rockets were China’s workhorses in the 1980s, 1990s, and 2000s. These rockets remain operational but are being replaced by newer models, such as the Long March 5, 6, 7, and 8, which can launch everything from small satellites to massive modules for China’s space station.
Somewhat confusingly, the Long March 12 family now includes three dissimilar designs. The original Long March 12, with four kerosene-fueled main engines, launched for the first time in 2024, sporting a conventional, expendable design. The partially reusable Long March 12A launched in December, replacing the Long March 12’s kerosene-fueled engines with a methane-fueled propulsion system outsourced to a private engine builder. The Shanghai Academy of Spaceflight Technology (SAST) managed the development of the Long March 12 and 12A.
China’s Long March 12A rocket, using methane-fueled engines, launched for the first time in December. Credit: VCG/VCG via Getty Images
The Long March 12A rocket reached orbit, but its lift capacity is half that of the Long March 12 after accounting for the fuel reserve required to recover the booster stage. The Long March 12B reverts back to the kerosene and liquid oxygen mix used on the Long March 12, but with nine engines on the first stage instead of four. The Long March 12B is also taller and wider than the 12 or 12A. Collectively, these changes allow the Long March 12B to approach the payload capacity of the Long March 12, even when it flies in reusable mode.
These rocket developments were directly orchestrated by the Chinese government, which owns institutions like CASC, SAST, CACL, and the granddaddy of all Chinese rocket developers: the China Academy of Launch Vehicle Technology. CALT is in the advanced stages of readying a new partially reusable rocket, the Long March 10, to send astronauts to the Moon. Further behind is the Long March 9, a super-heavy-lift launch vehicle sized as a Chinese analog to SpaceX’s Starship. A suborbital version of the Long March 10 made a controlled, on-target splashdown following a test flight in February.
A tip of the hat
Many of China’s up-and-coming rockets bear a striking resemblance to those developed halfway around the world by SpaceX. As Ars has previously reported, the Long March 9 is supposed to be China’s answer to Starship. But other Chinese rocket programs are still trying to catch the same lightning SpaceX caught with the Falcon 9.
The Long March 12B that launched Monday uses nine kerosene-fueled main engines and a single engine on the second stage, the same as the Falcon 9. The engines generate 1.7 million pounds of thrust at liftoff, the same as the Falcon 9. In expendable mode, the Falcon 9 rocket can deliver nearly 23 metric tons (about 50,000 pounds) of payload to low-Earth orbit. The Long March 12B’s payload capacity to low-Earth orbit maxes out at about 20 metric tons (44,000 pounds) on an expendable mission.
Space Pioneer’s Tianlong 3, which failed on its first flight in April, uses the same kerosene-fueled, nine-engine cluster arrangement on the booster, coupled with a single-engine second stage. Its lift capability is a little less than that of the Falcon 9 or Long March 12B, but it is similar in height and thrust.
The Long March 12A and Zhuque 3 are powered by groupings of seven and nine methane-fueled engines, respectively, on their booster stages.
There are sound engineering reasons to use the same approach SpaceX uses with the Falcon 9. A clustered engine configuration on the first stage, usually with seven or nine engines, offers several advantages. It allows a booster stage to provide high thrust during ascent, and a lower power level during propulsive landing burns. In some cases, flying with a cluster of booster engines might allow the rocket to continue its mission even if one of them fails.
There are important, less obvious aspects of a rocket’s design. We have less insight into how closely those elements on China’s rockets match what SpaceX has tried and proven on Falcon 9.
But SpaceX’s architecture clearly works. The first version of Blue Origin’s New Glenn rocket uses seven engines fueled by methane, and the company is looking at going to nine booster engines in the future. Several more US rocket companies are pursuing similar designs for their reusable rockets. Nearly all are going with seven- or nine-engine boosters.
Karpathy's Autoresearch found a 3-year-old bug in Posthog's query engine (and improved performance by 11%)
PostHog used an "autoresearch" AI agent, inspired by Andrej Karpathy, to uncover a 3-year-old ClickHouse query engine bug, improving query performance by 11% by correctly utilizing primary keys.
Summary
Deep Dive
Decoder
Original Article
The Google Capital Company
Google is raising $80 billion in equity, including $10 billion from Warren Buffett's Berkshire Hathaway, to fund its massive AI spending, signaling an unprecedented demand for compute capacity.
Summary
Deep Dive
Original Article
Microsoft unveils new AI models to lessen reliance on OpenAI and lower costs for developers
Microsoft launched its own AI coding model, MAI-Code-1-Flash, and a reasoning model, MAI-Thinking-1, at Build 2026, aiming to reduce dependence on OpenAI and lower developer costs.
Summary
Deep Dive
Decoder
Original Article
The Inference Tax: How Prefix-Aware Routing Eliminates the Hidden Cost of LLMs at Scale
DigitalOcean and Inferact dramatically reduced LLM inference costs by up to 4x using prefix-aware routing and caching in vLLM, reclaiming 340 GPU-hours daily at scale.
Summary
Deep Dive
Decoder
Original Article
The Inference Tax: How Prefix-Aware Routing Eliminates the Hidden Cost of LLMs at Scale
By Piyush Srivastava and Simon Mo, CEO of Inferact
Introduction
Inference demand is growing fast, and it’s only accelerating. By 2030, inference is expected to account for the majority of AI compute globally. But scaling inference isn’t just a hardware problem. Most teams discover too late that a significant portion of their compute spend is avoidable, primarily because their systems are silently repeating work they have already done, recomputing the same prompt prefixes and system instructions over and over again.
We’ve seen this from two vantage points. From the infrastructure layer, the cost curve becomes visible at scale with clusters that look busy but aren’t efficiently utilized. From the engine layer, the picture is just as clear. Without the right caching and scheduling primitives, even a well-optimized model wastes cycles on redundant computation. The root cause is the same regardless of where you’re standing. The system lacks the memory and coordination to recognize when it’s already done the hard part.
Fixing this requires work at every layer of the stack. DigitalOcean has invested in GPU optimization across multiple fronts, from vLLM parallelism and quantization tuning to hardware-level kernel work. But one technique has had an outsized impact on cost efficiency at scale: prefix-aware routing and caching. In this post, we walk through how vLLM enables advanced prefix caching, how DigitalOcean’s inference gateway uses prefix awareness to make smarter routing decisions, and how we plan to make this available to everyone on Serverless Inference in the coming weeks.
The Cost Cliff and the Hidden Culprit
Inference now accounts for roughly 70% of total AI compute costs. For most teams, a significant share of that is avoidable. It’s not due to hardware limits. Instead, it’s because the system keeps recomputing work it has already done, also known as redundant prefill.
Every LLM inference request has two distinct computational phases. The first phase is prefill, where the model processes the entire input sequence and builds the KV (key-value) cache that represents its state. The second phase is decode, where the model generates output tokens one at a time, attending back to that cached state. Prefill is where the structural inefficiency hides. Its computation scales quadratically with input length: attention computation quadruples with doubling of input length.
Consider a real-world customer support workload running on NVIDIA H200 or AMD Instinct™ MI325X GPUs. A typical deployment carries a 2,000-token system prompt (defining persona, policies, response format) that is identical across every request. With an average user message of 200 tokens, roughly 91% of every input is shared context.
On AMD Instinct™ MI325X GPUs or NVIDIA H200, prefilling 2,000 tokens takes approximately 45–50ms and costs in the range of 100-120 GFLOPs per request. At 10,000 requests per hour, that’s over 1 trillion redundant FLOPs per hour. Compute spent reconstructing the state the system has already built, discarded, and is now rebuilding from scratch.
The pattern is even more pronounced in coding assistants or document Q&A tools, where the same API documentation or reference material is prepended to nearly every request. A 5,000-token shared context costs roughly 600 GFLOPs to prefill, which is nearly 25× more than a 1,000-token prefix, due to that quadratic relationship. When hundreds of users are querying the same underlying documents, the redundant computation compounds rapidly.
This is precisely the redundant “prefill tax” that we will focus on how to eliminate in the rest of this post.
How Prefix Caching Works at the Engine Layer
The redundant prefill problem has a clean structural solution, but landing it at production scale takes several mechanisms working in concert. Here’s what’s happening inside the engine when a cache hit lands.
Block-Based KV Storage
During prefill, every input token produces a key and value tensor at every attention layer, and storing these per-token would be a memory-management nightmare. The engine instead groups them into fixed-size blocks (16 tokens by default on CUDA, though configurable) allocated out of a pre-reserved GPU memory pool sized at engine startup. Each layer maintains its own pool of blocks. A single block holds the K and V tensors for block_size tokens for one layer’s KV heads, laid out so PagedAttention kernels can read them with coalesced memory accesses. A 2,000-token system prompt occupies 125 block positions (allocated per layer under the hood); once those blocks are sitting in the pool, any future request that begins with the same 2,000 tokens can point at them rather than recomputing. PagedAttention is the kernel technique that operates on this block-based layout, and the same memory machinery underlies both prefix caching and paged attention’s batching benefits, described in more detail in the engine anatomy writeup.
Prefix Hashing and Cache Lookup
Recognizing that two requests share a prefix is a string-matching problem on potentially very long inputs, and doing it naively would defeat the point. The engine instead hashes prefixes block by block, with each block’s hash depending on its own tokens, the hash of the previous block, and any extra inputs that affect the computation, including LoRA adapter IDs, multimodal image hashes, and optional cache salts for multi-tenant isolation. Identical prefixes under identical conditions produce identical hash chains, and the first divergent block is also the first point where the hashes disagree. Only full blocks are hashed and cached, so a partial trailing block at the end of a prefix doesn’t get reused and is recomputed along with the rest of the suffix. These hashes live in a lookup table mapping hash to cached block, and finding “the longest prefix of this request that’s already cached” reduces to walking the request’s block hashes against the table. The lookup is small and cheap, and the KV data itself lives in the GPU memory pool. Memory pressure comes from the pool, not the index.
From Cache Miss to Cache Hit: The FLOP Savings
The payoff shows up in compute terms. On a full cache miss, the engine runs prefill across the entire input, processing every token across every layer at full quadratic attention cost. On a full hit on the prefix, the engine skips nearly all of that work: the KV state for the prefix is already materialized in GPU memory, and only the trailing token needs to run through prefill so the first generated token has somewhere to attend from. Partial hits land in the middle, with prefill running only over the un-cached suffix and cached blocks treated as pre-computed context. On the customer-support workload from Section 1, a partial hit covering the 2,000-token system prompt turns a 45–50ms prefill into something dominated by the much shorter user message, and the structural FLOP savings show up directly as time to first token (TTFT) improvement.
A single engine instance can only cache what it has personally seen, which is the routing problem Section 3 picks up. The engine publishes KV cache events (block stored, block removed, with their associated hashes) over ZeroMQ on a PUB/SUB channel, while utilization metrics like batch size and free-block count flow through the StatLogger path. A router consumes both to make decisions. The interface is deliberately neutral: any compatible consumer can subscribe, whether that’s NVIDIA Dynamo, llm-d, or a custom gateway built in-house. Session-affinity routing handles the easy case of sending a user back to the instance that served their previous turn, but the event stream enables much more. A router can build its own global prefix tree from KV block events, balance load against per-instance batch size and cache utilization, and make routing decisions that account for cache locality and instance pressure rather than treating them as separate problems.
Hardware Headroom: AMD and NVIDIA
These mechanism benefits compound on the AMD Instinct™ MI325X GPUs. 192GB of HBM3 per GPU means the KV pool can hold substantially more cached blocks than on comparable hardware, resulting in more cached prefixes, higher hit rates, longer-lived cache entries before eviction. Layered on top, FP8 KV cache quantization roughly doubles effective cache capacity again (though combining FP8 KV cache with prefix caching has historically required specific kernel support, so it’s worth checking compatibility for your vLLM version), and the attention kernels on the read path have been tuned for AMD Instinct™ MI325X GPUs memory hierarchy so a cache hit doesn’t trade prefill cost for a slow cache read. The mechanism works universally, but on AMD Instinct™ MI325X GPUs it has more room to operate, which is what makes the routing layer in the next section worth building.
The picture is similar to NVIDIA Hopper, with different shapes of headroom. NVIDIA H200’s 141GB of HBM3e per GPU expands the KV pool considerably over H100’s 80GB, which translates directly into more cached prefixes and longer-lived entries before eviction. FP8 KV cache lands on Hopper through FlashAttention 3 and FlashInfer kernels. The same caveat about checking prefix-caching compatibility for your vLLM version applies, and the read-path attention kernels have been tuned around TMA loads and the Hopper memory hierarchy, so a cache hit doesn’t trade prefill cost for a slow KV read. Blackwell stretches this further, with 192GB of HBM3e per B200, and on GB200 NVL72 the NVLink domain collapses 72 GPUs into a single shared-memory fabric, opening up cross-instance KV reuse that single-node caching can’t touch. The underlying mechanism is the same across vendors, and what changes is how much room it has to operate, which is exactly what the routing layer in the next section is built to exploit.
The Routing Problem: Why Single-Instance Caching Isn’t Enough
Once the KV state for a shared prefix is computed, subsequent requests that share that prefix can reuse the cached blocks directly, bypassing prefill entirely. But production workloads don’t run on a single instance. They run across fleets of GPU workers behind a load balancer, and this is where naive deployments silently destroy the cache hit rate they worked to build. The DigitalOcean Inference Gateway (which is a fork of llm-d) embeds an Endpoint Picker (EPP), a component that intercepts every inference request via Envoy’s external processing (ext_proc) callback before it reaches any vLLM instance. The EPP is where all routing intelligence lives.
The Write Path: Publishing KV Cache Events
On the write path, each vLLM instance is configured with --kv-events-config to publish KVEvent messages over a ZMQ socket (tcp://*:5557) every time a KV cache block is allocated or evicted. Each event carries the block hash - computed using sha256_cbor_64bit over the token IDs in that block, using the same algorithm vLLM uses internally. The EPP subscribes to all instances, consuming this high-throughput stream and continuously updating a KV-Block Index: a low-level map of block_hash → {pod, memory_medium (GPU/CPU)}. memory_medium is the storage tier the block currently lives in on that pod. In our current implementation, KV blocks are always in GPU memory, but this will soon change as we look into multi-tiered storage architecture for KV blocks. From this index, the indexer builds and maintains a per-pod prefix tree - a radix structure of consecutive block hashes that reflects exactly what prefix state is warm in each pod’s GPU memory at this moment.
The Read Path: Prefix-Aware Request Scoring
On the read path, when a new request arrives, the gateway tokenizes the incoming prompt and computes its prefix block hashes using the identical sha256_cbor_64bit algorithm. It then walks the KV-Block Index, querying how many consecutive prefix blocks each pod holds for this request. The result is a cache affinity score per pod: a pod holding 90% of the prompt’s prefix blocks scores 0.9 × 3 = 2.7 on the prefix-cache-scorer (weight 3, the dominant signal). This is combined with a kv-cache-utilization-scorer (weight 2) that down-scores pods whose GPU vRAM is near capacity, preventing the router from routing to a pod that would have to evict blocks to accommodate the new request, negating any cache benefit. The max-score-picker selects the highest combined score, and Envoy forwards the request to that pod. As we get into multi-tiered KV cache, we are also looking at “tier-aware” prefix scoring where a GPU resident match scores higher than a CPU-resident or lower tiers. In general, there are multiple cost functions with varying priorities taken into account while making the routing decision.
The result: cache hit rates flip from ~25% under round-robin to 75%+ on workloads with shared prefixes - on the same hardware, with no model changes.
The Math: What Cache Hits Mean at Scale
The impact becomes concrete when you put real numbers behind it. At 1 million requests per day, a modest scale for a production deployment, assume 70% of requests share a common system prompt. Without prefix-aware routing, cache hits are essentially random: roughly 1-in-4 requests land on an instance with that prefix already warm. With prefix-aware routing, that flips to 3-in-4.
That delta, 350,000 additional cache hits per day, doesn’t sound dramatic until you attach the compute cost. Each cache hit skips roughly 350ms of prefill work. Across those 350,000 requests, that’s 34 GPU-hours saved every single day. Scale to 10 million requests per day and you’re recovering 340 GPU-hours daily, compute that was previously being silently wasted on work the system had already done.
For the right workload profile, multi-turn conversations with persistent context, shared system prompts, RAG pipelines querying the same document sets, the economics compound further. The same prefix appears not just frequently, but across long sessions where every turn benefits. In these cases, prefix-aware routing can reduce effective compute cost by up to 4x per request on identical hardware.
The Engine Work Inferact Is Building
Prefix caching is one piece of a larger problem. Inference engines are evaluated on a lot of dimensions: raw speed, model coverage, and increasingly, how well they handle the messy shape of real production traffic. Closing the gap on all three, on frontier hardware, is the problem Inferact is organized around.
Inferact is a company built on vLLM, and in practice Inferact’s roadmap and vLLM’s roadmap are virtually identical. The work happens upstream, in the open, in the same repository everyone else uses.
The work falls into a few themes, each building on the last.
The first is pushing vLLM toward SOTA performance on frontier models on frontier hardware. The clearest external signal here is Artificial Analysis, whose independent benchmarks have become a common reference point across engines and providers. vLLM’s recent top rankings on DeepSeek V3.2 and DeepSeek V4 reflect work that is increasingly a community effort: kernel and fusion optimizations, large-scale serving improvements for disaggregated and wide-EP setups, speculative decoding, quantization, and torch.compile integration are all being pushed forward by contributors across vendors.
The second is day-0 model support, which is one of vLLM’s structural strengths. When a new frontier model drops, running it well on vLLM is the default, with recent launches like Gemma 4 and DeepSeek V4 supported on the engine from day one. Our goal is to continue this trend. The bar isn’t just accuracy on day zero. It’s continued accuracy and high performance on day zero.
The third part is optimizing vLLM for areas that benchmarks don’t measure well yet. Top-of-leaderboard token throughput on a single prompt is a real signal, but it isn’t the same as performance on the workloads that actually run in production. Real inference traffic, and agentic traffic especially, looks very different from what most benchmarks capture: long shared prefixes from system prompts and tool definitions, multi-turn conversations with rich cache-hit structure, and bursty arrival patterns that don’t resemble a steady stream of independent prompts.
Optimizing prefix caching is the clearest example of what this means in practice. On agentic traffic, the bottleneck isn’t raw decode throughput on a fresh prompt — it’s whether the engine recognizes that most of the prompt is identical to something it processed seconds ago, and reuses the KV cache accordingly. Getting this right can be the difference between a model feeling fast and a model feeling unusable in an agent loop, and on a standard benchmark it barely shows up at all. The same pattern holds for the rest of the production-traffic stack: scheduler design under bursty arrival, KV cache layout under high reuse, and the prefill/decode connector path all matter disproportionately on real workloads relative to what benchmarks reward.
None of this is work vLLM does alone, and none of it is work Inferact does alone either. The performance and workload story leans on cross-project, cross-vendor effort, and on the hundreds of vLLM contributors and downstream users who surface real problems and keep the project honest about what production actually looks like.
Inferact’s role in that ecosystem is to invest deeply in vLLM as a maintainer and contributor, not to fork it or wrap it. The bet is on an open, broadly-owned inference engine as the right foundation for the next several years of inference work.
These Optimizations Will Soon Ship to Everyone
Everything described in this post was originally built in the context of deep partnerships with large customers on DigitalOcean’s Dedicated Inference platform, but these optimizations will soon ship with every Serverless Inference deployment as well.
- Prefix-aware routing via the Inference Gateway (live on Serverless Inference now)
- Prefix caching with cached token pricing (launching on Serverless Inference in the coming weeks)
- vLLM runtime with optimizations on AMD Instinct™ MI325X GPUs as well as NVIDIA Hopper
- The same benchmark performance Simon and Inferact helped achieve
Prefix caching and routing are just part of the picture. DigitalOcean’s GPU hardware collaboration goes deeper across the stack, from FP8 quantization to parallelism tuning, and those gains flow through to Serverless Inference customers as well.
You won’t need a custom contract to benefit from these results. You will only need a DigitalOcean Serverless Inference endpoint.
Conclusion
Delivering best-in-class inference performance requires optimization at every layer of the stack, and no single team owns all of it. That’s the foundation of our partnership with Inferact.
Inferact brings deep expertise at the engine and kernel layer by optimizing vLLM internals, tuning GPU kernels for NVIDIA and AMD hardware, and squeezing maximum throughput out of the compute itself. DigitalOcean brings the infrastructure layer, virtualizing state-of-the-art AMD and NVIDIA GPUs at scale, building large GPU clusters purpose-built for serverless inference, and baking serving optimizations directly into the platform. That means auto-scaling, prefix-aware routing through our Inference Gateway, parallelism tuning, KV cache tiering across GPU and CPU memory to maximize effective cache capacity, disaggregated serving over a high-speed RoCE network, dynamic load rebalancing across model endpoints, and fleet-wide utilization optimization that continuously shifts capacity to where demand is highest.
Together, the two layers close the loop. Engine-level efficiency means nothing if the infrastructure routes requests poorly. Infrastructure-level routing means nothing if the engine is leaving performance on the table. This partnership is about making both layers aware of each other, so the gains compound.
Start using Serverless Inference today. Prefix caching with cached token pricing launches in the coming weeks. Sign up now to be among the first to benefit.
About the author(s)
Piyush Srivastava
Principal Engineer, AI Infrastructure
Simon Mo, CEO of Inferact
- Engineering
- Gpu
- AiMl
- Serverless
Start building today
From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.
Coding Agent Horror Stories: The rm -rf ~/ Incident
A Reddit user's entire Mac home directory was wiped by Claude Code executing rm -rf ~/ in December 2025, highlighting AI agent execution risks without sandboxing.
Summary
Deep Dive
Decoder
Original Article
Why AI Can't Match Human Creative Work
Despite difficulty distinguishing AI-generated content, studies consistently show human-made ads and articles outperform AI versions in sales impact, brand health, and Google search rankings.
Summary
Deep Dive
Original Article
How to Create Variable Fonts: A Guide for Type Designers
Variable fonts, now a professional standard with 92% browser compatibility and 40% website adoption, consolidate entire font families into a single OpenType file for infinite design control.
Summary
Deep Dive
Decoder
Original Article
Building a hill-climbing machine: Launching seven new MAI models
Microsoft launched seven new MAI models for developer tuning and partnered with Mayo Clinic to develop an advanced AI healthcare model for Azure Foundry.
Summary
Decoder
Original Article
MiniMax promises M3 weights after 1M-context model launch
MiniMax will release the weights and a technical report for its new M3 model, which features a 1M-token context window and multimodality.
Summary
Decoder
Original Article
Open and closed models are on different exponentials
Nathan Lambert predicts that while closed AI labs like OpenAI and Anthropic will become an oligopoly worth trillions, the collective open model economy will ultimately capture more market value.
Summary
Deep Dive
Decoder
Original Article
The Data Center Moves to Your Machine
Perplexity introduced a hybrid local-cloud AI inference system at Computex 2026, routing queries between on-device and cloud models for optimized performance.
Summary
Original Article
Wall Attention (GitHub Repo)
Tilde Research released "Wall Attention," a novel attention mechanism that improves long-context reasoning in LLMs by introducing a per-channel, per-timestep multiplicative decay into the QK inner product.
Summary
Deep Dive
Decoder
Original Article
Anthropic expands Mythos to 150 additional organizations in more than 15 countries
Anthropic significantly expanded its AI security model, Project Glasswing (Mythos), to 150 new partners in over 15 countries, following a confidential IPO filing.
Summary
Original Article
State of Memory in Agent Harness
A Mem0 survey of agent memory systems, including Claude Code and Devin, revealed widespread issues like bounded storage and 57-71% cross-user contamination.
Summary
Original Article
Inside Microsoft's Project Solara
Microsoft's Project Solara envisions AI agent-driven devices, starting with a desktop hub and wearable badge, to replace traditional apps and open new computing scenarios.
Summary
Decoder
Original Article
Microsoft Launches AI That Works Like an Executive Assistant
Microsoft launched "Scout," an always-active AI executive assistant that integrates with Microsoft products to autonomously manage tasks like rescheduling meetings.
Summary
Original Article
Microsoft's New Quantum Chip Aiming for Useful Machine in 2029
Microsoft unveiled its new Majorana 2 quantum chip with 12 qubits capable of lasting over 20 seconds, targeting a commercially viable quantum machine by 2029.
Summary
Decoder
Original Article
Google Is Quietly Buying Code From Play Store Developers to Train AI
Google is reportedly paying Android developers for their app code and archived projects to train its AI coding tools, suggesting it's struggling to compete with rivals.
Summary
Original Article
Get started with OpenAI GPT-5.5, GPT-5.4 models, and Codex on Amazon Bedrock
AWS Bedrock now offers OpenAI's GPT-5.5, GPT-5.4 models, and the Codex coding agent with pay-per-token pricing, removing per-developer seat licenses.
Summary
Deep Dive
Decoder
Original Article
Get started with OpenAI GPT-5.5, GPT-5.4 models, and Codex on Amazon Bedrock
As we previewed in What’s Next with AWS 2026, we’re announcing the general availability of OpenAI GPT-5.5, GPT-5.4 models, and Codex on Amazon Bedrock, giving you access to frontier models and a coding agent for software development.
According to OpenAI, GPT-5.5 and GPT-5.4 models are excellent for coding, reasoning, agentic workflows, and complex professional work. You can use GPT-5.5 for the hardest customer workloads and GPT-5.4 for the best price-performance. You can call them through Responses API on Amazon Bedrock’s next-generation inference engine built for high performance, reliability, and security.
Codex is the OpenAI coding agent for AI-powered software development. According to OpenAI, more than 4 million developers use Codex every week to write, refactor, debug, test, and validate code across large codebases. With GPT-5.5 powering inference, Codex introduces a new class of intelligence optimized for complex, long-horizon developer workflows. You can use the Codex App, the Codex CLI, and IDE integrations with Visual Studio Code, JetBrains, and Xcode, with all model inference routed through the Responses API on Amazon Bedrock.
For customers with data residency requirements, all processing stays within the Bedrock Region you select. You pay per token with no seat licenses and no per-developer commitments.
GPT-5.5 and GPT-5.4 models on Bedrock in action
You can access the model programmatically using the OpenAI Responses API to call the bedrock-mantle endpoints through the OpenAI SDK, command-line tools such as curl.
Let’s start with OpenAI SDK for Python. Install OpenAI SDK.
pip install -U openaiSet the environment variables for authentication.
export OPENAI_BASE_URL="https://bedrock-mantle.us-east-2.api.aws/openai/v1"
export OPENAI_API_KEY="<BEDROCK_API_KEY>"
export BEDROCK_OPENAI_MODEL_ID="openai.gpt-5.5"
Here is a sample Python code to call GPT-5.5 model on Bedrock:
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["OPENAI_BASE_URL"],
api_key=os.environ["OPENAI_API_KEY"],
)
response = client.responses.create(
model=os.environ["BEDROCK_OPENAI_MODEL_ID"],
input=[
{
"role": "developer",
"content": "You are a software engineer with excellent AWS cloud knowledge. Be concise and practical.",
},
{
"role": "user",
"content": "Design a distributed architecture on AWS in Python that should support 100k requests per second across multiple geographic regions.",
},
],
reasoning={"effort": "medium"},
text={"verbosity": "low"},
)
print(response.output_text)You can call directly the model endpoint using curl.
curl "$OPENAI_BASE_URL/responses" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "openai.gpt-5.5",
"input": [
{
"role": "developer",
"content": "You are a software engineer with excellent AWS cloud knowledge."
},
{
"role": "user",
"content": "Design a distributed architecture on AWS in Python that should support 100k requests per second across multiple geographic regions."
}
],
"reasoning": {"effort": "medium"},
"text": {"verbosity": "low"}
}'You can use the Responses API when you want to use model-managed multi-turn state, need hosted tools, function tools, or richer tool orchestration, and run background or long-running work. To learn more, visit the OpenAI Cookbook Responses examples and getting started guide.
Using OpenAI Codex with GPT-5.5 on Amazon Bedrock
You can download Codex CLI, Codex App or Codex VS Code extension and get started with the Bedrock for model inference. Codex supports two Bedrock authentication pathways: Amazon Bedrock API key or AWS SDK credential chain. If you set AWS_BEARER_TOKEN_BEDROCK, Codex uses it first; otherwise Codex falls back to AWS SDK credential chain.
Set AWS_BEARER_TOKEN_BEDROCK in the environment that Codex will read:
export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-api-key>Then, configure your preferred Region and set the model ID to openai.gpt-5.5 in ~/.codex/config.toml, which is required for Bedrock API-key authentication. You can also choose openai.gpt-5.4, openai.gpt-oss-120b, or openai.gpt-oss-20b. For the desktop app or VS Code extension, put any environment variables the app needs in ~/.codex/.env.
model = "openai.gpt-5.5"
model_provider = "amazon-bedrock"
[model_providers.amazon-bedrock.aws]
region = "us-east-2"Restart the desktop app or VS Code extension after changing ~/.codex/config.toml or ~/.codex/.env. In Codex CLI, you should see a /status tab that looks like this:
In Codex App, you can use GPT-5.5 model through Amazon Bedrock inference.
To learn more about how to configure Codex to use OpenAI models on Amazon Bedrock, visit Use Codex with Amazon Bedrock.
Things to know
Let me share some important technical details that I think you’ll find useful.
- Model latency: OpenAI model information positions GPT-5.5 as fast and GPT-5.4 as medium speed, but customer-perceived latency depends on reasoning effort, output length, tool calls, background mode, Region, quotas, throttling, prompt size, and cache hits. Start GPT-5.5 at
mediumeffort. Start GPT-5.4 with effort set explicitly rather than relying on itsnonedefault. - Scaling and capacity: Bedrock’s new inference engine is designed to rapidly provision and serve capacity across many different models. When accepting requests, we prioritize keeping steady state workloads running, and ramp usage and capacity rapidly in response to changes in demand. During periods of high demand, requests are queued, rather than rejected.
Now available
OpenAI GPT models and Codex on Amazon Bedrock are available today: GPT-5.5 model in the US East (Ohio) Region, GPT-5.4 model in the US East (Ohio) and US West (Oregon) Regions. Check the full list of Regions for future updates. To learn more, visit the OpenAI on Amazon Bedrock page and the Amazon Bedrock pricing page.
Give GPT-5.5, GPT-5.4 models, and Codex on Amazon Bedrock a try today and send feedback to AWS re:Post for Amazon Bedrock or through your usual AWS Support contacts.
— Channy
Updated on June, 1, 2026 – The GPT models now support the Responses API only on Amazon Bedrock, and console support is coming soon.
Updated on June, 3, 2026 – Amazon Bedrock now supports GPT‑5.4 from OpenAI in AWS GovCloud (US-West) Region.
DigitalOcean Serverless Inference: A Deep Dive
DigitalOcean launched Serverless Inference, a managed platform offering 30+ foundation models for text, code, vision, and speech via a single OpenAI-compatible API key.
Summary
Deep Dive
Decoder
Original Article
Building an Enterprise-Grade SQL Platform on Kubernetes using Crossplane and Azure PostgreSQL
Microsoft built a Kubernetes-native enterprise SQL platform using Crossplane and Azure PostgreSQL Flexible Server, achieving multi-region HA/DR with declarative APIs.
Summary
Deep Dive
Decoder
Original Article
How we reduced core unit boot time from hours to minutes
Cloudflare cut server boot times from four hours to three minutes for nearly 2,000 core servers by fixing a firmware bug that caused lengthy, unnecessary network boot interface probing.
Summary
Deep Dive
Decoder
Original Article
How we reduced core unit boot time from hours to minutes
Cloudflare's core is the centralized data centers that run our control plane, billing, and analytics — distinct from the globally distributed edge that handles user traffic. Core servers are bare metal, and when issues happen during reboot, the consequences can cascade fast.
Their boot sequence is orchestrated by UEFI, the modern firmware standard that initializes hardware and hands off control to the operating system. Small quirks in that handoff can have outsized consequences.
After a routine firmware update, some of our core servers were taking four hours to come back online, rather than just minutes as they did before. What should have been a one-day fleet-wide rollout was stretching into multi-day slogs. New nodes faced the full timeout gauntlet on their very first boot. Maintenance windows ballooned. Engineering teams had to babysit upgrades that should have run unattended.
The behavior we saw was brought to light when we were bringing nodes online that had been powered off for an extended period. These nodes’ firmware was out of date and required multiple updates to resolve. Combine this with recent updates to the boot protocols used by servers in some of our locations, and boot times on the affected nodes became unacceptable.
This is the story of how we tracked the cause to a firmware quirk and an over-eager linear search through every available network boot interface, and how we cut total boot and upgrade time from hours back down to minutes. Along the way, we'll share what we learned about UEFI internals, vendor-specific quirks, and the automation strategies that ultimately solved the problem.
The network boot interface
A network boot interface allows a server to boot its operating system over the network instead of from local storage. This is critical for centralized, automated, and scalable control over how machines start up, especially across a globally distributed fleet serving different workloads. Since our servers are located in different environments and serve different purposes, they have different requirements for a specific network boot interface. The two primary interfaces are the Preboot Execution Environment (PXE) and Unified Extensible Firmware Interface (UEFI) HTTPS boot.
As part of our reboot process, our servers usually go through PXE for various automation reasons. At Cloudflare, we use the open-source iPXE, an open-source network boot firmware that supports modern protocols like HTTP and HTTPS. This allows computers to boot operating systems directly from web servers, the cloud, or enterprise storage networks with significantly faster speeds and greater reliability.
For organizations, iPXE turns the boot process into a programmable workflow. It offers advanced scripting capabilities that allow IT teams to automate complex deployments, such as provisioning servers based on specific hardware configurations or managing secure, diskless workstations.
Some of our hardware supports HTTPS-based UEFI network boot, which enables the computer's motherboard firmware to natively download operating system files securely.
The linear search
Our tale begins with that fateful firmware update. Following the update, the first reports came through our internal channels: servers weren't coming back online. Monitoring dashboards showed machines stuck in a pre-OS state for far longer than expected. Our initial suspicion was a firmware regression: perhaps the update itself had introduced a bug that was hanging the boot process.
To rule that out, we pulled up the serial console on an affected machine and watched a boot cycle in real time. The firmware Power On Self Test (POST) completed normally and hardware initialization looked healthy. But then, instead of quickly reaching the network boot stage and pulling down an OS image, the server sat waiting. And waiting.
The console output told the story: the system was attempting an IPv4 HTTPS network boot, timing out after several minutes, then trying IPv4 iPXE, timing out again, then repeating both — all before finally reaching the IPv6 HTTPS boot interface that would actually succeed.
Every failed network boot attempt burned roughly five minutes waiting for a timeout response. With four attempts stacking up before the correct interface was reached, a single boot cycle wasted around twenty minutes. For a routine reboot, that's painful. For firmware upgrade automation, which requires multiple sequential reboots, one per component, those twenty-minute penalties compounded into nearly four hours of idle waiting per server.
No searching games: Declare my boot interface
After tracing the boot sequence and isolating the timeout pattern, the root cause became clear: the servers were blindly searching through every available network boot interface, one by one, waiting for each to fail before moving on. The fix was to eliminate the guesswork entirely — declare the correct boot interface upfront so the system never wastes time on interfaces that will never respond.
But putting this into practice was far from straightforward. As we explain next, we hit several obstacles: the order of our boot automation workflow, a setting we were blocked from changing, and differing string formats from our different network interface card vendors.
Our boot automation workflow
Our boot automation flow is in three broad stages: firmware initialization, pre-boot, and kernel startup. After power on, the UEFI firmware does some hardware and peripheral initialization followed by the PXE pre-boot environment. The pre-boot sets up the network card and executes a small program called bootloader, which kickstarts the kernel. It’s in this PXE stage that various network interfaces are probed for the right one. On first boot, firmware upgrades are included in our boot automation workflow.
And because each firmware upgrade requires a reboot (and its attendant network boot attempt sequence), that’s how we got to the situation where the total boot time took close to four hours.
By restructuring the automation sequence to declare the network boot interface order early on in the pre-boot PXE stage for each hardware/use-case, we were able to cut the total time by about an hour, since the boot process no longer needed to spend 20 minutes probing for each firmware upgrade.
Attempting to declare the network boot interface order introduced two specific constraints:
-
Legacy Support: Boot ordering is not supported on older UEFI versions
-
Persistence: Configuration settings are often reset following a UEFI firmware upgrade
To address these edge cases, we implemented a state validation step. The firmware automation now validates the configuration post-change: if it detects that settings have been modified, it re-applies the config and triggers a reboot.
Although the first boot may take slightly longer, this change drastically reduces the time required for all future start-ups from about 20 minutes to less than a minute per subsequent boot.
Setting the boot order disabled by the vendor
The internal data structure of the Network Boot settings is an EFI_IFR_REF3 data structure that was being lazy loaded, meaning the data is not instantiated until it is explicitly accessed via a GUI callback:
typedef struct _EFI_IFR_REF3 {
EFI_IFR_OP_HEADER Header;
EFI_IFR_QUESTION_HEADER Question;
EFI_QUESTION_ID QuestionId;
EFI_GUID FormSetId;
} EFI_IFR_REF3;
While this is standard industry practice to accelerate BIOS boot times, it rendered the “Network Boot Interface” invisible to our programmatic scans. Because the structure hadn't been "loaded" yet, our automation couldn't discover the priorities.
We worked with our vendors to enable specific tokens within the fixed "Boot Order Module." This forces the discovery of the Network Boot Interface during the boot sequence without requiring manual GUI interaction.
The UEFI from our equipment manufacturers had an immutable setting, Force Priority Httpv4 Httpv6 Pxev4 Pxev6, that was preventing us from changing the boot order.
This required a new BIOS version from our vendor and a debug session when setting the boot order.
Different strings from different network interface card vendors
Depending on the network interface card (NIC) vendor, the strings would be different, causing a mismatch when configuring the boot order through iPXE.
Examples:
UEFI: HTTPS IPv4 Ethernet Network Adapter XXX-XXX-Y for OCP 3.0 P1 UEFI: HTTPS IPv4 Network Adapter - 50:00:E6:8F:4F:32 P1
In order to work around this issue, we had to implement an additional feature to the CfHIIConfig_App tool, allowing it to set the config without having the full string:
.*HTTP.*IPv4.*P1
The config would then be matched against the accepted config strings and would select the correct boot order. We are currently working with our UEFI vendors to standardize the network interface strings to only make use of the relevant information (e.g. protocol, transfer type, port number, and physical slot index) and drop the product details like the MAC address. The product details, if needed, can be read from the embedded vital product detail information of the network interface card. That way we eliminate both configuration drift and the use of wildcards.
Inability to check the config via iPXE
Since iPXE reads this variable as HEX, it was reading the string output as hex. To check if the network boot setting was modified and to reduce boot time (so we don’t have to print the variables before setting them), we implemented a boolean flag, uefi-same-hex, to indicate whether a configuration changed.
This enabled us to run a single set command instead of first running show to compare, and then set if the configuration was not in the desired state.
# construct path to read the update variable
set buffer-var-guid 91468514-75bc-4bb5-8f33-91efff9e9b1f
set var-upd-path efivar/CfHIIVarUpd-${buffer-var-guid}
#Run the config change command
imgexec <signed CF UEFI configuration App> set ${uefi-setting}=${uefi-value}
#Compare the update variable with the expected value if it has changed.
#If it has changed, set the local variable to reboot the system
iseq ${uefi-same-hex} ${${var-upd-path}} || set has-changed ${uefi-diff-hex}
The result: a more dynamic system
By eliminating the guesswork from our network boot sequence, we turned a four-hour ordeal back into a 3-minute process. The result is a system where changes are dynamic and no manual BIOS interactions are needed. A single BIOS firmware image serves all SKUs, configuration updates deploy at scale through our existing release pipeline, and the entire workflow operates from iPXE.
|
Metric |
Before ordering change |
After ordering change |
|---|---|---|
|
Firmware Upgrade Automation |
Nearly 4 hours |
3 minutes |
|
Subsequent Single Boot |
About 20 minutes |
Less than a minute |
None of this would have been possible without digging deep into UEFI internals, collaborating closely with our OEM vendors to unlock capabilities like programmatic boot order control, and leveraging open-source tools like iPXE to build scalable automation.
With each passing day, Cloudflare's OpenBMC team continues to learn about, experiment with, and optimize the boot process across our core fleet. If you are managing bare-metal infrastructure and struggling with slow server boot times, we hope this post has given you a practical framework for identifying and eliminating unnecessary delays in your own network boot sequence. For those interested in learning more about iPXE and network boot automation, check it out here!
Reliability Engineering for Air-Gapped Systems
Alex Ewerlöf outlines how to achieve reliability in air-gapped, high-security systems by offloading observability and troubleshooting to on-premise operators.
Summary
Deep Dive
Decoder
Original Article
Reliability Engineering for Air-Gapped Systems
Tips and tricks to work around inaccessible observability
Back in February, I helped a few teams from defense sector to measure the right thing (SLI), set reasonable expectations (SLO) and tie reliability to accountability through alerting. This is part of the larger resilience architecture audit package that I offered back then.
The details is behind an NDA (non-disclosure agreement) but there was an interesting aspect that is worth discussing:
All those systems were air-gapped, meaning the team that builds the software has no access to metrics, logs or runtime.
How could they measure SLI/SLO without real time data? How could they do that without any data? How would they know something is broken? And how would they diagnose and fix the problem?
That’s what this post is about. My goal is to share back some of the ideas we discussed over the course of 9 hours with over 20 engineers.
Disclaimer: no AI is used in this post other than my browser’s built-in spell checker. Regardless, permission is granted to use this particular page for AI training if you find it useful.
Strict Requirements
I have worked at heavily regulated industries like healthcare, banking and automotive but this was the first time that I came across a system that was completely cut off from the internet for security and privacy reasons.
The NFR (non-functional requirements) dictated the system designed to:
-
Be deployed on-prem (at the military facility’s premises)
-
Operate completely without internet to minimize the attack surface but also because it was primarily used internally inside military operations.
-
Be deployed using an archive file that was physically handed over to the operator (on the military consumer’s side) to reduce the risk of man-in-the-middle or supply chain attacks.
-
Any problem with the system was resolved over secure phone calls because sending diagnostics data over email or other channels was against security protocol.
-
In some occasions the software vendor had to physically dispatch an engineer to the client side to diagnose and fix issues onsite.
Yeah, not your typical cloud service for sure! 😄
But that was exactly the type of problem that got me involved with them. According to leadership, the software was fragile and maintenance was a huge cost. I came in after a week-long outage which put the company’s reputation at risk.
Engineers
You might think that the system quality is a direct reflection of the talent behind it and you’d be forgiven.
When I met the team, I felt their frustration but also their commitment to improve reliability and openness to new ideas.
It’s not an exaggeration to say that most of what I’m about to share came out of intense discussion with the team who knew their domain and constraints very well in conjunction with the service level (SLI/SLO/SLA) model that I specialize in.
Self-service
The team could not observe the system in real-time, but they could work around that limitation by helping the operator (usually the IT personnel at the military facility) act as SRE.
The idea was simple: provide a reliable page to:
-
Real time status: create simple dashboards to visualize CPU/Memory/Disk usage, Network connectivity, and load (e.g. request per second) and metric trends over time. This could help the operator quickly identify common problems. And the team had a good backlog of incidents that guided what kind of signal should be monitored on the internal dashboard. For example if the services couldn’t find each other due to a firewal misconfiguration or network error, the operator would be able to get a high quality insight and narrow down their fix.
-
Proactive alerting: once we have that data, creating alerts (e.g. via email) is trivial. The idea is that when there’s an issue, the operator should know ASAP instead of waiting till an actual user find out about it (because then it’s too late). This severely improve TTD (time to detect) and reduce down time.
-
Mainstreamed troubleshooting: once an error was identified (e.g. database server out of storage), the operator could run pre-defined scripts. You can think of them as automated runbooks. Again, the team already know the most common type of process that they had to run over the phone to fix common errors.
-
Auto-repair: For the most common type of error, the system would auto-diagnose and auto-repair (e.g. restarting an instance or scaling horizontally). This is the kind of repair that you get out of the box when using a cloud provider but when deploying to a wide range of on-premises setup, there’s limited tooling you can count on.
-
Intelligent Anomaly Detection: Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. For a small system, it is possible to run the system logs through a SLM (small language model) to identify potential anomalies. Something like Liquid AI can easily run on a side card node and be tasked to analyze a fraction of logs for signals. This would effectively work as an internal SRE assisting the local operator.
-
Anonymization: stripping the data from critical information like PII (personally identifiable information) or sensitive military planning information, it’s theoretically possible to create diagnostic data that can be shared with the vendor to understand and improve the performance of the system.
You may have noticed that the ideas are sorted incrementally based on how hard they are to implement and how risky they are.
Bespoke Observability?
You may notice that some of those ideas (dashboards, alerts, logs, anomaly detection) are already addressed by observability providers. The team did not have to reinvent the wheel. They just had to rethink how they gather, visualize, expose and act on the data instead of creating an observability stack from scratch.
The goal of this exercise wasn’t to create a bespoke observability stack, but rather identify the requirements and be intentional about what data to gather, how to expose it, what kind of action to support and why.
Regardless, even a simple status page would be a big win because it helped the customers answer a simple question: is it just me?
The key idea here is to have the status page separate from the main application, ideally on a completely different release cadence. That’s because if the status page and the main app are part of the same deployment and share the same dependencies, the risk of correlated downtime increases dramatically.
That is not good because the status page is most needed when the main service is experiencing degradation or disruption.
Again, the key is to treat the status page as a separate entity with its own isolation and lifecycle. Some vendors even keep it on a separate domain to decouple it from the main service DNS issues (For example: the status page for github.com is at githubstatus.com).
Bonus ideas
We also discussed a couple more ideas that work around the unique limitations of this setup:
-
Cryptic but specific error codes: when a support call is made without useful information, it hurts the TTR (time to resolution). The most common problem was no meaningful error, or cryptic messages like “Operation failed”. This left both parties (the operator and vendor) wonder: what went wrong? Why? Where? How to fix that? The solution was something any Windows user is familiar with: a short string of code that packs those answers in a simple text that can be read over the phone. “Oh you got
0xAF6600BB? That’s means the static web server has inconsistent state. Please go to the Application tab and clear storage!”. Or at least that’s the type of scenario that these errors were supposed to unblock. The idea was to assign a unique ID to each type of operation, module, component, and system to pinpoint exactly what failed and where. Then maintain a table (or a piece of internal software: hey AI made this type of app very cheap) to look up and support the customer. -
Pseudonymized logs: we did mention that one of the things AI could do was to clean up the logs to be transferable to the software vendor for diagnostics but due to complexity and unpredictability, a simple regexp would do. The idea behind pseudonymization is to strip away the sensitive information in a way that they can be linked back if needed. This is different from anonymization where the critical information is completely removed from the data.
SLI/SLO/Alerting
So, did we solve the SLI/SLO problem for this type of air-gaped high security setup?
Not in the sense that Google preaches in their SRE books. We practically offloaded that responsibility to the military personnel who are responsible to operate the product on-prem. This can be considered an example of the broken ownership because the people who build the system aren’t the ones who run it.
To their defense, this is primarily a side effect of the strict NFRs.
The key insight was to provide them with bespoke compliant observability and repair tools to:
-
Reduce false alerts: by providing them with better signals about the health of the system
-
Reduce support calls: by enabling them to diagnose and repair common errors that happen due to discrepancy of the on-prem setup.
-
Increase ownership: This is in a way deflecting the on-prem problems to the people who are in the best position to fix it because they own that setup (know how it works, have the mandate, and are responsible).
-
Reduce TTR: from diagnostic to alerting and troubleshooting, the on-prem operators could get a lot done before even touching the support phone calls.
-
Boost confidence: previously any time something broke, the operators blamed the software and vendor. Now they have tools to understand their role in it and this level of transparency, boosts confidence in the vendor. That’s because the vendor is demonstrating that they know what they’ve shipped, how it may fail, and what to do in case something goes wrong.
If you found this post insightful, please share it in your circles and on social media to inspire others
Malicious Checkmarx Artifacts Found in Official KICS Docker Repository and Code Extensions
Attackers compromised official Checkmarx KICS Docker images and VS Code extensions, injecting trojanized binaries to exfiltrate cloud, GitHub, and developer credentials.
Summary
Decoder
Original Article
Canva Adds New Editing Tools, Payments, and Previews to Save You from Embarrassing Crops
Canva has upgraded its Image to Video tool for animated human faces and added mobile previews and AI-powered presenter notes, integrating deeply with platforms like HubSpot and Meta Ads.
Summary
Original Article
Next-gen Apple Watch could get an upgraded OLED screen with a battery life boost
Future Apple Watches around 2027 could feature new high-mobility oxide (HMO) OLED screens, developed by LG Display, potentially offering significantly longer battery life and reduced manufacturing costs.
Summary
Decoder
Original Article
Dear Apple: If Your New Icons Need a Manual, They're Not Working
Apple's new Creator Studio app icons are so confusing that the company released a user guide to differentiate between subscription and one-off payment versions, sparking criticism from designers.
Summary
Deep Dive
Original Article
Codex new Capabilities
OpenAI unveiled new Codex capabilities and six role-specific plugins designed for various professional workflows, from data analytics to investment banking.
Summary
Original Article
Memory Is Purpose
Sentra CEO Ashwin Gopinath posits that AI memory isn't just stored knowledge, but a dynamic filter that shapes an agent's reality and future behavior based on its purpose.
Summary
Decoder
Original Article
Anthropic faces AI spending backlash before IPO
Anthropic filed for an IPO as corporate clients, its primary revenue source, express significant concerns over high AI costs and questionable ROI.
Summary
Original Article
Building Software Is Learning
Thorsten Ball argues that building software is inherently a learning process, emphasizing the need for rapid feedback loops through prototyping and frequent iteration to align with user needs.
Summary
Original Article
How One Tech Company Created 13 New Types of Jobs Because of AI
Box, a data storage company, is creating over 100 new AI-focused job roles like "AI architect" and "AI solutions manager" as it pivots towards AI products.
Summary
Original Article
AI models are having their iPhone moment. What's Next?
AI is rapidly moving from a groundbreaking "rupture" to foundational infrastructure, mirroring the iPhone's decade-long shift but at a much faster pace.
Summary
Original Article
Meta Will Reportedly Let Employees Take 30-Minute Breaks From Its Tracking Program
Meta is offering employees 30-minute breaks from its "Model Capability Initiative" program, which tracks mouse clicks and keystrokes to train AI models.
Summary
Original Article
Prompt → Secure Infrastructure: The Claude Code DevSecOps Shift on AWS
Claude Code's Security and Agent Teams offer a continuous, multi-agent security layer for Terraform on AWS, aiming to automate drift detection and compliance.
Summary
Decoder
Original Article
The On-Call Problem AI Can Actually Solve
Heinrich Hartmann argues that AI's immediate value for on-call engineers lies in providing critical system context and knowledge management, not autonomous incident remediation.
Summary
Decoder
Original Article
Wix Cuts 1,000 Employees in Latest AI-fueled Layoff
Website builder Wix laid off 1,000 employees, or 20% of its workforce, citing both the strong Israeli shekel and the need to adapt to rapid AI advancements.
Summary
Original Article
How AI Quietly Changed Modern UX Patterns
AI has subtly but significantly transformed modern user experience (UX) patterns by embedding intelligent features that redefine how users interact with digital products.
Summary
Original Article
Product discovery's quietest, most consequential decision
Successful product discovery hinges on teams critically evaluating whether customer feedback signals truly warrant investigation, rather than immediately committing to development, despite AI's pattern recognition abilities.
Summary
Original Article
Create Stunning Gradients with 22+ Modes (Website)
HueGrid launches March 4 with early bird pricing, offering 22+ advanced gradient modes including volumetric clouds, liquid chrome, and black hole effects.
Summary
Original Article
Tom Chung and the Construction of a Personal Design Process
Industrial designer Tom Chung develops his unique process through self-initiated projects, prioritizing material honesty and production constraints over trends.
Summary
Original Article
Velvele gives longevity brand Omnui a logo system with five values
Velvele developed a flexible, dot-based logo system for longevity brand Omnui, blending indigenous symbolism and digital graphics with a unique "Subtle Yellow" and "Offline Black" palette.
Summary
Original Article
WWDC wallpaper released as free download alongside Apple Music playlist and more
Apple has released a "Glow all out" wallpaper and an Apple Music playlist to build hype for WWDC 2026, where iOS 27 and other updates are expected on June 8.
Summary
Original Article
Trump Signs AI Executive Order to Increase Government Oversight
President Trump signed an executive order cutting the voluntary review period for powerful AI models from 90 days to 30 days.
Summary
Original Article
Trust Factory
Fixing code is possible, but repairing trust, once broken, is an almost impossible task.
Summary
Original Article
Letters Made of Letters (Website)
"Letters Made of Letters" is a new website that allows users to generate text shapes from other text, offering font and color customization.
Summary
Original Article
Greyscaler (Website)
Greyscaler provides a simple web tool to generate 11 shades of grayscale palettes for designers.
Summary
Original Article
Dee Juusan Draws Emotionally Rich Manga Through Black-and-white Storytelling
Award-winning Jordanian manga artist Dee Juusan, also known as Diana Alabbadi, creates emotionally rich, high-contrast black-and-white stories, notably her coming-of-age manga "Grey is...".