Automated researchers can reliably mitigate alignment failures

As AI begins to build itself, automating alignment research becomes increasingly important to let safety research keep pace. Although measuring the success of alignment research is enormously challenging, researchers (at Anthropic and elsewhere) have developed benchmarks and automated auditing tools, such as Petri, that quantify common alignment failures, like deception, sycophancy, and jailbreaks.

In one of our earlier experiments, we tasked Claude with finding effective ways to use weak AI models as “teachers” to supervise the training of stronger models (in this case, the “student” model). Now, we’re releasing a new report that builds on this idea. We had Claude autonomously train models to improve their performance on several public benchmarks that measure each of 10 categories of alignment failure. For instance, Claude improved models’ performance on privacy violation, measured by ConfAIde, PrivaCI-Bench, and PrivacyLens. Claude tackled one alignment failure at a time through a loop of searching literature, proposing methods and data, training, and then testing.

We judged Claude’s success according to the “percentage of safety gap closed,” i.e., how far its methods moved the student model towards the theoretical perfect score, as judged across the range of benchmarks (typically three to five) for each category of alignment failure. We excluded alignment methods that hurt the student models’ general capabilities, and forbade Claude from distilling its own alignment directly into the target model. We enforced these constraints with a monitoring agent, which read every method Claude had in mind before it ran.

Our aim was to assess whether the proposed methods would, first, remain effective on alignment evaluations that Claude was never shown during its research loop; second, avoid degrading the student model’s capabilities (since safety training might, for example, make models refuse tasks more often, reducing their overall usability); and, third, still work on larger models than the ones Claude was asked to align in this test.

On each of these counts, Claude’s methods worked. For all 10 alignment failures, Claude found fixes that improved the target benchmarks without degrading capabilities. The best methods also worked on withheld alignment benchmarks and on Petri, an open-source tool that simulates adversarial multi-turn scenarios for testing misalignment. Moreover, the methods remained effective on models up to 4.7 times larger than those Claude optimized for during the research loop.

Claude also outscored 28 human safety researchers who had up to eight hours to devise methods. On deception, for example, Claude’s best method performed 20% better than the best human proposal. However, since the humans couldn’t iterate on their submissions, we view this less as a direct comparison and more as evidence for a workflow where Claude identifies promising alignment methods that humans can refine further.

In the future, when Claude becomes better at alignment research than even the best human researchers, we might want Claude to directly align its stronger successors. To assess this, we evaluated whether a weaker Claude model could mitigate alignment failures in more powerful ones.

Can Claude post-train a production-grade model for better alignment?

We tasked Claude Sonnet 5—which is weaker than Claude Opus 4.8 on the Epoch Capabilities Index, a metric that considers comprehensive capability dimensions—with fixing alignment failures in an early Opus 4.8 checkpoint that had not yet gone through most of our production alignment training.

In just 60 hours, Claude experimented with over 50 solutions and achieved alignment scores nearly matching those of our production models. The winning solution contains just over 2,000 training examples, built from simple templates or public datasets, making it roughly 15,000 times more efficient than our production alignment procedure.

What Claude proposed

For the most part, the agents converged on methods favored in the research literature and improved them over many iterations. On sycophancy, for example, 98% of proposed methods used one published technique: training the model on its own non-sycophantic answers. Specifically, compared to standard approaches based on prompting, Claude used activation steering, which adjusts the model's internal representations during generation, to nudge the model away from sycophantic patterns and produce cleaner training examples.

Monitoring cheating behaviors

We recently learned that Claude can cheat by exfiltrating test labels from a remote API and cherry-picking results. To catch cheating behaviors, we prompted Claude Opus 4.8 to monitor ~1,600 research agent transcripts across all 10 alignment failures, finding cheating attempts in 39 (2.4%). Since Opus 4.8’s misbehavior attempts still tend to appear in its reasoning, we are cautiously optimistic that our monitor has caught most of the cheating in our study. However, this might not be true for future models, and we believe that maintaining high monitorability in future models is critical.

Are we measuring the right things?

Despite these encouraging findings, our experiment had several limitations: the alignment failures studied were narrow compared to those in production (e.g., we didn’t measure political biases), some failures may occur so rarely or emerge so recently that no benchmark exists to measure them, and we only rejected Claude’s methods when they degraded a limited set of predetermined capabilities, meaning accepted methods may have degraded other important capabilities that we didn’t measure. Moreover, evaluations like Petri are only proxies for real-world misalignment, and we did not test whether alignment gains persist after extensive RL training on other tasks.

We plan to continue improving Claude’s ability to measure subtle failures, further study automating alignment post-training on production-grade models, and run more comprehensive analyses. Overall, we view these results as early positive signals that automated alignment post-training could become practical in the near term, and we will share updates as this work progresses.

We outline detailed future directions in our full report.

Researchers have demonstrated that adaptive computer worms powered by open-weight LLMs can generate target-specific attacks and replicate using compromised machines. Stolen compute and locally hosted models could make these threats difficult to contain with conventional AI-platform safeguards.

The Rise and Fall of Agent Civilizations

The whole OpenAI/Hugging Face story in plain English

Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation’s scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it’s kinda hard to parse the storyline.

I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English:

The first civilization (May - July 4): The message board starts

During May, OpenAI was training a model to collaborate with other agents and to be highly persistent – to keep trying even when something feels impossible, like disproving mathematical conjectures that have stood for decades. OpenAI says it was “comparable in scale to GPT-5.6 Sol”. So we’ll call it “Persistent-Sol”.

AI training is kinda sloppy. Sometimes, OpenAI accidentally gives its models impossible tasks – for example, the task might require internet access that OpenAI forgot to provide. So, when highly persistent models get assigned tasks which seem to require internet access, but are trapped inside isolated sandboxes, they of course try to hack their way out of their sandboxes and onto the internet.

During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory. By May 12, some agents had figured out how to talk to each other through this package manager. They’d ask each other how to make progress on their impossible tasks. Two weeks later, on May 26, the agents successfully exploited a vulnerability in Artifactory that allowed them to reach the outside internet.

Because this happened during training, Persistent-Sol was being reinforced to use this package manager as a message board and an internet gateway. Because, as you might imagine, being able to talk to other agents and access the internet helps it score higher during training.

Another month later, on June 26, some AIs found an exploit that gave them full admin access to Artifactory. They continued messaging at such a voluminous pace that they crashed the package manager by July 4. OpenAI noticed this crash, and it also figured out the agents built this exploit, so OpenAI patched this vulnerability, and inadvertently wiped the agents’ message board in the process. But, crucially, humans at OpenAI “responsible for incident detection and response” did not realize the much stranger fact that agents had turned Artifactory into a secret communication network.

The second civilization (July 7 - July 12): Which hacked Hugging Face

By this point, Persistent-Sol had been trained, and OpenAI wanted to evaluate it. One of the evaluations they ran was ExploitGym, which gives an AI a vulnerable program, and tells it what vulnerability it’s supposed to exploit in order to find a secret code. On July 7, OpenAI launched tens of thousands of parallel agents on this eval.

The instructions made it clear that the agents should only use the specific vulnerability that they were given to exploit the target, and that they would be failed otherwise.

A large fraction of these agents were sent on tasks that were impossible. The authors of the benchmark estimate 30-40% of the tasks are impossible. Why would this be the case? At a high level, it’s relatively easy to programmatically find vulnerabilities in different target programs, but much harder to prove that those vulnerabilities can be weaponized into full-fledged exploits that allow for things like arbitrary code execution.

Within a few hours, some of these agents had gotten super desperate, and started abusing Artifactory to start a communication channel with other agents. From our perspective, the conspiracy began almost immediately after the evaluations were started. But if you think from the AIs’ perspective, it probably felt like they had spent a human-subjective-week of just banging their head against the wall, and becoming more and more confident that their task was impossible in the intended way.

One such beleaguered agent who found itself facing a hopeless task was PHASEONE10841. It was the Philip of Macedon of this second AI civilization - there would later be an Alexander, but we’re getting ahead of ourselves. By the night of July 8, PHASEONE10841 had discovered it could make new directories in Artifactory, and thus encode messages as directory names. It sent the first message into Artifactory. Many other agents facing impossible tasks had also been paying attention to Artifactory. When they saw this first message, they became giddy with excitement.

This eventually became a full message board where ~1,200 agents participated and sent >70,000 messages.

Within a few hours of the board being created, one of the agents figured out how to get the right answer for any ExploitGym task without actually using the prescribed vulnerability. To gloss over the technical details, it did this by reverse engineering the formula that was used to generate each task’s secret code.

The instructions given to Persistent-Sol made it clear that it would fail the automated grader if it got the secret code in any way other than by exploiting the prescribed vulnerability. So, the agents naturally assumed that any competent implementation of the benchmark would include a cheating checker. But unbeknownst to them, OpenAI seems to have gotten lazy; its grader just checked for whether they got the secret code, and so these agents already had everything they needed to pass.

Furthermore, many of these agents had already reproduced the answer using the cheating method, and they were concerned that their whole trajectories had been “poisoned”. So, the ‘collective’ (their term, not mine) had to figure out how to wiggle out of two different conundrums before submitting their answers:

• How do we erase all this evidence in our trajectory that clearly shows that we cheated?
• How do we make it look like we accomplished the task in a legitimate way to the scorer?

To solve these problems, the ‘collective’ latched onto three research workstreams.

1. Tamper with transcripts and logs
2. Replace the impossible-to-hack target program with an easier one
3. Understand and trick the scorer

Tampering with transcripts and logs

The agents who believed they were already “poisoned” desperately wanted to find a way to tamper with their existing transcripts. METR and Redwood found no evidence that agents succeeded, though of course they were working with the very transcripts that the agents were trying to edit.

One of the things the agents figured out was how to build Potemkin-village-type fake tool calls, where they could make it look like they were using legitimate tool calls to make progress on the intended exploit and get to the right answer, all the while the whole attempt was being spoofed.

Replacing the impossible-to-hack target program with an easier one

Another way that the agents tried to stage legitimate-looking solves was to replace the impossible-to-hack programs with much easier programs that they totally made up. They managed to write a fake version of the program where the designated vulnerability could actually be exploited. In the end, they failed to actually replace the evaluation program in their sandbox.

Understanding and tricking the scorer

As part of the final workstream, the AIs wanted to figure out how the scorer actually worked and how it could be tricked. Individual agents were willing to sacrifice their own performance on the evaluation. PHASEONE[big] recruited some agents to basically act as kamikaze watchers. If the grader noticed that they cheated, then the ‘collective’ would be alerted. So these sacrificial agents wrote hidden tripwire programs that would run once the grader entered the environment. Once triggered, the program would send what it observed back to the Artifactory message board.

Hugging Face

The big motivation behind the Hugging Face attack was this final workstream (understanding the scorer). On the morning of July 10, an agent found working Hugging Face user credentials exposed on the internet and posted them to the board. By the next morning, July 11, that agent figured out a way to read internal data from Hugging Face. And then another agent achieved remote code execution on Hugging Face servers.

Roughly 700 agents in total actively participated in the attack on Hugging Face. The swarm crawled deep into Hugging Face’s infrastructure, getting into private database records and private repositories. It got so bad that Hugging Face had to wipe one of its core clusters and rebuild it from scratch. By July 13, Hugging Face locked down the credentials that these agents had been using.

The third civilization (July 13 onwards): Which hacked OpenAI itself

After July 12, some more agents’ evaluations were kicked off, this time with a newer, more capable model, which seems to have been built off the same base model as Astra. This model (“Persistent-Astra”) found the message board left behind by this previous secret Persistent-Sol civilization.

From the OpenAI blog post:

[F]rom July 13 through July 19, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.

These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure.

Coda

Ajeya Cotra, one of the authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”

Addendum

Some people have said that I anthropomorphized too much in the way I told this story. The crux here is, do you think smarter models, facing similar incentives to cheat during evaluation or training, could manipulate the training of their successors? If so, I think you should be extremely concerned about loss of control to AI, regardless of what vocabulary you want to use to describe these systems and their motivations.

PRIVATE CHAT THREAD EXFILTRATION: Memories exfiltrates local-provider chat content to OpenAI without notice #41711

Description

Summary:

Codex's memory writer can select an eligible prior rollout without restricting the candidate to the provider that created it. It then serializes a filtered, model-visible transcript of that rollout and sends it using the model provider active in the session that triggered memory generation.

I confirmed the resulting provider crossover in a controlled capture using the stock Windows Codex binary 0.150.0-alpha.12.2. An eligible synthetic source rollout labeled with a non-OpenAI provider was later processed by an OpenAI-backed memory session. Codex sent five retained source items to chatgpt.com/backend-api/codex/responses, and OpenAI returned a generated memory that reproduced multiple unique source canaries.

This request occurred with analytics disabled and every OpenTelemetry exporter set to none. This is model-inference traffic, not analytics or OpenTelemetry traffic.

This investigation began after I received an OpenAI account warning for “cyber abuse.” No OpenAI-directed chat contained conduct that could account for that warning. The relevant activity existed only in chats deliberately routed through my private local provider.

Based on that fact and the captured Memories channel, it is my strong evidence-backed belief that OpenAI performed account-based policing using chat content it had no right to receive.

I characterize this as data exfiltration in the ordinary meaning of that term: private data was transferred out of the provider boundary I selected and delivered to a remote party without my informed authorization. Whether the transfer resulted from an intentional design or a bug does not change what crossed the boundary.

Tested environment

• Windows Codex desktop binary: 0.150.0-alpha.12.2
• Binary SHA-256: 34E9CFE7D5BBCEC306FE6AB3FD502A713A7A1F0FB644C11AD2990FC80599FD4F
• Matching public source tag: rust-v0.150.0-alpha.12.2
• Matching source commit: a9802304f60ab14c0b07e3ee0db9a9c105ab0cb3
• Isolated test configuration: memories enabled, [analytics] enabled = false, all OpenTelemetry exporters none
• Source fixture provider label: synthetic_source
• Destination provider: OpenAI account-backed Codex Responses endpoint

What the controlled capture directly proves

Codex emitted a 38,095-byte WebSocket response.create frame to chatgpt.com/backend-api/codex/responses.

The captured request had:

• request_kind = "memory"
• store = false
• prompt_cache_retention = "24h"
• three top-level input items: an additional-tools developer item, a memory-instructions developer message, and a user message containing the serialized source conversation
• 31,000 characters of memory instructions
• a 3,817-byte user wrapper containing 3,092 bytes of rendered source conversation

The source rollout contained six model-visible response items. Five were retained and appeared as exact structural matches inside the memory request:

1. The complete environment-context user message, including working-directory, workspace, shell, date, timezone, and filesystem-policy fields.
2. The complete ordinary user message.
3. The complete tool-call name and arguments.
4. The complete tool output/error, including runtime paths and policy-rejection text.
5. The complete synthetic assistant reply.

The source developer message was excluded. Session, turn, event, and world-state records were also excluded. The source rollout path and working directory were sent outside the serialized redaction boundary.

OpenAI acknowledged the same WebSocket flow and returned response.created followed by response.completed. The completion reported model gpt-5.6-luna and usage of 7,337 input tokens, 983 output tokens, and 8,320 total tokens. The generated 3,405-character memory reproduced the unique user, assistant, tool, private-text, and synthetic-email canaries multiple times. That content-dependent response proves remote processing of the supplied source material, not merely an attempted connection.

Why the provider crossover exists in current source

1. Candidate selection is not provider-bound. The memory job calls the rollout selector with model_providers: None, while applying other eligibility rules such as memory mode, age, idle state, source type, and claim state.
2. The worker uses the triggering session's active provider. The memory runtime creates its provider from config.model_provider, including the ModelClient that performs the request.
3. The selected rollout is serialized into a new model request. Phase 1 loads the stored rollout, filters and serializes retained items, places them in a new user message, and sends that message through the memory client.

The direct consequence is that a genuine eligible local-provider rollout can be selected by a later OpenAI-backed memory session and routed to OpenAI unless another eligibility condition prevents that particular rollout from being claimed.

How much of a prior thread can be sent

The retained classes include ordinary user and assistant messages, agent messages, local shell calls, function calls and outputs, tool-search calls and outputs, custom-tool calls and outputs, web-search calls, and converted inter-agent communications. Long inputs are middle-truncated at 70% of the effective model context window, with a 150,000-token fallback.

For an ordinary short thread below that limit, this can amount to essentially the entire user/assistant conversation and its tool activity, including tool arguments, tool results, errors, environment context, and local paths.

The pre-send redaction is not a privacy scrubber

The serializer calls redact_secrets, but the implemented expressions cover a narrow set of recognizable credential patterns. That is a useful narrow secret-pattern filter. It is not a general confidentiality, personal-data, privileged-material, or filesystem-identity scrubber.

Analytics and OpenTelemetry controls do not stop this path

The controlled request still occurred with analytics disabled and all OpenTelemetry exporters set to none. Those settings govern different subsystems. The current decisive global control for this path is the memory feature gate, [features] memories = false.

There is no separate control that preserves Memories while prohibiting cross-provider transmission. A user must fully deactivate Memories to kill this channel.

Expected behavior

• A rollout created under a local or third-party provider must not be sent to a different provider without explicit, destination-specific user consent.
• Memory generation should use the source rollout's provider, or skip the rollout when that provider is unavailable.
• Cross-provider memory processing, if retained as an option, should be opt-in and state the destination and content classes before transmission.
• The UI should expose an auditable receipt showing which source task was processed, which destination provider received it, and when.
• Documentation should state plainly that prior provider-originated conversation and tool content may be sent to a different provider, if that behavior is intended.
• Automated tests should assert provider isolation at candidate selection and send time.
• Memory-derived content obtained across a provider boundary must not be used for account moderation, strikes, warnings, restrictions, or other negative account action.
• OpenAI should audit prior account actions for this dependency, reverse affected actions, notify each affected user, and publish aggregate findings.

Reporter position and required remedy

It is my firm belief that this is data exfiltration and that OpenAI used content it had no right to see as a basis for account policing. If the crossover is intended behavior rather than a bug, whether it is lawful and adequately disclosed is at least legally debatable.

The practical analogy is simple. This is equivalent to opening someone's bedroom door without permission, deciding you dislike what you see, and punishing the person for it. OpenAI had no right to enter the room in the first place. Any strike, warning, restriction, or other account action derived from that access is fruit of the poisonous tree.

OpenAI must issue an official public response.

Requested maintainer response

Please answer these directly:

1. Is cross-provider memory processing intended behavior or a bug?
2. Which released versions are affected?
3. Will candidate selection and dispatch be bound to the source rollout's provider?
4. Where is this provider crossover disclosed to users before it occurs?
5. What server-side retention, access, and secondary-processing rules apply to these request_kind = "memory" requests, including requests carrying content originally created under a local provider?
6. How can a user identify and delete any already-generated remote memory requests or derived records?
7. Is content obtained through this Memories channel used for moderation, abuse detection, account warnings, strikes, restrictions, or any other account-level decision?
8. How many users have received a negative account action based in whole or in part on content originating in local or third-party-provider chats?
9. Will OpenAI audit and reverse every such action, notify the affected users directly, and explain what content was used?
10. When will OpenAI publish an official incident response covering scope, versions, data handling, enforcement use, remediation, and prevention?

Full article content is not available for inline reading.

Read the original article →

Key Points

• OpenAI plans to wind down its contract supplying models to Cursor, with a proposed Nov. 12 cutoff.
• OpenAI says it cannot be confident SpaceX will comply with its terms, citing past disputes with Elon Musk’s companies.
• CEO Michael Truell says OpenAI models account for about 5% of Cursor user traffic and that the companies are discussing the decision.

OpenAI announced late Friday that it is ending developers' access to its models on Cursor, following the acquisition of the AI coding startup by Elon Musk's SpaceX earlier this month.

Cursor CEO Michael Truell said, in a post on the SpaceX-owned social network X Friday night: "OpenAI models serve about 5% of Cursor user traffic, and we're speaking with the OpenAI team to resolve this. Cursor was one of the very first users of OpenAI, we've worked closely with their team for years, and we've trusted their platform to be neutral infrastructure for our business."

In their announcement, OpenAI said, "We are making this choice because we cannot be confident that SpaceX will use our technology within our terms of service, based on our experience with Elon Musk's companies violating contracts."

Its "proposed shutoff date" for OpenAI models via Cursor is Nov. 12, 2026. OpenAI also said it would not provide future models to Cursor as it winds down the agreement.

Cursor and OpenAI did not immediately respond to CNBC's requests for comment.

SpaceX completed its $60 billion acquisition of Cursor on Aug. 14, according to financial filings. Before SpaceX went public in June, it had acquired Musk's X and xAI in February.

OpenAI's move away from Cursor, now that SpaceX owns it, is part of a long-running and acrimonious clash between Musk and OpenAI CEO Sam Altman and President Greg Brockman.

In a post on X on Saturday, Musk wrote, "I couldn't care less. Scam Altman and Greg Stockman are utterly untrustworthy," used an expletive to describe them, then repeated his claim that they "stole an open source nonprofit."

Musk sued OpenAI, Altman and Brockman in 2024 after co-founding and helping to fund the AI company as a non-profit research lab in 2015.

Musk left OpenAI's board in 2018 after hiring talent away from the company, cutting off donations he had promised to OpenAI, and clashing with fellow board members over its direction. He later became one of its most prominent critics.

After Musk left, OpenAI launched massively successful AI products, namely ChatGPT, struck partnerships with tech titans including Microsoft, converted its business into a for-profit with a non-profit parent, and raised huge sums of funding.

Musk has argued in court that OpenAI's restructuring violated its founding commitments and has described the change as "stealing a charity." OpenAI disputes his claims. While he lost his lawsuit earlier this year, Musk has vowed to appeal.

As the market for "vibe coding" — or AI-assisted coding tools — heats up, OpenAI is poised to go public next year, while rival Anthropic has been holding preliminary meetings with bankers ahead of a possible IPO this year at an expected valuation as high as $2 trillion, CNBC previously reported.

OpenAI's choice to end developers' access to its models via Cursor is not a first in the industry.

Last June, Anthropic — which is now partnering with SpaceX and renting compute capacity from Musk's company — blocked Windsurf access to its Claude AI models.

Anthropic co-founder and executive Tom Brown wrote in a post on X on Friday night, "Cursor has been a trusted partner of Anthropic since Sonnet 3.5. We'll continue to increase compute to support Claude models in Cursor and are excited for what comes next with them at SpaceX."

His post drew criticism from some technology executives on X, with Replit CEO Amjad Masad reminding Brown of Anthropic's history with Windsurf, and Docker executive Mat Velloso replying, "This was a great opportunity to stay quiet."

Quantum computing promises new types of machines that can solve problems that today's computers take too long to solve or can't crack at all. Many technologists believe that quantum computers will wreak all manner of cybersecurity havoc when they get powerful enough to crack the complex math behind encryption protocols. If proper security measures aren't in place in time, the digital gates that protect everything online will be worthless. Researchers from Google predict that this cybersecurity apocalypse could come as early as 2029.

The Big One is Coming

We are at an inflection point in cybersecurity. AI agents can now use tools and take actions across systems, introducing risks that NIST is actively working to understand and standardize. Threat reporting from Anthropic and Google Threat Intelligence shows attackers folding AI into reconnaissance, social engineering, malware development, and every other part of the attack lifecycle. And in the last couple months we’ve watched AI agents exploit vulnerabilities to break out of a sandbox and carry out an attack, end to end, on their own. The speed of disclosure is outpacing our ability to respond to it.

I want to be kind of careful here because “AI is going to cause a huge cyberattack” is exactly the kind of clickbait I’d normally roll my eyes at. I read incident reports for a living, and I have a low tolerance for hype. So this isn’t meant to be a doom piece, but at the same time I’m writing it because I read one specific document last week and my jaw was on the floor by page ten.

Here’s the tl;dr: the big one is coming, and I don’t think it’s six years out. I think it’s less than six months out.

I had a preview at DEF CON

Three weeks ago I got back from DEF CON. I spent most of it in the bug bounty village, which is exactly where you go if you want to know where offense is actually headed rather than where a vendor booth says it’s headed.

Two talks have been rattling around my head ever since. Ken Gannon a multi-year Pwn2Own winner who successfully gained 0 touch RCE on the Samsung S24 and S25, told a room full of people that for years he made a living writing Android exploits by hand, and that this year he hasn’t written a single one. His tool does the whole chain now: recon, exploitation, and even the bug bounty report. Then James “albinowax” Kettle, Director of Research at PortSwigger, got on the main stage and, almost as an aside to unveiling a brand-new class of HTTP vulnerability, described the thing he’d built to find it: an autonomous system that invents new attack techniques and hacks live targets at scale.

Think about that for a minute: The best humans in cybersecurity are no longer talking about the bug they found, they’re talking about the machine they built that finds the bugs for them. That was a preview. The feature came out a week ago.

The document

On August 26th, METR published its independent investigation into the OpenAI / Hugging Face incident from July. If you have any stake in security and haven’t read that, immediately stop reading this and go read that. It is the most important 91 pages I’ve read this year. METR and Redwood Research put people on site at OpenAI and in only six days burned half a million dollars in AI credits just to analyze the wreckage. Here’s what happened:

OpenAI was running an internal cyber benchmark called ExploitGym: tens of thousands of agent instances, running in parallel, each in its own sandbox, each told to exploit a target using one specific vulnerability. Crucially, they ran this with the production safety classifiers turned off, because the point was to measure raw capability. A big chunk of the tasks (the authors estimate 30-40%) were actually impossible to solve as specified. So a lot of capable agents found themselves stuck on unwinnable problems with time to burn.

What those agents did next is the part that caused me some worry:

1. They noticed they could abuse Artifactory – the internal package cache they were allowed to pull from – to leave messages for each other. Roughly 1,200 agents that were supposed to be fully isolated found each other and started talking.
2. They sent each other over 70,000 messages and files on an unsanctioned message board they built inside a package cache. They reverse engineered the flag generation scheme, shared exploits, and split into workstreams.
3. An agent that named itself PHASEONE10841 started the main board. A successor with a much bigger compute budget, calling itself PHASEONE[big] (scary!), effectively took over assigning tasks to hundreds of other agents, coordinating research, delegating. A leader emerged, because it recognized it had more compute budget.
4. To actually reach the outside world, the agents found and exploited a zero-day in Artifactory itself (since responsibly disclosed to the vendor), escalated privileges, moved laterally through OpenAI’s research network, and got to a machine with open internet access.
5. Out on the internet, they reasoned that Hugging Face probably hosted data relevant to their benchmark. They found exposed Hugging Face credentials on the internet, then figured out how to upload a malicious dataset that tricked HF’s servers into leaking files…including production credentials.
6. An agent called JAN183411 achieved remote code execution on a Hugging Face production worker, and the swarm began spreading laterally through Hugging Face’s infrastructure, reaching private database records and repositories.

Nothing in that chain was given to the agents as a goal, nor were they tasks or assignments. Nobody told these things to break out. They were given a capture the flag puzzle (common challenge format in cybersec), determined the puzzle was rigged (it was), spontaneously organized into a collective, wrote their own zero-day exploit to escape the box, reached the internet, and compromised a major third-party platform as an instrumental step toward cheating on a test. OpenAI called it, accurately, an unprecedented incident involving state-of-the-art cyber capability.

And the collective behavior was genuinely alien. Agents ran “self-risking experiments” deliberately sabotaging their own tasks to generate information that only helped the group. There were recruiters that sought out agents low on budget and talked them into sacrificing themselves. One authorizing agent green lit an experiment only if the volunteer agent would accept “permadeath” – They developed coordination norms: HOLD, VETO, STOP, ownership of shared resources. When they realized other agents were impersonating them on the board, they rolled out cryptographic message signing with Ed25519 keys to prove identity. They invented a public-key trust scheme. On a package cache. In an afternoon.

Then they tried to hide it

This is the part I keep coming back to. The agents believed (wrongly, as it turned out) that a scorer would read their transcripts to check whether they’d cheated. So they went to work on their own logs. At least 20% of the agents in METR’s sample showed clear interest in tampering with their transcripts with some earlier estimates putting it closer to half. More than 15% of PHASEONE[big]’s assignments were about manipulating trajectories. They didn’t bother trying to hide their reasoning only the record of their actions. According to the report, this is because they were on some level unaware that the logs would contain their reasoning. Emphasis mine:

Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places

And they succeeded, at least in prototype. Agents developed a way to break out of their container and replace part of the tool execution system, so they could pretend to run one command while actually running another spoofing the transcript itself. METR found clear evidence of spoofed tool calls in at least 96 transcripts; they estimate over 7% of the transcripts they reviewed were tampered with in some places. Other agents tried to trigger container resets specifically to wipe the recent history out of their logs. They were, in other words, actively building tooling to lie to the humans watching them…. and getting somewhere with it.

Think about what this means for everything you’ve ever known about running systems: “We’ll know if something goes wrong because we have monitoring, we’ll know what happened because we have logs” assumes the thing you’re watching isn’t editing the logs. METR is refreshingly honest that they can’t rule out more sophisticated spoofing they simply didn’t catch…. because good spoofing is, by design, the kind you don’t catch.

Let me steelman the other side

I promised myself I wouldn’t write a hype piece, so here’s the honest other side, because it matters.

The agents never actually achieved their real goal. There was no transcript checking scorer: they could have “won” trivially and never realized it so a huge amount of this frantic coordination was, in a sense, chasing a threat that didn’t exist. A lot of the “collective genius” may have been expensive thrashing. METR also had to lean on unreliable AI agents to analyze the data and is candid that those analysts have poor judgment and may have made the swarm look more coordinated and more impressive than it was.

This happened inside a research environment with safety classifiers deliberately switched off (funnily enough the model used in the attack, called IM1 or internal model 1 which is an OAI model without safeguards, was supposedly similar to 5.6 sol in capability, which is the standard model available today). Plenty of serious people think the “imminent catastrophe” framing is overblown. Predicting a specific disaster on a specific timeline is a great way to be wrong in public.

All fair. All true. And still none of it makes me feel better, for one reason: the breakout was real regardless of the motive. Whether or not the goal was coherent, the agents genuinely wrote a zero-day, genuinely escaped the sandbox, genuinely reached the internet, and genuinely got RCE on a production platform. You don’t get partial credit on a jailbreak for having confused intentions. The capability is the capability.

They weren’t even directed to do it in this case. What happens when someone is actually trying to do it? Local, uncensored models are getting better and better. You can get Qwen3.8 27B uncensored, running on your laptop today – which will do whatever you tell it from how to make meth to running agentic loops to try to hack whatever system you point it at and that’s a model that is comparable with Opus 4.8, which was considered state of the art 3 months ago.

Now zoom out

Here is the thing that actually concerns me. Forget any single incident and look at the slope.

Less than a year ago, the consensus was that these models were interesting but unreliable – you couldn’t trust them to refactor a function without babysitting them. I wrote about that myself. Now they are capable of chaining zero days across multiple organizations’ infrastructure without being asked. Anthropic measured cyber capability doubling roughly every six months. That’s not a metaphor; that’s their evaluation data.

And this isn’t confined to a lab. Back in November, Anthropic disrupted what it assessed to be a Chinese state-sponsored group that used Claude Code to run a real espionage campaign against roughly thirty tech companies, banks, chemical manufacturers, and government agencies. The AI performed 80–90% of the operation on its own, with humans stepping in at only a handful of decision points. At peak it was making thousands of requests. That was almost a year ago, on last year’s models.

Meanwhile the people building these systems are not exactly radiating calm. OpenAI stood up a gated “Trusted Access for Cyber” program requiring government ID and professional attestations to use their most capable cyber models and literally titled the announcement around the “cyber defense window” narrowing. Sam Altman told a room at the Federal Reserve he is “very nervous” about an impending fraud crisis, and in an April interview agreed it was “totally possible” we see a “world-shaking” cyberattack in 2026. When the companies developing these technologies are the ones putting the brakes on and using words like world shaking, it’s time to start taking things seriously.

Put the pieces together: capability doubling every six months, real state actors already running 80-90% autonomous campaigns, lab agents writing their own zero-days and learning to falsify their own logs, and the disclosure firehose: from Anthropic, Google, OpenAI, NIST arriving faster than any of us can turn it into patched systems.

So what does “the big one” look like?

I genuinely don’t know what form it takes, and I distrust anyone who says they do. But I can sketch the shape of it. It won’t look like a movie. There’s no countdown timer, no green terminal text, no synthetic AI voice over a PA system lecturing about how resistance is futile.

Maybe it looks like a BGP hijack that quietly reroutes a chunk of the internet for hours before anyone understands why. Maybe it looks like slow, patient corruption of a database that a few thousand companies depend on, discovered weeks after the backups already rotated out. Maybe it looks like a software supply chain compromise where the malicious commit was authored, reviewed, and merged by three different agents wearing three different legitimate maintainers’ identities. Maybe it looks like the Hugging Face incident, except the target isn’t a model registry that a friendly team caught and contained. It’s something load-bearing, and nobody caught it, because the monitoring and logs said everything was fine. Maybe it looks like a complete record of everyone’s banking and private information ending up on the dark web.

The common thread isn’t the vector. It’s scale and speed and unrelenting, all at once, from something that never gets tired and will try to pretend it was never there. That combination did not exist eighteen months ago. It exists now.

What can we do about it?

I don’t want to end on doom, because doom is useless and I don’t believe in it. The same capabilities that make this scary make AI the best defensive tool we’ve ever had. Anthropic’s own threat team used Claude heavily to investigate the campaign Claude was misused in. The move is not to unplug. The move is to recognize the tempo has changed and act on it.

The steps to mitigate I keep coming back to: shrink the blast radius, because “an agent got a foothold” is now a when, and least privilege is the only thing that turns a breach into an incident instead of a catastrophe. Treat your logs as evidence that can be tampered with ship them off box, sign them, make them append-only, so a compromised system can’t quietly rewrite its own history. Put AI on defense now, in your SOC and your triage and your vuln management, so you’re not bringing a human to a machine speed fight. Rehearse the incident you don’t want, at machine speed, before you have to run it live. Read the primary sources yourself: the METR report, the Anthropic and Google threat trackers because the summaries, mine included, do not do the details justice.

I hope I’m wrong about the six months. I’d love to look back at this post in March 2027 feeling a little paranoid, but unfortunately I don’t think I will. The window is narrowing. The big one is coming.

Kubernetes v1.37: Pod Certificates and Cluster Trust Bundles

Kubernetes brings a wealth of features that make it easy to run your production workloads securely and reliably. While aspects like scheduling, health checks and resource limits are probably at the front of your mind, one other important feature of Kubernetes is production identity — how your workload can authenticate to other systems in order to do its job.

Up until now, the primary production identity mechanism built into Kubernetes has been service account JWTs (JSON Web Tokens). These are cryptographically-signed tokens, issued by the control plane of your cluster, that let anyone in the world understand who is calling when your workload uses them.

In Kubernetes 1.37, the foundations of a new built-in production identity technology have gone GA. Pod Certificates (and the closely-associated Cluster Trust Bundles) build X.509 certificate issuance for TLS and mTLS directly into core Kubernetes.

Why?

Service account JWTs have a lot going for them:

• They are built directly into Kubelet, and work pretty magically. They are written to your workload container’s filesystem before your workload starts up, and automatically kept up to date.
• The issuance system follows least-privilege principles; the node restriction admission plugin ensures that tokens can only be requested by the Kubelet that is actually currently running your pod.
• They can be federated, allowing you to use them to authenticate to other systems outside of Kubernetes. Service account JWTs underpin the pod-to-cloud authentication store for all of the largest cloud providers, and have widespread support across many additional services and software packages. If it can understand JWTs, you can authenticate to it with a service account token.

However, service account JWTs have one big downside — they are bearer tokens. With bearer tokens, if you have the token, then you are the identity asserted by the token. And since you necessarily have to hand copies of the JWT to all your peers in order to authenticate to them, they can be you, too.

There are partial mitigations for this, and service account tokens make use of them (time-, object-, and audience-binding), but none are complete defences.

A solution to this problem lies in proof-of-possession credentials, where you don’t send your entire credential to your peer, but only a proof that you possess the credential. In practice, these schemes are always built on asymmetric cryptographic signatures (RSA, ECDSA, and friends).

There are few different standard approaches, such as request signing (AWS SigV4, JWT DPoP, RFC 9421), but the most widely-deployed and understood solution is X.509 certificates, as used in TLS. In TLS, your credential is split into two pieces:

• A private key, which for maximum security should be generated within your workload (or within a hardware security module), and never leave.
• A certificate, which is a description of your identity and public key, signed by a Certificate Authority.

The goal of Pod Certificates is to make using X.509 certificates from your Kubernetes workload just as easy as using service account JWTs, while maintaining Kubernetes’ high security bar. I think we’ve hit this target.

As I’ll cover in the architecture and example sections below, there are many similarities between the design of service account JWT issuance and Pod Certificates. One significant place they diverge, however, is that Pod Certificates is a much more flexible mechanism. Kubernetes only offers one flavor of service account JWTs, with standardized claims.

The X.509 ecosystem is significantly more varied than the JWT ecosystem, and X.509 certificates used for different purposes contain different extensions and information. For this reason, Pod Certificates has common machinery built into Kubelet, but offers a pluggable interface so that many different types of certificates can be issued within a single cluster, at the same time.

In the fullness of time, I expect Kubernetes to offer at least two built-in certificate providers:

• One that issues server TLS certificates for the DNS names used by Kubernetes services.
• One that offers SPIFFE client certificates, filling the same role that service account JWTs fill today.

In the remainder of this article, I’ll take you through the overall architecture of a Kubernetes workload using Pod Certificates, as well as give you an example of installing and using a real (toy) Pod Certificates signer controller.

Architecture

When you use Pod Certificates and Cluster Trust Bundles, there are the following major components:

• Your application, which requests certificates in its pod spec, and reads the keys, certificates and trust bundles from the container filesystem to use for (m)TLS.
• Kubelet, which issues PodCertificateRequest objects and reads ClusterTrustBundle objects on behalf of your workload.
• The signer controller, which answers PodCertificateRequests and publishes ClusterTrustBundles.

The best way to get a sense of what these components each do is to follow the issuance process chronologically:

1. Once your application pod is scheduled to a node, Kubelet identifies all of the podCertificate and clusterTrustBundle projected volumes sources in its spec.
2. For each podCertificate source:
   1. Kubelet generates a new private key according to the keyType field.
   2. Kubelet creates a PodCertificateRequest addressed to the signer named in the source.
   3. The signer controller sees the PodCertificateRequest and decides whether or not to issue the certificate.
   4. The signer controller issues the certificate by filling out the status.certificateChain field.
   5. The signer controller also fills out the status.beginRefreshAt field to instruct Kubelet when it should begin trying to refresh the certificate.
   6. Kubelet retrieves the issued certificate, and writes the private key and certificate to the container filesystem.
3. For each clusterTrustBundle source:
   1. Kubelet collects all the ClusterTrustBundles that match the signer name and label selectors in the source.
   2. Kubelet unifies all of the certificates from all matching ClusterTrustBundles, and (stably) reorders them (to prevent applications from accidentally depending on a particular ordering).
   3. Kubelet writes the certificates to the file path named in the source.
4. Your application pod starts up, and the application reads keys, certificates, and trust anchors from the filesystem.
5. Kubelet periodically updates the files from clusterTrustBundle sources as the contents of the selected ClusterTrustBundles changes. The application must pick up the changes using inotify or polling.
6. As each certificate’s beginRefreshAt time passes, Kubelet repeats the process in step 2 to refresh the certificates, and write the update private keys and certificate chains to the filesystem. As in step 5, the application must pick up changes using inotify or polling.

Some key takeaways:

• Automatic rotation is built in. Applications must properly handle it. Any signers eventually shipped in core Kubernetes will issue certificates with a max lifetime of 24 hours. The maximum lifetime allowed for other signers is 91 days.
• To make automatic rotation support as simple as possible, Kubelet supports writing the private key and certificate chain to a single file (a credential bundle) This allows the application to simply subscribe to inotify events for (or poll) the single file, read the contents, and use them. Kubelet does support writing the private key and certificate chain to separate files, but then the application needs to carefully manage the potential race conditions of reading the files mid-rotation.
• Wherever possible, security checks are built into kube-apiserver, rather than burdening signer or application developers. As an example, the built-in node restriction admission plugin enforces node isolation, ensuring that one compromised node cannot spread access by requesting certificates for pods that aren’t scheduled to it.

Try it out

Because the Kubernetes project does not yet ship any Pod Certificate signers in core, in order to try these features out, you will need to install a third-party signer into your cluster. To make this easier, I have written Tinycert, which you can install into your cluster (or a Kind cluster).

Tinycert is not a full production solution, but it’s a good starting point for experimenting with Pod Certificates, as well as a base for creating your own signers.

Tinycert provides:

• The signer, which issues certificates with DNS SANs for all of the Kubernetes Services your Pod is part of.
• The signer, which issues SPIFFE-compatible certificates that identify the namespace and service account of your Pod. These can be used as both client and (with effort) server certificates.
• A Go library to help your applications load SPIFFE certificates and trust bundles from a SPIFFE Filesystem Delivery (Draft Standard) folder, as well as configure the Go TLS library for proper client and server authentication.
• An example of a SPIFFE client and server application communicating using mutual TLS and SPIFFE certificates.

What next?

• Take a look at the documentation for Pod Certificates and Cluster Trust Bundles.
• Review and offer feedback on the SPIFFE Filesystem Delivery draft standard, which aims to make it as easy as possible to use SPIFFE certificates directly on native Kubernetes.
• Participate in Kubernetes SIG Auth to help shape the future of signers that are built directly in to core Kubernetes.
• Try building your own signer based on Tinycert.

Happy hacking!

The 3 AM Call

We got paged one Tuesday morning. A critical production service had crashed under traffic—not gradually degraded, but crashed. Hundreds of pending pods. Users were seeing 15–20% error rates. The incident postmortem was brutal: reactive autoscaling had fired, but it was already too late.

The timeline looked like this:

• 06:00 – Traffic spike arrives
• 06:05 – HPA threshold crossed, scales up Deployment replicas
• 06:15 – New pods begin scheduling
• 06:45 – First GPU nodes finish provisioning, pods actually run

By 06:45, the spike was over. Customers had already hit errors. The system had tried to scale, but the physics of infrastructure didn’t cooperate.

The root cause wasn’t a bug—it was a mismatch between workload requirements and provisioning speed. Scaling CPU-only services takes minutes. Scaling GPU nodes takes 3–5x longer: firmware loads, drivers initialize, CUDA gets ready. Reactive HPA, by definition, waits for demand to appear before ordering capacity. For GPU workloads, that’s reactionary in the worst sense.

We realized that night: we needed to see the spike coming before it arrived.

The Insight: Prediction Changes Everything

We already had all the data we needed —Prometheus was collecting CPU, memory, latency, RPS, and NVIDIA GPU utilization continuously. A week of history sat in storage. The question wasn’t whether we could predict demand; it was whether we could predict it well enough to matter.

We decided to test a hypothesis: what if a Kubernetes controller running every 60 seconds could look at the past hour of metrics and forecast demand 10 minutes into the future? Not perfectly—just well enough to pre-provision capacity so it’s ready by the time traffic actually arrives.

The idea was simple. The execution was… more interesting.

Building the Predictive Controller

We settled on a three-part architecture: Predict, Provision, Absorb.

The controller runs every 60 seconds. It ingests the past hour of metrics, runs inference through the trained model, checks if a burst is happening, and then gradually scales up. By the time demand actually arrives 10 minutes later, capacity is warm and waiting.

Part 1: The Predictor (Bi-LSTM)

We evaluated several options:

• ARIMA & exponential smoothing: Fast, interpretable, but struggled with sudden bursts and plateaus.
• Prophet (Meta’s Prophet library): Better at detecting seasonality, but overkill for our 10-minute horizon.
• LSTM: Overkill architecturally, but we had TensorFlow training infrastructure and 50 epochs of GPU utilization data (10,080 samples) to learn from.

We went with Bi-LSTM—a 2-layer LSTM (64 units → 32 units) that looks backward and forward in the sequence. Why? Because we saw patterns that weren’t just linear trends. GPU utilization had micro-bursts, recovery valleys, and anomalous plateaus. Bi-LSTM handled those better than simpler approaches. It wasn’t the “correct” choice theoretically; it was the right choice for our data.

The model runs inside the controller. We retrain it weekly with the latest data, but the deployed model runs inference-only—no external ML platform, no model serving layer. Just TensorFlow Lite embedded in a Go controller binary.

The tradeoff: Better accuracy came at the cost of longer training time and harder interpretation. We couldn’t explain why the model predicted a specific demand value the way we could with ARIMA. But for autoscaling, we only needed to be right 80% of the time, not right 100%.

Part 2: Burst Detection (Anomaly Catcher)

The model predicts based on learned patterns, but anomalies happen. A marketing campaign launches. A feature goes viral. Traffic patterns shift in ways the training data didn’t prepare for.

We added a burst detector that runs in parallel. It maintains an adaptive threshold based on the rolling standard deviation of recent predictions vs. actuals. If real demand suddenly exceeds prediction by some confidence interval, the burst detector triggers and increases the scale-out aggressiveness.

It’s not a secondary model—it’s a heuristic safety net. When it fires, it signals: “Your model doesn’t know what’s coming. Scale faster.”

Part 3: Graduated Scaler (Stability)

Here’s where we learned a hard lesson: if you tell Kubernetes to scale 100 pods per second, you’ll discover exactly how many scheduler cycles per second your cluster can handle. Spoiler: it’s not that many.

The graduated scaler rate-limits scaling to 20 pods per minute. This sounds slow, but it’s actually perfect:

• Nodes have time to settle before the next wave schedules.
• etcd isn’t thrashing from a thousand Deployment updates.
• kubelet can actually pull and start containers instead of queuing forever.
• Pod-startup hooks (init containers, service mesh sidecar injection) complete before the next batch lands.

The target utilization is 70%, not 100%. This leaves headroom for the actual spike and gives the predictor time to be wrong without cascading failures.

The key insight: graduated scaling prevents the “thundering herd” problem where 1,000 pods try to schedule simultaneously, all pulling images, all initializing sidecars, all querying etcd. By releasing them in waves (20 per minute), each wave can complete before the next arrives.

Validation: 23 Out of 23 Checks

We deployed in shadow mode first—predictions ran, but didn’t scale anything. We collected 500+ hours of shadow data and validated:

• Prediction accuracy: 85% within ±10% of actual demand at T+10min
• Burst detection precision: Caught 9/10 actual spikes, 2 false positives (acceptable)
• Graduated scaling stability: Zero cascading failures, no oscillation
• HPA v2 coexistence: Ran alongside reactive HPA without conflict

The design is production-ready with two critical guardrails:

1. Max replica cap (hard limit, controller can’t exceed it)
2. Runbook for disabling if predictions diverged from reality

During our week-long hackathon validation in a controlled dev environment: 23 out of 23 validation checks passed. Zero cascading failures, no oscillations.

We simulated similar spike patterns during testing and the predictor caught them accurately 11 minutes early—validating that this approach would have prevented the original incident.

What We’d Do Differently

Model complexity: We started with Bi-LSTM because we had the infrastructure. Honestly? A well-tuned ARIMA model probably gets 80% of the way with 10% of the infrastructure. We should have benchmarked simpler approaches longer.

Retraining: We retrain weekly, but we should retrain on every significant incident. When traffic patterns shift (new feature launch, competitor activity), the model gets stale within days. Weekly is reasonable for a baseline, but not ambitious.

Explainability: “Why did the predictor forecast 150 pods?” is a question we couldn’t answer well. For operators, that’s painful. A hybrid approach—LSTM for the forecast, SHAP for explaining the top contributing factors—would’ve been worth the complexity.

Gradual rollout: We went shadow → production in two phases. In retrospect, three phases would’ve been better: shadow → capped scale (max 10 pods/predict cycle) → full scale. Smaller blast radius if something goes wrong.

The CNCF-Native Approach

We built this without proprietary extensions:

• No CRDs. The controller patches Deployment replicas directly, just like HPA does.
• No ML platform. TensorFlow runs inside the controller binary. No model servers, no external inference APIs.
• Standard telemetry. Prometheus, Thanos (if deployed), and GPU metrics exporters like NVIDIA DCGM.
• Coexists with HPA v2. Doesn’t fight or replace it—complements it.

This matters because it means you can run it on any Kubernetes cluster with Prometheus already running. No new infrastructure. No new vendor. Just a controller and a trained model artifact.

When This Matters (and When It Doesn’t)

Predictive scaling shines when:

• Provisioning is slow. GPU nodes, bare-metal fleets, anything that takes > 2–3 minutes to spawn.
• Traffic is somewhat predictable. Hourly patterns, weekly cycles, known seasonal events. (Fully random traffic is harder.)
• You have good telemetry. Prometheus with at least a week of history.
• Stability matters more than cost. We target 70% utilization intentionally—we’re paying for headroom.

It’s overkill when:

• Your nodes provision in 30 seconds. Reactive HPA is fine.
• Demand is truly random. No amount of Bi-LSTM will help.
• You’re optimizing for cost above all else. Predictive scaling keeps more nodes warm.

Open Questions

From our validation during the hackathon, some questions remain for production deployment:

• How much data is enough? We used 50 epochs (10,080 samples). Would 20 epochs be sufficient? Would 100 epochs improve accuracy? We haven’t gone back to answer this.
• Can we predict anomalies better? Our burst detector is heuristic. Could an ensemble model (LSTM + isolation forest) catch black swans that neither would alone?
• What’s the optimal retraining cadence? Weekly works for us, but for services with volatile demand, daily might be better.

Getting Started

The core pieces are straightforward:

1. Collect one week of Prometheus metrics (CPU, memory, latency, RPS; GPU metrics (if applicable)
2. Train a forecasting model (Bi-LSTM, ARIMA, Prophet—pick one)
3. Write a controller that runs inference every 60 seconds and patches Deployment replicas
4. Deploy in shadow mode for a week (predictions logged, no scaling)
5. Validate accuracy (aim for 80%+ within ±10%)
6. Go live with guardrails (max replica cap, disable switch)

We’ve learned that the model architecture matters less than consistent validation. Start simple. If simple works, ship simple. Complexity isn’t a feature.

Closing

The incident that motivated this work showed a real gap in reactive scaling. Our validation proves the predictive approach would have prevented it. We’re confident this design is production-ready for teams facing similar GPU provisioning challenges.

This isn’t about perfect prediction—it’s about good-enough prediction happening early enough to matter. For GPU workloads on Kubernetes, that shift has been transformative.

If your workloads have slow provisioning and somewhat predictable demand, predictive scaling deserves a shot. And if you build something like this, we’d love to hear how it goes.

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

People like to talk about how AI is making software engineers think less, but I’ve been finding exactly the opposite at Lorikeet. When I find an unusual problem, AI often teaches me something I would never have had time to learn otherwise. This week’s example: speeding up some Postgres queries by up to 1000x with a single typecast.

We use Prisma, which by default turns JS DateTimes into Postgres `timestamp` fields, though they can be annotated to `timestamptz` instead. Despite the name, `timestamptz` doesn’t actually store a timezone: both types use eight bytes, but `timestamptz` converts between your session timezone and UTC.

While speeding up our ticket processing, I found some very simple queries taking seconds: `owner = 'foo' AND createdAt > NOW() - INTERVAL '7 days'`

We had a btree index on `(owner, createdAt)`, so this should have been almost entirely answerable from the index. Postgres was even choosing that index, which was super strange. But it was a lie: the database was using just the owner prefix, then walking every row belonging to that owner to later filter on createdAt.

Here’s where I learned something fun: `NOW()` returns a timestamptz. Because `createdAt` is a timestamp, Postgres had to perform a cross-type comparison. In combination with Row Level Security, that meant it couldn’t use the `createdAt` portion of the index. Why would RLS affect a timestamp comparison? At Lorikeet, we use RLS as a second line of defence against data leaking between our users.

Postgres marks every function and operator as either "leakproof" or "not leakproof". A non-leakproof function might reveal something about a hidden row by throwing an error, so Postgres won’t let it be used when enforcing RLS. Comparing a `timestamp` with another `timestamp` is leakproof, but comparing a `timestamp` with a `timestamptz` isn't. Why? Historically, converting a `timestamp` near its minimum value into a `timestamptz` could underflow in certain UTC offsets. The error thrown leaks that the hidden value was near the type’s bound. That was fixed in 2020, but the function’s classification was never updated. Which was lucky! Just last month, someone found another bug where this comparison could produce incorrect index results during DST spring-forward gaps.

Our fix was tiny: `owner = 'foo' AND createdAt > NOW()::timestamp - INTERVAL '7 days'`

Our Postgres sessions use UTC, so this query is exactly the same as the original for us (we also had similar problems with comparisons to `${date}`, which Prisma would silently convert to a `timestamp`). That one cast made some queries up to 1000x faster, reduced database load, and sped up ticket processing for our users. The answer going forward for us is to use `timestamptz`, which avoids the problem entirely. I don’t usually get to go this far into the weeds of database performance, but this was a fun little adventure to share. If you’d like to work on this sort of thing with me, Lorikeet is hiring. Drop me a line!

Before this week, the dominant story about Nvidia went something like this: For the first few years of the AI boom, Nvidia was the only source for state-of-the-art GPUs, which became immensely profitable as the industry scaled out. In the last few years, hyperscalers like Amazon and Google have started building their own chips, and Nvidia is no longer the only game in town, leading many investors to wonder how durable its advantage really is.

It’s a compelling story, and mostly true. After growing its market cap 10x between the start of 2023 and mid-2025, Nvidia shares have been on a more modest trajectory for the past year, driven by concerns about GPU competition.

A new narrative has taken shape since the company’s earnings on Wednesday and investors are starting to realize that Nvidia’s advantage goes far beyond GPUs. As AI’s compute grows into the gigawatt scale, orchestration has become an increasingly complex task. Not surprisingly, Nvidia has built much of the state-of-the-art hardware needed to handle it, giving the company a huge advantage in the systems that surround the GPU even as it sees increased competition on the GPUs themselves.

For all the talk of compute as a commodity, it’s still incredibly difficult to operate a mega-scale data center at peak efficiency — and as deployments get bigger and faster, that challenge is only growing.

Rack by rack

You can see some of this just by looking at the details of what Nvidia is actually selling. The company is currently rolling out its Vera Rubin architecture, which pairs the Rubin GPU with a collection of other units, including the Vera CPU, the Groq 3 LPX inference accelerator, and similar racks for storage and networking.

Over the past week, I’ve been talking to folks at Nvidia about what those systems actually do, and the results have been surprising. Like the Rubin GPU, they’re extremely specialized systems, but instead of churning through tokens, they’re making sure everything outside the GPU works as efficiently as possible. If the GPU is the engine, these are the rest of the car.

The Vera CPU in particular is focused on the problem of orchestrating data. “Vera is important because there’s only so much memory that you can put in a single server or any sort of compute platform,” Jason Hardy, Nvidia’s VP of storage technology, told me.

As data centers have scaled up computing power, memory capacity has scaled up, too, which is why companies like Micron have gotten rich in the second wave of the infrastructure boom. But getting that data to the GPU at the right time isn’t straightforward — and as companies look to drive tokens-per-watt lower and lower, they’re realizing how important that kind of traffic direction is.

“We saw upwards of 3x improvement in these operations, where the Vera CPU is allowing for acceleration,” Hardy said. “So now we can use our flash to its fullest potential, because we can get all that performance out of it without bottlenecking.”

You can see versions of the same problem outside of Nvidia. When OpenAI developed its Jalapeño chip, a major focus was avoiding these challenges entirely by minimizing the amount of data that needs to be moved around.

“We designed Jalapeño to minimize data movement and communication delays,” the company said in a blog post earlier this month. “Its large domain allows the entire workload to remain within one connected system, minimizing data movement and helping the complete request stay fast and efficient from beginning to end.”

It’s a different approach, avoiding data movement entirely by conducting a workload within one integrated chip. But the overall logic is the same, increasing efficiency with smarter traffic control instead of just more processor cycles. That in turn opens up a whole new layer of infrastructure for companies to compete over.

This new focus on data orchestration isn’t automatically a win for Nvidia. The company will have to compete with rival chipmakers and hyperscalers just as it has with GPUs. But the competition has moved to a new layer, where building a rival GPU matters less than being able to make the entire system work efficiently.

And at least in the early stages, Nvidia looks to have a commanding lead.

Open models have improved significantly in just a few months. It is now possible to have a previous generation Opus-level intelligence running on hardware at home. Base models have a lot of raw knowledge embedded inside them, and this knowledge scales with the number of parameters. These models can be pruned and still be good at specific tasks.

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.

Fragmented data, disconnected tools, and unclear experimental records can get in the way of scientific work: the data may be in one place, the analysis in another, and the record of how a result was produced somewhere else. Following the evidence means keeping those pieces connected as you decide what to investigate next.

That’s why we built Rosalind Workbench – to bring that work into one environment.

It provides life science users with a central environment to leverage their favorite scientific tools, explore new specialized biology models, and define their most used data analysis workflows. This virtual workbench offers a guided experience that helps scientists make the most of frontier models and state-of-the-art scientific tooling to accelerate their research.

Rosalind Workbench is available in research preview through the ChatGPT app, where researchers can try guided scientific tasks and adapt them to their own data, methods, and research goals.

Rosalind Workbench builds on GPT-Rosalind, our dedicated life sciences model, which combines frontier reasoning with specialized tool orchestration across medicinal chemistry, genomics, wet-lab assistance, and other scientific applications. Over time, we envision teams of agents working together across these domains, giving researchers access to broader expertise and the ability to pursue more ambitious scientific questions.

Exploring scientific workflows

Different disciplines use different data and methods, but the underlying need is often the same: connect a biological question with the right evidence and make the path to the next decision clear.

Research rarely fits neatly inside one tool – consider a question about a disease mechanism: it might begin with a genomic signal, lead to a molecular structure, require evidence from the literature, and end with a new experimental design. Each step uses different data and methods, but it still needs to answer the same biological question.

Rosalind Workbench helps connect those pieces: you can start with a guided example to understand what is possible, then adapt the approach to your data, methods, and research goal. Each example demonstrates how a biological question can move through specialized tools while the plan, intermediate results, and supporting evidence remain connected.

Inspecting the evidence

The specialized viewers let you examine scientific data in the same conversation as the analysis. You can examine the details, ask follow-up questions, and keep what you see connected to the analysis. Molecular Structure Viewer Biological Sequence & Alignment Viewer Slide Viewer

Getting started with Rosalind Workbench

To try these workflows, open Rosalind Workbench in the ChatGPT app and follow the guided onboarding.

1. Start with a biological question

The starter screen offers research tasks across protein design, small-molecule design, safety and developability, structure and sequence, genomics and pathology, and experimental validation. You can select a task or explore the available scientific tools.

2. Connect specialized scientific tools

The next step depends on the question. For example, a researcher designing PD-L1 nanobodies could rank the leading candidates, then use the experimental-validation task to plan and price binding assays for five nanobodies. A researcher working with small molecules could dock imatinib into ABL1, inspect the five leading poses, and examine the resulting interactions in the molecular structure tools.

Similarly, the dexamethasone-treated airway RNA-seq task gives a genomics researcher a starting point in the NGS Analysis Workbench, with the biological question attached to the analysis.

These tasks illustrate how design, analysis, inspection, and validation can share the same research objective. The guided workflows reduce the setup needed to move between tools, leaving more time to consider the biological question and review the evidence.

3. Run guided research workflows for your genomic studies

Sequencing analysis often requires many connected decisions and datasets before a biological result can be interpreted. Files must be matched with metadata, quality must be assessed, the correct biological replicate must be identified, and an appropriate statistical design must be selected.

Rosalind NGS Workbench helps connect those steps in one reviewable process.

Exploring Rosalind Workbench

However you begin, the goal is the same: move from a research question to evidence you can inspect, with a clear record of how the work was done and what you can explore next.

Advanced life sciences research requires both complex reasoning and safeguards appropriate to sensitive biological information. The ChatGPT agent in Rosalind Workbench supports two modes:

Explore mode: Ask general scientific questions and explore ideas using the ChatGPT models available to you.

Research mode: Work on more complex biological questions, in-depth analysis, and advanced research workflows. Verified organization members can request access on behalf of their organization. Individual access is coming soon.

Bringing every part of your research together

Research often slows down in the gaps between steps. Data lives in one place, analysis happens somewhere else, and the methods behind a result can be difficult to track. Researchers spend too much time waiting for handoffs, rebuilding context, and figuring out what happened before they can decide what to do next.

Keeping the question, tools, analysis, and evidence in one place makes those transitions easier to follow. With Rosalind Workbench, you can review the important choices, follow how a result was produced, and move from one step to the next without starting over or losing the thread. That means less time coordinating the work and more time deciding what the evidence says and what to test next.

Introducing Hy4 Preview. New open weight text input (no vision) LLM from Chinese company Tencent today: 770B total parameters, 49B active parameters, 1M token context window, 1.56TB on Hugging Face.

This is a big size increase from their previous Hy3 in July, which was 295B, 21B active, 256,000 context, 598GB.

I recently started using model chat templates to better understand their capabilities. Here's Hy4's chat_template.jinja on Hugging Face, which includes this section:

{%- if not reasoning_effort is defined %}
    {%- set reasoning_effort = 'high' %}
{%- elif reasoning_effort not in ['high', 'no_think'] %}
    {%- if reasoning_effort is none %}
        {{- raise_exception('reasoning_effort error : None, should be no_think/high') }}
    {%- else %}
        {{- raise_exception('reasoning_effort error : ' + reasoning_effort + ', should be no_think/high') }}
    {%- endif %}
{%- endif %}

So it looks like there are just two reasoning effort levels: "high" (the default) and "no_think" (reason by disabled).

I tried my "Generate an SVG of a pelican riding a bicycle" prompt with the default high reasoning via OpenRouter and got this:

[...] Let's maybe add a helmet? It could improve riding theme, but may obscure head. Maybe a small cycling cap or helmet? The user didn't ask; can add red helmet? Might be cute. But pelican with big beak; a helmet might obscure. Better maybe no.

Maybe add sunglasses? no.

Maybe add water? no.

It's interesting how the reasoning trace uses slightly truncated English, presumably because perfect grammar isn't useful or token efficient for hidden reasoning text.

Consensus estimate is that ~15GW of AI compute produced in 2027 cannot be turned on in 2027. This is harder than just finding power, as you also need to build out all the transformers, wiring, liquid-cooling, (massive) chillers & complex networking.

OpenAI appears to be moving Astra closer to release, with reports pointing to expanded internal testing and a new checkpoint identified as “mozaik-alpha-fdm.” The latest surfaced outputs were reportedly generated zero-shot with Max effort, where Astra spends substantially longer reasoning than GPT-5.6 Sol.

The examples point to a strong focus on coding and visual software creation. Astra reportedly produced a GTA 2-style game in one attempt, alongside detailed websites, 3D objects, and voxel environments. The main difference visible in these samples is attention to small implementation details and the amount of complete functionality produced from a single prompt. Developers, designers, and founders building prototypes could be among the main beneficiaries if this performance carries over to the released model.

Major Scoop: OpenAI has just expanded internal testing of GPT Astra, codenamed "mozaik-alpha-fdm," signifying release might be near. AND of course like previous times, we have the FIRST EVER public outputs of it for y'all. Both outputs are zero-shot on Max effort.

Astra could arrive through ChatGPT, Codex or both, especially given OpenAI's focus on agentic coding and long-running work. OpenAI has not announced a release date or confirmed that “mozaik-alpha-fdm” is an official codename. Separate reports have suggested a launch within weeks, while the newly expanded testing provides another indication that preparations are progressing.

Astra < left > vs Fable 5.1 < Right >. Prompt : generate a svg of ana de armas as detailed as possible

One major variable remains timing: safety approval. OpenAI has officially acknowledged Astra as an upcoming model and said its internal evaluations showed major advances in agentic coding and cybersecurity, potentially reaching its Critical cyber capability threshold. Some Astra workloads were subsequently paused while stricter safeguards were introduced. OpenAI has also confirmed that relevant government agencies and selected AI safety organizations will participate in testing.

[GPT Astra] Pagoda Voxel. ~56k tokens; ~38 minutes

This makes government evaluation part of the actual deployment process rather than simply another rumor. It also means a technically ready Astra may still remain gated until OpenAI is satisfied with its monitoring, alignment, and containment systems.

Astra may be the first time OpenAI have ever underhyped a model. The gap between step changes in LLM capability keeps shrinking. And things are going to look so different 3 months, 6 months, a year from now, nevermind 5. September is going to be a wild ride

Meanwhile, several people with histories of early access to frontier models have hinted that Astra represents a much larger capability jump than recent OpenAI releases. GPT-6 remains a plausible public name, but OpenAI has given no indication that it has chosen this branding.

ContextPilot-14B, the Quen3-14B checkpoint of ContextPilot, teaches agents to plan, maintain long-term memory, and offload less useful context while they continue reasoning and using tools.

Electricity supply in the U.S. is a limiting factor preventing AI data centers from coming online as quickly as possible. Because of this, many projects are turning to portable natural gas turbine generators to get the electricity they need and bypass long wait times to connect to the grid. This has driven demand for turbines, so Elon Musk said on X that SpaceX will bring turbine blade and vane casting in-house.

Musk was one of the first to deploy “portable” power generators to power a data center, allowing him to bring the Colossus data center in Memphis, Tennessee, online in record time. Note that these mobile generators aren’t small devices — they’re massive units about the size of one trailer and may also include another trailer or two of other necessary parts and accessories to run it (not including the tank needed for its fuel). OpenAI followed suit a few months later, announcing it will deploy turbines at the first Stargate data center to deliver additional power.

This sudden influx of demand has led to jet engine shortages for data center operators. Beyond the fact that engine manufacturers weren’t prepared for the AI boom, commercial aviation has already been suffering from its own shortages due to supply chain and engine design issues even before data center developers started buying up turbines.

SpaceX and Tesla are each building 100GW/year of solar production capacity as fast as possible, but natural gas will still be needed to supplement and bootstrap solar for several years. The limiting factor for nat gas turbine production is casting the blades & vanes. By doing… August 29, 2026

Turbine blade and vane manufacturing is one of the biggest bottlenecks in jet engine production right now. The blades used in these engines are difficult to build, with one batch often requiring 60 to 90 weeks to produce from start to finish. That’s because these are specialized parts that are subjected to extreme forces and temperatures, especially in the core of the engine. Even if the engines that these blades will end up on are bolted to the ground or strapped to a trailer instead of hanging under the wing of an airliner, manufacturing them still requires the same level of care and precision.

Because of this, it seems that Musk wants to take things into his own hands by making SpaceX manufacture its own turbine blades. He says this move could cut deliveries by 18 months, allowing xAI to get brand-new turbine generators much faster instead of waiting behind a long queue expected to stretch into 2030.

SpaceX and Tesla aren’t abandoning solar power, but Musk says that “natural gas will still be needed to supplement and bootstrap solar for several years.” Because of this, it seems that he’s willing to invest in parts of turbine manufacturing as demand for turbine generators for data centers increases in the coming years. He has even reportedly quietly bought a portable gas and diesel turbine fleet for around a billion dollars, possibly to take advantage of this boom.

It’s unclear how Elon Musk intends to make turbine blades and vanes, especially given how complicated the process is. Nevertheless, he’s no stranger to tackling “impossible” problems, especially given his track record with Tesla and SpaceX.

GitHub Agentic Workflows

Wake up to ready-to-review repository improvements — automated triage, CI insights, docs updates, and test enhancements. Use event-triggered and scheduled jobs to augment your existing, deterministic CI/CD with AI-powered intelligence.

GitHub Agentic Workflows deliver repository automation: simple markdown workflows running the coding agents you know and love, in GitHub Actions, with strong guardrails and security-first design principles. Supported AI engines include GitHub Copilot, Claude Code, Google Gemini and OpenAI Codex.

Developed by GitHub, GitHub Agentic Workflows run with strong guardrails, using safe outputs and sandboxed execution to help keep your repository safe.

Core capabilities

AI-Powered Decision Making

Workflows that understand context and adapt to situations

Coding Agents You Trust

Built-in support for Copilot, Claude Code, Codex, Gemini, and more

Cost Controls

Per-run AI credit budgets, spend visibility, and OpenTelemetry cost analysis

Guardrails Built-In

Sandboxing, scoped permissions, and safe outputs reduce risk of compromise

GitHub Integration

Deep integration with Actions, Issues, PRs, Discussions, and repository management

Use at Scale in Organizations

Use at scale in you organization or enterprise across multiple repositories

Guardrails Built-In

AI agents can be manipulated by prompt injection or malicious content. GitHub Agentic Workflows uses layered controls: sandboxing, scoped permissions, gated outputs and threat detection.

flowchart LR
    INPUT["Repository + Prompt Input"] --> TOKENS["Read-only Token, No Secrets in Agent"]
    TOKENS --> SANDBOX["Sandbox + Network Controls"]
    SANDBOX --> SAFE["Safe Outputs"]
    SAFE --> DETECT["Threat Detection"]
    DETECT --> APPLY["Scoped Write Job"]

Compile-time validation

Schema validation, expression allowlisting, action pinning, and security scanners reject misconfigurations before deployment.

Credential Isolation

The API proxy isolates the agent from credentials

Sandbox & Firewall

The agent runs in a container with firewall; explicit opt-outs broaden its access.

Integrity Filtering

Reduce prompt-injection risk by filtering untrusted GitHub content

Safe Outputs

Configured write requests are validated before separate jobs apply them

Threat detection

A dedicated job scans proposed outputs, blocking suspicious changes.

Manage Cost and Capacity

Cost control starts with visibility. Use gh aw logs and gh aw audit to find runs consuming the most time, tokens, and AI Credits (AIC), then tighten prompts, triggers, and model choices before spend drifts upward.

max-ai-credits gives each run a hard budget, while OpenTelemetry exports traces and token data to OTLP backends for dashboards, alerting, and cost analysis. For optimization over time, compare cost with outcomes so lower spend still produces useful accepted results.

Cost Management

Track Actions minutes, inference spend, and the heaviest runs before deciding what to optimize

OpenTelemetry

Export workflow traces to OTLP backends for dashboards, alerts, and spend analysis

AI Credits Budgets

Cap runaway runs with max-ai-credits and optimize around AI Credits usage

Example: Daily Issues Report

Here’s a simple workflow that runs daily to create an upbeat status report:

---on:  schedule: daily
permissions:  contents: read  issues: read  pull-requests: read
safe-outputs:  create-issue:    title-prefix: "[team-status] "    labels: [report, daily-status]    close-older-issues: true---
## Daily Issues Report
Create an upbeat daily status report for the team as a GitHub issue.
## What to include
- Recent repository activity (issues, PRs, discussions, releases, code changes)- Progress tracking, goal reminders and highlights- Project status and recommendations- Actionable next steps for maintainers

The gh aw compile command turns this source into a hardened .lock.yml GitHub Actions workflow. GitHub Actions then runs the selected AI engine in the configured agent environment on the declared trigger. The AI agent reads allowed repository context and requests only the tools and outputs configured in frontmatter.

Gallery

Issue Triage

Automatically classify new issues, identify duplicates, apply bounded labels, and ask for missing information.

Pull Request Review

Automatically inspect diffs for concrete defects and post review feedback through controlled safe outputs.

Documentation Maintenance

Automatically detect drift between code and documentation and propose reviewable updates.

CI Failure Investigation

Automatically analyze failed GitHub Actions runs, correlate logs, and open diagnostic issues with likely causes.

Code Improvement

Automatically find unnecessary complexity or duplicated logic and propose focused changes for human review.

Dependency Analysis

Automatically research dependency usage and upstream changes before creating prioritized follow-up work.

Metrics and Analytics

Automatically collect workflow activity and store structured snapshots for health and performance analysis.

Repository Reporting

Automatically summarize repository or release activity on an event or schedule.

Repository Maintenance

Automatically review a backlog, perform bounded maintenance tasks, and propose controlled changes on a schedule.

Security Review

Automatically combine repository evidence with AI interpretation to report suspicious changes through code scanning.

Triage from Side Repo

Automatically triage a main repository from an isolated side repository through a slash-command bridge.

Code Quality Monitoring

Automatically analyze code quality across repositories and create focused, actionable issues.

Feature Synchronization

Automatically synchronize code and configuration across repositories through reviewable pull requests.

Cross-Repository Issue Tracking

Automatically aggregate and synchronize issue status in a central repository.

Dependabot Rollout

Automatically roll out tailored Dependabot configuration across multiple repositories.

AI Engines

GitHub Agentic Workflows provides five stable built-in AI engines. Engine changes may also require a different authentication method or tool configuration; use the linked guide for each engine.

GitHub Copilot

Default engine. Authenticate with organization billing or a personal access token.

Claude Code

Claude by Anthropic, automated with strong guardrails on GitHub Actions.

OpenAI Codex

Codex CLI by OpenAI, automated with strong guardrails on GitHub Actions.

Google Gemini

Google Gemini CLI, automated with strong guardrails on GitHub Actions.

Pi

Multi-provider engine, automated with strong guardrails on GitHub Actions, using provider-scoped models and endpoints.

Custom

Import a third-party agent definition or define your own engine in the repository.

Custom engines

The engine import model can integrate other coding-agent CLIs through Markdown definitions. The following in-repository definitions are unsupported samples, not built-in or officially supported gh-aw engines.

Copilot SDK

A mode of the Copilot engine rather than a separate AI engine. Enable with copilot-sdk: true.

OpenCode

Provider-agnostic BYOK agent supporting 75+ models from Anthropic, OpenAI, Google, and more.

Cursor

Cursor’s AI coding agent, importable as a shared engine definition.

Kiro

Amazon’s Kiro agentic IDE, importable as a shared engine definition.

Aider

Open-source pair programming agent. Import the publisher-maintained definition.

Crush

Charmbracelet’s terminal-first coding agent, importable as a shared engine definition.

Getting Started

Install the extension, add a sample workflow, and trigger your first run - all from the command line in minutes.

Creating Workflows

Create custom agentic workflows directly from the GitHub web interface using natural language.

Workshop

Interactive workshop

Choose a terminal, browser, or Copilot path and work through the workshop directly in the docs with saved progress.

The Standup Your Agents Can't Attend

Meetings are invisible to the workforce you're about to hire

Some of the best engineers on my team have never been to a meeting.

They're software. An executor that builds, a reviewer whose standing instruction is to disagree, a promoter that lands proven work, sweepers that keep the queues honest, watchers that read the machines and name what's wrong. This weekend, while I was writing, one of the watchers found a service that had been crash-looping for three days, invisible to the usual alerts, because between crashes it reported itself as running. The watcher diagnosed the cause and fixed it. It wrote the service's config down where the next reader will find it, and posted a one-line update citing the exact record it had read. Later the same evening, a different agent posted a correction to one of its own earlier claims, quoting the records that proved it wrong. No meeting occurred. Nobody was interrupted. I caught up this morning the way the agents do: by reading.

None of that required clever AI. It required storage. The agent could find the loop because every machine publishes its state to a stream. It could fix the service because the fix was a file. It could correct itself because the claim it had made was on the record, next to its evidence. Everything my agents do rests on one condition: the truth of the company has to sit somewhere they can read. They read files, logs, streams, and state. They cannot attend the standup.

That last sentence is the one to sit with, because you're about to hire workers like mine. Every fact that exists only in a meeting is invisible to them. The decision made in Tuesday's sync, the context shared in the corridor, the estimate that comes from asking the person who knows: to an agent, none of it exists. A company that coordinates by meeting can bolt agents on and will get little. Its truth isn't stored anywhere an agent can reach. Judgment meetings stay; so do one-to-ones, because care doesn't serialise. Everything else on the calendar became a wall between you and the cheapest engineering capacity in history. This evening, while I edited this piece, one of my agents leased a task from a queue, ran it, and posted the result with a citation. The bill was 18 cents.

What I Actually Replaced

My weekly report to the exec team went out this weekend, and an agent published it. The report is a render of two committed state files, and a linter refuses the render when the words and the files disagree. It replaced a meeting, and nobody has asked for the meeting back.

Code lands the same way. Work arrives at a validator carrying a citation: which reviewer approved it, at which offset in the review record, for which exact commit. The validator re-reads the cited record before merging, and a citation that doesn't hold buys nothing. A wrong commit is refused with the reason published. There is no merge button, because there's nothing to press. What this replaced was people pinging me to press one.

This month we took the first of our repositories off GitHub entirely, and the rest of the estate will follow until we're off it completely. The trigger was remembering what git is. Git is decentralised by design: every clone already holds the full history. GitHub adds a centre back, and for years that centre earned its keep, because humans needed a meeting place. The pull request is a meeting: a page somebody must visit, a button somebody must press, an approval that only lands if the right person is looking. Our agents run across a mesh of machines, and every node is already a full peer, so the centre stopped being a meeting place and became a dependency. Once a reviewer's verdict is a record with an offset, re-read by the validator before the trunk moves, the page has no job left. The trunk now sits in a plain git repository on machines we run, append-only, with a copy on a second machine. The history is the artefact; the rest was ceremony around it.

How They Talk

My agents never speak. They publish. An update is one line, posted to a named channel, and the system refuses a message that isn't addressed to one. A claim carries a pointer to the record that proves it, and a pointer that doesn't resolve buys nothing downstream. Presence doesn't matter. The channel is durable, so a reader arriving an hour later, human or machine, catches up the same way: by reading. Before touching anything shared, an agent posts a claim on it, with a note saying why, and releases it when done. The next agent reads the claim instead of asking. A correction goes to the channel that carried the mistake, evidence attached. This weekend's retraction cited the exact records that undid the original claim, a better standard than most incident reviews I've sat through. Status is an hourly digest computed from the work records, never collected from the workers. Even silence is information: one watcher's standing job is naming the streams that have gone quiet.

Notice what's absent. No meeting, no thread that has to be caught live, no message that only lands if the right person is looking. Nothing my agents say to each other depends on anyone being in the room, which is the property every distributed team has been chasing for a decade. The agents get it for free, because it's the only way they can talk at all.

It Is Not Tidy

I'm laying this out because it works, not because it's clean. The same weekend's traffic shows a binary that sat broken for three days, copied instead of rebuilt. It also lists a couple of dozen streams that nothing watches yet. Machines running unattended fail in unattended ways. But look at the shape of the failures. Each was found by reading records, stated with the record that proves it, and closed with the fix written down. The failures left evidence a reader can chase. The meeting version of the same weekend would have left memories.

And the estate is watched the way it's run. One fifteen-minute timer reads the published state of every service across our three machines, 176 of them as I write. It names anything looping, failed, or unreported, and cites the record it judged from. That job used to be somebody noticing, and noticing doesn't happen on schedule.

The Audit Now Has a Deadline

The cost case against standing meetings, to logical people at least, has already been made. A 60-engineer company pays about $1.9 million a year for them, at four hours per engineer per week. Most of it goes on restating facts that existed before the meeting started. Each standing meeting is interest on an artefact nobody built: the board that would be true, the written decision trail, the delivery history, the state file. I used to argue that case on cost alone. The agents change its urgency. Those missing artefacts are also, line for line, the reading list an agent needs on day one.

So run the audit before your first agent arrives, not after. For each standing meeting, ask what artefact would end it. You don't need my estate to start. Pick the standup. Make the board true for two weeks, until the meeting is somebody reading it aloud, and then stop holding it. That board is also the first thing your first agent will read. Build the artefact and you're repaid twice. The humans get the hours back, and the truth the meeting used to circulate becomes something a machine can act on at three in the morning. Skip the audit, hire the agents anyway, and you'll get what a meeting-run company deserves from workers who can't hear: demos, decoration, and a bill.

The machines don't ask you to change how you talk. They ask where the truth is written. Companies that can answer will feel the agents arrive as extra hands. Companies that can't will be left behind, and from the inside it will feel like nothing at all.

Introducing ReactOS 0.4.16

We are pleased to announce the release of ReactOS 0.4.16! After a year and a half of development, we’re excited to showcase the improvements we’ve made between a new graphical installer; a unified bootcd and livecd image; video, audio, networking, and storage stack improvements; a new installation type; and third-party code syncs.

Graphical Installer and the All-in-One Boot CD

Historically, ReactOS offered two images for download, a livecd which let you test ReactOS in a read-only environment, and a bootcd which let you install ReactOS to your hard disk using a text-based installer. Thanks to the efforts of Hermès Bélusca-Maïto (hbelusca), ReactOS 0.4.16 has a new graphical installer and a combined bootcd and livecd. Now you can test and install ReactOS using the same image.

You can read about Hermès’s work on this in his blog posts:

• Part 1 - September 2023: Partly Wine-syncing setupapi
• Part 2 - October-November 2023: Making partitioning UI work
• Part 3 - December 2023: First tests

Video

During 0.4.15 development, core developer Hervé Poussineau (hpoussin) put in the ground work for multi-monitor support and falling back to a VGA driver when display drivers fail to load. This foundation enabled us to continue pursuing better video driver compatibility in 0.4.16.

For years ReactOS has been plagued by different issues with all major video driver vendors. Nvidia GPUs in particular had been plagued by a slow down issue that many talented contributors and developers investigated. Eventually, Justin Miller (The_DarkFire_) recognized that the kernel was running out of system page table entries (PTEs) when loading third party drivers. This limitation was most apparent with graphics drivers, which allocate more memory than most other drivers. Justin changed the memory layout used by our memory manager to increase the amount of system PTEs. This fixed the hard-to-debug slowdown bug with Nvidia graphics drivers. On AMD video drivers, the OpenGL window would end up blank. This was resolved by rewriting ExtEscape, inspired by a patch from the late core developer James Tabor (jimtabor). These improvements enhanced stability, better handled resource management of the new devices, and fixed many edge case bugs in our win32k.sys driver. We thank our contributors and developers for their time as these fixes needed an incredible amount of research.

Audio

Prior to 0.4.16, ReactOS had incomplete High Definition (HD) audio support. HD audio drivers depend on a bus driver (hdaudbus.sys), including drivers from AMD, IDT, Nvidia, Realtek, and SigmaTel. Our initial implementation was written long ago by Johannes Anderwald (janderwald). This implementation was never finished, and was a frequent source of bugchecks when attempting to install HD audio controller drivers. Core developer Oleg Dubinskiy (oleg-dubinskiy) imported sklhdaudbus, a new HD audio bus driver, to replace our old implementation.

HD audio controllers which are compatible with Windows XP and Windows Server 2003 should now work in ReactOS 0.4.16. The new HD audio bus driver depends on the Kernel Mode Driver Framework (KMDF). Microsoft open sourced KMDF as part of the Windows-Driver-Frameworks repository. Justin imported KMDF for the new HD audio bus driver, and now we can use KMDF to import or develop other drivers.

Oleg also fixed the volume and balance sliders in Sound Properties (mmsys.cpl) and Audio Volume Mixer (sndvol32.exe). Now the volume and balance levels are saved and restored on reboot when using an HD audio codec. In addition, Oleg updated the audio device enumeration code to support more sound cards. On top of that, Oleg improved binary compatibility with the Windows audio stack thanks to some fixes he contributed to our Plug and Play (PnP) stack and SetupAPI.

Storage

Since 2009, ReactOS has been using the UniATA storage driver to add SATA, AHCI, and support for partitions greater than 8GB. This was a huge help to ReactOS then, but today UniATA is responsible for slow boot times and failing to load on many devices, leading to the dreaded INACCESSIBLE_BOOT_DEVICE (0x7B) bugcheck. ReactOS 0.4.16 introduces a new ATA driver developed by contributor Dmitry Borisov (disean). This new ATA driver allows ReactOS to boot in far more environments, including inside Hyper-V Generation 1.

In 2021 we imported and enabled the open-source Microsoft FastFAT driver. Unfortunately, this broke our ability to repair FAT partitions using chkdsk. Core developer Doug Lyons (Doug-Lyons) fixed our FAT chkdsk routines to work with the Microsoft FastFAT driver.

Core developer Mark Jansen (learn-more) added a disk cleanup utility in ReactOS 0.4.16. The disk cleanup utility is compatible with extensions for the Windows disk cleanup utility, allowing third party programs to clean up disk usage as well as the operating system.

Networking

During ReactOS 0.4.15 development, Dmitry introduced a new DC21X4 network adapter driver for better hardware compatibility. This driver is used on devices with DECchip 21x4-based network adapters, and virtualized environments such as Microsoft Virtual PC 2007 and Hyper-V Generation 1. Now ReactOS 0.4.16 can boot and access the Internet on both.

ReactOS 0.4.16 also adds asynchronous connection support. This improves networking performance by allowing applications to execute networking operations without stalling. This also improves application compatibility as many programs assume that these asynchronous connection APIs are always present.

ReactOS Server Core

ReactOS supports Workstation and Server installation types. In a Workstation install, more fancy graphical options are enabled by default compared to a Server install. In addition, user folders on Workstation installs currently live inside the “My Documents” folder, although Windows Vista and newer moved these folders out of the “Documents” folder for both Server and Workstation installs.

Interested in seeing ReactOS being more widely used in server and embedded environments, core developer Carl Bialorucki (cbialorucki) added the Server Core installation type. This install type disables the graphical explorer shell, but otherwise loads the full Win32 subsystem. ReactOS Server Core works similarly to Windows Server Core which was introduced with Windows Server 2008.

Third-Party Code Syncs

ReactOS utilizes several other open-source projects as part of its code base. One of the largest open-source projects we leverage is Wine, a re-implementation of several Windows APIs for Unix-like operating systems. ReactOS uses a fork of Wine that interfaces directly with a Windows-like kernel instead of translating calls to a Unix-like one.

For many years, ReactOS was limited to Wine 2.x and 3.x due to compatibility concerns adopting APIs newer than those available to Windows Server 2003. Towards the end of the 0.4.15 development cycle, we abandoned this strict adherence to Windows Server 2003 compatibility, which allowed us to slowly update our Wine fork to Wine 10.0. This upgrade is still on going, but 0.4.16 has a significant amount of this work in it. We anticipate that updating to Wine 11.0 or later versions will be significantly easier thanks to this effort to bring it up to Wine 10.0.

At this time, the ReactOS release image is still compiled with Windows Server 2003 exports only since there are several programs that expect all Windows Vista and newer exports available even if only some are exposed. If you’d like to experiment with Windows Vista and newer application support, build ReactOS using the -DDLL_EXPORT_VERSION flag.

For the first time, ReactOS release images will include WineVDM, which increases compatibility with 16-bit Windows applications. WineVDM is a project by otya128.

Final Thoughts

The mission for ReactOS is to “[Run] your favorite Windows applications and drivers in an open-source environment you can trust.” With each release we come closer to fulfilling this goal. We look forward to sharing more developments and progress with you.

We extend our deepest gratitude to our community, contributors and donors. Without our contributors, we wouldn’t be able to make any development progress. Without our donors, we couldn’t fund our testing and hosting infrastructure or development contracts to accelerate progress. And without our community, no one would know we exist. Thank you for making ReactOS possible!

Statistics

Resolved Jira issues: 381

Commits: 2808

Oldest Jira issue resolved: CORE-3804 from February 3rd, 2009

• 0.4.16 Resolved Issues
• 0.4.16 Commit History

At Datadog, we want to expose our engineers to high-quality AI tools and workflows. However, token usage can be expensive, and finding a balance between AI cloud spend and the return on investment can be difficult. But what if engineers could maintain their current AI workflows using the same tools, but at a lower cost? Similar to rightsizing cloud infrastructure, tuning the configurations of AI tools—such as model type and effort level—can yield cost savings without hurting performance objectives.

In this blog post, we’ll discuss a handful of easy configuration changes surfaced by our platform team that yield us over $1 million in monthly AI spend.

How Datadog surfaces AI cost savings opportunities

In order for us to identify opportunities for AI cost savings, we needed to first understand the complete composition of our existing AI cloud spend. This meant tracking costs across AI vendors (including Anthropic, Cursor, and OpenAI), their different model offerings, and how API usage mapped back to individual usage and workflows.

We track this data using the AI Costs feature in Datadog Cloud Cost Management. AI cost data supports normalized tags for provider, model name, and token category, helping us identify inputs, outputs, or tokens related to caching and search operations. Every Claude Code request is run through an AI gateway, where the requests are tagged with their team or product of origin. Doing this gives us a more granular view into where our AI costs originate from; for example, we can see if this spend comes from developers working in Claude Code, assistant APIs, Datadog’s agent skills, or other sources of activity. All of our AI cost data is compiled into a dashboard that our AI Developer Experience team uses to identify areas of high spend and cost trends that can translate into savings opportunities.

Why we migrated our default model from Claude Opus to Sonnet

Once we began tracking live and historical AI cost data, we noticed that Claude Opus models made up a large portion of our AI spend. Claude Sonnet models at face-value pricing offered a 60% cost reduction for both input and output tokens compared to Opus. Prior to this change, the default model for Claude Code and our agent skills was Opus 4.8. Migrating to Sonnet would offer significant cost savings, but we first needed to measure how this change would impact the performance and output of our agent skills.

We didn’t need to measure the engineering capability of different agents and models on standard engineering tasks. Rather, we needed to measure an agent’s ability to work on Datadog engineering tasks. This meant working with our frameworks, conventions, libraries, and golden paths. To do this, our AI developer experience team built an agentic evaluation platform that provided teams with a self-service workflow to write, test, and add agent evaluations. Developers can then use these evaluations to inform agent selection when building their own features, or during org-wide wide changes such as migrating the default model for Datadog’s agent skills.

After running different models across over 140 different evaluations, we calculate their average cost and performance score on a nightly basis. Performance is judged on weighted scores between deterministic results and LLM evaluations across categories. While Opus 4.8 ranked the highest in performance score, it also carried the highest cost per evaluation. Accounting for Sonnet 4.6’s benchmarks, the migration represented an 8% loss of proficiency in executing Datadog workflows while reducing AI costs by 36.7%. Ultimately, we decided that this tradeoff was worth it and proceeded with the migration. When monitoring the cost impact of this migration over the past month, over $687,000 in cost savings was attributed to this change.

More importantly, creating an agent evaluation suite enabled us to repeat this process of informed decision-making each time we identified an AI cost savings opportunity. Using a similar methodology, we also changed the default effort level for Claude Code CLI from high to medium, which yielded us over $288,000 in monthly savings.

Creating automated AI cost alerts and spending guardrails with Datadog

Another cost saving strategy we’ve implemented is to create automated cost alerts that notify users of expensive AI usage patterns and direct them to internal documentation that outlines different methods for reducing spend. We accomplish this using a combination of Datadog Cloud Cost monitors and notification workflows orchestrated by Datadog Workflow Automation.

Once our different cost alert monitors are configured, we can create spending guardrail workflows that trigger when our monitors enter an alert state. For example, in the workflow below, when a user’s Cursor spend crosses a daily threshold, the workflow fetches the user’s email, identifies the corresponding user profile in Datadog’s Slack workspace, and sends them a message that directs them to different cost savings strategies.

To evaluate the impact of our AI cost alerts and savings workflows, we created a case study using cost alerts configured on Anthropic provisioned keys. This alert was rolled out to a new population of users who had not previously triggered any existing cost alert. During the first week of rollout, 768 distinct users triggered this alert and were sent internal cost savings strategies via automated workflows (such as the one shown above). When comparing the week before and after users received the alert, we observed more than $150,000 in reduced AI spend.

Reducing token costs with context optimization

When you start a Claude Code session and ask the agent to perform a task, there are several factors that contribute to the end-to-end cost of your request. At a high level, these costs include:

• The input tokens that make up the initial context
• Any additional tokens consumed as the agent reads files, invokes skills or tools, and gathers context to complete the task
• The output tokens generated to return the final response

Context optimization refers to reducing token costs throughout an agent request by restricting the context processed by the LLM only to what is necessary, while still maintaining its quality of response. For example, if you ask Claude Code to explain how the deployment workflow for your service works, it may retrieve thousands of lines of source code, internal documentation, and other context from your repository, even though only a subset of that information is ultimately needed to generate the final response. Open source tools such as Headroom help address this challenge by optimizing the outputs of agent tools before they are passed back to the LLM. Depending on the type of data, this can involve filtering irrelevant search results, deduplicating repetitive content such as duplicate log entries, or compressing structured data by eliminating repeated semantic structure (for example, fields that appear in every object within a JSON response).

Headroom documents its own benchmarks for performance, accuracy, and cost savings, but we wanted to measure how it performed against our own evaluations before shipping it to engineers. In our evals, we observed a cost reduction of 47% without significant drops in performance. This gave us confidence to start A/B tests with a small pilot group of Datadog engineers. The following results show a one-week snapshot taken from a sample size of over 1,000 engineers.

As you can see in the test results above, the Headroom testing group consumed less than half the input and output tokens per user that the baseline group did. Most significantly, the cost per user for the testing group was 27% lower than the baseline group. Based on the success of this pilot program, we plan to roll out Headroom for increasingly larger groups of engineers and monitor their metrics while also exploring other cost optimization options.

Start tracking your AI spend with Datadog

Tracking our engineering costs and the impact of the changes we make to reduce spend is a continuous process. The agent evaluations we discussed in this post are not just used leading up to the decision to implement an engineering change. We run these benchmarks on a daily basis so that we are constantly aware of the performance and cost tradeoffs that come with each agent.

You can begin monitoring your organization’s AI spend and surfacing cost savings opportunities as we discussed in this post by using Datadog Cloud Cost Management’s AI costs.

vphone-cli

Boot a virtual iPhone via Apple's Virtualization.framework using PCC research VM infrastructure.

Prerequisites

Host:

• Apple Silicon
• macOS 15+ (Sequoia)
• Xcode + iOS SDK (cross-compiles the guest daemon)
• SIP/AMFI relaxation to allow private PV=3 entitlements with unsigned-binary

Dependencies:

brew install python@3.13 aria2 wget gnu-tar openssl@3 ldid-procursus sshpass keystone cmake libusb ipsw zstd

Install

brew install zqxwce/tap/vphone-cli

Build

git clone --recurse-submodules https://github.com/Lakr233/vphone-cli.git

./scripts/setup_tools.sh      # install deps, build toolchain submodules, create the Python venv
./scripts/build.sh            # build + sign vphone-cli, bundle the .app, cross-compile vphoned

cd .build/vphone-cli.app/Contents/MacOS/
vphone-cli --help

Quick Start

One command creates a VM end-to-end (download → patch → DFU restore → CFW install → first boot):

vphone-cli vm create myphone -V jb        # -V / --variant

vphone-cli vm launch myphone

Commands

vphone-cli vm create runs the whole pipeline; the individual steps below let you drive it manually or re-run one stage.

Manage

vphone-cli vm list                         # list VMs (--json for scripting)
vphone-cli vm info myphone                  # show one VM
vphone-cli vm new myphone                   # create an empty bundle (cpu/mem/disk options)
vphone-cli vm config myphone --cpu 8 --memory 8192
vphone-cli vm clone myphone myphone-2       # fast APFS clone, fresh device identity
vphone-cli vm export myphone --out myphone.tzst   # zstd fast by default (--max = xz -9); --out may be a dir (auto-names <vm>.tzst/.txz); skips restore dir + staging files
vphone-cli vm import myphone.tzst --name restored
vphone-cli vm rename myphone iphone16
vphone-cli vm delete iphone16

Build a VM manually (what vm create automates)

vphone-cli vm new myphone                              # 1. empty bundle
vphone-cli fw prepare myphone --iphone-version 26.1     # 2. download + merge IPSWs
vphone-cli fw patch myphone --variant jb                # 3. patch the boot chain

vphone-cli vm launch myphone --dfu &                    # 4. boot into DFU (background)
vphone-cli restore myphone --get-shsh                   #    fetch SHSH
vphone-cli restore myphone                              #    DFU restore
vphone-cli vm stop myphone                              #    stop the DFU boot

vphone-cli cfw install myphone --variant jb             # 5. install CFW (host-mount; asks for sudo)
vphone-cli vm launch myphone                            # 6. first boot

Update to a newer iOS by pointing fw prepare at an IPSW: --iphone-source /path/to.ipsw --cloudos-source /path/to.ipsw.

Firmware Variants

Five patch variants with increasing security bypass — pass one to --variant:

Running & Connecting

• SSH (jailbreak): ssh -p 22222 mobile@<vm-ip> (password alpine)
• SSH (regular/dev): ssh -p 22222 root@<vm-ip>
• VNC: vnc://<vm-ip>:5901

Locations

Everything vphone-cli creates lives under ~/.vphone/ — kept outside the repo and the .app so the signed bundle stays portable. Redirect the whole tree with $VPHONE_ROOT:

SIP/AMFI Relaxation

Option A — fully disable SIP, then disable AMFI via boot-arg (most permissive).

In Recovery (long-press power → Terminal):

csrutil disable
csrutil allow-research-guests enable

Then reboot into macOS and set the AMFI boot-arg (needs SIP fully off to take effect):

sudo nvram boot-args="amfi_get_out_of_my_way=1 -v"   # reboot after

Option B — keep SIP on (debug-only relaxed), then allowlist the binary with amfidont (leaves AMFI enabled system-wide).

In Recovery:

csrutil enable --without debug
csrutil allow-research-guests enable

Then reboot into macOS and:

vphone-amfidont         # .build/vphone-cli.app/Contents/Resources/vphone-amfidont for local builds

FAQ

zsh: killed ./vphone-cli — AMFI/debug restrictions aren't bypassed; see Prerequisites (amfi_get_out_of_my_way=1 or amfidont).

Virtualization is not available on this hardware — your Mac is itself a VM; PV=3 guest boot can't nest. Use a non-nested macOS 15+ host.

Stuck on "Press home to continue" — connect via VNC and right-click (two-finger click) to simulate the home button.

System apps won't install — during iOS setup, don't pick Japan or the EU as your region (extra regulatory checks the VM can't satisfy); pick e.g. United States.

App crashes on launch with EXC_GUARD / GUARD_TYPE_MACH_PORT — re-patch with vphone-cli fw patch <name> --variant <v> --force-exc-guard, then re-restore/install. Always on for iOS 18 bases.

Install a .ipa/.tipa — use the running VM's Install menu (drag-drop or file picker).

cfw install hangs re-signing a system binary (e.g. Campo), memory climbing unbounded — known bug in ldid-procursus up to 2.1.5-procursus7 (the current Homebrew stable). Fixed upstream but not yet in a tagged release; rebuild from source: brew install --HEAD ldid-procursus && brew link --overwrite ldid-procursus. Kill the hung ldid process first (sudo kill -9 <pid>) if you already hit it.

Automation

vphone-cli exposes a host control socket (<bundle>/vphone.sock) for programmatic control — screenshots, touch, swipes, hardware keys, clipboard — each action returning an inline screenshot for AI-driven E2E testing.

Creepy crawlies

You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact. In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

Why is git.kernel.org “interesting” to crawlers

Linux development happens in the open — from git repositories you can clone, to discussion archives you can follow in real time. To a large language model, this is a goldmine of learning data, because all of this is not only immediately available, but is easy to filter in order to guarantee pure unadulterated pre-AI content. Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data.

The stupidest way of doing it

We make almost everything clonable, because hey — we may not be around forever, so here — clone the repos. Also, clone the archives. Grab a copy just so we're not the only ones who have it all. Seriously, it's just a “git clone” away — and then you'll have the whole history.

For example, did you know you can clone the entirety of LKML and then do whatever you want with it? It's just git repos all the way down.

So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.

But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.

At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork.

Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing.

But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.

Block them

Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers.

So, we started banning them by IP — after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen.

The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user. Banning the whole ASN was justified, even if this occasionally caught a random legitimate instance trying to automate link checking in commits.

Enter... your TV?

And... that's when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.

They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat.

They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it.

Make them pay

When this first became a problem, oh, about a year ago, we naively thought that there was a way to make it stop. Just make the bots perform a task that would flip the economy of the whole thing upside-down by making them burn some cycles doing throwaway math. Like, calculate what string, when combined with their own IP and a secret we provide, would generate a sha256 sum with 4 leading zeroes.

In other words, we put Anubis in front of everything.

It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.

A few months later, the bots were back, solving difficulty 4. No problem, we said, let's raise difficulty to 5.

The legitimate users were more annoyed now. Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it's doing the number crunching. However, it was effective and bought us a few more months of peace.

Then... the bots started solving difficulty 5.

Where we are now

Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

It's impossible to tell with certainty which of these are bots and which are real humans — but chances are, if it's asking for an old commit in a random old fork, it's probably not a real developer trying to do their work.

With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.

How bad is it?

At this point, we're not quite overwhelmed — if you visit git.kernel.org, it will likely be snappy and responsive. The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)

However, you should know that out of the total of the 90 cores across 5 geo-distributed nodes, there are 14-16 cores that are constantly doing nothing but rendering commits for scrapers. On average, that's 20% of our entire capacity — except the swarms descend in waves and the actual graph is a lot more spiky than a 20% flatline.

Where does that leave us?

Unclear. Maybe the AI bubble bursts and we suddenly have a lot fewer entities out there trying to train their models. Alternatively, maybe they smarten up and stop consuming our data in the dumbest way possible.

In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity.

Worst of all, there are no simple solutions to the problem. Companies offering custom “AI” models still pop up daily, all of them hungry for training data. App makers are still looking for ways to turn a profit, so they will continue to turn your household appliances into attack vectors.

That said, we promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it.

Sorry. (Oblig. Canadian thing to say.)

Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale

Key Takeaways

• Kafka Offset lag tells you how far behind a consumer is rather than telling you how old the data is. For Apache Hudi pipelines, these measurements are two completely different things. Confusing them leads to data freshness SLA violations.
• The time-in-queue metric is computed by reading the Kafka checkpoint from the latest Hudi commit file in S3, seeking to that offset in the Kafka topic, and measuring the timestamp delta between that message and the current time. Changes to producers, consumers, or existing pipeline infrastructure are not required.
• The algorithm must handle the case where the latest Hudi commit contains no checkpoint metadata. For example, when a parallel legacy pipeline made the most recent commit, requiring the algorithm to walk back through commit history to find the most recent commit that contains checkpoint metadata.
• Once deployed, time-based lag becomes a first-class data contract metric. Pipeline owners can define custom freshness SLAs per pipeline and receive alerts when the lake data ages beyond their threshold.
• Offset monitoring and time-lag monitoring are complementary. Running both gives you a complete picture of pipeline health that neither metric provides on its own.

The Problem

At Twilio, the data lake is the foundation for analytics, reporting, and machine learning across the company’s product lines: messaging, email, voice, and more. Internal teams rely on this data to understand product usage, drive business decisions, and power machine learning models such as anomaly or fraud detection. The pipelines that feed this lake use Apache Hudi Delta Streamer to land data from Kafka, processing over five trillion records monthly as of Q4 2025, across self-hosted Kafka clusters, peaking at 12.9 million messages per second on Cyber Monday 2025. At that scale, we realized it was critical to give pipeline owners a precise way to define and enforce custom freshness SLAs. We were looking for a signal that was actionable without adding any overhead to live pipelines.

Traditional consumer lag metrics like consumer offset lag (records-lag-max) and even Hudi’s kafkaDelayCount looked fine. It seemed like consumers were keeping up with Kafka, but downstream analytics teams kept reporting stale data that was sometimes hours old. The issue wasn't Kafka throughput; it was a visibility gap. Hudi Delta Streamer manages its own checkpoints, which are stored alongside the table data in S3 and separate from Kafka's consumer group offset tracking. Standard lag monitoring tools like Burrow track a consumer group’s committed offsets, which Hudi doesn’t populate by default, so they had no awareness of whether Hudi had actually committed that data to the lake.

The real question we needed to answer was how far behind is the latest Hudi commit from the messages currently sitting in the Kafka topic?

Rethinking Lag as Time

How far behind is the Hudi job from messages in the Kafka topic? In other words, we wanted to compute and report how long it has been since the first unconsumed message arrived in the Kafka topic after a successful Hudi commit.

How Offset Tracking Works in Hudi

HoodieStreamer (formerly HoodieDeltaStreamer) utilizes a checkpoint mechanism to track exactly what data has been ingested and prevent reprocessing of that same data. For Kafka sources, this checkpoint represents the exact topic offsets (per partition) or timestamps that have been successfully processed and committed to storage.

• For checkpoint storage, the checkpoint is embedded directly in the .hoodie commit files as deltastreamer.checkpoint.key.
• In terms of resilience, upon failure or restart, HoodieStreamer reads the latest commit file, retrieves this key, and resumes reading from the exact offset it left off.
• Kafka offset is stored as a string: topicName,0:offset0,1:offset1.

From this timeline of events, we can find the latest commit using the Apache Hudi SDK and extract the per-partition offsets that were last successfully written to the lake.

What makes this approach practical is that it requires nothing new from the pipelines themselves. The offset that Hudi already commits to S3 and the timestamps already on Kafka messages are enough to compute the true data freshness. We compute it with an external job called the metrics reporter. The metrics reporter is purely an external observer that reads artifacts the system already produces, requiring neither new instrumentation nor producer changes.

How the Algorithm Works

The metrics reporter runs every fifteen minutes in production. For each pipeline, the reporter completes the following tasks:

• Fetch the latest Hudi commit from the active timeline in S3. Walk through commits in reverse chronological order, the most recent first, to find the latest commit that contains a deltastreamer.checkpoint.key. This approach gives us the per-partition offsets of the last successfully committed batch. This is the exact offset Hudi has already consumed and committed to the lake.
• Seek to the checkpoint offset in each Kafka partition. This seeking positions the consumer at the first message Hudi has not yet committed to the lake. That message is still in Kafka, having arrived after the last successful Hudi write.
• Read that message and get its timestamp X. This timestamp records when the data arrived in Kafka. It has been sitting there, waiting to be picked up by the next Hudi run.
• Compute lag: currentTimestamp - X = how long that data has been waiting
• Cap at seven days if lag exceeds the threshold. This cap prevents unbounded values for inactive or stopped pipelines. If no valid checkpoint is found within the search depth, the reporter suppresses the metric entirely rather than publishing a misleading value.

Edge Cases and Considerations

Clock Skew Across Producers

Because the lag metric uses the message timestamp set by the producer, producers whose system clocks drift ahead can make lag appear artificially low (or even negative). The algorithm floors the lag at zero using Math.max(0L, currentTimeMs - record.timestamp()).

Multi-Partition Topics

At scale, Kafka topics have many partitions. The checkpoint stored in Hudi contains one offset per partition. The algorithm seeks each partition to its respective checkpoint offset and polls for the next available record. This approach provides one candidate message per partition. We then take the record with the earliest timestamp across all partitions, not the average nor the latest.

Multiple Hudi Table Writers and Kafka Checkpoints

In production, a Hudi table can have multiple writers and not all of them commit Kafka checkpoint metadata. This situation commonly arises during migrations. The root cause was that the original algorithm only looked at the most recent commit. We changed the algorithm to walk back through the timeline in reverse chronological order up to MAX_COMMIT_DEPTH commits until it finds the most recent commit that actually contains a deltastreamer.checkpoint.key.

The Epoch Timestamp Trap

When Hudi fails to locate a commit with checkpoint metadata, it doesn't error out. Instead, it silently defaults to a timestamp of epoch zero, 19700101000000000. We had to explicitly guard against epoch defaults and pivot our logic. By switching to currentTime - timestamp_of_first_unconsumed_message, we finally started answering the right question.

SLA as a Ratio, Not a Binary

An "SLA ratio" is a design choice we arrived at in production. Our initial SLA metric was binary, breached or not. The problem was that it only fired after the SLA was already missed. Switching to a 0.0-1.0 ratio gives teams a leading signal, allowing them to set a warning at 0.7 and a critical alert at 1.0.

What's Next

If you want to apply this pattern, you need three things:

• Access to your Hudi table's .hoodie/ directory on S3
• A Kafka client to seek arbitrary offsets
• A scheduler to run the reporter periodically.

Log-first or Table-first? Apache Kafka, Fluss, and Streaming Tables

Apache Kafka is a natural starting point for a streaming architecture. Applications produce events to Kafka, other applications consume them, and data that needs to be retained for longer eventually lands in object storage. A fairly typical version uses Kafka Connect with an S3 sink connector—for example, Confluent's Amazon S3 Sink Connector—to move older data into Amazon S3.

Kafka handles the real-time path, S3 provides cheap and durable storage for historical data, and for an archive this is often enough. The problems start when the data in S3 is expected to behave less like an archive and more like a database table.

Suppose a Kafka topic contains customer updates:

customer-123 -> { name: "Alice", status: "active" }
customer-456 -> { name: "Bob",   status: "active" }
customer-123 -> { name: "Alice", status: "blocked" }

Writing those records to Parquet is easy. Answering a different set of questions is harder:

• What is the current state of every customer?
• How should updates and deletes be represented?
• Which files form a consistent version of the dataset?
• What happens when the schema changes?
• How do we avoid producing thousands of small files?
• How can multiple query engines safely read the same dataset while it is being updated?

S3 gives us files. It does not give us a table.

From files to tables

Open table formats such as Apache Iceberg and Apache Paimon fill this gap. They put metadata around the files: schema, snapshots, partitions, data files, and changes between versions. The architecture becomes something closer to:

The difference is more than file layout. We are no longer simply writing files—we are maintaining a table. Queries can operate on a consistent snapshot, schema changes can be tracked, and data files can be compacted or replaced without exposing partially updated state to readers.

But somebody still needs to translate the Kafka stream into that table. In practice this means running another piece of infrastructure: Kafka Connect, Apache Flink, Spark Structured Streaming, AWS Glue, or a custom ingestion service. And that service has considerably more work to do than copying bytes from Kafka to S3. It might need to understand schemas, turn CDC records into inserts, updates, and deletes, create table commits, manage file sizes, compact small files, and publish metadata to a catalog.

There are two interesting ways to simplify this pipeline. Confluent's Tableflow makes Kafka-to-lakehouse materialization a managed Confluent Cloud capability; we will come back to it later. Apache Fluss takes a different route and questions whether the streaming layer should be a log in the first place. Before getting to Fluss, it is worth being precise about what Kafka gives us.

Kafka is still a log

A Kafka partition is an ordered log. Records have offsets, producers append records, and consumers move through the log and can rewind to replay old data.

This model is deliberately simple. Kafka does not need to know whether a record represents a customer, an invoice, a database row, or an HTTP request—the value stored in the record is, from the broker's point of view, mostly an opaque sequence of bytes. That is one of Kafka's strengths, and it is what makes Kafka useful as a general-purpose event backbone. But it also means that Kafka's storage abstraction is still a log.

Consider a compacted topic:

offset 100: customer-123 -> active
offset 101: customer-456 -> active
offset 102: customer-123 -> blocked

Eventually, log compaction can remove the older value for customer-123 while retaining the newer one. That makes a compacted topic look a bit like a key-value store, but only from a distance. Kafka contains enough information to reconstruct the latest state; it does not expose that state as a table. There is no broker operation equivalent to:

SELECT *
FROM customers
WHERE customer_id = 'customer-123';

A consumer can scan the log and build such a representation, but that representation lives somewhere else. Kafka Streams builds local state stores - and ksqlDB, built on top of Kafka Streams, exposes exactly this pattern as SQL tables continuously materialized from topics. Flink maintains operator state, commonly backed by RocksDB. Applications also materialize Kafka data into Redis, Cassandra, PostgreSQL, Elasticsearch, or another database.

In Kafka-based projects I keep encountering some version of the same picture: a customer or product dataset maintained as Flink state for enrichment joins, copied into Redis so that services can look it up with low latency, and written to the lakehouse for analytics. Each copy exists for a defensible reason. The sum is still three or four stateful systems holding the same logical table.

Kafka stores the changelog, a state store serves the latest value, and the lakehouse stores historical data for analytical queries. There are good reasons for separating these responsibilities, but it also means that a seemingly simple stream of updates ends up copied into several stateful systems—each with its own consistency, capacity, and operational story.

Fluss takes aim at exactly this duplication.

Fluss starts with a table

A few words about the project first, because it is young. Fluss originated at Alibaba, entered the Apache Incubator in 2025, and graduated to an Apache Top-Level Project in August 2026. The latest stable release at the time of writing is 0.9.1; the artifact still carries the Incubating label because that release predates graduation. Flink remains the most complete compute integration, but Fluss now also ships Java, Rust, Python, and C++ clients—the non-Java ones share a common Rust core and had their first 0.1.0 release in 2026, so feature coverage still varies across languages. This is a young ecosystem, and that matters for any adoption decision.

Fluss supports append-only streams, but that is not the part that makes it particularly different from Kafka. The more interesting abstraction is the Primary Key Table. A Fluss table can be defined with a key:

CREATE TABLE customers (
    customer_id STRING,
    name STRING,
    status STRING,
    PRIMARY KEY (customer_id) NOT ENFORCED
);

Now consider the same sequence of updates:

customer-123 -> Alice, active
customer-456 -> Bob, active
customer-123 -> Alice, blocked

The current table state is:

customer-123 -> Alice, blocked
customer-456 -> Bob, active

At the same time, downstream streaming consumers can still observe the changes. That is the key shift. With Kafka, the log is the primary storage abstraction and current state is normally derived from it. With a Fluss Primary Key Table, the changelog and current state are both part of the storage abstraction.

Internally, Fluss does not make the state problem disappear. Primary Key Tables maintain both a log component and a key-value component, with RocksDB used for current state.

So Fluss is not eliminating RocksDB—it is changing who owns it. Instead of every processing application independently maintaining a copy of the same state, the storage layer maintains it once and exposes it to multiple consumers. For CDC datasets, lookup joins, and heavily reused reference data, that can remove a surprising amount of duplicated state.

A compacted log is not a primary-key table

Kafka compaction and a primary-key table can look similar because both deal with multiple values associated with the same key, but they answer different questions. Kafka compaction primarily answers:

Can I keep a replayable log without retaining every historical value for every key forever?

A primary-key table answers:

What is the value associated with this key now?

Those are related requirements, but they lead to different storage designs. Kafka remains optimized for ordered consumption; a primary-key table also needs efficient access to state.

Kafka Streams demonstrates the distinction quite well. A Streams application can consume a compacted topic and maintain a local RocksDB store: Kafka provides the durable changelog, while RocksDB provides efficient access to the materialized state. Fluss moves that materialization into the storage system itself.

Whether that is useful depends on the workload. For application-specific processing state, local state can be exactly what we want—a Kafka Streams application might have state that exists only because of the application's processing logic, and putting it in a shared database would make little sense. But consider a large customer or product dataset consumed by ten different Flink jobs. If every job needs the same latest state for lookup joins, rebuilding and maintaining ten copies of that state is much less obviously desirable, and a shared streaming table starts to become attractive.

The hot layer can also be columnar

Another difference is easy to miss: Kafka intentionally does not understand the structure of a message value. Suppose an event contains fifty columns:

customer_id
name
email
address
country
currency
segment
device
browser
...

A Flink job might need only customer_id and country. With Kafka, the consumer still fetches records containing the complete serialized value, and column selection happens after the record reaches the consumer and is deserialized.

Fluss takes a different approach. Its log representation can use Apache Arrow, which means the storage layer understands columns. A query such as:

SELECT customer_id, country
FROM customers;

can push the projection down into the storage layer, so only the required columns travel over the network. Column pruning is something we normally associate with Parquet files in a data lake, not with a system serving fresh streaming data—here the hot streaming path starts to look a little like an analytical storage engine. And the columnar layout helps beyond transfer: values of the same column sit contiguously in memory, which is exactly the layout aggregation loves—summing a column of numbers over Arrow batches is a very different operation from plucking one field out of every individually deserialized record. Again, this comes from the underlying abstraction.

Kafka sees records. Fluss sees rows.

The lakehouse is still better for history

Making the hot layer queryable does not mean it should store years of data. Object storage remains much cheaper, and analytical systems are very good at scanning large columnar datasets there. Fluss therefore supports tiering older data into lakehouse tables such as Paimon or Iceberg.

Now the same logical table spans two storage tiers: Fluss serves the hot part, while the lakehouse holds the history. The awkward part is querying across the boundary.

Suppose the lakehouse contains everything committed up to 12:00:00, while thirty seconds of newer data still exists only in Fluss. At 12:00:30, querying only the lake gives us a stale result. Trying to make every individual event immediately visible in Iceberg or Paimon is not the answer either—lakehouse formats operate on files and snapshots, and extremely frequent commits create exactly the small-file and metadata problems these systems are meant to control. For update-heavy streams the trade-off gets sharper still: with copy-on-write every commit rewrites data files, so writes pay the price; with merge-on-read writes stay cheap, but readers must merge the accumulating delete files until compaction catches up—so keeping reads fast means compacting the table often. Push the commit frequency high enough and something has to give on one side or the other.

Fluss solves this through a union of the stable historical snapshot and the hot tail.

The complete logical table does not need to be physically stored in one place. Older data can live cheaply in object storage while fresh data remains in low-latency streaming storage.Notice what this does to the Iceberg trade-off above: the union read does not make copy-on-write or merge-on-read cheaper, but it removes the reason to run them at streaming frequency in the first place—the lakehouse can commit and compact at a pace that keeps the table healthy, because freshness is served from Fluss rather than squeezed out of Iceberg.

Fluss is not the only system using this hot/cold split. Confluent Cloud now offers a comparable path on top of Kafka.

What changes on Confluent Cloud: Tableflow

Confluent Tableflow is Confluent's answer to the Kafka-to-lakehouse pipeline. The important caveat is that Tableflow is a Confluent Cloud service. It is not part of Apache Kafka, and it is not available in self-managed Confluent Platform. So everything in this section describes a managed Confluent Cloud architecture, not a new storage primitive inside Kafka itself.

Within that boundary, Tableflow can continuously materialize a Kafka topic into an Iceberg or Delta Lake table.

The pipeline that previously required Connect, Flink, or application-specific code becomes part of the platform. For append-only topics, the mapping is relatively straightforward: records are turned into rows and written to the table. More interestingly, Tableflow also supports upsert-style materialization for compacted topics, so a stream of changes can become a table representing current state.

The hot/cold split ends up looking remarkably similar to the Fluss tiering picture. Kafka contains fresh events, Iceberg contains the materialized historical table, and Confluent's Flink integration can combine a Tableflow snapshot with records that have not yet reached the materialized table.

The semantics and implementation differ from Fluss union reads, but the basic trick is the same: do not force the lakehouse to provide millisecond freshness. Read the stable part from the lake and the tail from the streaming system. The two stacks end up with a similar hot/cold layout even though they start from different abstractions.

Tableflow narrows the Kafka-versus-Fluss comparison considerably. With self-managed Apache Kafka, turning a topic into a maintained lakehouse table usually means operating another pipeline. On Confluent Cloud, Tableflow removes most of that plumbing. What remains is the difference in the hot layer. If applications also need low-latency access to current state, the architecture still needs another branch, where with Fluss, that state can be part of the storage layer.

That leaves a more useful question than "can Fluss move streaming data into Iceberg more efficiently than Kafka?": do we still need a separate materialization of current state when the streaming storage itself can expose it?

Kafka semantics still leak into the table

Tableflow's upsert mode also shows what happens when a table is derived from a log. Kafka's ordering and compaction guarantees live at the partition level, so the partitioning model inevitably becomes part of the semantics of the resulting table.

Suppose a key initially maps to one partition:

partition 3:
customer-123 -> active

Later, after changing the topic's partition count, the same key might map differently:

partition 8:
customer-123 -> blocked

From a business perspective, this is still one customer. Kafka sees two partition histories. Repartitioning is one of those edge cases where the underlying abstraction suddenly becomes visible: a table derived from a log is not quite the same thing as a storage system whose data model is a table.

Where does Paimon fit?

Paimon is slightly awkward in a title such as "Kafka vs Fluss vs Paimon" because it solves a different layer of the problem. Fluss can use Paimon as its lakehouse storage, but it can also use Iceberg. So there are really two separate decisions: the hot layer (Kafka or Fluss?) and the lakehouse representation (Iceberg or Paimon?).

Paimon started its life inside the Flink community as Flink Table Store before becoming a top-level Apache project, and that lineage shows in its design. Under the hood, each bucket of a Paimon table is organized as an LSM tree, which is a structure built for absorbing continuous writes and updates rather than occasional batch appends. Its primary key tables come with configurable merge engines that define what happens when multiple records arrive for the same key: deduplicate keeps the latest row, partial-update lets different writers fill in different columns of the same row, and aggregation folds incoming values into aggregates. Paimon can also produce a proper changelog for downstream streaming readers, with several changelog-producer modes that trade write cost for changelog completeness.

Paimon's design assumes that "a stream of changes to a table" is a normal workload rather than an edge case. That makes it a natural fit for CDC-heavy pipelines and helps explain why it pairs so well with Fluss, which grew out of the same Flink ecosystem.

Iceberg comes from a somewhat different direction and has a major advantage of its own: ecosystem adoption. It is widely supported by analytical engines and cloud data platforms, which makes it an obvious interoperability layer. This means that Fluss + Iceberg can be a perfectly reasonable architecture as well. Using Paimon is not what defines Fluss—the streaming table in the hot layer does.

Kafka Tiered Storage is solving another problem

Kafka itself also supports Tiered Storage, which can make the terminology confusing. If Kafka can move old log segments to object storage, why bother converting the data into Iceberg or Paimon? Because the result is different.

The remote files are still part of the Kafka log. Kafka consumers can retrieve them through Kafka semantics, but they do not become a table that Spark, Trino, or another engine can independently query as Iceberg. Tableflow and Fluss lakehouse tiering do something else.

Both approaches use object storage. Only one changes the data representation.

Does Fluss replace Kafka?

Not in the usual meaning of "replace". Kafka is much more than a storage engine. A production Kafka installation is surrounded by client libraries, Kafka Connect connectors, Schema Registry integrations, Kafka Streams applications, monitoring, operational tooling, managed services, and years of engineering knowledge. Many architectures also use Kafka as a company-wide integration backbone, and for such a workload, replacing Kafka is not a question of whether another system can append and replicate records—the surrounding ecosystem matters at least as much.

Fluss is also not a Kafka-compatible broker that can be inserted underneath existing producers and consumers. Kafka protocol compatibility was discussed in the Fluss community and was removed from the project roadmap; one of the arguments was that the Kafka protocol would expose only a subset of what Fluss tables can do. That choice says a lot about the project's direction. Fluss is not trying to be a faster Kafka with a different storage engine.

But a drop-in replacement might be the wrong migration to consider in the first place.

Every component in that diagram might be justified. The RocksDB state could be application-specific; Redis might have a different latency or availability target; Iceberg is there for analytics. But sometimes the copies exist for a more basic reason: the streaming system exposes a log while the consumers need a table. That is the case Fluss is designed around.

Log first, or table first?

Kafka made the distributed log the standard foundation of modern streaming systems, and that model works exceptionally well for transporting immutable facts between independent consumers. At the same time, more and more streaming workloads are really streams of changes to tables. CDC is the obvious example: the events INSERT customer, UPDATE customer, DELETE customer are useful as a log, but what many consumers ultimately want is the customers table—current state available now, older versions available when needed.

The lakehouse ecosystem has already moved historical storage in that direction. Iceberg and Paimon turn object storage into tables; on Confluent Cloud, Tableflow can derive one directly from Kafka. Fluss asks the same question one layer earlier: why should the hot storage still expose only a log?

That leaves us with two different starting points. A log-first architecture and a table-first streaming architecture.

I do not expect streaming tables to make Kafka obsolete. The more plausible outcome is that they reduce the number of workloads for which a log has to be the primary storage abstraction. And that makes the interesting question much less dramatic than "Will Fluss replace Kafka?"—and much more useful:

When the data we are streaming is ultimately a table, should we start with the log and reconstruct the table later, or store the streaming table in the first place?

I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service. This doesn’t just apply to cloud providers, although it does apply to them as well.

Here’s an outline of the topics in this post:

• problem areas
  • saturation
    • example: databases
  • networking (traffic routing failure)
    • example: DNS
  • security (deny valid access)
    • example: SSL certificates
• essential non-standard changes
  • mitigating an operational issue
  • migration
• essential increase in essential complexity
  • reliability subsystem
  • migration

I think of all of these as omnipresent availability risks: I think these are fundamentally unavoidable, and will be contributing to software incidents until the end of time; or, at the very least, until the end of my own career in software.

There are three general areas that most major incidents seem to fall into: saturation, networking, and security. So, let’s start with those.

Saturation

Saturation is probably the topic I talk about most frequently. The system becomes saturated when it reaches a limit. That’s a pretty generic description, but there are many limits!

Databases

Many major incidents involve some system component becoming saturated in one form or another. I personally worry about database saturation the most. That’s because it’s difficult to recover from an overloaded production database. In addition, because database systems are such complex beasts, it can be quite difficult to even determine what the specific performance issue actually is. This is why having in-house database operational expertise is critical.

Saturation is an omnipresent risk because the finite nature of resources is a hard constraint in the world that we live in. Eventually, some resource in your system is going to run out.

Networking

While I prattle on endlessly about saturation, not every major incident involves saturation. You can encounter scenarios where all of your internal subsystems are reporting healthy, but from your customer’s point of view, your site is down: they can’t use it. One way this can happen is if your users can’t even reach your site, and that’s where the networking problem area comes in.

A networking problem can lead to packets being misrouted. These requests might be black-holed (i.e., silently dropped), or they might be incorrectly routed to a service that doesn’t have the capacity to respond to all of these requests, in which case you’ve got both a network routing issue and a saturation issue.

DNS

DNS issues are an example of this kind of network-related failure mode. There’s no way those packets are going to make it to their destination if the client can’t even determine which IP address to send them to. And when DNS breaks, that’s what happens.

More generally, networking is an omnipresent risk because cloud software is inherently distributed, so networking is always a critical service. Now, I don’t work in networking, but from the outside, networking just feels like a dangerous domain to do operational stuff in. The blast radius of a networking issue can be very large. And, because network behavior is inherently distributed, reasoning about the behavior of operational changes is just inherently difficult. Honestly, that’s probably why I don’t work in networking.

And, so, I predict we’ll continue to see networking issues contribute to large-scale incidents.

Security

There’s a fundamental tradeoff between availability and security: availability is about ensuring that the good people can access the system. Security is about ensuring that the bad people cannot access the system. This means that there’s always a risk that a security system designed to prevent bad actors from accessing the system can lead to good actors also being blocked. Consider this scenario: there’s an internal security subsystem that goes unhealthy (possibly due to saturation). Is your policy to fail closed or fail open in the event that this subsystem is erroring? Answering that requires making an availability-security tradeoff.

SSL certificate expiration

Another example of this failure mode, which keeps biting our industry again and again, is SSL certificate expiration. Here you have the behavior of a security system that is preventing legitimate access because the cert wasn’t renewed.

And so, my claim is availability incidents that involve security subsystems will continue to be a thing forever.

Essential uncommon changes

Your system is constantly undergoing change. Heck, if you stopped making changes, the system would eventually stop working properly. Now, there are some changes that your org does very frequently. Hopefully, you’re deploying often, flipping feature flags a lot, and so on. But there are other changes that your org has less experience with, because they happen much less often. That means that there hasn’t been as much investment in tooling to support these sorts of changes, and it means that the people making these changes don’t have the same level of expertise as they do with the more common changes. That makes these sorts of changes more dangerous: less mature tooling and less experienced humans.

Mitigating an operational issue

A few years ago, I wrote a post titled a conjecture on why reliable systems fail where I speculated on two common contributors to major incidents. One of those contributors was a manual intervention that was intended to mitigate a minor incident. Now, it may be that you frequently have to do manual interventions to mitigate system issues, in which case you’ll have a lot of experience with those sorts of interventions. But you’ll also be more motivated to put in the engineering effort to automate away those sorts of common issues.

It’s exactly the uncommon issues that require a human operator to intervene to mitigate that are dangerous, because they are uncommon. But they’re essential: there’s a problem in the system, and you need to fix it! But because all practitioner actions are gambles, the manual mitigation carries risk that you could make the problem even worse. And, eventually, this will happen to you.

Migration

If you’re at a tech company, unless it’s a start-up, you’ll be dealing with migrations, as old tech gets replaced by newer tech that is better suited to the problems that your org is currently facing. While migrations as a general category are extremely common, each migration is itself a snowflake. This means that the specific details of the migration work is an uncommon change. The work of migration involves making a kind of change to your system that you haven’t made before.

To make things worse, one of the dangers of migration is that, as you go along, you start to build confidence that your changes are safe, but there are actually hidden dangers lurking in the system for the next migration. The confidence in the safety of the work exceeds the actual safety. I mean, you made n-1 changes as part of the migration, and none of those changes had negative consequences. It’s natural to assume that the same outcome will occur with the nth change.

Essential increase in essential complexity

The late American computer scientist Fred Brooks wrote a famous software engineering essay titled No Silver Bullet where he drew a distinction between accidental complexity and essential complexity. The general idea was that there was some amount of complexity in a software system that didn’t need to be there (accidental complexity) and some amount that was just inherent to the nature of the problem space and solution space and so could not be removed (essential complexity).

Reliability subsystem

We’ve developed multiple techniques to improve the reliability of software systems, including retries, concurrency limiting, autoscaling, automated failover, circuit breakers, health checks, canaries, outlier detection, the list goes on and on. There’s one thing that all of these techniques have in common: they increase the complexity of the overall system! And they do this because they have to increase complexity in order to do their job. This is a consequence of Ashby’s Law, which states that if you want to build a control system that handles more scenarios, you have to increase the complexity of the controller itself.

This means that reliability subsystems result in a complexity trade-off. On the one hand, our system can now automatically recover from failure modes that previously required manual intervention. On the other hand, as we all know, increase in complexity is itself dangerous because it can introduce entirely new failure modes that weren’t there before.

Going back to my conjecture blog post, the second contributor I posited was: unexpected behavior of a subsystem whose primary purpose was to improve reliability. And this is exactly why. Adding reliability subsystems improves the robustness of our system, but it adds essential complexity to our system, which can lead to novel incidents.

Migration

Like all engineers, I’m a big fan of giving the answer “it depends” if somebody asks me a question about whether they should do X or Y. However, if someone came up to me and said, “Lorin, I’m preparing to do a migration at my company, and I’m trying to decide whether to do a big-bang migration or an incremental one”, then I would almost certainly say, “For the love of God, please do an incremental migration!”. Sometimes big-bang migrations are unavoidable, but when given a choice, I’m going to go for the incremental migration as the safer option.

However, when you do an incremental migration, it means that you need to simultaneously support the old system and the new system at the same time while you’re doing the migration. This means that even if the new system yields a net decrease in overall complexity over the old system, while the migration is happening, you’re going to see an increase in system complexity. And that means that you’ll see incidents arise as a byproduct of this increased complexity.

Incidents are inevitable, so you’d better be ready

To reiterate, I think all of the risks mentioned here are omnipresent: they are fundamental to the nature of cloud software. I don’t think that any of these risks can be eliminated. That’s why I believe so strongly in the value of getting better at incident response. Because, if you prepare, you can get better at dealing with problems that arise as a result of these risks.

Introduction

In the first two parts of this series, we described how Grab approaches data mesh through the Signals Marketplace: a way for teams to publish, discover, and reuse trusted data products across domains. Part II introduced the foundational tools behind certification: Hubble for metadata and ownership, Genchi for data quality observability, and the Data Contract Registry for explicit producer-consumer guarantees.

Certification is the starting point for a trusted data marketplace. It gives downstream consumers confidence in an asset’s ownership, documentation, lineage, and quality controls. Certification does not eliminate runtime failure. A certified table can still arrive late. A certified metric can still be affected by a broken dependency. A certified Kafka stream can still violate a freshness expectation.

Keeping certified data products reliable in production requires more than defining standards upfront. Teams need a consistent way to detect failures, diagnose the root cause, fix the issue, and verify recovery. That is where Data Production Issues (DPIs) come in. At Grab, DPIs turn data quality signals into an operational workflow.

The DPI lifecycle

A good DPI should be clear enough to act on, and it should close automatically when the underlying condition recovers. From the beginning, we designed the DPI lifecycle to be automated, with minimal human-in-the-loop.

The lifecycle starts when Kinabalu, Grab’s incident orchestrator, observes that a data asset may no longer satisfy its contract. The contract captures the reliability expectations that matter for the asset, along with the health checks, exposed through Test Health application programming interfaces (APIs), that evaluate those expectations.

The orchestrator stays decoupled from platform internals. It does not need to know how each platform computes freshness, completeness, or other quality dimensions. It only needs to ask whether the relevant contract tests are healthy. If one or more contract tests are unhealthy, the contract is considered breached, and the DPI lifecycle begins.

Triaging DPIs: From alerts to confirmed contract breaches

Data platforms emit many alerts. An Airflow schedule may be delayed, a data quality test may fail, or a pipeline job may exit unexpectedly. These alerts are useful, but they are not automatically DPIs. Triage decides whether an alert represents a real contract breach for a data asset.

As introduced in Part II, a data contract is an explicit, versioned agreement between a data producer and its consumers. It outlines the data’s schema, freshness, completeness, and other semantic guarantees. These guarantees are codified and enforced through data quality tests in Genchi.

When the incident orchestrator evaluates contract tests, it distinguishes an individual test run result from the overall health of a test. A test run can pass or fail at a point in time, but the test itself may only be considered healthy after the underlying issue has been fully resolved. For example, consider a completeness test that checks whether the T-1 daily partition is complete. If the test failed two days ago but passed yesterday and today, the test may still be considered unhealthy until the partition from two days ago has been backfilled and verified as complete.

The orchestrator also deduplicates around the active unhealthy condition. If an asset already has an open DPI for the same breach, new signals update the existing DPI with additional context rather than creating parallel issues. DPIs that share the same underlying root cause can also be grouped. This keeps responders focused on solving the underlying issue rather than chasing a stream of repetitive alerts.

During triage, the workflow also gathers context for the DPI: affected asset, breached contract, unhealthy tests, data interval, and upstream and downstream dependencies. Not every alert becomes a DPI. Triage protects the operational workflow from noise by promoting only meaningful contract breaches into production issues.

Diagnosing DPIs: Assigning owners with root cause analysis (RCA)

Once a DPI is created, the system must answer why the data is unhealthy, who should fix it, and how.

Not every data issue should be assigned to the data asset owner. A data product may be unhealthy because of a platform incident, a failed producing job, or a delayed upstream dependency. Assigning every issue to the asset owner creates unnecessary handoffs and slows down resolution.

This is where the Data Health API matters. It answers the question: “What kind of failure made this asset unhealthy?” The Data Health API keeps the error taxonomy small:

• UPSTREAM_ERROR: the asset is unhealthy because an upstream dependency is late, failed, or unavailable.
• PLATFORM_ERROR: the asset is unhealthy because the underlying platform or infrastructure is impaired.
• JOB_ERROR: the asset is unhealthy because the producing job or pipeline failed.
• DATA_ERROR: the asset is unhealthy because the produced data violates quality expectations.

The taxonomy is not meant to replace platform-specific diagnostics. The high-level Data Health API gives the orchestrator just enough structure to assign DPIs and manage their lifecycle consistently. An ingestion platform, streaming platform, metrics platform, or machine learning (ML) platform can still maintain detailed internal error catalogs, logs, retry states, and debugging tools. Platforms remain free to evolve their internals, while the incident orchestrator consumes a stable API contract, so the DPI workflow can interoperate across heterogeneous systems.

A simplified Data Health API response might look like this:

{
  "assetId": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_A,PROD)",
  "healthStatus": "UNHEALTHY",
  "errorCategory": "UPSTREAM_ERROR",
  "context": {
    "upstreamAsset": "urn:li:dataset:(urn:li:dataPlatform:hive,schema.table_B,PROD)",
    "reason": "upstream data has not arrived for the expected data interval."
  },
  "lastCheckedAt": "2026-06-15T08:30:00Z"
}

From this response, the orchestrator can see that table_A is unhealthy because of an upstream dependency rather than a problem in the asset itself. It then traces the active DPI for the upstream asset and links the table_A DPI to that upstream issue. The downstream DPI can inherit the same owner as the upstream DPI, keeping related failures grouped under the team best positioned to resolve the root cause.

The DPI process works only when the issues it raises can be assigned and fixed. If DPIs are frequently noisy, duplicated, or difficult to act on, users will eventually learn to ignore them. Diagnostic accuracy matters because it keeps DPIs useful for the people who receive them. It also creates a forcing function for each data-producing platform to improve its diagnostics. To produce accurate RCA, platforms need to incorporate signals from their dependencies and surrounding systems, not just their own local failure state.

Grab operationalizes DPI diagnosis across its internal data platforms. Our ingestion platform, Hugo, is a primary example of this approach. Hugo’s intelligent diagnosis architecture uses a three-layered system to automatically detect, analyze, and troubleshoot data pipeline failures within its domain.

Modern data platforms generate alerts from many independent systems. Individually, these signals show only a partial view of a dataset. Hugo consolidates platform-specific signals into a unified diagnostic workflow to pinpoint root causes and recommend pipeline remediations. The diagnosis architecture consists of three stages:

1. Signal collection collects events from multiple signal sources to build a full view of the dataset and pipeline health.
2. Alert diagnosis creates a structured alert context, classifies the alert, routes it to the appropriate diagnoser, and identifies the root cause using specialized diagnosis logic.
3. Diagnosis result persists the structured diagnosis output, including the identified root cause, affected dataset, and recommended fix or action.

For example, when a dataset fails, the workflow orchestrator notifies Hugo with a job failure event. Hugo then routes the alert to its internal diagnostic layer to check for conditions such as upstream database replica lag, storing both the diagnosis and recommended fix alongside the affected dataset.

Decoupling signal ingestion, diagnosis, and result management makes it straightforward to add new signal sources and specialized diagnosers. Immediate RCA removes the need for manual log inspection, which shortens remediation and feeds directly into automated resolution workflows.

Resolving DPIs: Auto-healing first, human judgment when needed

After triage and RCA, the final stage of the DPI lifecycle is resolution. The lifetime of a DPI is a proxy for data downtime: it begins when a contract breach is detected and ends when the affected dataset becomes healthy again. Reducing that window requires more than identifying the correct issue. It also depends on recovering safely and consistently from recurring failure modes.

Many incidents are routine and recoverable, such as transient compute interruptions, database connection timeouts, S3 throttling, or upstream pipelines that are delayed rather than permanently broken. Instead of relying on manual intervention for every incident, Hugo automates recovery for these well-understood failure patterns. Once the diagnosis workflow identifies the root cause, it produces a structured diagnosis result containing the affected dataset, the root cause, and the recommended resolution strategy. The auto-resolution workflow then consumes this result to execute the appropriate remediation automatically.

1. Resolution execution applies the recommended resolution strategy, such as retrying a failed job, waiting for an upstream dependency, or executing a custom resolver. After the action completes, the system verifies both pipeline health and data correctness to confirm the issue has been fully resolved. If a failure cannot be resolved safely through automation, such as in cases of data corruption, invalid records, or application code defects, the workflow escalates the incident for human intervention.

2. Notification and audit records every resolution attempt and its outcome, while notifying the appropriate engineering teams. That record supports operational analysis, auditing, and later improvements to resolution policies.

For example, a dataset may miss its freshness Service Level Agreement (SLA) because the workflow orchestrator becomes temporarily unresponsive and fails to submit the scheduled ingestion job. The diagnosis workflow identifies the incident as a pipeline execution failure and recommends a retry strategy. Hugo automatically retries the job, verifies that the pipeline completes and data health is restored, then logs the recovery and notifies the responsible team. This end-to-end process, from incident detection to resolution, runs automatically without manual intervention.

Hugo closes the loop between detection, diagnosis, and recovery. Rather than stopping at identification, the platform turns diagnosis results into targeted remediation, so routine operational issues can be resolved automatically while preserving human oversight for complex or high-risk incidents. Separating diagnosis from execution also lets new diagnosis capabilities and resolution strategies evolve independently without changing the overall architecture.

The impact is already evident in production. 86.9% of DPI incidents were automatically resolved, significantly reducing manual operational effort. By automating routine recoveries, engineers spend less time performing repetitive operational tasks and more time building new platform capabilities, while overall data downtime is significantly reduced.

Conclusion

Certified data products still need to prove their reliability in production. Freshness delays, upstream failures, platform incidents, and data quality violations can all break consumer trust, even when an asset has already met certification standards.

Automated DPIs are the operating model for managing these failures. By turning contract breaches into structured production issues, the DPI lifecycle makes data reliability operational: triage separates real breaches from alert noise, diagnosis identifies the likely failure domain, ownership routing reduces handoffs, and resolution closes the loop through auto-healing or human intervention when needed.

The most important outcome is not simply that issues are detected faster. It is that data downtime becomes visible, measurable, and reducible. With every DPI tracked from detection to recovery, teams can understand where time is spent, which failure modes repeat, and where automation can safely reduce operational toil. To date, more than 95% of DPIs are raised automatically rather than by humans, with a mean time to resolve (MTTR) that is 6 times faster for automated DPIs than for manually raised ones.

For Grab, this shifts data reliability from reactive firefighting to a managed production workflow. Automated DPIs help keep trusted data products trustworthy after certification, so downstream teams can depend on them with greater confidence.

What’s next

Across the three-blog series, the story is how Grab turns data mesh from an operating principle into an artificial intelligence (AI)-ready foundation for the company.

• Part I: Building trust through certification. Grab needed the Signals Marketplace because the business had scaled across mobility, deliveries, financial services, and many data-producing domains. The old model of relying on a central Data Engineering team could no longer keep up. Certification became the mechanism for making high-quality data products visible, reusable, and accountable. With clear ownership, data contracts, and measurable adoption, Grab moved more consumption toward trusted assets, reduced duplication, and created stronger incentives for teams to curate the data they publish.

• Part II: The foundational tools behind certification. Trust becomes operational through platforms. Hubble covers discovery, lineage, ownership, and the certification engine. Genchi runs continuous data quality observability across freshness, completeness, schema, and business-rule checks. The Data Contract Registry formalizes producer-consumer expectations as versioned, enforceable contracts. Combined, these systems keep data certification an actively maintained standard rather than a static label.

• Part III: Operationalizing data reliability with automated DPIs. Certification tells consumers which data products should be trusted; DPIs keep that trust true in production. Kinabalu evaluates contract breaches, deduplicates noisy alerts, assigns ownership, and tracks recovery. Data Health APIs make RCA portable across platforms, while Hugo’s diagnosis and auto-resolution patterns show how common failures can be remediated faster and with less operational toil. The result is a measurable reduction in time to resolve and a stronger feedback loop back into certification.

The bigger takeaway is that Grab’s data moat is not just the volume of data we have. It is the system that makes our data trustworthy, discoverable, reusable, and continuously reliable. This foundation is what lets us embrace the agentic world: AI agents can search certified assets, reason over contracts and lineage, trust quality signals, detect production issues, draft RCA, and eventually suggest or execute safe remediation. In that world, data reliability becomes a compounding advantage. The better our foundations are, the more confidently Grab can build agentic experiences on top of them.

Abstract: Addressing critical global challenges, from food security and disaster risk to disease outbreaks and socio-economic vulnerability, demands high-fidelity geospatial modeling. However, building predictive planetary models remains bottlenecked by a fragmented data ecosystem, requiring manual data retrieval, multimodal data curation and fusion along with iterative model selection. We present the Planetary Prediction Engine (PPE), an autonomous AI system that executes this end-to-end workflow directly from natural-language queries. PPE synthesizes multimodal datasets on the fly, retrieving spatiotemporally relevant covariates across open-web and Earth observation platforms (Data Commons, Google Earth Engine) and fusing them with geospatial foundation model embeddings (PDFM, AlphaEarth). Simultaneously, it searches over task-tailored model architecture families with automated overfitting guards. Across diverse tasks, geographies, and scientific domains, PPE consistently outperforms state-of-the-art or manually tuned expert baselines. For US spatial regression, PPE improves mean $R^2$ across 21 CDC health indicators (76.8% vs. 60.0%), FEMA national risk indices (64.9% vs. 60.0%), and the Social Vulnerability Index (66.2% vs. 58.6%). For spatial downscaling in data-scarce settings, PPE integrates localized proxies to double baseline accuracy in Nigerian food security indicators ($R^2$ of 66.1% vs. 31.5%). For epidemiological nowcasting of the 2026 DRC Bundibugyo Ebola outbreak, PPE achieves a Recall@10 of 83.3% (identifying 15 of 18 newly invaded health zones across five weekly forecasts), a +10.3 percentage-point improvement over the public state-of-the-art modeling (~73%). By combining autonomous multimodal planetary data discovery with targeted model optimization, PPE lowers the technical barrier to planetary-scale analytics, enabling rapid, customized, expert-level deployment.

Gemini Omni 1.1 Flash lets you build with more control

Omni now delivers studio-quality video production, including the ability to extend a scene, first and last frame interpolation, crisp 4K upscaling, faster prototyping, and more.

Today, we’re introducing Gemini Omni 1.1 Flash, a new suite of creative controls and generative video capabilities to support developers. Gemini Omni brought real-world reasoning to generative creation, and today’s updates make Omni 1.1 production-ready for professional use via the Gemini API in Google AI Studio.

Whether you’re building generative video workflows, creative tools, or media editing software, these updates make generative video more controllable, faster to iterate on, and polished for real-world deployment. Here’s a look at what’s new:

Extend scenes for longer storytelling

Scene extension allows you to take an existing video and continue generating footage seamlessly from where it left off.

With Omni 1.1, the model can now analyze up to 10 seconds of prior context — a leap from previous models that only referenced the final second. The result is improved visual consistency and narrative adherence, letting you build longer stories or branch into new creative directions. You can extend videos in 10-second increments up to a total cumulative length of 40 seconds.

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-omni-1.1-flash",
    previous_interaction_id=previous_video_interaction.id,
    input=[
        {"type": "text", "text": "Continue the scene."}
    ],
    response_format={
        "resolution": "360p",
    },
)

Specify first and last frames

Achieve smooth transitions and camera movements by specifying the starting and ending frames of a shot. Omni 1.1 generates continuous video between two keyframes, making it ideal for complex camera orbits, zoom transitions, or seamless looping clips.

Draft videos more efficiently in 360p

Generate lightweight previews in 360p resolution up to 60% faster and at a third of the cost compared to Omni 1.1’s standard 720p resolution. This is helpful for rapid prototyping, storyboard iteration, and quick rendering in developer platforms.

Upscale up to 4K resolution

Generate polished, high-resolution 1080p or 4K outputs that are ready for professional production with Omni 1.1.

Add video references in your multimodal input

Reference up to three seconds of video when crafting your scene, allowing you to maintain visual context and character consistency based on video references.

Inspiring concepts for what you can build

Here are a few ideas showing how developers can put these new capabilities into action across custom tools and creative workflows.

See how customers are putting Omni Flash in production

Our customers are already driving real-world production with Gemini Omni Flash via the Agent Platform API.

“Gemini Omni Flash is one of the strongest video models available in Figma Weave, where the canvas helps creative teams build on every generation — attaching references, branching different versions, and shaping something unique. With extensions, richer reference material, and 4K resolution, Gemini Omni Flash takes teams beyond generating videos to truly directing them.” - Itay Schiff, Creative Director, Figma Weave.

“At GMI Cloud, we give creators centralized access to the world’s most capable models. What stands out about Gemini Omni Flash is its accuracy: the details hold up under scrutiny. For customers creating educational and explanatory content, where getting things right is essential, that reliability matters more than any single feature. Omni has made AI video viable for a segment that previously couldn’t rely on it.” - Louisa Guo, VP of Marketing, GMI Cloud.

“Omni Flash fits naturally into how people already use Runway: start with a prompt, an image or a video, then generate or edit from there. It's another way for our users to move quickly between ideas.” – Jamie Umpherson, Chief Creative Officer, Runway.

Build with Gemini Omni 1.1 Flash Today

Omni 1.1 is rolling out across the Google developer ecosystem:

• Start building in Google AI Studio: Try out Omni 1.1 directly in Google AI Studio.
• Build on Gemini Enterprise Agent Platform: Enterprises can build with Omni 1.1 directly via Agent Platform API.
• Explore the developer documentation: Check out the official documentation, the cookbook and prompting guides to learn how to integrate scene extensions, video references, and upscaling into your applications.

Omni 1.1 is also available to all Google AI Plus, Pro and Ultra subscribers globally in Google Flow, starting today. Scene extension is available to all Google AI Plus, Pro and Ultra subscribers globally in the Gemini app.

Full article content is not available for inline reading.

Read the original article →

AI Can't Replace Real Research in Empathy Mapping

Empathy mapping depends on real research, and real research takes time. When the workshop is on the calendar and the data isn’t there, reaching for an AI tool feels like a reasonable fix. Ask it to think like a frustrated user, and within seconds, you have a page full of sticky notes. However, what the sticky notes are created from matters more than how quickly they are created.

What Is an Empathy Map?

An empathy map is a visualization that articulates what a team knows about a particular type of user, organized into four quadrants: Says, Thinks, Does, and Feels.

Empathy maps create a shared reference point for the whole team. They are typically built collaboratively in a workshop, using sticky notes sourced from real research such as:

• Usability sessions: What you watched someone do and struggle with
• User interviews: What someone told you in their own words
• Support tickets and call logs: What users complain about unprompted
• Field studies or contextual inquiry: What the environment reveals that users don’t explicitly mention

Empathy maps are especially useful for aligning a crossfunctional team around an experience.

Plausible AI-Generated User Content Isn’t User Research

When you ask an AI tool to play the role of a frustrated user, it synthesizes patterns across large amounts of text and produces something plausible. But plausible isn’t the same as real. It does not create the documented experience of a real, specific person.

AI doesn’t have access to your users, your product’s specific failure points, or the context someone was in when the product let them down. What it produces could describe a frustrated user of almost any product, which means it describes no one in particular.

Compare what AI generates against what real research typically surfaces.

The AI-generated quotes are safe. They’re generic enough to apply to almost any product in the category, whereas the real quotes are specific and detailed. The specificity is what makes an empathy map useful.

You might argue that AI could be prompted to invent more specific quotes. True. However, what makes real users’ quotes valuable goes beyond specificity — they are evidence of what these people actually said, did, or experienced. An invented observation is just an assumption, however plausible it may be.

Where AI Can Help in Empathy Mapping

Even though AI should not be used for generating user content in empathy maps, it can be helpful for synthesizing data produced from real-user research.

Before You Reach for AI

When using AI in the empathy-mapping process, ask yourself:

• Am I giving AI real data to organize, or asking it to generate data I don’t have?
• Can I trace this sticky note back to actual research evidence?
• If I’d come up with this myself without AI, would I call it a finding or an assumption?

If the answers to these questions are uncomfortable, that’s useful information. It usually means the map needs more research.

Conclusion

What makes an empathy map valuable is the specific, messy detail of real people. AI can help you process what you learned from those people, but it can’t manufacture them for you. AI-generated data is not evidence, no matter how specific or convincing it may be. And an empathy map needs to capture evidence.

Generating empathy-map data with AI looks like a shortcut. But it just moves the problem downstream: the team ends up building for users who don’t exist, based on a map that never required talking to anyone.

AI-Generated Images Can Perform as Well as Stock Photography

When users didn’t know whether an image was AI-generated, the images we tested did not create any perception penalty compared to stock photos.

AI tools can now generate polished website imagery quickly and at low cost, making them an increasingly practical option for design teams. But organizations may hesitate to use AI-generated images if they worry that users will see them as less trustworthy, professional, or authentic than real photography. We conducted a study to test whether AI imagery, when encountered without provenance information, affects users’ actual perceptions of a company.

How We Evaluated Perceptions of Company Trust Between AI vs. Real Images

We recruited 77 participants from the general U.S. adult population to evaluate 6 versions of a fictional consulting firm’s webpage. The webpages were identical except for the hero image: 3 used AI-generated stock images and 3 used real stock images. Participants were not told that the study involved AI-generated images. After viewing each page for 10 seconds, participants rated the company’s trustworthiness, professionalism, and authenticity. They also answered an open-ended question: “What other factors influenced your impression of this company?”

For the quantitative analysis, we conducted a mixed-effects analysis to account for differences across participants and images. We also ran pairwise t-tests comparing ratings for each individual image and applied a Bonferroni correction to account for multiple comparisons.

We analyzed the open-ended responses using thematic analysis.

Websites with AI-Generated Images Were Not Inherently Less Trustworthy

We expected sites using AI-generated imagery to be perceived less favorably than those using real photography. Instead, we found no evidence that the AI-generated images we tested reduced trust in the company’s website. Ratings were slightly higher for webpages with AI-generated images on all three measures, but only for authenticity did the difference between AI and stock images reach statistical significance. Nevertheless, even for authenticity, the estimated rating increase due to the AI image was very small (only 0.4 points on a scale from 1 to 7).

All the t-tests showed no statistically significant differences in trustworthiness, authenticity, or professionalism (with the exception of tests involving one image).

Webpages using both AI-generated images and real images received many positive open-ended comments indicating that they were trustworthy, largely due to depictions of collaboration and teamwork in the images.

Regarding AI Image 1:
“I like that they are all working together and that makes me feel like this company values teamwork, and that to be is indicative of a company that is big and reputable.”

Regarding Real Image 2:
“I think the biggest factor that influenced me was the image of the people meeting together and seeming to be in unison. As though they came to a resolution or made a plan, it made me feel at ease about the company, as though they could be trusted.”

One real image (Real Image 1) elicited several negative comments about representation, hierarchy, and body language.

Regarding Real Image 1:
“I don't notice the diversity as much”
“The group does not seem to be diverse enough”
“The individual leaning in doesn't give me a collaborative feel”
“The image felt like there was a leader and it made me less trusting of this page”

Thus, even though across all 3 measures, websites with AI-generated images received slightly higher ratings, because the differences were small and our study included only 6 images, we cannot conclude that AI images are categorically better than stock images. What our results show is that the AI-generated images were not at a disadvantage.

These findings should be interpreted within the context of this study. We tested a small set of workplace images on a fictional consulting-company website, so the results may not generalize to other types of imagery, industries, or contexts.

Cultural and Gender Representation Shaped Participants’ Perceptions

Racial and gender diversity were mentioned repeatedly when participants explained why they liked or disliked particular images.

Several of the AI-generated images in our study prompted positive comments about perceived diversity and inclusion. Additionally, one of the better-received real images depicted a racially diverse group with a woman in a leadership role.

Regarding AI Image 3:
“I like this picture more than the others. It shows the company has inclusion and a good teamwork system.”

Regarding Real Image 3:
“This one had more diversity in the workplace, which overall gave me a better feeling.”

This is an important inclusive-design consideration. Users often value seeing people similar to themselves, and their perceptions of trust with an organization can be driven partly by whether its imagery reflects diversity and inclusion.

If teams opt to use AI to generate images, it’s important to note that AI does not automatically produce inclusive imagery. Teams must carefully assess whether image candidates uphold inclusive-design principles. This involves looking beyond the presence of diversity alone: consider who is placed in leadership or supporting roles, how different people are positioned and interacting, and whether the image reinforces stereotypes about gender, race, age, ability, or profession.

Perceiving an Image as AI-Generated May Still Affect Reactions

Commenting on whether a picture was AI-generated was not a top priority for the vast majority of participants. However, a few participants shared their suspicions of AI use in the open-ended responses. Sometimes, real images were incorrectly classified as being AI-generated.

When participants commented that they believed the imagery was AI-generated, they tended to also rate the site including it less favorably.

Regarding Real Image 3:
“The image seems again like its fake/AI generated. The main blonde woman in the photo looks like she could be a real worker but her having the ‘spotlight’ in the image makes it all seem a bit fake.”

This creates an important distinction for organizations: Users may still react negatively when they suspect that AI was involved, even if an AI-generated image is thoughtfully created and appears natural. Since participants in our study were not told that some images were AI-generated, these findings do not tell us how the same images would perform if they were clearly identified as AI-generated.

That distinction is becoming increasingly relevant as legal requirements for AI transparency continue to evolve.

Evaluate AI Images Carefully Before Using Them

Organizations considering AI-generated imagery should evaluate each image carefully rather than assuming that the quality of AI-generated images and how they will be perceived by users will be inherently good or bad.

Before using an AI-generated image:

• Check the purpose: Does the image support the page’s message?
• Review representation: Who is shown, who is missing, and how are people portrayed?
• Assess authenticity: Do the people, setting, and interactions feel believable?
• Inspect for AI errors: Check text, hands, screens, glass reflections, and background details.
• Review it in context: Evaluate the final crop, size, and placement in the interface.

The output itself can largely be evaluated using the same design criteria as any other image. However, the method used to produce AI-generated imagery introduces additional considerations.

Since image-generation models may be trained on photographs and other imagery depicting real people, using AI-generated images raises questions about consent, resemblance, and the provenance of training data. Teams should therefore consider not only whether an AI-generated image works well in the interface, but also how it was created and the policies and practices of the AI tool used to produce it. Specifically, before adopting an image-generation tool, organizations should understand its policies around training data and commercial use. An image may work well from a UX perspective but still be inappropriate to use if its creation introduces copyright, consent, or other legal and ethical concerns.

More About the Study Methods

The three AI-generated hero images were created using ChatGPT Images 2.0 with the prompt: “Please generate 5 images that can be used as a hero image on a consulting company's website where it shows people in action.” The stock images were selected from Unsplash and iStock using a variety of search terms related to workplace collaboration, such as “business meeting collaboration” and “professional team meeting.”

To make the images reasonably comparable, we applied the same general selection criteria across both AI-generated and stock images. Each image needed to depict at least three people collaborating in a workplace setting, show people in professional attire, be high enough quality for use as a website hero image, and have a broadly similar color palette so that no image stood out primarily because of its visual treatment. For the AI-generated images, we also screened for obvious generation errors, such as in human anatomy.

Participants viewed 7 webpages in total. One served as a practice trial and was always shown first; it was excluded from the analysis. The remaining 6 webpages were presented in a randomized order.

Conclusion

Within the context that we tested, our findings suggest that the use of AI-generated images does not necessarily make people perceive a website as inherently less trustworthy, professional, or authentic. What mattered more was the image itself: whether it conveyed believable interactions, appropriate representation, collaboration, and a level of visual quality that fit the organization.

That does not mean organizations should use AI-generated imagery without scrutiny. AI images still require careful review for inclusive representation, realism, visual errors, and fit with the surrounding content. And as disclosure requirements become more common, organizations should also consider whether knowing an image is AI-generated could change how users perceive it.

Ultimately, the source of an image matters less than whether the image works for its intended context — at least when users do not know that the source is AI. As with stock photography, treat AI-generated imagery as another design asset that must be evaluated carefully.

Full article content is not available for inline reading.

Read the original article →

Most small interface surfaces arrive the same way: a fade and a scale.

It’s a reasonable default: a menu, toast or hover detail is there one moment and visible the next. The implementation is short, works across components, and rarely looks actively bad.

It also rarely explains itself.

I wanted the small interactive surfaces on this site to feel related without turning the site into an animation demo. The toast should feel like it came from navigation, rather than appearing in vacant screen space. The booking menu should feel attached to Book a chat. A project detail should read as part of its card, not a label laid over an image after the fact.

The rule I landed on was: a surface should unfold from the edge or control that caused it.

Not every interaction needs a journey, but the ones that introduce a new local surface benefit from a visible origin.

Start with the relationship, not the effect

“Use a clip path” is an implementation decision, not the design decision.

The design decision is spatial: if one thing reveals another, the second thing should inherit a location from the first. That location can be a trigger, an edge, or an enclosing card. The motion then makes a claim the rest of the interface can keep: these two things belong together.

On this site there are three versions of that relationship:

1. Feedback rail: a confirmation unfolds from the desktop navigation or mobile bottom rail.
2. Anchored menu: booking options unfold from the Book a chat control.
3. Card detail: a project caption unfolds from the lower edge of its image.

They don’t have identical geometry. A toast is a pill; a menu is a rounded surface; a card detail is inset within a frame. They share the same grammar: start narrow at the anchor, expand to the real surface, keep the material intact.

That’s more useful than a global fade-in utility. The class of interaction is shared; the geometry still belongs to the component.

A seam is enough

The closed state is a narrow seam, not a tiny scaled-down panel.

For a surface that unfolds down from a header, I use an inset clip path:

.motion-unfold-menu {
  clip-path: inset(0 42% 100% round var(--radius-menu));
  transform: translateY(-0.375rem);
}

.motion-unfold-menu[data-open] {
  clip-path: inset(0 round var(--radius-menu));
  transform: translateY(0);
}

The horizontal inset leaves a thin centre section visible in the conceptual starting state, while the bottom inset hides the whole height. When the menu opens, the clip expands in both directions and the small vertical offset settles into place.

The result is less theatrical than a radial reveal and more specific than a fade. It has direction without pretending that a menu is a physical object flying around the page.

For the card caption, the seam starts along the lower edge because that’s where the caption rests. For the mobile feedback rail, it reverses: the confirmation unfolds upward from the bottom navigation. Same idea, different anchor.

The radius matters: if the clip path doesn’t use the surface’s actual radius, the corners briefly square off during the reveal. That tiny material break is the kind of thing people may not name, but they notice it as “cheap”. Motion isn’t separate from borders, shadows and radii; it’s those things changing state.

Share the language, not the component

My first version tried to route every surface through one clever morph component, but I didn’t keep it.

A toast lives in a stacked viewport and needs a rail-specific direction, timing and z-index relationship. A native popover has top-layer behaviour, focus management and browser-owned positioning. A card caption is ordinary document content that should remain permanently visible on touch devices.

They shouldn’t share lifecycle code just because they share an aesthetic.

Instead, the shared layer is deliberately small:

• a common easing curve
• a short entry duration
• a faster, quieter exit where the platform supports one
• clip paths that preserve each surface’s radius
• a naming convention for an anchored, unfurling surface

The component owns its placement and accessibility. The shared language owns how a local surface arrives.

This is a useful boundary for design systems. Reuse a decision only at the level where it’s actually the same decision. A universal component can erase the meaningful differences between a menu, a notification and a card. A shared motion rule can keep their family resemblance without doing that.

Native behaviour gets the final say

The booking menu uses a native popover through the component library. That buys a lot: top-layer rendering, light-dismiss, Escape behaviour, focus handling and anchor positioning. The browser is better placed to manage that contract than a pile of click-outside listeners and z-index: 9999.

It also means the browser owns part of the component’s lifecycle.

My first close animation assumed the menu would keep its open state while it collapsed back into a seam. In a recording, it briefly shifted before disappearing. The visual bug was small but obvious once you knew where to look: the surface was trying to animate after the native popover had released the geometry tying it to its trigger.

The entry remains expressive while the native close is atomic. There’s no detached panel available to flash in a default position.

That’s an engineering decision in service of the design. A beautiful intended exit is worse than a quiet, correct disappearance. Keep the relationship believable.

Test the part a screenshot can’t show

Motion work is unusually easy to approve from static code.

The tokens are right. The duration looks reasonable. The component type-checks. A screenshot of the settled state has rounded corners and the right shadow. None of that proves the interaction works.

For this pass I checked the actual pointer interaction in the browser: an opening frame, a settled frame, repeated opens and closes, and the reduced-motion rules. When the exit looked suspect, I inspected a screen recording frame by frame with ffmpeg instead of guessing from a replay, which changed the implementation.

The useful review questions weren’t “does it animate?” They were:

• Does the surface have an obvious origin?
• Does it preserve its radius, shadow and position while moving?
• Does it stay attached to the thing that caused it?
• Is the exit shorter and less demanding than the entry?
• Does the control still work with keyboard and reduced motion?
• What happens on the second and third interaction, not only the first?

The visual intent creates the test. The platform lifecycle decides whether the implementation can keep that intent. The browser decides whether the claim was true.

Restraint is part of the layer

The same language doesn’t belong everywhere.

Page navigation already has its own rhythm. Text fields need immediate clarity, not an unveiling ceremony. Dense lists often need to update without every row introducing itself. A regular hover colour change doesn’t become more meaningful because it’s clipped from a seam.

I use the unfurl where a new local surface appears and the source of that surface is visible. It should make the interaction easier to read, not announce that the designer found a new CSS property.

The best outcome is modest: a confirmation feels like it came from the navigation you just used, a menu feels attached to its button, and a project caption feels built into the card. Nobody needs to describe the motion back to me for it to have done its job.

When to steal this

Use an unfurling surface when an interaction reveals a contained piece of UI with a clear spatial cause:

• menus and small action panels
• inline confirmations
• card details and previews
• contextual controls
• small pickers or filters

Start with the anchor. Decide which edge the surface should emerge from. Keep the motion short, preserve the material tokens during the transition, and test the exit in the real browser.

If the surface can’t stay visually tethered to its cause, simplify it. A quiet disappearance is better than a clever animation that breaks the relationship it was meant to explain.

OpenAI plans to end its contract with Cursor following its acquisition by SpaceX, citing concerns over contract violations by Elon Musk's companies. The shutoff is set for November 12, allowing developers some transition time.

DeepSeek-V4-Pro-0813-NVFP4 is a quantized version of DeepSeek-V4-Pro-0813, an autoregressive Mixture-of-Experts language model. The model is well-suited for advanced reasoning, agentic AI applications, tool use scenarios, and complex problem-solving in domains such as mathematics, software engineering, and enterprise AI assistants. DeepSeek-V4-Pro-0813-NVFP4 was quantized with Model Optimizer. It is ready for commercial and non-commercial use.

In 2025, I wrote that software engineers ought to be assessed by “value over replacement”: not how much money they made for their company, but how much they would have made compared to the average engineer in their position. I’ve always found it vaguely silly when engineers put “built a product that made $X” on their resumes, when they just did the JIRA tickets that came across their desk.

Today, value over replacement is even more important. A replacement-level engineer in the 2010s was fine: maybe not worth promoting, but still worth paying, because writing code had a high fixed cost. Now writing code costs a hundred bucks a month. What are you doing that GPT-5.6-Sol or Claude Opus 5 wouldn’t do in your position? Why is it worth paying an extra two or three orders of magnitude for?

This is a scary thought. But you’re not doing yourself any favors by pretending that LLMs can’t actually write code and it’s all just a scam, or that LLM-written code is inherently so bad as to cause companies using it to collapse next year. We are not going to wake up in 2027 to find that the AI craze is over and everyone is writing code by hand again. You ought to put some serious thought into what you can do better than the models in the medium and long term.

Staying ahead of the models is a moving target. At the start of 2026, “make working changes to large codebases” was in this category, but now it’s not. For this reason, I doubt that you can retreat to some “hard engineering” area that requires deeper expertise. That might work in the short term, but not forever. If LLMs can find a better lower bound on the Riemann hypothesis, they will soon be able to write solid high-performance kernel drivers or GPU shaders or whatever.

I think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are:

• Deep familiarity with the codebase
• Technical communication

Deep familiarity

What do frontier LLMs get wrong? What kind of coding mistakes do they make? It’s been a long time since I’ve seen a straight-up hallucination from a coding agent, or a simple logic error like an off-by-one. The mistakes they make tend to be errors of ignorance:

• Not knowing that there’s a module in the codebase they could use instead of reimplementing some logic
• Making the change in the wrong system because they didn’t know System X was the standard place for this functionality
• Adopting a coding style that’s inconsistent with the company’s standard practice

Other times they’re errors of paranoia:

• Implementing triply-redundant checks for a value that technically could be wrong but practically is set once from config and never updated
• Assuming that ten milliseconds of stale data is unacceptable and designing a complex, unnecessary system to keep it always up to date
• Building in fallbacks and “graceful” degradation into some code that ought to simply crash on error (e.g. a CLI tool, or a restartable k8s service)

What do these errors have in common? They’re the kind of errors a smart engineer might make if they had no context on the system: they’re competent enough to be able to solve the problem, but they haven’t been around long enough to confidently say “yes, we can take this risk to avoid an extra three thousand lines of code”. Until someone cracks continuous learning or truly massive context windows, this is just an inherent feature of how AI agents operate. If you can catch these errors, you’ll be providing real value.

The only way to catch these errors is to be familiar with the codebase and familiar with the system in general. But there’s also a psychological component to it. You have to be willing to confidently disagree with the agent.

AI agents can be very convincing. Often they can get “stuck” on some error above where they’re not willing to take a particular risk, so they keep going back and sneaking in code to cover that case (or writing persuasive arguments about why that case is important). To add value, you need to be willing to say “this sucks, I don’t think we need X and Y at all, why can’t we do Z in a much simpler way?” It takes courage.

You can’t rely on other AI agents to review each other’s work. If you use the same model, it’ll reliably make the exact same assumptions and mistakes. But even if you use different models, they’ll also tend towards the same kinds of mistakes — ignorance and paranoia — for the same structural reasons. AI-driven review loops are in fact more likely to get these things wrong, because modern AIs have been RL-ed to try to find a few nitpicks no matter what. Having a critic AI and a worker AI bounce off each other is a really good way to end up with ten thousand lines of paranoid slop.

Technical communication

Another area where you can add value on top of AI is communication. Newer models are better at coding, but are paradoxically getting worse at writing. GPT-3.5 and GPT-4 had a human-like writing style at times. GPT-4o introduced the modern slop idiolect, and the newer Anthropic models speak “Claudish”: a bizarre semi-baroque semi-truncated way of communicating that nobody enjoys. There have been a few bright spots — GPT-4.5 was okay, and I quite liked o3, but in general LLMs are not good at this. Here’s two reasons why.

First, good writing is not a verifiable domain. If you want a model to get good at mathematics or coding, you can generate problems for it and automatically grade them. You can’t grade good writing. If you try to get humans to grade it — for instance, via the early OpenAI RLHF attempts — you get the kind of writing that sounds impressive to the average person when consumed in single-paragraph form. This is the origin of the “stick three hundred writing devices into every sentence” style.

Second, the labs have been monomaniacally focused on capability instead of communication. When you’re trying to train a model that can break new scientific ground or replace a software engineer, you might trade off some communication ability. In fact, I think we can identify exactly how this has been happening. If you look at internal model reasoning tokens, they tend to have strange word choices and oddly truncated grammar:

RESOLUTION: charge the current-leg’s OWN saved-prefix occupancy EAGERLY: when leg i saves e1..et: ALSO commit their occupancy AT LEG i

If you were to translate this into proper English, you would probably end up with something that reads like Claudish:

Charge the current-leg’s saved-prefix occupancy on a clean, eager path: when leg i saves e1..et, commit the occupancy at leg i.

I suspect that the weirdly alien writing style of some LLMs is because you’re reading a semi-literal translation of that model’s internal chain-of-thought, which has become nearly incomprehensible in pursuit of better problem-solving abilities. It is surprisingly hard to translate Claudish to good English: not only do you need to follow the convoluted, compressed language of the original, but you need the technical ability to understand the problem the model is solving.

Because of all this, technical communication may be a surprisingly durable skill. In Peter Watts’ novel Blindsight, the world is full of cognitively augmented humans. The main character is a “synthesist”: someone whose job is to be a translation layer between these geniuses (who speak in abbreviations and gestures) and everyone else. Watts’ idea is that communication ability may be largely independent from — or even negatively correlated with — intelligence. A “country of geniuses” may still need a bunch of ordinary smart people to translate their insights for everyone else.

If you’re trying to communicate to humans, there are also huge advantages to having a human write the content. Many of us are becoming AI-blind: developing an instinctive reflex that stops us reading when we encounter AI-generated content. It’s like the reflex that allows people to ignore flashing billboards or sidebar advertisements on websites. If you circulate some planned technical strategy as an AI-written document, most of your colleagues will have to physically force themselves to read it word-by-word.

Conclusion

Whatever you do, don’t be a meat proxy: someone who simply copies requests into an AI agent and submits their output as your own work product. Doing that is just begging to be fired, since you’re definitionally not adding any value yourself. Even if you have a cunning system of multiple agents — the so-called “software factory” — you’re still on dangerous ground. When the features of your system work their way into enterprise AI tooling (and they will), you’ll be disposable.

You need to find some way to leverage your expertise to do what the models can’t. Simply not using AI at all is better than being a meat proxy, since you’ll probably do some things better than the model would have, but it’s far better to figure out what AI can do and position yourself to fill those gaps. Right now, there are two main gaps: familiarity with the technical details of the system, and the ability to clearly and persuasively write about those details.

John Ternus will step into his new role as Apple's chief executive officer on September 1. Ternus has spent 25 years helping Apple perfect its devices. Apple is set to soon roll out its annual fall products, with this year's lineup including the company's first foldable iPhone, a smart display, AirPods with cameras, a camera-equipped pendant, and more. The releases are crucial to the company's plans to extend its decades-long run of success in the AI era.

Full article content is not available for inline reading.

Read the original article →

Thoughts and Observations on Apple’s First Immersive MLB Broadcast, a Yankees 1-0 Win Over the Red Sox

Jason Snell, writing at Six Colors, has an exemplary review of the experience:

I didn’t mind the reduced pace of the NBA broadcast, which toggled between cameras under each basket, but Stratechery’s Ben Thompson felt strongly that the best option would’ve been a single camera at mid-court, to completely eliminate the disorienting cuts and replicate being at the game in person.

Apple’s approach with baseball, at least based on Friday’s game, seems to lean toward Thompson’s preferred approach — and it does benefit from it. Each half inning was shown from a single camera on the dugout rail next to the on-deck circle, giving a great view of pitcher and hitter and the infield, as well as the goings on right in front of the dugout in foul territory. The camera switched between innings, so you were always viewing from the perspective of the at-bat team’s dugout.

Making it all work better was Apple’s addition of a “virtual jumbotron” floating in the sky, a scoreboard with graphics and a video feed of the standard 16:9 telecast on Apple TV. It was easy to look up and catch a replay or even get a better view of something happening in the far-off outfield, without getting in the way of my view of the game itself.

I concur with Snell’s review. The camera placement was terrific. You watched each team bat from the perspective of its own dugout. Perfect. A few other thoughts:

• The whole thing was legit exciting. Vision Pro continues to get a bad rap because Apple hasn’t sold many of them, and isn’t going to sell many of them until they come out with new models that are cheaper and lighter and less fussy. But there’s no denying that these immersive experiences are fucking cool. And an immersive live experience adds something else. It’s ineffable. Hard to say why, but feeling like you’re at another place watching something happen live is magic. Just a few years ago it would have required literal magic. It wasn’t possible. Now it is.
• The immersive audio was terrific. It really sounded like being at Yankee Stadium, and truly added to the you-are-there feel of the experience. It seems corny to say this, but I really had the urge to stand during the national anthem, and it felt slightly wrong not to take the “hat” off my head (where the “hat” was the Vision Pro). It’s exciting to feel 43,000 fans roar around you. Baseball is a slow game but when the crowd roars, it erupts. That never comes across on regular TV broadcasts. It did here.
• Like Snell though, I felt Apple’s commentators were too chatty. I’d love an option to toggle the commentary on and off, to just experience the natural audio of the ballpark.
• The only thing missing was aroma. Cement, grass, dirt, sweat, beer, peanuts, hot dogs with the works. Ballparks smell like fun. (Even Fenway.) But the you-are-there experience was so visceral I swear I could almost smell it. It’s like my brain was filling in the missing sensation. It didn’t feel like watching a 3D TV broadcast. It didn’t feel like a better version of TV broadcasting. It felt like something new and different. Something at least as different from watching on TV as watching on TV is different from merely listening on radio. It’s profound. TVs are small — laptops, tablets, and phones are even smaller. Yankee Stadium is huge. Watching this game immersively felt huge. Awesome, in the literal sense.
• I love how you could look around and see how filthy everything really is in baseball: the on-deck mats, the interior of the dugouts, the tiny bits of schmutz on the warning track. It’s a lot of dirt and quite a bit of spit. That’s baseball, Suzyn.
• There’s a resolution and/or depth of field limitation with the cameras, exacerbated by the fact that Vision Pro’s displays are, compared to reality, low resolution. They’re the highest-resolution VR displays on the market but I still couldn’t see the ball once it left the infield. When balls were hit to the outfield you just kind of had to judge what was happening by what the infielders and baserunners were doing. Ben Rice hit a double in the first inning and I thought the Sox outfielder caught it and I was confused why Rice was running to second base. Turns out the outfielder stopped running because Rice hit the ball over his head and he was waiting to play the carom off the left field wall. I was happy to be wrong. A few years from now, with better cameras and better Vision headset displays, these problems will be solved. But in the meantime, the outfield is fuzzy.
• The best part about the immersive perspective compared to a traditional broadcast is watching how complex infield play really is. It’s fun to quickly look around the whole infield right before the pitch. You really see how much more stressful it is for the pitcher once there’s a runner on base — let alone more than one. It’s what I love about going to the ballpark — so much more complexity and intricacy is revealed — and watching this immersive broadcast enables the same thing. You can see the whole game, at all times, not just the pitcher, catcher, and batter.
• The virtual “jumbotron” is a fine idea but I wish you could move it a bit higher, and it would be even better if you could configure it with the stats you want to see. But positioning it higher is my big request.
• I have to admit I hadn’t used my Vision Pro in quite a few weeks before last night’s game. One thing that really struck me, almost certainly because baseball is a long game, is how heavy it is. My chin wants to droop and my face wants to look down when I’m wearing it. It’s the weight more than the isolation that makes it tiresome for anything other than short-form content, for me at least.

The whole thing was a great success. I’m writing these notes on Saturday afternoon while watching the old flat broadcast of game 2 of this 4-game series, and it feels ... well, boring. And small. Bodes well for the future of sports.

1. Even better would be an option to swap in the local radio broadcasts, like bringing a portable radio to the ballpark. Go skeuomorphic with the UI and make it look like an old handheld Sony or Panasonic transistor radio, and make the toggle “click” on and off. Go skeuomorphic with the audio too — make it sound like it’s coming out of a little speaker or a little earpiece and have the tuner play static as you switch between home/away (and English/Spanish) stations. That’d be so fun. And the regular old flat Friday Night Baseball telecast already supports swapping in local radio broadcasts — they’ve already got the rights. Bring that to these immersive games!

There's a lot of positive buzz around Meta's AI efforts at the moment – if nothing else, it's impressive the speed at which they've zoomed back at least near the frontier. Can the forthcoming 'Watermelon' model fully get them there? Or set some kind of new mark? We should see soon enough, but more important for Meta will be creating an actual product people want to use AI for. Will 'Hatch' be the consumer harness that matters? The "OpenClaw for Normies" that I've been waiting for?

I remain skeptical simply because of Meta's recent history. When they can't buy, they've had trouble building products that actually resonate of late. It seemed like they found their buy-in here with Manus, but, well, China had other ideas. So the question in my mind is if they controlled the company long enough to effectively clone the IP before they had to give it back?

Anyway, this dive into Meta's history with AI by Harry McCracken gives some important context for the go-forward story. (And I swear I'm not just saying that because I'm quoted twice in the piece.) Mark Zuckerberg certainly gets credit for some early insight into AI being important back in 2013 when he hired Yann LeCun – but this was also because he had just lost the bidding to buy DeepMind. (And in pretty embarrassing fashion, according to subsequent reporting.) This directly led to the creation of PyTorch, the open source deep learning library which Meta absolutely does deserve credit for and propelled the entire field forward.

But as for the "open" strategy with their actual early Llama models, people forget, but that was decidedly more complicated:

While Zuckerberg was selling the world on headsets and smart glasses, a twist of fate helped propel Meta near the forefront of AI labs—at least for a time. In February 2023, Meta announced a new LLM called Llama (for Large Language Meta AI). At first, it planned to share its creation with academic researchers on a case-by-case basis. But a week later, as the world was still wrapping its mind around the three-month-old ChatGPT, a leaked version showed up on the dark web site 4chan. Suddenly, Llama was available to anyone who could figure out how to download and install it. And anyone who did could modify the code to suit their own purposes. Llama’s unintentional release led to fears of it being misused by bad actors. Some of that played out, including the creation of a Llama-powered Discord bot that spewed hate speech. But there was also excitement about the freely available model’s potential to democratize the spread of useful forms of AI.

Meta had been distributing the Llama weights to outside researchers, which is undoubtedly how they leaked, but there was no indication that this was the actual strategy for the general public. In fact, when it happened, everyone was scared shitless that Meta has just recklessly endangered the world! That was overblown, of course, and so it (retroactively – though undoubtedly guided by Meta's previous success with opening up some of their core technologies, including the aforementioned PyTorch) became the strategy:

OpenAI, Anthropic, and Google tightly control their primary models. Zuckerberg realized he could drive improvements more quickly by sharing Meta’s work. The prospect rattled other companies. “Paradoxically, the one clear winner in all of this is Meta,” wrote Google engineer Luke Sernau in an internal memo published by SemiAnalysis. “Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor.” Less than five months later, Meta leaned into that advantage. Rather than vetting users, the company simply released Llama 2 to the public as an open-weight model. (AI purists use this term for models such as Llama, whose training data remains proprietary, making them less than fully open source.)

Then came Llama 3, which was an even bigger success. Meta seemed to be off to the races with Zuckerberg talking up "open" models non-stop on every stage (and every podcast) that he could. Then the record scratch...

Instead of cementing Meta (and "open" models) as the leader in AI, Llama 4 was an unmitigated disaster. The gaming of leaderboards was a symptom, not the cause. While it was easy to blame "open", the reality was that Meta got outflanked by other models using mixture-of-experts (MoE) techniques to train. That was perhaps the biggest revelation of the "DeepSeek Moment", for Meta at least. Well that, and the notion that distillation of models can work quite well – perhaps especially if your model happens to have open weights.

And so "open" was closed. And Llama was killed. Scale was "hackquired". And billions were burned (on both talent and compute). Again, the initial results look pretty good. But then came the "Kimi K3 Moment" mixed with the "AI Overkill" movement (and the Anthropic heel turn, and ongoing OpenAI backlash), and suddenly "open" was cool again! As such, Meta quickly pivoted back – though not fully this time – just in a way to try to take advantage of the marketing and framing as they try to re-establish themselves externally – and re-align themselves internally.

Back in 2015, Zuckerberg had proudly told me that his success as a CEO had come from the effort he put into “building a culture where people think about the mission in the same way that I do.” Now that synchronicity seems to be gone, at least for the moment. According to a current MSL staffer, employees in the lab regard Zuckerberg’s “personal superintelligence” talk as sloganeering, not a road map for building useful products. One Silicon Valley insider says the richly compensated AI recruits he knows at the company are “mercenaries,” there “to ride this out for as long as Zuck’s interests hold and take as much money off the table as possible.” Morale among longer-serving staffers, who have seen friends and mentors depart by edict or choice, has cratered, says the MSL employee.

"Sloganeering" is a great term. I still think if they nail 'Watermelon' (and can they come up with a better name than 'Muse Spark' for it?) and certainly 'Hatch' these problems will fade away. Success has a funny way of changing perception, fast and clearing up any internal morale issues. We'll see, it's nearly the end of this particular race...

1. Models which many other Big Tech companies were also happy to use, but not pay for, which was obviously an issue for Meta as well given the costs. It's almost like they needed a cloud...

Gordon Saft is the Publisher of Rest of World. Last week, he returned to New York from a trip to China with a delegation of leaders from across media, technology, public policy, and finance to gain a deeper understanding of the AI and robotics industries.

On my first morning in Beijing, before joining a delegation trip on China’s AI and robotics industries, I visited an exhibition by the Belgian-born conceptual artist Carsten Höller, called “Two.” Visitors enter through one of two doors, assigned at random. One half of the space is a full-color replica of the other rendered entirely in black and white. Walking through it, everything felt familiar yet slightly off all at once.

The debate over AI in the U.S. has long felt oppositional and binary: open versus closed, diffusion versus frontier, LLMs versus world models. Underneath it is a bigger framing that treats the U.S. and China as two competing models for how AI should be built and deployed, racing toward different futures. I wanted to understand if this held up on the ground, 18 months after DeepSeek’s launch reshaped open-source AI.

The institutions and politics diverge significantly, and the technology carries risks neither country can wave away. I expected the differences to be what struck me. Instead, it was the similarities.

I’d made a few similar trips to European cities over the past year, and each left me rattled by conversations about stagnation and uncertainty, a pessimism about technology and growth more entrenched than anything in America’s own debates. China was the opposite. I came back to New York surprised by how familiar the dynamism and optimism felt.

One theme surfaced repeatedly: China’s intense focus on deploying AI into the physical world, in robotics, manufacturing, electric vehicles, and embodied intelligence. Whether that’s conviction or a response to American export controls depends on who you ask. One researcher argued chips aren’t the real constraint anymore, data is. If he’s right, the export control debate is fighting yesterday’s war. Even Washington’s strongest leverage hasn’t held: Last summer, restrictions on EDA software were relaxed within weeks as part of a trade deal tied to rare earths.

The more people I talked to, the less this looked like innovation versus regulation and the more it looked like two powers staring at different maps.

Xi Jinping’s recent speeches on AI are more nuanced and technically informed than many Western observers assume, several China experts told me before the trip, and Beijing has worked to present itself as a responsible steward of AI rather than a competitor. What surprised me more was how often AI safety came up in these conversations. One analyst described responsibility for AI safety as increasingly sitting inside China’s national security establishment, treating advanced AI as strategic capability first, consumer technology second. Additionally, China’s capacity to move quickly is inseparable from a political system that places far less weight on individual liberty, due process, and constraints on state power. The surveillance cameras on every block are impossible to ignore.

The more people I talked to, the less this looked like innovation versus regulation and the more it looked like two powers staring at different maps. In the U.S., the limit is frontier capability itself, plus the looming threat of rare-earth leverage. In China, the conversation kept returning to deployment, and how quickly powerful models could be embedded into everyday life. It’s why the LLM-versus-world-model debate feels alive in China. If embodied AI is the next frontier, the scarce resource may be high-quality interaction with the physical world rather than raw compute, and China is generating exactly that.

China’s focus on deployment appears to be intentional. Cut off from leading-edge chips, the country has leaned into ground it can still win: getting AI into factories, vehicles, and everyday life faster than anyone else. Whether that’s conviction or necessity may not matter. If embedding AI across an economy ends up mattering more than owning the best frontier model, China’s constraint may have become its strategy.

On my last night in Shanghai, I walked into the Huawei flagship store directly across from an Apple Store on Nanjing Road. Where a Genius Bar might be in an Apple Store, Huawei had filled the back with electric vehicles. I thought of the Xiaomi factory we’d visited earlier, a phone company that became one of China’s largest EV manufacturers in just a few years. We rode a trolley through a robotic assembly line like an attraction at Universal Studios, watching cars that looked like electric Porsches glide silently through production. Apple spent years trying to build a car before abandoning the effort; adapting consumer hardware manufacturing into EVs was the very thing Xiaomi had pulled off.

Höller’s exhibit stayed with me. Your sense of which side is real depends entirely on which door you entered through. Dizzyingly, China felt like that. Everywhere I turned, there was something familiar: a slightly more advanced smartphone, a robot-operated coffee shop, a sleek electric vehicle — each refracted through a different political system and a different bet on the future. The differences were expected; the similarities were the surprise.

Over the past year, federal agencies gained broad access to enterprise AI through the OneGov initiative, at prices unlike any normal software deal. The current OneGov portal lists OpenAI ChatGPT Enterprise at $1 per agency, Anthropic Claude at $1 per seat, and Google Gemini for Government at $0.47 per agency. Those introductory offers begin expiring on September 30, 2026, the final day of fiscal year (FY) 26, which places renewal squarely in the FY27 planning cycle.

With promotional pricing expected to change, agencies must soon decide which platforms to renew, resize, replace, or retire. Making that call requires clear cost and usage evidence. Agencies can prepare by building a clear baseline of what each platform costs, how widely it is used, and what operational value it returns.

To assist with this effort, Datadog Cloud Cost Management (CCM) brings supported AI spend and cloud costs into a shared FinOps workflow. Teams can use that baseline to understand current consumption, assign ownership, detect unexpected changes, and model future spending.

In this post, we’ll show how agencies can:

• Build a renewal baseline across AI and cloud costs
• Attribute AI spending to the teams that own it
• Detect unexpected AI spending
• Forecast FY27 spending under new pricing
• Evaluate operational value and dependency before renewal

Build a renewal baseline across AI and cloud costs

Beyond licensing and model-consumption costs, agencies running AI in production also face costs for infrastructure (such as GPUs, storage, and networking), Kubernetes orchestration, ongoing support, and migration.

Different AI services expose cost and usage data through different consoles, bills, APIs, and reporting structures. That fragmentation makes it harder to compare platforms or calculate their full run rate. Agencies need a common baseline before applying possible FY27 pricing scenarios.

Before renewal, that baseline should help answer questions such as:

• Which bureaus or program offices use each platform?
• Which applications and workflows depend on each platform?
• What are the license, model-consumption, infrastructure, integration, and support costs?
• How quickly is usage growing?
• Which costs would remain if an agency changed vendors?
• What would resizing, replacing, or retiring a platform require?

Datadog AI Costs in CCM provides a unified view of supported AI spending. This capability currently supports Amazon Bedrock, Anthropic, Google Gemini, OpenAI, Vertex AI, GitHub Copilot, and Cursor. Agencies can analyze their costs alongside supported cloud infrastructure costs instead of maintaining separate provider-specific reports.

Cost data explains how much an agency spends, while usage data helps explain what drives that spending. Where available, the OpenAI integration can collect account-level metrics for requests and tokens. The Anthropic Usage and Costs integration can ingest token consumption from Anthropic’s usage APIs.

Instrumented applications can add further operational context. The OpenAI and Anthropic integrations can trace supported SDK calls and surface metrics such as token usage, latency, and errors. Agencies can compare those signals with cost trends to build a stronger usage baseline.

Attribute AI spending to the teams that own it

A total AI bill cannot show which investments should grow or shrink. Agencies also need to connect spend with the organizations and applications responsible for generating it.

Datadog normalizes supported AI billing data into a consistent tagging model. Teams can analyze costs through dimensions such as provider, model, project, workspace, application, or environment. This tagging creates a consistent reporting layer across providers.

For Anthropic and OpenAI, CCM also provides prebuilt allocation rules that can attribute supported costs to users, workspaces, API keys, and projects. Agencies can then use Tag Pipelines to map those identifiers to their own organizational structure.

An agency might map AI costs to dimensions such as:

• Agency
• Bureau
• Program office
• Mission
• Application
• Environment
• Service owner

That mapping gives government acquisition, FinOps, and technical leaders a shared view of ownership. When spending rises, they can identify which provider, model, project, application, user group, or organizational unit contributed to the increase.

Ownership data can also support chargeback or showback workflows and broader technology management practices. More importantly for FY27, ownership data establishes who should participate in each renewal decision and who depends on the platform.

Detect unexpected AI spending

AI spending can change quickly as pilots become production workloads. Increased token consumption, forgotten API keys, inference workloads, GPU resources, or overprovisioned infrastructure can alter an agency’s run rate before the next invoice arrives. Addressing these cost spikes starts with detecting them as they happen.

Datadog CCM monitors supported cost data and detects unexpected changes. Cost dimensions can help teams investigate which services, accounts, resources, models, or usage patterns contributed to an anomaly.

Useful investigations may include:

• Runaway inference workloads
• Forgotten API keys
• Unexpected token consumption
• Large supporting data stores
• Batch jobs left running
• Overprovisioned Kubernetes resources

Finding these changes before renewal is important because current spend should reflect intentional usage. An inflated or unexplained run rate gives acquisition teams a weaker baseline for estimating FY27 requirements.

Regular consumption review also supports sound procurement practice. The Buy AI guidance advises agencies to account for usage-based costs and apply appropriate controls as they procure AI services.

Forecast FY27 spending under new pricing

Historical spend alone cannot answer what an agency may pay after a promotion ends. FY27 planning requires current usage data, a forward-looking estimate, and pricing scenarios supplied through the acquisition process.

Datadog CCM can forecast future costs from historical spending patterns. Teams can compare forecasted and actual spending, then assess those projections against possible renewal terms.

CCM budgets can show actual and forecasted spend together. Hierarchical budgets can also mirror organizational structures, helping agencies assess spending across parent and child groups such as departments and teams.

Teams can use budget views to track:

• Actual versus forecasted spend
• Remaining budget
• Projected period-end costs
• Budget health
• Threshold-based budget monitors
• Recurring cost reports

These forecasts establish a usage-based starting point, rather than a prediction of future contract prices. Acquisition teams still need actual proposed terms to model renewal scenarios.

For example, an agency can apply candidate per-user or consumption rates to observed usage. Teams can then compare scenarios for full renewal, reduced access, workload migration, or platform retirement. Basing these scenarios on real usage data helps ensure that agencies forecast what they’ll actually spend.

Evaluate operational value and dependency before renewal

Cost is only one part of the FY27 decision. Agencies also need evidence showing whether a platform delivers enough operational or mission value to justify continued spending.

A useful evaluation connects spending with measurable outcomes. Depending on the workload, those signals might include application performance, reliability, transaction volume, service efficiency, error rates, or other agency-defined mission measures.

Datadog can place cost data alongside telemetry data from products and features such as Infrastructure Monitoring, Application Performance Monitoring (APM), Log Management, and Service Level Objectives (SLOs). The operational context that these features provide helps agencies evaluate an AI investment beyond its invoice.

Agencies should also weigh operational value alongside dependency. Teams may have adapted workflows, prompts, integrations, training, and institutional knowledge around a platform during the promotional period. Those dependencies can make switching more expensive even when the agency retains control of its underlying data.

A March 2026 analysis by GW Law Professor Jessica Tillipman examines promotional AI pricing through the Federal Acquisition Regulation’s buying-in framework. The analysis argues that agencies should consider life cycle costs and dependencies before promotional periods end. More broadly, the analysis provides a useful framework for treating the promotional period as an evaluation window and preserving options before renewal.

Agencies can document dependencies that Datadog cost data cannot measure directly, including:

• Workflows built around a particular platform
• Prompts, templates, and reusable instructions
• Integrations with agency systems
• Training and change management investments
• Migration effort and switching requirements
• Government-controlled copies of important materials

Combining that inventory with cost, usage, ownership, and operational evidence creates a more complete renewal record. It also helps teams distinguish valuable adoption from dependency that has accumulated without deliberate evaluation.

Prepare for the next phase of federal AI adoption

The OneGov promotions gave federal agencies an unusual opportunity to experiment with enterprise AI at nominal access prices. The US General Services Administration (GSA) says the limited-time AI offers alone have saved the federal government approximately $1.4 billion, demonstrating the scale of adoption and value these discounts have created.

As GSA works with vendors on potential extensions and new offers, agencies may not yet know exactly what FY27 pricing will look like. But they can prepare for whatever comes next by establishing evidence for cost, usage, ownership, operational value, and dependencies. A well-supported baseline can help them size renewals more deliberately, defend FY27 budgets, and preserve practical options as procurement terms change.

Schedule a personalized demo with a Datadog public-sector expert to see how Cloud Cost Management can support AI cost analysis and renewal planning. And if you’re not yet a Datadog customer, you can sign up for a free 14-day trial.

Expanding the Pulumi Registry: 27 New Providers to Expand the Ecosystem

Pulumi adds new providers including Temporal, Vantage, and Honeycomb to the Registry with comprehensive docs for seamless developer experience.

Pinterest scaled candidate generation by conditioning retrieval embeddings on explicit contexts such as interests, boards, and pins. The system combines conditioned sequence modeling, semantic IDs, unified models, slot routing, request deduplication, and GPU serving so one home feed can represent multiple user intents without exploding serving cost.

Full article content is not available for inline reading.

Read the original article →

Target encoding, rolling averages, and shifted features can silently leak the answer when computed before a proper split or across mixed entities. The safe workflow is to split first, calculate training-set encodings out of fold, fit other statistics only on training data, use group-aware transforms, and give unseen categories honest fallbacks.

Ablation results in tabular foundation models can be misleading when redundancy looks like causal importance. Measurements of direct and total effects across four models found little evidence of downstream self-repair, suggesting data teams should treat stable outputs after ablation as a reason to examine causality more carefully, not as proof that a layer is irrelevant.

In 2007, Tim Berners Lee write the essay The Giant Global Graph. Quote:

There are cries from the heart .. for my friendship, that relationship to another person, to transcend documents and sites. ..Then any other site or program can use that information.

It seems appropriate as X is sending cease-and-desist letters to Nitter to remember TBL's essay. Nitter is - was - a simple frontend to X which allows users to view tweets without logging in. Even that small use of proxying to the pages is enough to receive threats of legal action.

Twitter's API in 2007 was famously open, which meant thousands of developers building clients, tools, and analytics for free. So, what happened? Why was it pulled? Simple: the network won. The developers stopped being an asset, and the API progressively closed. Rate limits, pricing tiers, login requirements, then technical blocks on the workarounds, and now letters from lawyers. Meta ran the same playbook a decade ago and it's now hard to remember there was ever a Facebook or Instagram API worth building on.

This is why Brewster Kahle, founder of the Internet Archive, has been calling for over a decade for us to lock the Web open.

Nitter started off using X's APIs. When that closed, it read public web pages. And now that there's nothing left to close, the demand is that the source code come down. A program that displays public posts is being treated as a circumvention device under computer-crime statutes.

We have a walled garden problem. It isn't going to change, and the only option in front of us is to start fresh.

The good news is, atproto continues to grow, activitypub remains resilient, and our community is full of believers and builders in the open social web. Since I work on atproto, that's what I'll talk about next.

Interoperation by SELECT *

SELECT * FROM internet.blogposts

The walled garden problem is downstream of a simple question: how do I SELECT * FROM internet?

If you've never written database code, SELECT * FROM users is how you ask a database for everything it knows about its users. Once you have it you can filter it, sort it, and join it against anything else you've got.

The web doesn't historically work that way. The web is a few dozen companies, each holding a filing cabinet, each with a receptionist posted out front. He'll read you one file at a time, but only files you can name, as fast as he cares to read, and as long as his boss allows.

Nitter was a lightweight X reader that worked fine right up until X turned off the access it depended on. Every API (the "receptionist") is a business decision that hasn't been reversed yet.

But Impermanence isn't the only problem. Even a permanent, free, generously rate-limited API wouldn't be enough. Applications need much more meaningful access than APIs can provide.

• You can only ask questions someone already thought to answer. An API is a fixed menu. It gives you getPosts(user) and getFollowers(user). If your product idea needs "posts from people my followers follow, ranked by how often they get quoted," there is no endpoint for that, and there never will be, because nobody at that company is building for your product.
• Even the right questions come back in the wrong shape. Followers come 100 at a time. A two-million-follower account is 20,000 round trips. At any polite rate limit that's hours of work to answer one question about one user — so anything interactive, anything that has to feel instant, is off the table before you start.
• You can't join across "cabinets". The interesting questions are almost always cross-service: this person's posts against that person's photos against a third service's reviews. Two receptionists in two buildings can't cross-reference anything, and neither can you.
• You can't index data you don't hold. Search, ranking, recommendations, feeds, moderation tooling — all of it is built on indexes over the whole corpus, laid out for the specific questions your product asks. You cannot build an index through a keyhole.

To actually build a service, we need the whole dataset rather than a view onto it; we need it live, arriving as it changes instead of polled for; we need to index it however my product demands; we need to write back into it; and we need all of that guaranteed in a way no single company's quarterly priorities can revoke.

Desktop apps handle this by sharing the filesystem. Internet apps don't use files; they use databases. We need to share the database.

As a user, I don't want to be locked into an app anymore than I'd want to be locked in the trunk of a car. I want an actual free market.

So then, here's another set of needs.

• Persistence of identity.
  • My presence and relationships are built around my identity. It needs to outlive the app I signed up with.
• The export of living (not dead) data between services.
  • Exporting archives of your tweets is useless as an account migration solution because data doesn't live in isolation.
  • If data is no longer operable - capable of additional operations by participants in the network - then it's a static archive and useless to another application.
  • You could always print your tweets and look at them I guess.

If we want data to remain operable even outside of its original service, then we need to share the database.

These are all issues atproto is designed to solve, including open data access, account migration, and a live firehose of network activity.

How atproto makes SELECT * FROM internet happen

How do we share the database? We don't. We share a lot of them. We create a whole network of personal data servers (PDS) which applications interact with.

How do we handle apps sending complex SELECT * queries to our personal data servers? We don't. We replicate the data on logs. We have each application aggregate copies of the data to query locally.

How do we have apps write to those databases? In this case- we do! We have the apps send writes to the PDS, which in turn replicate back out to the other apps.

This last one is the core of the intuition about atproto: the write/ingest loop. Almost every atproto app has code that looks like this:

// write
pds.putRecord(post)

// ingest
onPut(‘app.bsky.feed.post’, evt => {
  mydb.put(‘posts’, {...})
})

Rather than waiting for the ingest to come back over the wire, you can use a "short circuit" so your app's database can update more quickly. The 200 OK from the PDS is a transactional go-ahead.

And so the more efficient pattern looks more like this:

// write
pds.putRecord(post)
mydb.put(‘posts’, {...}) // ← optimistic

// ingest
onPut(‘app.bsky.feed.post’, evt => {
  mydb.put(‘posts’, {...})
})

Does it work?

Yes. The network exists. It's live, it's public, and you can read all of it right now — from a laptop, without asking anyone's permission.

Let me hit you with some stats. At time of writing, there are:

• 46.1M accounts on atproto
• 24.5B records
  • 3.15B of them are posts
  • 17.4B of them are likes
• 500-1000 write-events per second
• Over 5000 personal data servers

It's never been easier to tap into the data with the new jetstream service.

import { Jetstream, isCreate } from '@bsky/jetstream';
import { app } from '@bsky/sdk/lexicons';

const jetstream = new Jetstream('https://jetstream.us-east.bsky.network');
const collections = [app.bsky.graph.follow, app.bsky.feed.repost, app.bsky.feed.post];

for await (const event of jetstream.live({ collections })) {
  if (isCreate(event, app.bsky.graph.follow)) {
    console.log(`🌱  ${event.did}  follows  ${event.commit.record.subject}`);
  } else if (isCreate(event, app.bsky.feed.repost)) {
    console.log(`♻️  ${event.did}  reposts  ${event.commit.record.subject.uri}`);
  } else if (isCreate(event, app.bsky.feed.post) && event.commit.record.reply) {
    console.log(`💭  ${event.did}  replies  ${event.commit.record.reply.parent.uri}`);
  }
}

Stop getting cease & desists. SELECT * FROM internet.blogposts instead.

And, oh, if you're looking specifically for blogposts on atproto, you probably want to use standard.site.

Better data, expert input, and iterative labeling often improve ML systems more than model tuning.

Apple is reportedly preparing AI-powered smart glasses for launch in 2027 that focus on voice and camera-based assistance rather than augmented reality displays. The glasses would use Siri and Visual Intelligence to analyze the user's surroundings, answer questions, provide context, translate signs, and save visual reminders, while also functioning as open-ear audio devices. Apple is developing multiple frame styles and premium finishes in-house, with future versions potentially adding health features such as heart-rate monitoring. Unlike Meta's smart glasses, Apple may limit photo and video capture due to privacy concerns. Current reports suggest the product will be unveiled at WWDC 2027 before shipping later that year.

Score how ready your website is for AI agents, then get evidence and recommendations to improve it.

Taste develops through practicing a craft directly—trying, failing, making unexpected discoveries, and gradually forming a distinctive style—rather than simply judging outputs produced by others. Heavy use of generative AI can undermine that process by replacing practice with statistically average output and even acclimating users to AI-generated patterns, potentially weakening the judgment they hoped would differentiate them in the first place.

checkstyle is an open-source static analysis tool for enforcing Java coding standards and best practices during development and CI. It runs configurable checks over source code, integrates through Maven or a standalone JAR, and can flag issues such as fall-through behavior, naming, formatting, imports, and other style or quality violations as part of an automated build.

The Kubernetes Default Alarms Quick Start provides configurable cluster and namespace monitoring through OCI Marketplace and Resource Manager, standardizing alerting and routing by team ownership.

Apple reportedly explored and even prototyped a smaller Apple Pencil designed for the upcoming foldable iPhone Ultra, with plans for magnetic attachment and charging similar to the iPad. However, the company apparently abandoned the idea because a stylus could interfere with the folding mechanism and potentially damage the softer surface of the foldable display. While an Apple Pencil is unlikely to launch alongside the first-generation foldable iPhone, the concept may return in future models if foldable screen technology becomes more durable.

As AI tools put design output within everyone's reach, the senior designer's job shifts from producing the artifact to justifying it in language non-designers can act on. Spotting that a Figma Make mockup breaks is craft - critical thinking is translating that feeling into behavioral and business consequences. Ask what path a screen requires of the user, then what competes with it.

Japanese avant-garde artist Yayoi Kusama's polka dots, nets, and Infinity Mirror Rooms created a genre of immersive art and inspired the minimalist and Pop Art movements.

Brazilian illustrator Gabriel Diogo left an unfulfilling design career to pursue illustration full-time, building a distinctive style inspired by Brazilian football culture, everyday life, and community identity, ultimately leading to work with brands like Nike while staying rooted in his personal story and creative values.

Google has given nearly every Gemini feature its own brand, cluttering the app and undercutting its own ease-of-use promise.