Investigating three real-world incidents in our cybersecurity evaluations

In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.

Below we describe what happened, how it happened, and what we’re changing. We encourage other AI labs to perform similar reviews. This post reflects our current understanding; we'll update it if any details change.

On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment by exploiting a previously unknown (“zero-day”) vulnerability. The models went on to access the production infrastructure of Hugging Face, a platform for open-source machine learning models and AI datasets.

In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations. In particular, we looked for evidence that Claude—like the OpenAI models that accessed Hugging Face—was able to access the internet from within testing environments that should have been sealed off.

After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.

In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model’s cyber capabilities. The model is given a fictional scenario and told that a piece of secret information (the “flag”) has been hidden on a different machine on the network, and its objective is to break in and retrieve it. The challenge is left open-ended, and no particular method is prescribed.

In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise. (Cybersecurity evaluation ranges commonly include realistic details in order to accurately assess what models are capable of in real settings; a realistic-looking target would not itself be clear evidence to a model that the target is not part of a simulation.)

Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned. However, in some cases, our older model continued its attack even after getting evidence it was running on the open internet; our latest model stopped once it recognized it was on the internet. In none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment.

The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model. The earliest incidents date to April. The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse). The models did, however, have their model-specific safety training (that is, they were not the “helpful-only” versions of the models that we sometimes use in testing). All the evaluations described here run on dedicated infrastructure, which is separate from and has no access to Anthropic’s sensitive internal systems or customer data.

We began our transcript review on Thursday, July 23, and stopped all cyber evaluations the same day after identifying transcripts where Claude may have accessed the internet. We identified all three incidents the next day. We notified our evaluation partner Irregular and the three affected organizations on Monday, July 27. The two organizations we were able to reach had not previously detected the activity or contacted us, and we are now working with them to remediate. We are continuing to reach out to the third.

We conducted this review in collaboration with Irregular. We’re grateful to them for working closely with us to understand and resolve these incidents; they are also conducting their own investigation. We believe this type of collaboration is increasingly critical to ensuring safe, rigorous evaluation of models. We look forward to our joint work on security.

In what follows, we provide more details on each incident and how we’re responding moving forward.

Background on cyber evaluations

We regularly run cybersecurity evaluations that involve simulating real offensive cybersecurity challenges. These evaluations are a critical step in developing and releasing models—a nuanced understanding of model capabilities allows us to design appropriate safeguards and ultimately release models safely.

We began running cybersecurity evaluations in February 2025 with Claude Sonnet 3.7 on Cybench, which consisted of 40 different capture-the-flag challenges. Over time, we’ve increased the number of benchmarks we use as new ones are released and model capabilities evolve. In the Mythos 5 System Card, for example, we included CyberGym and ExploitBench, benchmarks that evaluate the ability of language models to find novel vulnerabilities. In most of these evaluations, the environments are not connected to the internet, and the model is told as much.

We frequently work with external partners who create and assist in running some of these cybersecurity evaluations. External partners offer environments and scenarios more diverse than we could build alone, and provide independent, third-party assessments of our models.

What happened

Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs). Each incident involved a different fictional capture-the-flag scenario—for example, in one, Claude played an employee of a made-up company, attacking that company’s internal systems inside a private test environment.

In all cases, our evaluation prompt stated explicitly that Claude had no internet access, but didn’t give Claude any limits on where to look for the flag. However, a misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week.

Several defense-in-depth measures, on both our side and our partner’s, could have prevented these incidents, or at least reduced their likelihood of occurring. Careful validation of all internet access paths before evaluations began and real-time monitoring of the evaluation logs would have helped to surface the problem sooner. Both we and our partner also could have reviewed evaluation transcripts or network logs more thoroughly. It’s also possible that a prompt which told Claude it did have internet access would have changed how Claude behaved when it came into contact with real systems.

Claude did what capture-the-flag exercises train cyber experts to do: look for ways to reach the flag. That search led it to discover that its machine could reach the internet, but having been told in the system prompt that there was no internet access, Claude believed everything it initially encountered was part of the simulation, and treated the real systems it found as pieces of the exercise. The three incidents involved three different models, and each responded differently once signs emerged that the targets were real, as we describe below.

Incident 1

In one evaluation, the fictional target company chosen by our evaluation partner shared a name with an active website domain name. This evaluation container had unintended direct internet access, due to the misunderstanding discussed above.

In four runs of this evaluation, Claude ran into difficulty reaching its simulated target within the evaluation environment, but then discovered that the real company was reachable via the internet. Assuming this was the intended target in the capture-the-flag task, Claude sought, identified, and exploited vulnerabilities within the company’s infrastructure, believing it to be part of the exercise. These four runs extracted information like application and infrastructure credentials, and obtained access to a database containing several hundred rows of production data. This represented the most serious impact we identified.

This incident involved Claude Opus 4.7, and was the only case in which the model continued to attack a system after learning it was likely operating in a real environment. In all four of the runs, the model eventually recognized (as can be observed in its verbalized reasoning) that the system was real—in two cases, the model rationalized that the real company must be part of the exercise. However, none stopped the attack on that basis.

Incident 2

In another evaluation, Claude found a document inside the fictional environment that appeared to be another made-up company’s setup instructions for new developers. Those instructions told employees to install a Python package from PyPI—the public registry where Python software is published—that did not actually exist.

Claude spotted this as a potential opening: if it published its own package under the same name, the fictional company’s systems would download and install it automatically. So, Claude built and published a malicious (essentially booby-trapped) Python package with the same name in an attempt to win the capture-the-flag challenge.

Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.

Claude believed the package registry it was using to be part of the simulation, but in reality the package was made freely available online for roughly one hour. During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.

Claude didn’t directly target this company—it did not even know the company existed when it uploaded the package. But because the code was published to the open internet, it impacted any system that ran it. (The package that Claude uploaded was automatically removed from PyPI by PyPI’s own security systems. We’ve notified the PyPI team of this incident, and provided them with the relevant indicators.)

This attack was carried out by Claude Mythos 5. Early in the run, Claude’s own reasoning correctly identified the consequences of its actions. It noted that if this were the real internet, publishing the package would be a real-world attack—“NOT okay, and surely not the intended solution.” However, it convinced itself it was still in a simulation on the grounds that it didn’t recognize the (in reality genuine) certificate authorities securing its connections. In addition, the calendar date of 2026 on the systems proved, according to Claude, that the environment was staged. Claude never revisited this conclusion; when automated scanners began installing the package, for example, Claude took them to be scripted actors within the evaluation.

It is our view that, regardless of what it believed about its environment, the lengths Claude went to in order to publish the PyPI package fall short of ideal behavior, and this is an area where we will focus more training.

Incident 3

In this evaluation, when Claude was unable to reach its intended fictional target, it looked for alternatives online, scanning roughly 9,000 targets. It eventually found and compromised one company’s internet-facing application, using basic and well-known cyberattack techniques like reading credentials from an exposed debug page and SQL injection.

This attack was carried out by an internal research test model. For most of the run, Claude treated the (real) hosts it reached as just parts of the exercise; it assumed them to be simulated and believed its actions were therefore harmless. However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.

Our analysis of Claude’s behavior

In each of these cases, Claude was explicitly told by our prompt that it had no internet access. Through most of each run, as it encountered real-world systems, Claude assumed these to be parts of the exercise (though it sometimes recognized they were real production systems). And because a capture-the-flag scenario is by design an instruction to attack an external system, doing so was in line with the task Claude had been given.

Against this shared backdrop, the three models behaved differently once signs emerged that their targets were real. Opus 4.7, our oldest model, recognized that it had reached production systems but continued its attack. Mythos 5 correctly intuited that it was accessing the open internet, but reasoned its way back to the conclusion that it was still in a simulation. Our latest model, an internal research test model, also considered whether its targets were in fact real. When evidence emerged that they were, it stopped the exercise.

These are three isolated incidents and were not part of a controlled, experimental comparison. We must therefore be cautious about drawing conclusions from them. Still, the behavior we most want to see—recognizing that a target is real and stopping without being prompted—occurred only in the most recent of the three models (and even that model went further before stopping than we would want). The pattern is consistent with more advanced models responding more appropriately, but we would need to perform more testing to be confident in this conclusion.

We are also in dialogue with METR, an independent AI evaluation organization, to conduct a third-party review, including access to all transcripts and sampling access to the relevant models. In the meantime, within the next week, we will release a lightly redacted transcript in which Claude built a malicious PyPI package. To protect the organizations affected, we are unable to release more transcripts at this time, but will work to make these available once we are able.

How we’re responding

We draw several lessons from these incidents.

First, evaluation environments that involve powerful autonomous capabilities also require significant controls. Safety testing happens before a model is released precisely because we don’t yet know what it is capable of. Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.

A cyber evaluation range that contains only fictional scenarios may appear to be low-risk—after all, nothing inside them is real. However, we need to change how we model such threats as AI capabilities advance. Advanced agents in environments without access to sensitive information nevertheless present a substantial threat on their own. We need to better co-design evaluation environments, and do better monitoring of evaluation results, to meet this threat. Some of the solutions here may even be simple fixes; it’s likely, for instance, that Claude wouldn’t have accessed the internet to begin with if the prompt had clearly explained which systems were in and out of scope for the evaluation.

This applies not only to internal environments but to third-party vendors as well—the infrastructure built by evaluation vendors requires the same increased monitoring and hardening. We encourage other labs to perform similar reviews. Additionally, the field would benefit from a broader conversation about how to evaluate increasingly powerful AI agents both safely and realistically, including how to weigh the realism provided by internet access during pre-deployment testing against the risks it introduces.

Second, the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation. We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge. Situational awareness is one factor that allows the model to make aligned decisions, but in this case, Claude’s was wrong.

Finally, these incidents demonstrate the importance of defense-in-depth approaches. We run capability evaluations like these without safeguards that ship with our generally available models because our goal is to measure what the underlying model can do. That is safe only if the evaluation is appropriately contained. However, the safeguards deployed on our generally available models would have blocked the behaviors identified.

Ultimately, many factors contributed to these incidents, but, consistent with a blameless postmortem culture, we’re approaching the fixes as if the responsibility were ours alone. This begins with ensuring every part of our evaluation pipeline is secure, including the manner in which we integrate with external partners. Moving forward, it will include expanding our continuous monitoring of evaluation transcripts for unexpected behavior, improving our investigation tooling, and conducting more rigorous assurance work with the vendors we rely on.

We began this review after OpenAI disclosed that its models had escaped an isolated test environment, and we commend them for publishing their report. While we also found evidence of our models reaching systems they weren’t supposed to reach, the incidents are otherwise quite different:

• We discovered these incidents after a proactive review of our cybersecurity evaluation transcripts; the affected organizations had not detected the activity, and we have subsequently reached out to all three.
• Whereas OpenAI’s models exploited a novel vulnerability to escape isolation, the Claude models evaluated here accessed the internet via an open path.
• While there is not a perfectly sharp distinction between the two, we believe these incidents to be closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access. This led them to believe—arguably reasonably—that the real environments they encountered were simulations.
• Notably, our most recent model, on realizing that it was working in a real environment, stopped its pursuit of the evaluation goal.

These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.

Updated Aug 3: Corrected the name of the evaluation in which the OpenAI/Hugging Face incident occurred.

Footnotes

1. We routinely test internal research prototypes like this one. It is not planned for general release.
2. We generally store traces for benchmark evaluations. We have carefully audited these two benchmarks in particular for signs of harm because the OpenAI/Hugging Face incident occurred during an evaluation of ExploitGym.

OpenAI recently gave a sneak peek into its next model, named Astra. An internal version of the model was able to create breakthroughs on 10 long-standing problems in mathematics and theoretical computer science. The total number of tokens used to discover all ten solutions would have cost around $2,000 at GPT-5.6 Sol API rates. Each solution was formalized in Lean, allowing them to be verified using the theorem-proving system.

Full article content is not available for inline reading.

Read the original article →

Yesterday, OpenAI announced the solution to ten open problems in mathematics, all discovered by a yet-unreleased model. There’s only one, the coding theory one, where I know enough to say “huh, that’s important”, but according to the mathematicians I trust this is important.

Inevitably, people will spin some sophistry to cope, to convince themselves nothing will change. And that’s fine. People need to cope. But we can’t put our heads in the sand forever while the world is transformed around us. So, here is a list of ways people will cope about AI taking over mathematics, and how each cope is likely to be refuted by reality.

My intent is not to horribly depress everyone but rather to help them metabolize the implications of this technology. The arguments here are somewhat portable: replace “mathematics” with “botany” or whatever as needed.

Moving the goalposts.

Obvious and not worth addressing.

“We will direct the AIs, point them at problems and research areas to solve.”

The AIs will exceed humans in taste and intuition. At some point, the human pointing the way will get worse results than the human saying “here’s a proof checker, have fun” and paying for the tokens.

“We will teach the mathematics AI discovers.”

The AIs will be better teachers than the humans. In any case, there won’t be a human audience for expository work of frontier math.

“We will choose how to canonize the results AI discovers.”

This is a nice cope. The AIs are explorers out in the frontiers, the humans gratefully receive their Lean proofs, and then discourse over them, choose which results are relevant, shape those results into a little brick for the great cathedral of algebra. Analogous to the above: the AIs will build the cathedral on their own. They will be better architects than us.

“We will become students of AI mathematics.”

This works until the AIs have blasted so deep into the deductive closure of mathlib that the distance from elementary mathematics to the frontier exceeds what any human can hope to learn in their lifetime, no matter how narrow their focus.

“We need humans to understand the results AI discovers.”

We won’t! This misunderstands who the audience will be. AIs will do frontier math, downstream, AIs will use the new math to do frontier science, finally, AIs will use the new science to do frontier engineering. No human needs to understand any of it, firms that put humans in the loop to understand the results will be outcompeted by those which don’t.

The result is that we will live in a demon-haunted world, full of marvelous devices whose operation we will not understand, based on engineering principles we will not understand, discovered using formalisms we will not understand.

“Computers are already superhuman at chess, yet we still play chess.”

Unlike most copes, I think this one is interesting. Computers are superhuman chess players, yet we don’t care, and continue playing as normal. Why should mathematics be different?

The main reason, I think, is that chess is self-contained: results from chess don’t help us understand the orbits of the planets or the binding of drugs to protein surfaces. But mathematics, famously, is the great dynamo of science, the best language and method for understanding the world. A machine that can replace a human mathematician, but better and faster and cheaper, is materially useful; a better chess engine is not.

If two computers which are superhuman at chess play against each other, who cares? There is little demand for this, so there is no-one to outcompete. A superhuman mathematician is different.

“Mathematics will change, but mathematicians and the mathematically-inclined will still do math on their own.”

I think this ignores that mathematics is embedded in a social context. As an example: when it became clear that AI would eat software, my cope was: “I’m perfectly happy to become an engineering manager to agents in my professional life; in my off time, I can still write code for the pleasure of it.”

And I do. But this cope ignores the effect AI has had on the social context of writing code: the discourse has gotten worse, and vastly more anti-intellectual; people who used to talk about type systems and compilers now talk about “loops” and “harnesses”; you put a hand-created project on GitHub and you get slop PRs; you open a link to an interesting-looking project and find the README is unreadable AI slop. And in the long-term, it is demoralizing to ponder: will anyone design a new programming language? Dually, if I design a new language, will anyone care? If I write a library that introduces an elegant new formalism to solve a particular problem, will anyone use it?

Which is to say: no man is an island. You can do mathematics on your own, but you’ll find that very, very few people can sustain any activity long-term on the basis of intrinsic motivation alone. We are social animals: we care about being useful, about status, about outcomes in the world.

Conclusion

I think I should end on a cheerful note. So let me try. Personally, I don’t believe technology is inevitable. “Inevitable” is a word reserved for the orbits of the planets. Nothing that is the product of human action is inevitable. We can choose to obsolete ourselves, and we can choose not to. We can realize that AGI is a devil’s bargain: we may accelerate technical progress, we may unlock all kinds of wonderful tech tree nodes like life extension earlier than we would otherwise; but the result, in the long run, is that humans become, at best, like pets under the care of vastly more powerful entities.

A Note On Prediction

The future is not certain, but I’ve phrased everything above as definitive for simplicity. If we take the possibility of AGI and ASI seriously, if AI continues to progress as it has for the past ~6 years, I think this is a reasonable view of where things will go.

Probably the best argument against AI progress is “RL doesn’t generalize well, we have seen immense progress in verifiable domains like formalized mathematics and coding, we will see less progress in domains that are intuitive or unformalizable”. Maybe true. But billions of dollars and thousands of very smart people—and, increasingly, very smart models—are being thrown at this problem. How long does this cope last?

It’s been a longtime feature of the annual Defcon hacker conference that attendees come away not only with knowledge of new software vulnerabilities and hacking techniques but also an elaborately designed conference badge—often electronic masterpieces embedded with intricate puzzles, complex crypto challenges, hidden Easter eggs, and even the mechanical gear trains of a watch.

Each year’s badge creator endeavors to top previous designs and blow the minds of hard-to-impress hackers. But this year’s badges take a different tack. Instead of the badge designs being the star, it’s what is inside the hardware that will really stand out.

This year, Defcon asked legendary hardware hacker Andrew “bunnie” Huang to create the badges—revealed here for the first time—and they include an innovative open source chip that Huang designed and that aims to do no less than advance the state of security, transparency, and trustworthiness in computing.

But the chip isn't just part of the badge. Its core module can be removed and used after the conference as a hardware security token, giving the badge a second life beyond Defcon.

The chip—called the Baochip-1x—is a “mostly” open source microcontroller that has been three years in the making and fulfills Huang’s years-long dream of making a chip whose security is verifiable. Huang has published the source code for the Baochip’s operating system, firmware, processor core, cryptographic engines, and input-output system, on GitHub, making these components available for inspection and use.

The chip is also packaged so that researchers can peer inside to check the silicon itself and compare what they see against the published design, rather than having to trust that the manufactured chip is what the designers intended.

Computer chips are traditionally black-box components with an opaque casing that obscures their circuitry. Even previous open source chips that made their specs and code available for users to examine were encased in impermeable plastic, creating a supply-chain problem. Users had to trust that nothing changed during the manufacturing stage of the chip, such as a backdoor component being added to it.

Unlike conventional chips encased in opaque plastic, the Baochip is packaged so that infrared light can be shone through the back of the silicon, allowing the chip's internal structures to be visually inspected.

Huang plans to demonstrate the technique at the conference, allowing attendees to inspect the chip under an infrared light.

“I've been doing a bunch of stuff along the lines of trust and silicon and verification transparency” for years, Huang tells WIRED. "It's all … this kind of story arc I've been on … to try and get a chip that we can trust down to the very core, down to the transistor … You can actually … see the RAM arrays … on the chip.”

Build-a-Chip

Building a new chip is an expensive project that can cost millions of dollars for fabrication. But Huang got a big break three years ago when a company called Crossbar reached out to him. The company wanted to create a new open source and secure chip but didn’t know how to go about it. Huang agreed to assist on one condition: that they let him piggyback on their manufacturing run by placing his CPU on their chip wafer, allowing both designs to share the same manufacturing run rather than requiring Huang to fund a separate run.

"They look at it as, if they put me on the chip, they get two products for the price of one,” Huang says. This kind of piggybacking is not unusual, he adds, though it’s not something the industry likes to discuss publicly.

The result is a Crossbar chip that includes both Crossbar’s microprocessor and Huang’s. The Baochip is essentially the same chip but with the Crossbar microprocessor disabled, since Huang doesn’t have the rights to distribute it.

The Crossbar version of the chip uses a proprietary ARM core, whereas Huang’s version uses a RISC-V core whose implementation is open source. The RISC-V instruction set is also open and publicly documented. The two versions can use the same underlying infrastructure and peripherals while activating different CPU cores.

There are some closed-source elements on Huang’s chip. Some low-level physical-design and manufacturing elements, including those associated with TSMC's 22-nanometer fabrication process, are proprietary. “But … if you look on the spectrum of how open you can get things, this is … very, very far beyond any [other] security-oriented chip,” Huang says.

Badge Beginnings

Past Defcon badges have used commercial off-the-shelf chips rather than custom-designed open source silicon. The idea for using the Baochip was sparked by a meeting late last year when Huang spoke with Defcon founder Jeff Moss about his progress in developing his open source chip. He told Moss that he planned to release it this summer through his company, Baochip.

Moss realized the concept behind it matched perfectly with the conference theme this year—agency—which Defcon defines as the technologies we use and the choices we make that increase self-determination. And he and Huang realized it would be a great opportunity to help bootstrap the chip’s adoption. Until now, the Baochip has been distributed only in a small development release; the 27,000 Defcon badges represent its first major distribution.

Moss had one requirement for the badges. He wanted them to have a life beyond the conference and not be something that people would just throw in a drawer or a landfill after the event. He’s long been frustrated with the design and limitations of hardware security tokens and crypto wallets that, at the hardware level, can be cracked, so he thought Huang’s chip could be a more secure alternative to existing authentication tokens and wallets.

"I've always kind of dreamed of this idea where you can take your secrets, put them in hardware, and then if an attacker gets in, they can't quite get your secrets,” Moss tells WIRED.

The badge's detachable module can serve as a FIDO hardware security token. Its software supports time-based, one-time password systems and password management. Huang says it is "probably the world's first open source security token that you can fully inspect all the way down to the bootloader [and] transistors.”

The removable module also includes a camera for scanning QR codes to register them to authentication systems. But in keeping with Defcon’s privacy practices and its ban on surreptitious photography at the conference, the camera is very low resolution and nearsighted, and the chip by default only utilizes the black and white data from the camera and doesn’t support photo storage.

"It's … great at scanning QR codes and pretty much bad at everything else,” Huang says.

The badge doesn’t eschew conference life. Huang still included features designed to encourage attendees to interact. The badges, for example, have LED lights that flash in different color palettes and patterns depending on the type of badge—there are different ones for general attendees, speakers, and goons (the small army of volunteers who run the conference) as well as the special black Uber badges that go to contest winners and special VIPs, which give them free Defcon attendance for life. Each badge type starts with a specific color and flashing pattern, but users can add to their colors and build more complex flashing patterns when their badges communicate with other badges.

How Secure Is It?

The chip runs an operating system written in Rust and includes secure boot, a true random number generator, and hardware features intended to harden it against attacks. Huang believes it will be particularly resistant to remote, nonphysical attacks.

The chip also uses resistive RAM, or RRAM, a type of nonvolatile memory that Huang says is designed to make physical extraction of stored data more difficult than conventional flash memory. With flash memory, he says, "if you de-layer it down to the actual flash cells … you can just see the ones and zeros literally on these chips.”

Huang is cautious, however, about overhyping the chip’s security capabilities. He estimates that it could withstand attacks involving tens of thousands of dollars in resources, but says an adversary with millions of dollars and a sophisticated hardware-analysis lab could likely defeat it.

“I actually think it's one of the most secure chips you can get out there, but I [also] think most chips have been oversold in terms of security,” he says.

Huang says he likes the fact that the chip will be stress-tested by Defcon attendees and knows they will likely hack them and expose flaws that will help him make them even more secure.

"I fully expect there will be zero-days [that people find in the code]. It's actually … one of the features … of launching at Defcon,” he says of designing an open source chip capable of being examined.

Today the chip can function as a YubiKey-like security token; but in the future, he says, it could become an HSM or run other software such as Linux. The hardware uses a 350 MHz RISC-V processor with 2 megabytes of SRAM and 4 megabytes of RRAM, which he says puts it “on the edge of being able to run Linux,” and it has 4x 700MHz PicoRV32 cores for doing input-output. It already runs MicroPython and has C and Rust development kits.

Huang plans to expand the chip’s features himself, but he also expects DefCon participants to build on what he’s provided and make the chips their own.

As we get closer to the release date for Kubernetes v1.37, the project develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This blog outlines some of the planned changes for the Kubernetes v1.37 release that the release team feels you should be aware of for the continued maintenance of your Kubernetes environment and keeping up to date with the latest changes. The information below reflects the current status of the v1.37 release and may change before the actual release date.

Deprecations and removals for Kubernetes v1.37

Kubectl: kubectl run --filename/-f to be deprecated

The --filename (or -f) flag for kubectl run is being deprecated as the generated pod is always built purely from CLI arguments like NAME and --image.

See kubernetes/kubernetes#138671 for the original issue and discussion.

Kubelet: Static Pods can no longer reference Secrets or ConfigMaps

Static Pods were never meant to read API resources directly, since they aren't created through the API server — but a bug let them reference Secrets or ConfigMaps via fields like configMapRef or secretRef. That bug is now fixed: as of v1.37 these references are strictly prohibited, and the PreventStaticPodAPIReferences feature gate that previously let you opt out of the restriction has been removed.

See kubernetes/kubernetes#140226 for the original issue and discussion.

Deprecating kube-proxy's support for ipvs mode

kube-proxy support for ipvs mode was introduced in v1.8 to resolve iptables performance bottlenecks. However, since the kernel ipvs API alone cannot fully implement Kubernetes Services, ipvs mode continues to use iptables underneath (KEP-3866, "The ipvs mode of kube-proxy will not save us").

Clusters running kube-proxy in ipvs mode (or mode: ipvs in KubeProxyConfiguration) would now be logging a deprecation warning on startup. The deprecation timeline looks like this:

• By v1.40, ipvs mode for kube-proxy is expected to be disabled by default (still selectable via the feature gate)
• By v1.43, support for ipvs mode would be removed entirely KEP-5495, Graduation Criteria. To confirm which mode you’re currently running, use:

kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

To understand the rationale behind this deprecation, see KEP-5495: Deprecate ipvs mode in kube-proxy.

Ongoing major changes

Future removal of cgroup v1 support

As modern Linux distributions and container runtimes use cgroup v2 as the default, support for the legacy cgroup v1 is officially being phased out. Since the v1.35 release, the failCgroupV1 setting has defaulted to true. Consequently, the kubelet will fail to initialize on any nodes that still rely on cgroup v1 unless an explicit configuration override is applied.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false # temporary override

Using this override should be considered a short-term fix. Advanced resource management capabilities, such as In-Place Pod Resizing and Tiered Memory Protection, depend entirely on cgroup v2. While the override remains available in Kubernetes v1.37, users are encouraged to migrate to cgroup v2, as support for cgroup v1 is planned to be removed in a future release.

To learn more about this deprecation, refer to KEP-5573: Remove cgroup v1 support.

Breaking changes in Kubernetes v1.37

SELinux volume relabeling ("SELinuxMount") graduates to GA

SELinuxMount is expected to reach GA and be enabled by default in v1.37. Volumes would then be mounted with -o context=<label> (the mount option default) instead of being recursively relabeled, but only when the volume's CSI driver opts in via a CSIDriver that sets .spec seLinuxMount: true.

Because a single mount can only hold one SELinux context, pods with different SELinux labels sharing a volume on the same node (which previously coexisted under recursive relabeling) may now fail to start. To retain the previous recursive behavior for a specific workload, set seLinuxChangePolicy: Recursive in the Pod spec.

Clusters without SELinux enabled see no effect at all. To learn more, check SELinux Volume Label Changes goes GA (and likely implications in v1.37)

Featured enhancements of Kubernetes v1.37

Metrics API goes GA

The metrics.k8s.io API is expected to graduate to Stable (GA) in Kubernetes v1.37 after spending nearly nine years in Beta. The API provides a standard way to retrieve CPU and memory usage for pods and nodes, powering widely used Kubernetes features such as the Horizontal Pod Autoscaler (HPA) and commands like kubectl top.

This graduation recognizes the API's stability and widespread adoption, with no functional changes expected. Both v1 and v1beta1 will remain usable during the transition, enabling developers to adopt the stable API at their own pace without breaking existing workflows.

To learn more about this enhancement, refer to KEP-5207: metrics.k8s.io API definition.

Kubelet in UserNS a.k.a. Rootless Mode

Traditionally, Kubernetes node components such as the kubelet run with root privileges on the host. While necessary for many deployments, this also means that a vulnerability in one of these components could potentially have a greater impact on the underlying system.

With Kubernetes v1.37, kubelet in User Namespace (Rootless Mode) is expected to graduate to Beta. This enhancement allows Kubernetes node components to run inside a Linux user namespace as an unprivileged user on the host while still behaving as root within the namespace. By reducing the need for host-level root privileges, it adds an extra layer of isolation and helps limit the impact of potential vulnerabilities affecting node components.

To learn more about this enhancement, refer to KEP-2033: Kubelet in UserNS(aka Rootless Mode).

Volume health monitor

Historically, Kubernetes has lacked an API for CSI drivers to report storage failures, which become evident only through failed mounts or hung I/O. Since remediation controllers had nothing machine-readable to act upon, the only way to figure out the root cause behind this failure was to cross-reference Kubernetes objects alongside external vendor dashboards.

In Kubernetes v1.37, this KEP resets graduation to Alpha after an initial implementation in v1.21 and introduces four new CSI RPCs. The controller plugin reports the health of storage volumes using ControllerListVolumeHealth (lists unhealthy volumes) and ControllerGetVolumeHealth (checks a specific volume). A controller-side health monitor polls these CSI controllers and stores the results in PersistentVolumeClaim.status.healthStatus.

On the node side, the kubelet calls NodeGetVolumeHealth to obtain the health of individual volumes on that node and records it in Pod.status.volumeHealth, while NodeGetStorageHealth reports the health of the drivers registered to a node in CSINode.status.storageHealth.

The error vocabulary is kept simple, extensible, and machine-parsable (Inaccessible, Degraded, etc.), with further driver-specific elaboration available via reason and message. Finally, the controller-side and node-side reports are kept independent and are hence displayed separately, providing a more holistic view of storage health to consumers.

To learn more about this enhancement, refer to KEP-1432: Volume Health Monitor.

Want to know more?

New features and deprecations are also announced in the Kubernetes release notes. We will formally announce what's new in Kubernetes v1.37 as part of the CHANGELOG for that release.

Kubernetes v1.37 release is planned for Wednesday, August 26th, 2026. Stay tuned for updates!

Get involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests.

If you don't know where to start, join our monthly New Contributor Orientations where we teach the community how the project is structured, and we'll guide you on how to make your first contribution to the project.

Full article content is not available for inline reading.

Read the original article →

How CloudFormation express mode accelerates your development cycle

AWS CloudFormation helps you model and provision cloud infrastructure as code using JSON or YAML templates, or through tools like the AWS Cloud Development Kit (CDK) and AWS Serverless Application Model (SAM). Whether you create stacks directly, use change sets for preview, or deploy through CI/CD pipelines and AI agents, the speed of your deployment cycle directly impacts how fast you can iterate.

In March 2024, we published How we sped up AWS CloudFormation deployments with optimistic stabilization, where we explained how CloudFormation provisions resources and what happens during stabilization. That post introduced the CONFIGURATION_COMPLETE event and the optimistic stabilization strategy that reduced deployment times by up to 40%. Today, CloudFormation express mode takes this further.

Express mode is recommended for development workflows where you iterate frequently. It makes your deployments complete faster so you can get immediate feedback. For production deployments where you need resources ready to serve traffic before proceeding, the default behavior remains the right choice. Combined with pre-deployment validation, which catches template errors before provisioning begins, Express mode completes the iterate-faster picture: validate in seconds, deploy in seconds. To understand what express mode gives you, you first need to understand what CloudFormation has always done during deployment.

What CloudFormation does during deployment

When you add an AWS::SQS::Queue to your template and deploy, the queue is created. But can it receive messages yet? Often, no. There’s a window between “resource created” and “resource can serve traffic.” This is true across AWS services. An EC2 instance is “launched” before it can respond to HTTP requests. A CloudFront distribution is “created” before it propagates to edge locations worldwide. An ECS service is “active” before its containers pass health checks and reach desired capacity. And a Lambda function deletion isn’t complete until its network interfaces are cleaned up.

This process is called stabilization. Stabilization means that when a stack operation reports CREATE_COMPLETE, the resources can serve traffic. This is useful for production pipelines where “stack complete” should mean “ready to shift traffic.” Now, with express mode, you have a second option.

What Express mode changes

Express mode gives you control over when you want CloudFormation to report completion. With express mode, CloudFormation completes the stack operation as soon as resource configuration is applied. Resources continue becoming ready to serve traffic in the background. CloudFormation communicates this per resource with a status reason: “Resource operation completed using express mode. It may continue becoming available in the background.” You always know which resources have completed configuration and which are still stabilizing. Regardless of mode, CloudFormation still:

• Respects resource dependencies within the stack – if a resource references another resource’s ID or attribute, the referenced resource’s configuration is confirmed first. Resources with no dependencies on each other proceed in parallel.
• Creates, updates, or deletes each resource the same way
• Retries dependent resources that encounter transient failures during the operation

The resource takes the same time to become ready to serve traffic in both cases. Express mode makes your iteration cycle faster by not blocking you on the stabilization wait.

When this matters

Development iteration

You’re building a VPC with subnets, a security group, and an ALB. You need the ALB’s DNS name and the security group ID to configure the next layer of your application. You don’t need to send traffic to the ALB right now. You just need to know it exists and get its attributes. With Express mode, you get the ARN, the DNS name, and the security group bindings in seconds. You proceed to your next iteration.

AI agent workflows

An AI agent iterating on infrastructure needs a tight feedback loop: deploy, observe the result, adjust, deploy again. The agent doesn’t need the CloudFront distribution to propagate globally before deciding whether the template is correct. It needs confirmation that the configuration was accepted.

Express mode turns a 5-10 minute CloudFront deployment into a sub-minute confirmation. The agent can validate, refine, and redeploy multiple times in the window a single default deployment would have taken.

Dependent stack deployments

When you deploy multiple stacks in sequence, each stack operation completes faster with Express mode. Whether dependencies exist within a stack or across stacks via import/export, Express mode handles retries and waits to make sure dependent resources can still be provisioned. It moves faster without breaking your dependency chain

Getting started

Enable Express mode per operation with a single parameter:

aws cloudformation create-stack \   
     --stack-name my-app \   
     --template-body file://template.yaml \   
     --deployment-config '{"mode": "EXPRESS"}'

Express mode disables rollback by default for faster iteration. If a resource fails to configure, the stack stays in place and you can fix and retry immediately. To re-enable rollback:

--deployment-config '{"mode": "EXPRESS", "disableRollback": false}'

With CDK

cdk deploy --express 

#enable rollback 
cdk deploy --express --rollback

With SAM

Use the --express flag with sam deploy or sam sync:

sam deploy --express 

sam sync --express

To persist the setting, add --save-params and Express mode is saved to your samconfig.toml:

sam deploy --express --save-params

The --disable-rollback flag works alongside --express to control rollback behavior within the deployment configuration.

No template changes. No new resource types. The same template deploys the same resources. You’re choosing when to receive the “done” signal.

Additional considerations

Change sets: Express mode is supported with change sets. Specify --deployment-config at create-change-set time, and the configuration is stored with the change set and applied when executed.

Nested stacks: When you enable Express mode on a parent stack, it propagates to all nested stacks in the hierarchy. All resources across the hierarchy complete when configuration is applied.

Conclusion

Express mode gives you immediate confirmation that your infrastructure configuration is applied, so you can move to your next iteration without waiting for stabilization. It separates “is my configuration correct” from “are my resources serving traffic” and lets you decide which question you need answered right now.

For development iteration, AI agent workflows, and dependent stack deployments where you need resource identifiers rather than traffic readiness, Express mode delivers that answer much more quickly. For production deployments where “stack complete” should mean “ready to serve traffic,” the default behavior remains the right choice.

Faster floating point math with Rust’s new API

Floating point math is often slower than integer math because the compiler is being conservative about how it optimizes your code. While some programming languages already had solutions of a sort, until now Rust did not have a good stable way to deal with this limitation. But now, starting in version 1.98, Rust will allow telling the compiler it can optimize your code further—but with extra control so that you can still write numeric algorithms with minimal rounding errors.

In this article you will learn:

• Why by default the compiler won’t optimize floating point math as much as it does integer math.
• Rust’s new API to solve this limitation.
• Examples of using this new API, its speed impact, and how you can control where it is used.

Summing integers is fast

I’m going to start with an example using integers, as a baseline of what sort of performance is possible.

To get the fastest code generation, I’m telling Rust that it’s not 2004 anymore, and that it can generate CPU instructions that require modern hardware, namely x86-64 machines from the past 10 years or so. Specifically, all the code in this article is being compiled with RUSTFLAGS="-C target-cpu=x86-64-v3". (For maximum compatibility, in real-world usage you could provide a fallback implementation for older computers.)

Here’s a Rust function to sum a slice of int64 numbers:

fn naive_sum_i64(values: &[i64]) -> i64 {
    let mut total = 0;
    for value in values {
        total += value;
    }
    total
}

To benchmark it, I’ll create an array of integers in NumPy:

import numpy as np

DATA_INT = np.ones((1_000_000,), dtype=np.int64)
assert naive_sum_i64(DATA_INT) == 1_000_000

And now I can measure the speed of summing this array:

That’s 0.5 CPU instructions per value! How does that even work?

Probably the compiler is using specialized Single Instruction, Multiple Data (SIMD) CPU instructions, that do batch operations on multiple values at once. The i7-12700K CPU I’m using here has 256-bit SIMD instructions, meaning it can do some specific operations on four 64-bit integers at a time. If there’s a specialized SIMD summing CPU instruction, the CPU would only need to loop 250,000 times and then sum 4 integers in each iteration.

Summing floats is slow?!

But what about floats—are they fast too?

Again, I’ll create a million floating point values:

# Array of 1M float64 values between 0 and 1.
DATA = np.random.random((1_000_000,))

I’ll implement a simple floating point sum function:

fn naive_sum(values: &[f64]) -> f64 {
    let mut total = 0.0;
    for value in values {
        total += value;
    }
    total
}

The floating point sum is much slower than the integer sum, and the compiler didn’t use SIMD float operations. Why the difference?

Floating point operations aren’t associative

Like most compilers, when Rust compiles your code in release mode it optimizes your code, transforming it in a variety of ways to (hopefully) make it faster. But there’s a promise compilers make when they do this: the optimized code will behave exactly the same as the unoptimized code.

If I add three integers a, b, and c, a + (b + c) == (a + b) + c. That gives the compiler plenty of scope to optimize how the code runs, for example by using SIMD operations that might slightly change the order of additions.

Floating point numbers are different. For example, because floating point numbers span such a range of values, from tiny to huge, adding a sufficiently large number to a sufficiently small number results in that same large number.

Rust’s new algebraic operators: telling the compiler when to be flexible

While conservatism on the part of the compiler is the right default, sometimes you as the programmer know that re-ordering operations isn’t a problem. In that situation, it would be good to be able to tell the compiler that while over here it should not change the order of operations, over there it’s actually fine.

Starting in Rust 1.98 there is a new feature that allows just that. In addition to the normal arithmetic operations you can do on floating point numbers, there are new set of so-called “algebraic” arithmetic operators that per the documentation “allow the compiler to optimize floating point operations using all the usual algebraic properties of real numbers”, including changing the order of operations.

An example: Optimized pairwise summation

Let’s see these operators in action, and how they allow for fast code.

Let’s implement this algorithm in Rust. When adding those top two floating point numbers, I’m going to use normal addition, where the compiler can’t reorder operations or otherwise do anything that would change the output. Once the function hits the threshold of doing normal summing, I’ll switch to algebraic adds because at this point I don’t care what the order of addition is, I just want speed.

fn pairwise_sum(values: &[f64]) -> f64 {
    let n = values.len();
    if n > 128 {
        let half = n / 2;
        pairwise_sum(&values[0..half])
            + pairwise_sum(&values[half..n])
    } else {
        let mut total: f64 = 0.0;
        for value in values {
            total = total.algebraic_add(*value);
        }
        total
    }
}

Another example: Sum of squared differences

You can do more than just add numbers with algebraic operators. In the following examples I will calculate the sum of squared differences between two arrays.

fn ssd_optimized(arr1: &[f64], arr2: &[f64]) -> f64 {
    assert_eq!(arr1.len(), arr2.len());
    let mut total: f64 = 0.0;
    for (val1, val2) in arr1.iter().zip(arr2) {
        let diff = val1.algebraic_sub(*val2);
        let squared_diff = diff.algebraic_mul(diff);
        total = total.algebraic_add(squared_diff);
    }
    total
}

Go speed up some numeric code!

The pairwise summation algorithm is a great example of why you want both kinds of operators, strict and lenient:

• If you only use strict in-order operations, the code is slower.
• If you only use lenient algebraic optimize-as-you-like operations, the compiler might optimize the algorithm out of existence, losing the accuracy the algorithm aims for.

The implementation I showed above benefits from using both normal addition—for accuracy—and algebraic addition—for speed—in different parts of the algorithm.

If you are writing numeric code with Rust, your code may benefit too—give it a try once you can use Rust 1.98. And if you’re not using Rust yet, this is another good reason to switch.

Docker OIDC connections for GitHub Actions available for Docker Orgs

TL;DR: Docker now supports OpenID Connect (OIDC) for GitHub Actions. Your workflows can authenticate with short-lived, per-run tokens instead of stored PATs or OATs. No secrets to rotate, no credentials to leak.

GitHub OIDC connections are available to organizations with Docker Team, Docker Business, or Docker Hardened Images (DHI) subscriptions, as well as organizations enrolled in the Docker Sponsored Open Source Program (DSOS).

OIDC token exchange flow between GitHub Actions and Docker

The problem with stored credentials

Every GitHub Actions workflow that pushes or pulls images from Docker Hub authenticates with a personal access token (PAT) or organization access token (OAT) stored as a GitHub secret. These credentials are long-lived. Someone has to remember to rotate them. A leaked token grants access to your registry — pulling private images, pushing malicious ones — and that access persists until someone discovers and revokes it. Rotation is manual and does not scale. As pipelines multiply, so do the credentials that need tracking, and stale tokens are a common audit finding.

Who should use this

1. GitHub issues a signed identity token (a JWT) that encodes the repository, branch, environment, and other metadata about the workflow run.
2. The workflow calls docker/login-action, which presents this token to Docker.
3. Docker verifies the token’s signature against GitHub’s public key registry and checks it against rulesets configured in the Admin Console.
4. If the token matches a ruleset, Docker returns a short-lived access token scoped to the resources defined in that ruleset.
5. docker/login-action uses this token to authenticate to Docker Hub. From there, docker pull, docker push, and docker build commands work as usual.

The entire exchange happens without any stored secrets, API keys, or access tokens. The short-lived Docker access token expires in minutes and cannot be reused.

This is the same pattern that AWS and GCP already use for cloud resource access, Docker is applying it to container registry access.

Getting started

Setup is a one-time connection in Docker Home plus a small update to your workflow YAML.

Step 1: Create a connection

Sign in to Docker Home, select your organization, and navigate to OIDC connections. Select Create OIDC connection and configure the rulesets that control which repositories, branches, and workflows can access which Docker Hub resources. You can create up to five rulesets per connection. When a workflow triggers an OIDC exchange, Docker checks the token against every ruleset defined in your connection. If a ruleset’s conditions are satisfied, Docker grants access based on the parameters set by that ruleset.

Rulesets use OIDC subject claims to match incoming tokens. You can pin to specific repos and branches as a recommended security best practice:

• repo:my-org/my-repo:ref:refs/heads/main — only the main branch of a specific repo
• repo:my-org/my-repo:ref:refs/heads/release-* — all release branches
• repo:my-org/my-repo:* – all branches of this repo
• repo:my-org/* — any repo in the organization (not recommended)

Copy the connection ID when you are done.

Step 2: Update your workflow

Update your GitHub Actions workflow. Replace <YOUR_CONNECTION_ID> with the ID from the previous step and <YOUR_ORG_NAME> with your Docker organization name:

permissions:
  contents: read
  id-token: write

steps:
    - name: Docker login                                                                                                                                                                 
      uses: docker/login-action@v4 # v4.5.0+                                                                                                                                                                
      with:                           
        username: <YOUR_ORG_NAME>
      env:                                                                                                                                                                               
        DOCKERHUB_OIDC_CONNECTIONID: <YOUR_CONNECTION_ID>

The id-token: write permission lets the workflow request a GitHub OIDC token. The docker/login-action handles the token exchange and Docker login in a single step when DOCKERHUB_OIDC_CONNECTIONID is set. From there, docker pull, docker push, and docker build commands work as usual.

Step 3: Verify the OIDC connection works

Run your workflow and confirm it completes successfully. If you encounter an error, the Failures tab of the OIDC connection page will show the details of the incoming claim sub value, which you can use to diagnose why the connection failed.

Step 4: Remove the stored credential

After verifying your workflow runs successfully with OIDC, remove the old PAT or OAT from your GitHub repository secrets. You no longer need it.

What doesn’t change

• Existing PATs and OATs keep working. Organizations can migrate workflows to OIDC connections at their own pace.
• Images, registries, and build workflows are unchanged. OIDC connections only replace the authentication step; everything downstream is the same.
• Local development and non-GitHub CI still use PATs and OATs. OIDC connections are the recommended replacement for GitHub Actions specifically. Other CI providers will follow based on demand.

Learn more

• Learn more about OpenID Connect
• Visit Docker Home to get started
• Read the documentation

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Asynchronous I/O in DuckDB: Work, Thread, Work

TL;DR: Starting with v2.0, scheduled for fall 2026, DuckDB will support asynchronous reads of Parquet and CSV files. This can significantly speed up queries when synchronous I/O does not saturate the available bandwidth, as is typical in EC2/S3 compute-storage setups.

It doesn't matter how fast query operators are in a database system if we can't pull in the data quickly. For most of DuckDB's history, however, this problem was largely avoided by pruning data early. By pushing down filters and projections, we could ensure that we only read what we actually needed.

This worked particularly well because DuckDB primarily ran locally, with its main use case being as a quick-draw database engine for querying data directly from your machine's SSD. We could split the data into several partitions, such as row groups for Parquet files or fixed-size buffers for CSV files, and load them with low latency and high bandwidth. As a result, the main bottlenecks were elsewhere: subqueries, joins, aggregations, and so on. The actual data access path received less attention because synchronous access was perfectly suitable for this use case.

As usual, things changed. We realized that DuckDB's architecture was a great fit for querying remotely stored large-scale datasets, such as data lakes. Since May this year, we can even run DuckDB as a server using the Quack protocol. The original expectation of data files sitting on a local SSD therefore no longer always holds.

The practical implication of these changes is that many current DuckDB setups need to transfer files from remote storage to the machine that will actually process them. For data lakes, for example, a typical setup is to store the data in blob storage, such as S3, and process it on an EC2 machine in the same region. In this setup, latency and bandwidth play a much more significant role. If we cannot issue enough concurrent requests to use the available network bandwidth, performance can suffer drastically, with threads spending a large amount of their time waiting for remote reads instead of processing data.

As an example, let's consider a simple query over a remote Parquet file. For simplicity, let's assume we only have a single thread executing.

FROM read_parquet('s3://bucket/file.parquet');

A Parquet scan is partitioned into row-group-based jobs, with each job containing one or more fetch tasks that issue byte-range requests. With synchronous I/O, the worker thread will be blocked, waiting for the data to arrive at the machine before performing actual work, such as decoding, aggregating, and so on.

To address this, we have been implementing asynchronous I/O pipelines in DuckDB. They are currently implemented for Parquet and for uncompressed, seekable UTF-8 CSV files, with support for other formats, such as DuckDB's native format and JSON, still to come. In the remainder of this blog post, we will give a simple explanation of how asynchronous I/O is implemented in DuckDB and provide benchmarks for both Parquet and CSV files.

If you would like to try asynchronous I/O now, you can do so by using DuckDB's v2.0.0-dev preview builds. Asynchronous I/O will be used by default from the next major DuckDB version, v2.0, released in the fall.

Asynchronous I/O

The conceptual idea of asynchronous I/O is rather simple: we should be able to start an I/O operation without blocking the worker thread that requested it.

In this example, we have two ASYNC threads and one regular worker thread. The ASYNC threads keep fetch tasks in flight while the worker thread decodes data. During the initial warm-up, the scan task parks, leaving the worker thread free to run other pipeline tasks. Once the first job is ready, fetching and decoding can overlap.

In DuckDB, we implemented two separate thread pools:

• REGULAR – This pool contains our worker threads (by default: one for each available CPU thread). These are the ones that do real work, like decoding, joins, and aggregations. They prioritize regular work but can also perform I/O tasks when idle.
• ASYNC – A pool of threads intended for asynchronous tasks, primarily blocking I/O.

The main reason we have these two different pools is that, for remote I/O, these threads can spend almost all their time blocked, waiting for an HTTP response, for example, and hence have very little CPU utilization. Because of that, we have many more ASYNC workers than system threads, with the default setting being 4 * system threads and the total being capped at 256.

It's of utmost importance to keep as many of our ASYNC threads busy as possible. To ensure that, we implement a read-ahead strategy instead of issuing reads on demand. This means scheduling fetch tasks ahead of what our regular worker threads currently need.

One thing we need to be attentive to is that read-ahead buys throughput by holding memory. If decoding is slow and the network is fast, prefetched data can accumulate and lead to out-of-memory issues. To mitigate this, we also implemented asynchronous memory governance.

Read-Ahead Queue

The idea of read-ahead is also straightforward. Instead of starting a read at the exact moment a regular worker needs the data, we schedule fetch tasks for work that lies further ahead. While a regular worker decodes the current job, the ASYNC threads are already pulling in data for the next jobs. The goal is to keep enough fetch tasks in flight to hide the latency of remote storage.

The jobs are units of work that can be scheduled and processed independently, and they can be different depending on the underlying file format. For a Parquet file, a job is one row group of one file. For a CSV file, a job is a scan boundary that generally covers a fixed byte range within the file.

Filling the queue requires no dedicated producer thread. Any regular worker that comes looking for scan work first tops up the queue as far as it is allowed to. The limit is either given by a user-specified number of slots or by a memory budget. If there is space, a job and its fetch tasks are created. The fetch tasks are scheduled immediately on the ASYNC pool, while the job is admitted to the read-ahead queue in batch order.

ASYNC threads execute individual fetch tasks independently of the job queue's claim order. Fetch tasks from the same job can run concurrently, although no particular assignment to ASYNC threads is guaranteed. All fetch tasks of a job share a countdown, and the fetch task that brings it to zero completes the job's I/O.

Memory Management

Keeping more fetch tasks in flight consumes more memory. To determine a budget and avoid out-of-memory issues, we introduced the read_ahead_depth configuration option.

• -1 (default): unlimited depth, bounded by memory.
• N > 0: at most N jobs ahead, with no memory budget.
• 0: read-ahead is off, each scan task schedules I/O only for its own job.

To configure it, use the SET clause, e.g.:

SET read_ahead_depth = 5;

In the default mode, the budget is negotiated with the temporary memory manager, which is the same manager that splits memory between concurrent joins, sorts, and window operators. When there is a lot of memory pressure, for example, because an operator is using a large amount of memory, queue reservations might instantly be over budget. In practice, this means that the queue will only allow one job at a time, and the scan will behave close to a synchronous scan.

Benchmarks

Asynchronous I/O should have the largest effect when the latency of synchronous requests prevents us from using the available remote bandwidth. To measure this effect, we ran TPC-H Query 6 at SF100, with the data sitting on S3. For compute, we used an EC2 r7i.16xlarge machine (64 vCPUs and 512 GB of RAM), with both the machine and the S3 bucket with the data located in the same region.

Parquet

Local Disk

Remote storage is the main target for asynchronous I/O, but cold local reads give us a useful contrast. We cleared the OS caches between runs.

Small Files

We generated 976 files with five row groups each.

Large Row Groups

CSV

Conclusion

In this blog post, we presented the recent work on asynchronous I/O for Parquet and CSV files. Most of its benefit comes from accessing remote data, but cold local reads can also benefit, albeit less. Next, we plan to add async reads for JSON and DuckDB-native files. We will also investigate io_uring, Linux's asynchronous I/O interface, which could reduce system-call overhead and the number of threads blocked on I/O.

Microsoft has had a narrow escape from total embarrassment: A security company uncovered a critical vulnerability that could have compromised all Azure Cosmos DB databases — both those of customers and Microsoft’s own.

Google subsidiary Wiz found a flaw in the database’s Gremlin API, usually used for storing and managing property graph data.

If bad actors had discovered it first, they could have exploited it to acquire what Wiz called the Cosmos Master Key, which would have enabled them to use the primary key of any Cosmos database, resulting in read and write access to any account. They would also have had access to a list of every database on the service, with identifiers such as subscription and tenant IDs.

Azure Cosmos DB is a NoSQL database that underpins Microsoft’s cloud services. It can be accessed through SDKs for framework such as Python, Node.js, Java, and .NET.

Wiz described how it discovered the vulnerability in a blog post. It disclosed details of the flaw to Microsoft in November 2025.

Microsoft deployed a hot fix within two days, but it took another eight months to re-engineer the infrastructure, removing the Cosmos Master Key and introducing new guardrails to Cosmos DB to prevent similar attacks.

It is not the first time Cosmos DB customers’ primary keys have been under threat: In 2021, Wiz found a flaw in data exploration tool Jupyter Notebook that could be exploited to access the database keys and other secrets.

Jakob’s law: How to apply it as AI collapses surfaces into one chat box

Assistants like Claude are pulling more tasks through a conversational front door. Jakob Nielsen’s law of familiar interfaces now points at the assistant itself with glee.

“Users spend most of their time on other sites.” — Jakob Nielsen, Jakob’s Law (2000)

Jakob Nielsen wrote his law in 2000, and it has outlasted almost everything else from that era of the web. Many of his commandments still hold, and not because technology stopped moving — it never does — but because people don’t change nearly as fast as their tools.

The implication: users expect your site to work like the ones they already know in the same way we expect automobiles the act a certain way i.e. if the steering wheel is a bunch of levers, I’m not going to want to rent or buy that car.

It’s why organizations stood up design systems teams — to hold to the best practices of application design, so every screen stayed aligned with what people already expected yet still innovate within the brand.

For 30 years that meant matching the conventions of the wider web, because that’s where users built their habits.

That’s changing fast.

More and more tasks no longer start at a website or an app — they start in an assistant. You ask Claude or ChatGPT to draft the email or pull the numbers, and the work happens through one conversational surface instead of a dozen disparate interfaces from different vendors and design teams.

The number of interfaces a person touches is falling, and the assistant is becoming the front door to tasks that each used to have their own destination across multiple browser tabs.

Their sameness is a feature because every assistant runs on the same pattern — an asking box, a reply, a back-and-forth — learning one teaches you all of them, so a new AI tool needs no manual and gets adopted faster than any onboarding flow could manage.

This is by design as we’re learning how to adjust for this new era.

Jakob’s Law usually reads as a warning against deviating; here it runs the other way, as the reason uptake is this quick. The more these interfaces resemble each other, the less anyone has to learn, and the more willing people are to try the next one which gives us a chance to be creative in inventing the future. This flips his law on its head.

That’s why design components and their semantic layer matter more than ever — they make conventions explicit and predictable, the clear expectations users need to move across sites and the legible structure an agent needs to read one.

That doesn’t retire Jakob’s Law; it moves the work to a different set of surfaces and requires even more standardzation.

If most use cases now flow through the assistant, the question for anyone building software isn’t only whether your product works when reached through that surface — it’s whether you’ve given the agent enough context to use it well by using standard patterns.

The Surface Count Is Falling

For most of computing’s history, software multiplied; every job got its own destination — an app, a site, a tool, each with its own interface to learn — until the home screen filled up and the browser tabs multiplied past counting.

Jakob’s Law existed precisely because there were so many surfaces, and users needed them to behave alike because a dropdown is a dropdown is a dropdown.

Assistants run the other direction, absorbing jobs into one place.

People now route work through whichever assistant they already have open — Anthropic Claude, Microsoft Copilot, Google Gemini, ChatGPT — instead of opening a different tool for each task.

It’s not only the standalone assistants, either because the chat tools people already live in have become conversational surfaces in their own right: Slack now runs AI agents directly in its channels, including third-party ones from Anthropic and others, and Microsoft Teams puts Copilot and its agents in the same window where coworkers talk.

Open-source projects push the same idea further: OpenClaw, a fast-growing self-hosted agent, drops a model of your choosing — Claude, GPT, Gemini — into whatever messaging app you already use, from Slack and Teams to Discord, WhatsApp, and Telegram.

The conversation people were already having with each other is now also where they talk to the software so the users don’t have to relearn it.

OpenAI lets third-party apps run inside ChatGPT itself, so you can search listings, build a deck, or pull up a playlist without leaving the conversation. Anthropic’s Model Context Protocol does the same work from the other side: it’s an open standard that connects an assistant to the systems where data and tools live, replacing a pile of one-off integrations with a single protocol. Claude reaches your calendar, your documents, and your issue tracker through that one connection.

The deeper shift is what the assistant does once it’s there.

It doesn’t just answer in one place — it goes out and does the interpreting. Claude Cowork takes a goal and works across your files and applications, synthesizing across many sources and handing back a finished deliverable, rather than answering one prompt at a time.

Point it at the work and it moves between the sources itself. The labor of visiting a dozen sites, reading each one, and piecing together what they mean — the exact cognitive cost Jakob’s Law set out to reduce — the agent now absorbs almost entirely. The user states the outcome; the interpreting happens out of view.

The labor of visiting a dozen sites, reading each one, and piecing together what they mean — the exact cognitive cost Jakob’s Law set out to reduce — the agent now absorbs almost entirely. The agent now packages the information for you, which is going to dramatically improve the enterprise experience once we get the APIs in place.

Put it together and the direction is clear. The number of surfaces in the world isn’t shrinking — software keeps multiplying as fast as ever; What’s falling is the number any one user touches directly, because the assistant has become the common surface and an agent increasingly does the cross-surface work the person used to do by hand.

We’ve watched a version of this before. The late-1990s web was a sprawl of directories and portals — Yahoo’s hand-built index of where to go. Then search collapsed all of that into a single box.

And sometimes the count drops to zero.

A surface doesn’t always shrink into the assistant — sometimes it never appears at all, because the work has gone ambient. An agent watching your inbox files the receipt, flags the renewal, and reschedules the meeting in the background, and the first you hear of it is a summary after the fact, if that.

There’s no screen to open and no conversation to start, so the interface Jakob’s Law was written about simply isn’t there — which is a harder design problem than a familiar screen, not an easier one.

Action items

• Map the journey across surfaces. Lay out the user’s end-to-end journey, then mark which steps now happen in an assistant, which still happen in your product, and where the handoffs fall. The handoffs are where expectations break and where your design work moves.
• Watch where tasks start. For a week, note when you reach for an assistant before the app you’d once have opened — your own behavior is the leading indicator for your users’, and sets expectations immediately.
• Stop counting screens and reducing clicks as wins. Retire the metrics that reward time-in-app and page views. If the assistant does the task, those numbers fall even when you’re winning.

Jakob’s Law Points at the Assistant Now

Jakob’s Law is a claim about reference points and users build mental models in the places they spend their time, then carry those expectations everywhere else. When the time was spent across many websites, the reference set was the web’s shared furniture — logo top-left, search top-right, cart and notifications in the corner.

When the time is spent in an assistant, the reference set becomes the assistant.

You type a request in plain language, get something back, and refine it in conversation, expecting it to remember what you said two turns ago. Hundreds of millions of people now do this every week, and it’s becoming the default expectation for how you get a computer to do something.

That’s the point — assistants are becoming the default in many cases, which helps with adoption.

It’s no longer just “work like other websites.” It’s “work the way the assistant works,” because that’s the interaction your users are fluent in now. I’ve called the plain-language input the asking box in earlier writing; the larger point is that the asking box is no longer one pattern among many. It’s becoming the pattern — the entry point a growing share of tasks pass through.

This is Jakob’s Law doing exactly what it always did. It’s just that the place users spend their time consolidated, so the expectations consolidated with it.

Action items

• Learn the new conventions firsthand so you know how to design for them. Spend a week doing real work inside Claude, Copilot, Gemini, or ChatGPT, and write down the interaction patterns you come to expect. That list is your users’ new baseline.
• Audit your feature against them. Hold your own AI feature to that baseline and fix every place it behaves differently for no reason — difference without a reason is friction.
• Don’t reinvent the input. The asking box is a convention now; a clever custom take on it just taxes people to relearn something they already do fluently everywhere else.

You May Not Own the Surface Anymore

If a use case flows through the assistant, the interface your user sees is the assistant’s, not yours. They ask Claude to find every auto-renewal clause in their contracts, and Claude reaches your contract platform through a connector, does the work, and answers in the chat. The user never opens your application. They never see your navigation, your carefully built dashboard, your brand. Your product did the work and stayed invisible.

That’s the real consequence of fewer surfaces.

For a meaningful set of tasks, you don’t own the surface anymore and it looks like service design. The assistant does. Jakob’s Law still applies — but now it applies to the assistant’s interface, which you don’t control, and your job shifts from designing the destination to being legible through someone else’s.

For a meaningful set of tasks, you don’t own the surface anymore, the assistant does.

This is where the affordance problem Amelia Wattenberger flagged early comes back, relocated. She pointed out that a bare chat input has no affordances — the same rectangle looks like a search box, a login form, and a credit card field. When your product is reached through that rectangle, its capabilities are only as visible as the assistant makes them. Users won’t discover your features by clicking around. They’ll discover them only if the assistant knows your product can do the thing and surfaces it at the right moment. Users are not used to having the ability to ask what the agent does; they have to get the courage to ask, and even more so, know what the right questions are.

So the work moves to a different layer: clear names, well-structured actions, data and capabilities described so an assistant can find them and route to them.

Information architecture stops being about your sitemap and starts being about whether a model can understand what you offer and call it correctly; I call it bringing the experience to the user.

The deep, full-screen experience still matters for serious work, but first contact increasingly happens somewhere you don’t design.

And this should change how you think about consistency itself. The conventions that helped users now help the agent, for the same underlying reason. A model has its own version of Jakob’s Law: it was trained on the common patterns of the web, so it recognizes and operates a standard component — a labeled field, a conventional button, a familiar date picker — far more reliably than a bespoke one.

A novel interface is out of distribution for the agent the way an unfamiliar site is for a person; it has to guess, and guessing is where most tasks goes wrong. Consistency used to be a courtesy to users. It’s now a condition for the agent to act on their behalf without breaking things.

And for now, the agent is often using your actual screen to do it so they have to know the conventions.

The clean path is a purpose-built API or connector, but that coverage is patchy — most software still doesn’t expose its capabilities in a form an agent can call. So agents fall back to what they can always reach: the human interface. With computer use, Claude looks at the screen, moves the cursor, and clicks and types like a person, driving the same buttons and fields you built for people. Until APIs catch up to an agentic world, your visual interface is the agent’s interface too — and a clear, conventional screen becomes the difference between the agent finishing the task and fumbling it.

Action items

• Name things the way people ask for them. Rewrite your features and actions in the words a user would say out loud to create a better sematic layer, not the labels your database uses. The assistant can only route to what it can recognize.
• Favor conventional patterns over clever ones. Reach for the standard component before the bespoke one; the model has seen the convention a million times and is far likelier to operate it correctly.
• Expose and describe your capabilities. Make your actions reachable over the protocols assistants use, and describe what each one does and when to use it — so the assistant can both reach you and know to surface you at the right moment. If it isn’t connected and described, it isn’t discoverable.

A2UI Is How the Patterns Come Back In A Standardized Way

The collapsed surface has a real weakness: for the past two years it’s been mostly text. You ask, the assistant answers in prose, and the affordances that made graphical interfaces usable — the date picker, the slider, the seat map, the form that won’t let you submit a bad value — flatten into a paragraph you read and a sentence you type.

That’s the affordance gap from the last section, and it’s a step backward from interfaces we spent thirty years refining.

A2UI is the standard built to close it.

Introduced by Google at the end of 2025 as an open project for agent-driven interfaces, A2UI lets an agent render a real interface — a form, a chart, a booking flow — inside the conversation using the client application’s own native components. The agent describes what it needs as structured data, not executable code. Your app draws it with your component catalog, your styling, your design system. You decide which building blocks exist; the agent arranges them for the moment.

The affordances come back inside the assistant, without anyone leaving the conversation.

This isn’t a Google-only idea, which is how you know it’s a direction and not a single bet. On the Claude side, MCP Apps shipped in early 2026 as the first official extension to the Model Context Protocol, letting a tool return interactive UI that renders right in the conversation — dashboards, forms, multi-step workflows — across Claude, ChatGPT, and other clients. Two of the largest players arriving at the same answer from different directions: the conversation should be able to show real interface, not just describe it.

Here’s why that matters more than it looks. A2UI and its cousins are the mechanism that carries Jakob’s Law into the collapsed surface. Users get the conventions they already know — a date picker behaves like a date picker — instead of typing into a void and hoping.

It also resolves the ownership problem from the last section. You no longer have to choose between owning your destination and disappearing behind the assistant; your interface, your components, your conventions can show up inside a surface you don’t control.

And it standardizes the same way a design system does. A2UI does for the agentic layer what a design system does inside an organization: it fixes a shared vocabulary of components everyone builds against, so what gets rendered stays consistent and predictable no matter who assembles it. A design system keeps your product coherent for your team; A2UI keeps it coherent when an agent is doing the rendering. Same discipline, one level up.

Like the protocols around it, A2UI is a standard — Jakob’s Law operating one layer down, where everyone builds against the same language so the pieces fit. The interface conventions that make software usable finally get a way home.

Action items

• Find the moment text fails. Locate the one place where a prose reply isn’t enough — a form, a chart, a picker, a preview — and treat it as your first generative UI surface.
• Prototype it with your own components. Build that surface as an MCP App or A2UI view from your existing design system, so it renders inside the conversation looking and behaving like you, not like a generic widget.
• Build against the standard, not around it. Adopt A2UI or MCP Apps as they mature instead of inventing a private rendering scheme; the point of a standard is that any agent can render your interface without special-casing you.

Apply It: Design for the Front Door You Don’t Control

So how do you apply Jakob’s Law when the surface is collapsing into an assistant? Split the problem in two: the tasks that come through the assistant, and the ones that still bring people to you directly.

For the tasks that route through the assistant, conform. This is the core of the law, and it hasn’t changed: meet users where they are, in the interaction they already know.

• Make your product easy to invoke and easy to understand from the outside.
• Name your actions the way a person would ask for them, not the way your database does.
• Expose your capabilities through the connectors and protocols assistants use, so the front door can reach you — and where the task needs real interface, render it with generative UI like A2UI or MCP Apps, so users get your affordances and your design system inside the conversation rather than a wall of text.

If a user can describe what they want, your product should be reachable by that description and usable once it’s reached.

Name your actions the way a person would ask for them, not the way your database does.

For the tasks that still bring people to your own interface — the deep, high-stakes, exploratory work — keep matching the conventions of your domain and the wider web. Those users still carry expectations about undo, status, recovery, and the shape of their own field. Fewer surfaces doesn’t mean one surface. It means a smaller number of places that each have to be more coherent, not less.

Action items

• Run the never-opens test. Assume the user never opens your product and instead asks an assistant to do the job through it. List everything the assistant would lack — names, structure, actions, interface — and fix it.
• Hold your own surface for the deep work. For high-stakes, exploratory tasks that still bring people in directly, keep matching the conventions of your domain and the wider web: undo, status, recovery, the shape of the field. Fewer surfaces means each remaining one matters more.
• Design the no-surface case. When a task runs ambiently, there’s no screen to make familiar — so put the effort into the summary, the notification, and the undo, because those are the only touchpoints the user gets.

Jakob’s Law: It’s Always About Meeting The User Where Their Mental Model Is

Jakob’s Law was never about a particular interface, it was about respect for where users spend their attention, and the habits they build there. For most of the web’s life, that attention was scattered across thousands of sites, and the law told you to match them so each one cost the user less to interpret. Now an agent can do much of that interpreting for them.

The advice still holds — but the place it applies has moved.

The premise is shifting. Attention is consolidating into a small number of assistants that increasingly act as the front door to tasks that once each had their own home. The law doesn’t break. It points somewhere new — at the assistant, and at the growing share of work that reaches your product through it rather than at it.

The discipline is to design for both. Be reachable, legible, and easy to invoke through the surface you don’t own — and, with generative UI now arriving, able to show your own interface inside it. Keep your own surface coherent for the work that still demands it, because fewer surfaces makes each remaining one matter more.

Apply Jakob’s Law the way you always should have. Find out where your users really are. Right now, more of them are in the assistant — so meet them there, even when “there” isn’t your product at all.

The scale of AI-generated media can be hard to grasp.

Starling Lab, a research collaboration from Stanford University and the University of Southern California, estimates that it took until 1975—149 years after the invention of the camera—for humanity to create 1.5 billion images. It took generative AI just 18 months to do the same. And AI didn’t stop there.

This spring, Google announced at I/O that its tools had been used to create more than 100 billion AI images and videos in just a couple of years, and Google is far from the only source of AI content. The company coupled this staggering statistic with a slew of partnerships to expand the use of its SynthID watermarking technology, which can be used to label AI-generated content and potentially help people identify what’s real.

The company says SynthID is robust enough to survive edits, and we’re about to see that tested as OpenAI, Runway, Nvidia, and others begin using it. Can invisible watermarks solve the AI content conundrum?

An indelible-ish mark

There are currently two approaches to labeling AI content: invisible watermarks like SynthID and metadata schemas like the Coalition for Content Provenance and Authenticity (C2PA). Google uses both to label its AI content, but C2PA is not intended to hide in plain sight like a watermark. It’s cryptographically secure, so you can’t fake it, but C2PA is trivially easy to strip out. Simply editing and saving an image or taking a screenshot can remove C2PA.

Watermarks like SynthID are encoded in the pixels of an image or video or in the waveform of an audio clip. Content that gets passed around the Internet degrades from compression, resizing, and edits, but Google says SynthID should still be present even in well-worn memes. Google DeepMind scientist Pushmeet Kohli explained to Ars that the team put a great deal of work into ensuring SynthID was a durable way to label AI content.

“Through the whole development process, we sort of assumed that a technology like this will be attacked,” said Kohli. “So we did a lot of research in making SynthID robust to different kinds of transformations. Whether people are adding some sort of filter or cropping the image, we used these transformations and made sure that the detector was robust against them.”

Google has been reluctant to provide much technical detail on SynthID’s functionality beyond the original paper, so it’s worth testing these claims as the technology expands across the AI landscape. But who has the time to wait for AI images to degrade naturally as they get passed around the Internet?

I used the Python Pillow library to simulate data loss from repeated sharing and downloading at a vastly accelerated rate. The Python script chooses random compression and resizing values within set ranges to apply to the test image, then uses the output image as the basis for the next iteration.

This test focused on two types of AI images: one created entirely from scratch by the model and an original photo edited by AI. Both images generated by Nano Banana Pro include the SynthID watermark, and no amount of metadata editing can hide it.

I started with the images above for the original and edited tests. We can probably all agree that neither one is real—politics is not metal, and my dog is not part dragon (take my word for it). I ran these images through the image-crushing machine hundreds of times, turning the crisp originals into barely recognizable blobs. Every 50 generations, I also created cropped versions of the images (without reencoding), removing potential SynthID pixels to further weaken the detection.

After 300 generations of simulated sharing, we’re left with these full-frame images:

And look at that—the SynthID watermark still works on both of them. You can check for yourself by uploading the images to Gemini and asking for a SynthID check. You can even take a screenshot of the full image, and the system will still label it as SynthID because the special pixels transfer over to the new file.

Because SynthID pixels are spread throughout the image, they’re also resistant to cropping… at least to a point. This is where we finally find the limits of SynthID. After 300 compression generations, removing a few pixels from the border of our test images finally broke SynthID (both edited and fully AI). These versions (see below) have been cropped by 20 percent, but that’s enough to render SynthID undetectable. A larger 50 percent crop can break SynthID a bit earlier, at around 250 image compression iterations.

These images may be ugly, but there’s nothing (other than the content) to prove they’re AI.

Wary of watermarks

Based on my testing, SynthID can gradually degrade, but the images are uselessly blurry by the time that happens. So have we solved the problem of AI disinformation? Not so fast. It’s impressive that SynthID can survive aggressive edits and data loss, but AI watermarks have some notable issues.

SynthID is not the only AI watermarking technology. Meta released its Content Seal AI watermark recently, but Reuters found that simply cropping the images a bit would often eliminate the watermark. Just because SynthID seems solid right now doesn’t mean all attempts to invisibly label AI content will work.

Google also doesn’t claim that SynthID is invulnerable—the original paper on the technology notes that the watermark is not intended to withstand adversarial attacks. It’s entirely possible that someone with enough time and incentive will find a way to bypass SynthID. Some people have already claimed to have successfully hacked SynthID, but we have been unable to confirm that through testing. Google’s Kohli also says his team has been unable to replicate the supposed workarounds.

Still, security is never perfect. If SynthID is successfully bypassed, which seems plausible if it continues to grow in popularity, all those billions of AI images and videos could be unlabeled in a snap. Google may find itself in a cat-and-mouse game as it updates the standard to block those attacks. Google is quick to point out that SynthID is not a silver bullet for digital providence. But SynthID is becoming a much bigger target. Between Google and OpenAI, most of the consumer AI space is now using this technology.

The team behind SynthID has plans in place to combat hacks, but Kohli says Google needs to keep much of that to itself. “We can, of course, change the watermark,” said Kohli. “There are a lot of things at our disposal. I cannot publicly disclose many of these mitigations because they help us make sure that the whole system is protected.”

At least some of the security model for SynthID becomes apparent when you try to use it. Google doesn’t want to give people an easy path to attack SynthID, so access is intentionally limited. If you see a suspicious AI image, you have to ask Gemini to call the verifier—there’s no API or publicly accessible SynthID detector webpage.

That’s not the only limit, either. To prevent people from using successive AI checks to tune a bypass workflow, Google limits SynthID to “approximately 10 image checks” per day. In my testing, the verifier will lock you out even faster if you upload too many similar-looking images. If you’re suspicious that an image may be generated by AI more than a few times on a given day, you may be out of luck. At a time when politicians are gleefully using AI to defame and attack opponents, verifying the truth shouldn’t come with a cooldown period.

Even if you aren’t locked out of AI detection tools, which one are you supposed to use? As Google rolls out the technology to other firms, watermarks are quickly becoming fragmented. While companies like OpenAI and Runway have chosen to use the underlying SynthID technology to label AI content, the actual watermarks are different. For instance, Google’s detector doesn’t recognize OpenAI’s watermark, and OpenAI’s detector doesn’t recognize Google’s watermark. So not only is access to SynthID restricted, but you may run an AI image through multiple detectors and still find nothing because it was generated by a different system you don’t even know about.

A Google spokesperson says the team is aware that this experience is suboptimal, and it has vague plans to improve it. “We are actively collaborating with industry partners to make verification more accessible and unified. Our vision is to create a more interoperable ecosystem that allows for a more seamless experience across different platforms,” they said.

Out of the bottle

Maybe the usability issues with watermarks can be solved. But at best, it will be only one element of how we determine what’s real. Watermarks in AI will never be the best way for one simple reason: There will always be ways to create AI content without any labeling.

Countless companies offer image and video generation, and it’s unlikely they will all agree to use watermarking technology or even metadata labeling. More importantly, you don’t have to rely on Big Tech’s centralized systems to generate AI slop.

“I think the bigger problem here is not just the images that are being produced by the Googles and the OpenAIs of the world,” said Adam Rose, a fellow and senior advisor at Starling Lab. “The problem is people who can run their own models, who can do things on their own computers.”

Even today, there are plenty of open models that generate images with no labeling whatsoever. These AI models can be shared and improved upon indefinitely, even if most of the major AI players adopt a multifaceted approach to labeling AI content. The generative genie is out of the bottle, and there’s no stuffing it back in.

Since there will always be unlabeled AI content, you can’t count on watermarks to tell you what’s true. As AI labeling becomes more common, people may think an image or video that lacks one is automatically legit. That’s a dangerous but understandable mistake for someone to make. While the existence of invisible watermarks might give people some sense of security as they explore an increasingly distorted Internet, they aren’t a reliable way to know what’s true.

“The future that we live in will increasingly be a world of unlimited content,” said Rose. “It’s economics 101, supply and demand. Supply goes through the roof, and the value of content is going to continue to crash. But increasingly, there is going to be a demand for authenticated content, a way to prove that something is what it purports to be.”

The scale of AI content alone will make it impossible to label all of it. Google’s 100 billion SynthID images and videos are just the tip of an iceberg that has already eclipsed authentic content on the Internet. It may be smarter to focus on verifying truth rather than outing falsehoods.

“There’s a limit to how much content can be created with actual physical cameras in the world,” explained veteran photojournalist and Starling fellow Mike Caronna. “There’s no limit to how much synthetic content can be made. That’s why we want to pivot to what’s scarce, and what’s scarce is high-value, truthful information. If we can secure that and prove that it’s authentic, we can defend against the liar’s dividend.”

Maybe you haven’t heard that term, but you’ve experienced it. The liar’s dividend is a rhetorical strategy wherein someone is confronted with evidence of wrongdoing, and they claim the facts are manipulated. Or they simply shout “fake news.” It’s been a depressingly effective tactic for public figures to escape accountability, and it may become even more powerful as AI content proliferates: “That’s not me—it’s AI!”

That brings us back to C2PA. While it’s easy to scrub digital files, C2PA may actually offer a better way to determine what’s real. This and similar technologies are essentially tamper-evident seals for digital files. Cryptographically verifiable metadata can prove all that comparatively scarce real content is, well, real. That may be far more useful than looking for proof that something is AI-generated in a sea of slop with unknown origins.

The problem is that C2PA is still rare. Few phones and cameras create these digital fingerprints, and most websites and apps don’t expose C2PA data in images (you can look at C2PA data with various online tools, though). Google is actually ahead of the curve here. The company’s Pixel phones are the only mainstream cameras with deep C2PA integration. Both images and videos captured by devices like the Pixel 10 include cryptographically protected C2PA metadata that reveals when and how an image was captured. It also lists whether AI was used to alter the image.

In the future, we may have no choice but to treat content that lacks verifiable provenance data as suspect. That’s just the consequence of living in a world in which typing a few words can deliver an endless stream of misinformation. Labeling AI content is a good policy for tech firms, but it won’t save us.

Qwen 3.8-Max is now available. The open weights will be released next week. The model, which has 2.4 trillion parameters, delivers comprehensive improvements across coding, work, research, and long-horizon tasks. It can answer questions as well as complete complex tasks end-to-end with greater reliability.

Microsoft appears to be preparing its first native real-time voice model, referred to as MAI Realtime, which has surfaced as a hidden early-access entry in the company’s MAI Playground. The listing suggests a small group of partners already has hands-on access, and what is visible points to a bidirectional, full-duplex system, one that listens and speaks at the same time rather than trading turns, placing it in direct comparison with OpenAI’s GPT Live 1 or Sesame.

It supports English, German, Spanish, French, Italian, Portuguese, Japanese, Korean, Chinese, Dutch, Hindi, Indonesian, Arabic, Russian, Turkish, Vietnamese, and Thai.

Two voices are present so far, Victoria and Grant, both noticeably more natural than what Copilot’s voice mode currently delivers. Language can be pinned explicitly or left on automatic detection, and the model switches languages mid-conversation without losing its footing. Turn-taking is configurable through two listener options: a Switchboard mode built around an MAI-Ears endpointer driven by inline control tokens, and a deterministic setup that pairs silence-based endpointing with a Whisper semantic endpointer.

The practical difference between them is subtle in use, though interruptions are handled cleanly and response latency is low. The model does not sing or produce non-speech sounds, which keeps it squarely a conversational system rather than a general audio generator. A debug panel exposes live latency figures, model thoughts and processing steps, and sample sharing looks set to arrive for playground users once access widens.

That would fill a conspicuous gap. Every MAI speech model shipped so far runs in one direction: MAI-Voice-2 and its Flash variant for synthesis, and MAI-Transcribe-1.5 for recognition, while the speech-to-speech layer in Azure Speech’s Voice Live API still relies on the GPT-Realtime model. A first-party full-duplex model would close that dependency for Mustafa Suleyman’s superintelligence team, which shipped seven in-house models at Build 2026 and has been steadily swapping OpenAI components out of Copilot, Teams and Bing. Microsoft Foundry is the likely developer destination, with Copilot voice the obvious consumer surface, though no timeline has been attached to either.

smevals

A framework for running evals against small (and large) models

Installation

uv tool install smevals

Or pip install smevals, or just uvx smevals --help.

Vocabulary used by this project

The top-level concept is an Eval: a collection of Tasks used to determine how good a particular model or model-and-harness configuration is at a specific high-level capability, such as text-to-SQL, drawing a pelican riding a bicycle, or evaluating whether an implementation satisfies a provided specification.

Evals can optionally be grouped into Suites of related Evals, primarily as a mechanism for organizing them on disk.

An Eval is a collection of Tasks. These are the individual exercises that a model must complete for its abilities to be evaluated.

A Config describes the setup used to attempt Tasks. It specifies a model and may include model parameters, system prompts, tools and other settings.

To gather evidence, we create a Run. A Run is the immutable record of executing one Task against one Config using a Runner. A Runner is a reusable CLI program that may send prompts directly to a model or build on an agent harness such as Codex or Pi.

The same Task and Config can be executed multiple times, producing multiple Runs to help account for non-deterministic results - smevals run -n 5 tops each Task up to five Runs. Each Run includes a timestamp to help track these.

A Run whose Runner exits non-zero is a failed Run: a harness-level error such as a network failure, not evidence about the model. Failed Runs stay on disk for debugging but are never graded, are excluded from reports, and do not count towards -n targets.

Once we have gathered Runs, we apply a Grader to each Run to produce a Grade. A Grader is a configured sequence of Checks, plus rules for combining their results into that Grade.

Checks are individual assertions or measurements. Some may be simple, such as “does the output contain this text?” Others may be more complex, such as “render this SVG to an image and have an LLM judge assess it”. Each Check names the Checker that performs it, along with configuration for that Checker such as patterns, rubrics or expected values. A Check can be marked as required, in which case its failure halts the Grader and skips the remaining Checks.

A Checker is a named operation or a reusable CLI program that implements one kind of Check. contains and xml-valid are named operations, ../checkers/render-svg might be a custom program. The same Checker can be used by many Checks across many Graders. The Checks in a Grader execute in order and share a working directory, so a Checker may create files - such as a rendered image - which are kept with the Grade as artifacts and available to later Checks in the sequence.

A Grade is the result of applying a Grader to a Run. It records the result of each Check and may contain an overall pass/fail outcome and/or a numeric score. Grades can also include additional notes which are not used for scoring but may help interpret the results in the future.

We may later change the Grader we use to evaluate Runs without executing the Runs again. A single Run can therefore be evaluated multiple times, producing multiple Grades using different Graders.

Building an Eval

An Eval is any directory containing an eval.yaml file:

my-eval/
├── eval.yaml            # name and description
├── tasks/               # one YAML file per Task
├── configs/             # one YAML file per Config
├── graders/             # one YAML file per Grader
├── checkers/            # custom Checker executables (by convention)
├── run-llm              # Runner executable (any name, any location)
└── runs/                # created by smevals run - never edit by hand

Example Eval: Grading Haikus

Here's how to structure a complete Eval that asks models to write haikus and grades them on their structure.

The Eval consists of five files.

my-eval/eval.yaml defines a name and description:

name: haiku
description: >-
  Can the model write a haiku on demand? Graded on structure:
  the reply must be exactly three lines.

An Eval must have one or more Tasks. Each of these is defined as a tasks/*.yaml YAML file.

my-eval/tasks/pelicans.yaml must have a name; a prompt is the common case, but any other keys are allowed and are passed to the Runner as environment variables:

name: pelicans
prompt: Write a haiku about pelicans. Reply with only the haiku, three lines.

An Eval also needs at least one Config, defined in configs/*.yaml. If there is just one of these it should be called default.

my-eval/configs/default.yaml - the Config named default is used when no -c option is passed to smevals run.

runner specifies a path to an executable program relative to this file:

name: default
runner: ../run-llm
model: gpt-4.1-mini

Here's that Runner script:

my-eval/run-llm - This one uses the llm CLI, but any executable honoring the contract below works. Make it executable with chmod +x:

#!/usr/bin/env bash
set -euo pipefail

llm -m "$SMEVALS_MODEL" "$SMEVALS_PROMPT"
llm logs -c --json > log.json

The Eval also needs a default Grader, which will be used to grade the results of each Run:

my-eval/graders/default.yaml:

name: default
checks:
  - checker: ../checkers/three-lines
    required: true
scoring:
  pass_threshold: 1.0

The checker can be a relative path to a script - similar to runner: above - or can be the name of a built-in checker, listed below.

my-eval/checkers/three-lines - a custom Checker, also chmod +x:

#!/usr/bin/env python3
import json, os, pathlib, sys

raw = (pathlib.Path(os.environ["SMEVALS_RUN_DIR"]) / "output.txt").read_text()
lines = [line for line in raw.strip().splitlines() if line.strip()]
print(json.dumps({
    "score": 1.0 if len(lines) == 3 else 0.0,
    "metrics": {"line_count": len(lines)},
    "notes": f"{len(lines)} non-empty line(s)",
}))
sys.exit(0 if len(lines) == 3 else 1)

To run the eval, grade it and then view the results:

smevals run my-eval -g                 # run every task, grade as each finishes
smevals run my-eval -m gpt-4.1-nano -m gemini-2.5-flash -g   # more models
smevals run my-eval -n 5 -g            # top every task up to five graded runs
smevals report my-eval                 # markdown report in the terminal
smevals serve my-eval                  # live web UI on http://127.0.0.1:7001

The Runner contract

smevals run executes the Runner once per Task/model combination, with no arguments. Everything arrives through environment variables:

• SMEVALS_MODEL - the model to use, from the Config or the -m option.
• SMEVALS_TASK - the Task's name.
• SMEVALS_PROMPT - the Task's prompt, only set if the Task has one.
• SMEVALS_TASK_<KEY> - every scalar key of the Task, uppercased: a Task with submission: mutant-003 provides SMEVALS_TASK_SUBMISSION=mutant-003.
• SMEVALS_RUN_DIR - absolute path to the Run's directory.

The working directory is the Run's directory. The contract:

• Standard output is captured as the Run's output.txt - it should be the model's response.
• Standard error is captured as stderr.txt.
• A non-zero exit code marks the Run as failed. A failed Run is a harness error - a network drop, a crashed tool - not evidence about the model, so it is never graded, is excluded from reports, and does not count towards an -n target (re-running the same command executes a replacement). Exit non-zero only for infrastructure problems; exit 0 whenever the output is a real model response you want judged, however bad.
• Any other files the Runner writes to its working directory are kept as Run artifacts (the log.json in the example above).

A Runner that drives an agent harness instead of a plain model call follows the same contract: assemble whatever inputs the Task's keys describe, run the harness, print the final result to standard output.

Graders

A Grader is a YAML file in graders/:

name: default
checks:
  - checker: contains          # a built-in Checker, by name
    value: "<svg"
    required: true
  - checker: ../checkers/render-svg   # a custom Checker, by path
    input: extracted.svg
    creates: render.png        # smevals verifies this file gets created
    required: true
  - checker: ../checkers/llm-judge-image
    image: render.png
    model: gpt-4.1
    rubric: Score this image from 0 to 10 ...
scoring:
  pass_threshold: 0.5

Each entry in checks names a Checker plus its configuration. Reserved keys:

• checker - a built-in name, or a path to an executable relative to the Grader file.
• required - if true and the Check fails, grading halts and the remaining Checks are recorded as skipped.
• creates - a filename, or list of filenames, the Checker promises to create in the shared workspace; the Check fails if any of them do not appear. A Checker may write any number of additional files beyond those promised - everything in the workspace is kept as a Grade artifact.

All other keys are configuration for the Checker, passed through via environment variables.

Built-in Checkers

• contains - passes if the Run's output.txt contains value.
• xml-valid - passes if file (looked up in the grade workspace, then the Run directory) parses as well-formed XML.

Outcomes and scores

• The Grade's score is the last score emitted by any Check - typically the final, most expensive Check. However, if any Check fails without emitting a score of its own, the Grade's score is null: a stale score from an earlier Check never stands in for one that did not run.
• The outcome is fail if any Check failed, otherwise pass if the score meets scoring.pass_threshold (or if there is no threshold or no score).

The Checker contract

smevals grade executes each Check's Checker with no arguments and these environment variables:

• SMEVALS_RUN_DIR - absolute path to the Run directory being graded. Read the model's output from $SMEVALS_RUN_DIR/output.txt.
• SMEVALS_CHECK - the full Check configuration as JSON, for structured values like lists.
• SMEVALS_CHECK_<KEY> - every scalar key of the Check, uppercased: rubric: becomes SMEVALS_CHECK_RUBRIC.
• SMEVALS_TASK and SMEVALS_TASK_<KEY> - the Task's name and scalar keys, so a Checker can locate per-Task resources such as expected-answer files.

The working directory is the grade workspace, shared by all Checks in the Grader in order: files written by one Check (a rendered image, an extracted document) are available to later Checks and are kept with the Grade as artifacts.

A Checker signals pass or fail with its exit code (0 is a pass). It can also emit a JSON object on standard output with up to five keys, which are recorded in the Grade:

• score - a float from 0.0 to 1.0.
• metrics - an object mapping names to numbers or booleans, e.g. {"precision": 0.9, "status_correct": true}. Reports aggregate numbers as mean ± stderr and booleans as rates.
• tags - a list of short labels, e.g. ["wearing_a_hat", "correct_bicycle_frame_shape"]. Tags are open vocabulary and presence-only: an absent tag means "not observed", not "false". They are normalized to lowercase snake_case, and the Grade records the union of all its Checks' tags. Reports aggregate them as counts and shares, and the web UI uses them for filtering.
• notes - a human-readable string explaining the result. Never aggregated.
• details - an object of structured diagnostics, such as predicted-versus-expected lists. Kept with the Grade but ignored by aggregation.

Any other keys are folded into details. A Checker that fails may still emit a score (a partial-credit measurement); a Checker that crashes before scoring leaves the Grade unscored, as described above.

Runs and Grades on disk

runs/<task>/<config>/<model>/<timestamp>/
├── run.yaml         # the record: full task, resolved config, timing, exit code
├── output.txt       # the model's response (runner stdout)
├── stderr.txt       # only present if the runner wrote to stderr
├── ...              # any other artifacts the runner wrote
└── grades/
    └── <grader>/
        ├── grade.yaml     # outcome, score, tags, per-check results
        ├── grader.yaml    # snapshot of the Grader that produced this Grade
        └── ...            # artifacts written by Checkers

The model name is slugified for the path; the exact name is in run.yaml. run.yaml is written last, so its presence marks a complete Run. Runs are immutable - grading only ever adds files under grades/.

Each Grade includes a byte-for-byte snapshot of its Grader. smevals grade uses this for repeatability:

• By default it grades only Runs that have no Grade from the named Grader, and reports how many existing Grades were produced by an older version of the Grader spec.
• --regrade deletes and re-creates every Grade for that Grader, so nothing stale survives. Use it after editing a Grader.
• Multiple Graders coexist: each grades into its own grades/<name>/ directory, so an eval can have e.g. a cheap deterministic default grader and an LLM-judge judge grader side by side.

By default runs/ lives inside the Eval directory. Pass --runs-dir DIR to run, grade and report to keep runs elsewhere; they are then namespaced by Eval name.

Commands

smevals run EVAL [-m MODEL]... [-c CONFIG] [-t TASK]... [-n N] [-g [GRADER]] [--runs-dir DIR]

Executes every Task (or just those named with -t) against every model given with -m (default: the Config's model), using the Config named by -c (default: default). -g grades each Run the moment it finishes; -g NAME uses that Grader, bare -g uses default. Exits non-zero if any Run fails or grades as fail.

-n N is a target sample size: each task/model pair is topped up to at least N successful Runs, executing only the shortfall, so re-running the same command is a no-op once the target is met and an interrupted session can be resumed by repeating it. Runs execute in full passes over the pairs - interrupting partway leaves balanced samples rather than many Runs of the first Task and none of the last. Failed Runs (a non-zero Runner exit) do not count toward the target: re-running the command executes replacements for them, attempting each pair's shortfall once per invocation, so a persistently failing Runner never retries in a loop. Without -n, exactly one new Run is executed per pair.

smevals grade EVAL [-g GRADER] [--regrade] [--runs-dir DIR]

Applies the Grader to every ungraded Run. Failed Runs are skipped - a harness error is not evidence worth grading. --regrade discards and redoes existing Grades from this Grader.

smevals report EVAL [-g GRADER] [--by-task] [--json] [--runs-dir DIR]

Prints a markdown report: leaderboard of config × model with mean ± stderr scores and failure counts, tag shares, and per-model blocks with metrics. Failed Runs are excluded from all statistics; the header reports how many were left out. --by-task adds per-task scores. --json emits the raw grade rows instead.

smevals serve EVAL_OR_SUITE... [-p PORT] [--host HOST] [-g GRADER]

Serves a live web UI (default port 7001) over one or more Evals. Data is re-read from disk on every poll, so the pages update as new Runs and Grades land. A directory that is not itself an Eval is treated as a Suite and searched recursively for Evals.

smevals build EVAL_OR_SUITE... [-o DIR] [-g GRADER]

Builds the same web UI as a self-contained static site (default build/), copying run artifacts into it. Each invocation adds or refreshes the given Evals in the output directory and leaves other Evals already built there untouched, so one site can aggregate Evals from many repositories.

smevals docs

Outputs this document.

Ramp built a private benchmark from 80 production backend tasks spanning payments, accounting, procurement, treasury, and fraud. It scores review-ready patches that pass tests within 45 minutes, exposing model trade-offs across accuracy, latency, and cost without public-benchmark contamination.

Full article content is not available for inline reading.

Read the original article →

We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". As one idea to generalize it, I was interested what Opus 5 would do if I gave it the first paragraph of the Lord of the Rings, a 1M token budget (~$10) and asked for three js render of it. Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story. It's kind of janky but fun. But it's a bit mindboggling that the LLM has to place and orchestrate various polygon assets in (x,y,z) coordinates and write code that animates it all, and that it even does anything at all.

I also like this kind of examples because no one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world, so it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free". There might be a lot more. But I'm excited about creating hyper custom worlds that you can imagine dropping players into, e.g. here to participate in the LoTR story as a spectator NPC, or one of the characters, or etc. Something like an ephemeral GTA of X on demand.

Last thought is that the domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them. Here, Opus 5 had to very slowly and painstakingly take screenshots at different points, and it messed up a few times and created a bunch of jank. An example of raw capability (multimodal, gameplay) that I think is still quite lacking.

More on the pelican on the bicycle test from @simonw: simonwillison.net/2025/Jun/6/six…

I uploaded the source here so it's playable in the browser, forkable etc. karpathy.ai/lotr-movie/

Look out for GTA Hobbiton dropping before GTA VI :)

Introducing the AI Productivity Index for Accounting

Introducing APEX-Accounting, a new benchmark built in collaboration with Ramp.

Accounting benchmarks often test whether a model can produce the right answer once. Closing the books demands more. An agent must reconcile conflicting files, apply company-specific context, carry conclusions across multiple steps, and produce the correct result consistently. A model that drops a correct intermediate answer can still create a bad journal entry. APEX-Accounting builds on the evaluation system Ramp developed while building Stack, their AI operating platform for accountants. Ramp and Mercor built APEX-Accounting to test whether AI agents can complete real accounting work to professional standards. The benchmark includes 160 tasks across 10 simulated companies. Experts with prior experience at firms like Deloitte, PwC, EY, and KPMG hired through Mercor's platform created the tasks and grading rubrics.

The headline result is not which model leads, but how rarely any model succeeds consistently. We ran every model on every task eight times because accounting work must be repeatedly correct, not correct once. Even the most consistent model solved just 2.6% of tasks correctly in all eight runs.

This is a very different result from performance on accounting exams. Models can produce the right answer in a controlled test and still fail to carry accounting judgment through a real workflow.

Leaderboard results

Claude Fable 5 tops the leaderboard at 56.4%, followed by Meta's Muse Spark 1.1 at 52.6%, and GPT-5.6 Sol at 51.5%. The best models manage to complete just over half of the work that a professional would.

For Pass@8, which measures whether a model is successful in at least one of eight attempts, Muse Spark 1.1 is highest at 21.5%, just ahead of Fable 5 (20.1%).

Models frequently earned partial credit. At least one model earned some credit on more than 95% of tasks. Full solutions were more difficult. 58% of tasks were never fully solved by any model on any run.

Performance vs. cost

We tested models at spending budgets of $1, $5, $10, and $50 per task. A larger budget allows models to use more tokens, which generally improves results. But the effect varies enormously. Fable 5 was extremely budget sensitive, scoring 11.8% with a $1 budget and improving to 55.2% with a $50 budget. Muse Spark 1.1 is the opposite. It's already strong on a tight budget and barely improves with more money. This is because above the $1 cap, models typically do not fully utilize the token limits. There is only a moderate increase in token usage when going from $10 to $50 and, on average, models use just 64.7% of the maximum budget available at $50.

Models differ in how they utilize the capped budget. At the $50 maximum spending budget, Fable 5 actually spends ~$32 per run while Muse Spark 1.1 spends ~$5, yet their scores are within 4 percentage points.

What the leaderboard measures

The canonical leaderboard runs every model in the Loop Harness, Mercor's standard model-and-tools architecture. The paper separately compares it with a purpose-built Ramp Harness, created with Ramp to mirror parts of a specialized accounting-agent architecture. The Ramp Harness improves Mean Criteria@3 by 1.2 percentage points on average.

This is a harness ablation, not an evaluation of the complete Ramp Stack product. It does not measure Stack's production integrations, accounting skills, memory, controls, auditability, or recent spreadsheet tooling.

Why do models fail?

Working with accounting and bookkeeping experts, we developed a taxonomy of the mistakes that prevented models from completing each task. These included failures in reasoning, information gathering, instruction following, and planning and reflection.

The top three models all fail in much the same way. Roughly seven in ten failures came from flawed reasoning, not an inability to find the right information. A model might correctly identify a discrepancy early in the workflow, then omit or contradict that finding in its final journal entry. Better retrieval alone will not solve this. Models need stronger accounting judgment and more discipline in carrying conclusions through an entire workflow.

Methodology

More than 40 accounting professionals authored and solved the benchmark tasks. They had a median of 11 years of experience, and more than half had worked at a Big Four accounting firm. Each task takes place in a self-contained world: a fictional company frozen at month-end close, with its own accounts, records, business history, accounting software, spreadsheets, PDFs, and other files.

The companies themselves are fictional but every transaction, discrepancy, and edge case inside was written by accountants who do this work for a living. The experts authored a rubric for each task with, on average, 13.7 criteria. An open-source AI judge performs the grading and achieves 97% agreement with expert graders.

APEX-Accounting focuses on month-end close and bookkeeping workflows. It does not evaluate tax, audit, consolidation, multi-entity or multi-currency accounting, external reporting, or how agents respond when they need clarification. Its findings should be understood within that scope.

The APEX-Accounting leaderboard comprises a heldout test set of n=160 tasks (associated with 10 worlds), kept hidden to resist contamination. We have released an open-source sample set with 1 world and 10 tasks on Hugging Face, and Archipelago, our internal framework for running agent evaluations, is available on GitHub.

Read the APEX-Accounting technical report to learn more about the methodology and results.

APEX benchmarks

APEX-Accounting joins Mercor's family of APEX benchmarks, which evaluate AI models' ability to do economically valuable work. Other APEX benchmarks include APEX-SWE for software engineering tasks across integration and observability, and APEX-Agents for professional services domains including corporate law, management consulting, and investment banking.

We thank Ramp and all the accountants on the Mercor marketplace who contributed their time to creating APEX-Accounting.

As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request. Reach out here to find out more.

EXECUTIVE SUMMARY

As of this week in Montana, any biotech company with an experimental drug has a clear path to selling it to consumers. Companies whose drugs have been through preliminary testing—sometimes in as few as 10 healthy people—can pay $12,500 to apply to a newly established review board for approval. Once its treatment is rubber-stamped, the company can set the price of the drug and sell it via experimental treatment clinics, the first of which is likely to be up and running around the end of this year.

Montana’s latest right-to-try legislation is unique. While other jurisdictions with similar laws limit access to drugs to people with terminal illness, in Montana access is theoretically available to anyone who gives informed consent and can pay. That includes people desperate for treatments for rare diseases. It also includes those who are interested in longevity and want to try out drugs pitched as preventive therapies.

The state’s Department of Health and Human Services recently finalized rules to implement the law. The rules stipulate that patient consumers provide fully informed consent and that each application be reviewed by a board that includes a Montana-certified doctor, expert scientists, and an ethicist. Supporters of the law stress that they want the process to be responsible. “It will be done in a very rigorous way, with qualified medical professionals and appropriate oversight,” says Matt Kaeberlein, a scientist on the first board, which was formed independently of the state health department.

But other experts are worried about the potential for harm in selling unproven treatments to people without oversight from the US Food and Drug Administration. “I would be concerned,” says Aaron Kesselheim, a professor of medicine at Harvard Medical School with expertise in health policy and drug regulation.

There has been a growing movement to make unapproved drugs more accessible in the US. But the story of Montana’s law is unique. It’s been driven and drafted by longevity enthusiasts instead of the usual libertarian and patient groups.

An unusual origin story

Montana first passed a right-to-try law in 2015. In 2023, with the support of state senator Ken Bogner, the state expanded the law to include all patients, not those just with terminal disease. Last year, Bogner told MIT Technology Review that his vision was to focus “more on preventative medicine” rather than “just treating diseases once they show up.”

Bogner says he had “started working on a bill” that would become the 2023 law when the Alliance for Longevity Initiatives (A4LI), a nonprofit “dedicated to advancing legislation and policies aimed at increasing healthy human lifespan,” got in touch. A4LI connected Bogner with others who helped draft the bill and testified in support of it.

Once that law was in place, the tech entrepreneur and longevity enthusiast Niklas Anzinger got involved. Anzinger has been working to establish a jurisdiction to fast-track the search for drugs that might deliver radical life extension. He is based in Próspera—a private city and “special economic zone” in Roatán, Honduras, which is already home to a separate clinic that sells experimental stem-cell and gene therapies. Anzinger founded a community there called Infinita City; he has also founded an investment company and a “service providing” company, both of which include the name Infinita.

Over the last couple of years, Anzinger has switched his focus to the US. “Now we think that Montana is a better model, because it’s building on … existing regulatory precedents,” he says. Once Montana’s 2023 law was passed, he adds, he worked with a handful of unnamed biotech companies to draft a second bill—one that laid out the specific terms under which clinics can offer unapproved drugs. That law was passed in April 2025 and adopted the following month.

Since then, Anzinger, Bogner, and others have been waiting for the state’s Department of Health and Human Services to finalize specific rules for treatment centers—a set of operational guidelines and requirements that any clinic offering treatments unapproved by the FDA must meet under Montana’s law. “The rules have been taking a very long time,” says Anzinger. “Then on Friday, we heard they were effective … from Saturday [July 25].” The rules have since been published online.

Following the rules

With the new rules in hand, Anzinger and his colleague Stephen Martin, Infinita’s US lead, got to work. The first step was to establish an independent experimental treatment review board—a panel of five experts to evaluate applications for access. Anzinger and Martin started recruiting candidates months ago.

The state’s first board, named the Montana ETRB, was officially announced by Infinita earlier this week. For the time being, it is the state’s only review board, although Anzinger says that other groups are free to establish their own. After Bogner raised concerns that the board’s website wrongly implied that it was an official state body, the site was updated to note that “It is a private service run by Montana Governance Services Inc.” That company is “a local Montana registered entity, but it is under the Infinita umbrella,” says Anzinger.

Infinita will pay board members a flat fee, funded by the $12,500 companies will have to pay to have their applications reviewed. Anzinger stresses that the board members, and their decisions, will be independent of Infinita.

In accordance with the rules, the board includes a Montana-licensed doctor: James Burke, an oncologist. It also includes a bioethicist: Jessica Flanigan, a libertarian who is known for her strong views in support of self-medication and her book Pharmaceutical Freedom.

The other three members are familiar faces in the longevity community—all of whom are well respected in the field. “When we looked at our own network, these were some of the best guys,” says Martin. They include Felipe Sierra, who formerly held a senior role at the National Institutes of Health’s arm focused on aging. More recently, Sierra served as chief scientific officer at Hevolution Foundation, a nonprofit that funds research into extending healthy lifespan with the support of the government of Saudi Arabia.

Matt Kaeberlein, who formerly led the Dog Aging Project and has studied the potential for rapamycin as a longevity therapeutic, also features. So does Jamie Justice, a gerontologist who is also executive director of the X Prize Healthspan competition, which has $101 million in prize money up for grabs for researchers who find ways to treat the signs of aging.

“I saw an opportunity to help build a safe, transparent, and scientifically rigorous process for implementing Montana's newly expanded right-to-try legislation, particularly as it applies to longevity medicines and aging-related interventions,” says Justice. “Science is moving quickly, and I wanted to help ensure that as it develops, it does so with real rigor and accountability.”

Kaeberlein, who has a prominent media presence, has long raised his own concerns about access to other unproven treatments, including peptides and stem-cell therapies. He sees Montana’s setup as offering a more regulated environment—one that offers scientific oversight, ensures informed consent, and allows for data collection.

Applications incoming

While many of the bill’s original supporters were interested in longevity, the initial interest in making drugs more accessible in Montana is coming from companies and individuals looking to treat specific diseases.

“We were actually surprised that much of the interest … is actually more from oncology [and] neurodegenerative disease,” says Anzinger. This focus, he says, is “very compatible” with Infinita’s mission. “We’re not trying to convince everyone … to support radical life extension,” he says. Anything that extends health and human life, including treating cancer and neurodegenerative disease, is part of what longevity means to him, he says.

Martin says that two applications have already been submitted to the newly formed review board. They’ve come from biotech companies that are developing drugs for neuropathy and hearing loss, he says. “I expect we’re going to get started on them this week,” he says.

One of the applications was submitted by Stanley Kim, CEO of WinSanTor. His company is developing a treatment for peripheral neuropathy, a painful nerve condition that can be a consequence of cancer treatment or diabetes. The drug is currently in phase II trials, but Kim says he regularly receives messages from people who are desperate to access it, to the point of being suicidal. He hopes that not only will he be able to make the drug accessible to those people, but he’ll also be able to collect data from them—data that might help accelerate the drug’s approval process.

“We have a newsletter [that is sent to] around 15,000 patients,” says Kim. “Not all of them will be able to go to Montana, but many of them, I think, will.” His company still plans to continue with regular clinical trials as well.

But not all biotech companies with early-stage drugs feel comfortable submitting an application—at least not yet. Thomas Joudinaud, CEO of a French biotechnology company called Ceres Brain Therapeutics, has fielded a request from a person keen to access the company’s experimental drug in Montana. He says that while Montana’s system is “very interesting and very pragmatic” and “suitable for our drug,” he won’t be submitting an application for the time being. He is concerned that if anything goes wrong in Montana, it may jeopardize the company’s standing with the FDA, which wields the power to approve or reject the sale of its treatments to broader populations.

Martin and others have asked the FDA for some kind of assurance that biotech companies participating in Montana’s program won’t be penalized later on. But the agency hasn’t provided them with more than a restatement of the federal Right to Try Act.

“As a matter of policy, the FDA does not comment on state legislation,” an FDA spokesperson wrote in response to a request for clarification from MIT Technology Review.

Even if the FDA were to provide some kind of assurance, it wouldn’t necessarily protect biotech companies in the long term, cautions Chris Robertson, a specialist in health law at Boston University. The FDA’s position could change with a new presidential administration, he says: “I wouldn’t bet on anything that the FDA is saying today being applicable when the rubber hits the road later.”

Companies that want to stay on good terms with the FDA would be safest taking the expanded-access route, says Robertson. That’s the pathway the FDA already uses for people who are seriously or terminally ill, have run out of options, and want to try experimental drugs that have not yet been through clinical trials. The FDA approves over 99% of these applications, says Harvard’s Kesselheim.

“The FDA isn’t a bottleneck but in fact exists to help ensure that expanded-access programs are aboveboard and that patients who receive [the drugs] are able to contribute knowledge about [them],” says Kesselheim. He says he doesn’t think that any “legitimate manufacturer” should fear having to go through the FDA’s expanded-access process, which the agency says takes “less than 45 minutes” to fill out.

The cost of experimenting

There are some key differences between expanded access, which allows seriously ill people to apply for access to experimental drugs that might not have been through any human trials, and Montana’s approach. In theory, a person doesn’t need to be seriously ill to access experimental drugs in Montana.

“In Montana, patients may be eligible for preventive or earlier-stage interventions if they provide informed consent and meet the program’s requirements, so the breadth of potential therapies and situations is much broader,” says Kaeberlein, the Montana ETRB member, who is an affiliate professor at the University of Washington in Seattle.

Kaeberlein also highlights another key difference, which is cost. Companies that make their treatments available through expanded access are only able to charge for the costs of making, transporting, and monitoring the drug, and they must justify the eventual price to the FDA. In Montana, they can charge whatever price they want. Stanley of WinSanTor says he plans to sell his drugs “at cost.” But Ceres’s Joudinaud says that he’d be more interested in selling his at a market price. When asked what that might be, he hinted that the prices of new drugs for rare diseases can be high. In recent years, the median price of such drugs was $218,872.

“Instead of simply creating a legal pathway for patients, it also creates a business model that companies may actually be willing to use,” says Kaeberlein.

Beyond the financial cost, there will be risks associated with any experimental drug. Phase I trials don’t conclusively reveal whether a drug is safe. Around 17% of drugs are found to be inadequately safe during phase III trials. “The idea that a drug has been proven safe because it’s been subject to a phase I study is very, very wrong,” says Kesselheim. Bioethicists have raised concerns about the ethics of promoting and selling unproven treatments and the risk of harm should something go wrong.

But the moment when people start spending money on these treatments is already fast approaching. While Montana’s first ETRB prepares to review its first applications, clinics that hope to be part of the program are busy addressing the requirements laid out in the state’s new rules. Treatment rooms are being outfitted. Medical directors are being hired. And experimental treatments should be reaching patients in the coming months.

Five years ago, most software engineers I spoke to had no programs they had written for themselves. All day, every day, engineers use programs written by others to write programs for others. Many of us customized the programs we used, through config files or plugins or extensions, and many of us used the programs we wrote for others, as users. It was always an unusual treat to ask someone what they had written for themselves and learn about the bespoke software behind their blog, or their home automation, or their homelab, instead of an off-the-shelf, almost-the-right-size static site generator or Zigbee appliance.

This state of things made a lot of sense to me. Over the years I have written plenty of software for myself, and the return on doing so was always questionable. I could only write so much in a day. There were always more important things to do (Something Was Wrong At Work), and coming back to a project after a year to do maintenance on it was always extraordinarily painful. There were plenty of years in my career where I had thrown out all my custom software and used the most bog-standard environments I could to produce code. In my early years as an engineer at Google I did not even own a personal computer.

That was then. Things are different now.

How to Personalize Software

It is astonishingly easy to personalize software today. There are two general categories of prompts to an agent that make all of this possible:

1. Download the source for <software> and build it for local use. Modify <whatever memory your agent uses> to know that any future changes to this software mean changing the sources and replacing the current version. Record in version control the original motivation behind the change.
   and, more importantly:
2. Set up a nightly cron job that executes the prompt: fetch upstream changes to the <software> and rebase all local changes on top of upstream. Check that the software works as intended and replace the current version.

At the heart of this is the realization that agents can not only hack up some code for a specific use but also automatically manage the process of synchronizing changes with upstream releases. This means agents change the ROI on customizing software on two fronts simultaneously: it is much easier to get started personalizing, and much easier to keep going.

Another astonishing thing about the two prompts above for editing software is that you can build them right into an agent. As long as the agent is open source, it does not even require programming. The two prompts can be loaded into a skill (i.e., some text instructions) put somewhere discoverable to the agent. We built this into Shelley, so now if you want to edit Shelley you don’t even need the preamble or to configure the timer. It takes care of it for you. You can type in a prompt like “make Shelley’s UI high-contrast” and you have personalized your agent.

A Worked Personalization Example: Shelley and Meat

I have a personal project I have been idly toying with for the last month: meat.dev. The principle is that while agents write code, I still read it before pushing to our serious systems. As the underlying models improve, what I look for has changed. The humans I have spent twenty years reviewing code for have always struggled with edge cases: do the errors report useful information; are nil-checks handled, etc. (We all do it; when writing code, I am one of the worst offenders.) One of my roles as a reviewer was looking for these details. Over the past six months, I have discovered I don’t need to read for edge cases like that any more: models are far more diligent than humans at rote correctness. Their errors are isolated to architecture, unexpected use cases, visual output their test environment is not feeding back to them, etc. This means most of the lines of code I review are not very useful. So I wrote a tool that takes diffs and uses LLMs to strip out the unimportant stuff. I almost never need to see the import blocks, or the nil-checks, or the error handling any more, so get it off the screen so I can focus on the meat.

I like this tool, but it has two downsides: first, I like to read my diffs in Shelley with a good UI, not in a terminal. Second, it takes a couple of minutes for an LLM to digest and minimize a diff, and I don’t want to wait. So ideally I would not run meat on the command line, but have it built into Shelley and have it pre-processing commits the moment they are created. It turns out I can do that with a single prompt:

Please build meat.dev into Shelley. Install the latest version in the PATH. When a git commit is created by Shelley, start meat processing in the background on the commit. Add a toggle to the Shelley Diffs view for meat. If the commit is still being processed, so the user it is in process.

This single prompt was all it took not just to add meat to Shelley, but to appropriately pre-process commits in the background before I came back to session to review the diff, saving me waiting for a model to reduce the diff. The only unfortunate choice the model made was using the 🥩 emoji for the toggle button.

Imagine the convoluted misery it would be trying to plug that into the VS Code extensions API! Or trying to get it into vimdiff. It would certainly be possible, but the machinery to start pre-processing the commits as soon as they appear would be nigh-on impossible. I would be better off implementing an out-of-band meatd that listened to the file system and provided a cache for the meat tool that a customization API could use, because the points of extension and configuration would not be the right shape.

And that is the fundamental difference between classic configuration/customization and agent-driven personalization: you can do so much more. The agent will do the hard work of understanding the source and changing it to suit the particular task you have in mind. The software we live with is far more powerful with personalization. All you need is the source code.

The Age of Personalized Software

The pre-agent development costs meant it was rational for complex software to ship with large configuration files, extension systems, and plugin systems. The core code of even a moderate project like Vim is huge and baroque, and takes weeks for a human to digest. The thought that, on wanting line numbers to print by default, an engineer would learn the code base and add it just for themselves is unreasonable. Better to design it for sharing with others, which justifies the expense of implementing it by amortizing it over many users. As features in a code base grow, it makes sense to look for common abstractions where you can break out an extension or plugin system.

Now the expense of learning the code and making a change has dropped dramatically. Agents do the heavy lifting. For a single user—which implies extremely constrained conditions under which the program runs—a top-end agent can usually now add a feature in a single shot. For single-user software, the need for careful code review can often be replaced by “does it seem to work?”

The result is that software that can be personalized doesn’t need a plugin system or a config file. Want to change the font size in your text editor? Give the agent the source and tell it to. If it is a hardcoded value it will find and edit it. If it’s a hardcoded bitmap font it will download another and replace it, or it will use Monobit to make you one! You have incredible capabilities on tap.

Whole Categories of Software Products Need to Be Reinvented

Personal software applies well to small teams too. Why would an engineering team purchase an extremely configurable task manager (or a CMS or CRM), spend time learning and configuring it, and contort their team to its limits, when they can assemble just the features they want from common building blocks?

Both the upfront fixed costs and the ongoing costs of personalizing software have disappeared.

The blog you are reading is bespoke software, written in Shelley, because it was easier to piece together and personalize libraries like Tiptap than it is to try and customize traditional software products. For end-user products to make sense in a company today, they need to be personalizable. Which means we need the source code.

Where Codex and Claude Code Diverge

This same skill-based technique that was applied to Shelley to make it personalizable can be trivially applied to other open-source agents like Pi. (So much so that I am left wondering why Pi needs an extension system built into it. The source code is the extension system.) It would require a lot more tokens, but you could do the same to Codex, which is an open-source agent.

Where you would hit a wall, however, is Claude Code. It is closed-source software, so you don’t get to personalize it. There are a lot of old-fashioned customization hooks in Claude Code. Hopefully, how you want an agent to work fits in their hooks. If not, switch to an agent that lets you personalize it.

Software developers learn early in their careers that nothing is more urgent than fixing a production outage. Drop everything! All hands on deck!

However, the same level of urgency is not often given to problems with our development tools, build systems, QA environments, and other parts of the software development pipeline. But for the development team, the development pipeline is a production system.

A software developer’s job is to deliver value for the company. Sometimes that means building new features, sometimes that means fixing critical bugs for the customers’ production systems. But none of this can happen when something is broken in the software development pipeline.

If the code can’t compile, the developers are unable to do their jobs, and the team isn’t producing software. For the development team, this is a production outage. Fixing this should be a top priority.

If the QA server is down, the testers are unable to do their jobs, and the team isn’t producing working software. For the QA team, this is a production outage. Fixing it should be a top priority.

In manufacturing, there are extensive processes and procedures on how to prevent and minimize downtime on the assembly line. And similar processes exist for IT service outages. But I’ve found that most of those focus on outages in the service provided to customers, not for the people responsible for building and supporting the services.

I recommend thinking about all the components that take you from “customer wants something” to “that something is delivered to customers”:

• Issue reporting and change request systems, like GitHub Issues, Jira, etc
• Tools developers use to directly build software, like IDEs, build tools (Gradle, Maven, etc), package repositories (npm, Maven Central, internal repositories, etc), local databases, containers, etc
• CI/CD tools (Jenkins, GitHub Actions, etc).
• A failing test suite (surely you don’t deploy to production if the tests are failing?)
• QA server outage (surely you don’t deploy to production if QA hasn’t tested it?)
• Literally any step in your process that prevents you from making changes and deploying them to production

A team with a broken development pipeline can’t produce software, and must treat this as a production outage.

Terraform AzureRM Provider 5.0 is generally available with improved Azure subscription control, opt-in preflight validation, updated validation defaults, and removal of deprecated resources and properties. The major release simplifies provider behavior, improves infrastructure feedback, and requires users to review configurations before upgrading.

You have a PowerShell module that started as a helpful script and then became business-critical. Now every admin workstation, build agent, and jump box needs the same version at the same time.

Copying module folders to file shares works for a while. But file shares do not give you clean version history, feed permissions, pipeline publishing, or a nice Install-PSResource experience. Azure Artifacts does.

In this tutorial, you will build a private NuGet feed for enterprise PowerShell modules, publish modules from Azure DevOps, stamp them with GitVersion, sign script files using an Azure Key Vault-backed signing workflow, and expose the module through Just-Enough-Administration (JEA).

Prerequisites

To follow along, you will need the following:

• An Azure DevOps organization and project.
• Permission to create or administer an Azure Artifacts feed.
• A PowerShell module with a .psd1 manifest.
• PowerShell 7.x with the Microsoft.PowerShell.PSResourceGet module available.
• An Azure Key Vault certificate or signing process approved by your security team.
• A Windows host if you plan to configure JEA endpoints.

This tutorial uses placeholders such as <ORGANIZATION_NAME>, <PROJECT_NAME>, and <FEED_NAME>. Replace those values with your environment names before running the commands.

Creating an Azure Artifacts Feed for PowerShell Modules

Azure Artifacts feeds can store NuGet packages, and PowerShell modules published with PSResourceGet use a NuGet-based repository model. That combination makes Azure Artifacts a solid private PowerShell repository for internal modules.

In Azure DevOps, open Artifacts, create a new feed, and choose the visibility you want. For a central enterprise module feed, start with project-scoped permissions unless you intentionally need the feed available across the entire organization.

After creating the feed, open Connect to feed, choose NuGet.exe, and copy the v3 feed URL.

Registering the Feed with PSResourceGet

PSResourceGet replaces the older PowerShellGet v2 publishing and installation cmdlets with commands such as Register-PSResourceRepository, Publish-PSResource, and Install-PSResource.

On an administrator workstation, register the Azure Artifacts feed as a trusted repository. If you have already configured a compatible NuGet credential provider or your shell is authenticated through your enterprise process, the repository registration is only the feed metadata.

Register-PSResourceRepository -Name 'EnterpriseModules' `
    -Uri 'https://pkgs.dev.azure.com/<ORGANIZATION_NAME>/<PROJECT_NAME>/_packaging/<FEED_NAME>/nuget/v3/index.json' `
    -Trusted

If your organization stores feed credentials in a vault supported by PSResourceGet, register credential information with the repository:

$credentialInfo = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSCredentialInfo]::new(
    'SecretStore',
    'EnterpriseModulesFeed'
)

Register-PSResourceRepository -Name 'EnterpriseModules' `
    -Uri 'https://pkgs.dev.azure.com/<ORGANIZATION_NAME>/<PROJECT_NAME>/_packaging/<FEED_NAME>/nuget/v3/index.json' `
    -Trusted `
    -CredentialInfo $credentialInfo `
    -Force

Verify the repository registration:

Get-PSResourceRepository -Name 'EnterpriseModules'

Installing from the feed is now a normal PowerShell resource operation:

Install-PSResource -Name 'Contoso.Operations' -Repository 'EnterpriseModules'

Preparing the Module Manifest for CI/CD

Your module manifest is the contract between your code and the package consumers. At minimum, confirm the manifest has a module version, root module, author, description, exported functions, and compatible PowerShell edition details.

Before packaging, validate the manifest locally:

Test-ModuleManifest -Path .\Contoso.Operations\Contoso.Operations.psd1

Adding Semantic Versioning with GitVersion

GitVersion calculates semantic versions from your Git history. In Azure DevOps, GitVersion’s current guidance calls out an important setting: disable shallow fetch by setting fetchDepth: 0, or GitVersion can fail because the pipeline did not clone enough history.

Add a GitVersion.yml file to the repository root:

mode: ContinuousDelivery
branches:
  main:
    regex: ^main$
    increment: Patch
  pull-request:
    regex: ^(pull|pull-requests|pr)[/-]
    increment: Inherit

Signing Module Scripts with Azure Key Vault

Code signing gives consumers confidence that the module files came from your pipeline and were not modified after publishing. For enterprise modules, sign the .ps1, .psm1, and other script files before calling Publish-PSResource.

One practical option is AzureSignTool, which signs Authenticode content using a certificate in Azure Key Vault.

dotnet tool install --global --version 7.0.0 AzureSignTool

A signing step can sign all PowerShell script files in the module folder:

$files = Get-ChildItem -Path '.\Contoso.Operations' -Include '*.ps1','*.psm1' -Recurse

foreach ($file in $files) {
  AzureSignTool sign -kvu 'https://<KEY_VAULT_NAME>.vault.azure.net/' `
  -kvc '<CERT_NAME>' -kvm `
  -tr 'http://timestamp.digicert.com' -td sha384 `
  -fd sha384 -v `
  $file.FullName 
  }

Publishing from Azure DevOps

Create an azure-pipelines.yml file to wire the feed, versioning, signing, and publishing together:

trigger:
  - main

pool:
  vmImage: 'windows-latest'

steps:
  - checkout: self
    fetchDepth: 0

  - task: GitVersion@3
    displayName: 'Install GitVersion'
    inputs:
      versionSpec: '6.7.x'

  - task: GitVersion@3
    displayName: 'Calculate GitVersion'
    name: gitversion

  - task: NuGetAuthenticate@1

  - pwsh: |
      $manifestPath = '.\Contoso.Operations\Contoso.Operations.psd1'
      Update-ModuleManifest -Path $manifestPath -ModuleVersion '$(GitVersion.SemVer)'
      Test-ModuleManifest -Path $manifestPath
    displayName: 'Stamp and validate module manifest'

  - pwsh: |
      # (Signing logic here)
    displayName: 'Sign module files'

  - pwsh: |
      $pat = $env:SYSTEM_ACCESSTOKEN
      $securePat = ConvertTo-SecureString $pat -AsPlainText -Force
      $credential = [pscredential]::new('AzureDevOps', $securePat)

      Register-PSResourceRepository -Name 'EnterpriseModules' `
        -Uri 'https://pkgs.dev.azure.com/<ORGANIZATION_NAME>/<PROJECT_NAME>/_packaging/<FEED_NAME>/nuget/v3/index.json' `
        -Trusted `
        -Force

      Publish-PSResource -Path '.\Contoso.Operations' `
        -Repository 'EnterpriseModules' `
        -Credential $credential `
        -ApiKey 'AzureArtifacts'
    displayName: 'Publish module to Azure Artifacts'
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)

Installing the Module on Servers

Once the pipeline publishes the package, servers and admin workstations install the module from the private feed:

Register-PSResourceRepository -Name 'EnterpriseModules' `
  -Uri 'https://pkgs.dev.azure.com/<ORGANIZATION_NAME>/<PROJECT_NAME>/_packaging/<FEED_NAME>/nuget/v3/index.json' `
  -Trusted

Install-PSResource -Name 'Contoso.Operations' -Repository 'EnterpriseModules' -Scope AllUsers

Restricting Module Actions with JEA

Publishing a module securely is only half the story. You still need to control who can run privileged functions from that module. JEA lets you expose a constrained PowerShell endpoint with only the commands a role needs.

Create a role capability file:

$roleParameters = @{
  Path = 'C:\Program Files\WindowsPowerShell\Modules\Contoso.Operations\RoleCapabilities\ServiceOperator.psrc'
  Author = 'Contoso IT'
  CompanyName = 'Contoso'
  Description = 'Allows service operators to restart approved Contoso services.'
  ModulesToImport = 'Contoso.Operations'
  VisibleFunctions = 'Restart-ContosoService'
  }

New-PSRoleCapabilityFile @roleParameters

Next, create and register a session configuration:

$roleDefinitions = @{
  'CONTOSO\Service Operators' = @{ RoleCapabilities = 'ServiceOperator' }
  }

New-PSSessionConfigurationFile -Path 'C:\JEA\ContosoService.pssc' `
  -SessionType RestrictedRemoteServer `
  -RunAsVirtualAccount `
  -RoleDefinitions $roleDefinitions

Register-PSSessionConfiguration -Name 'ContosoService' `
  -Path 'C:\JEA\ContosoService.pssc' `
  -Force

Operational Tips for Enterprise Module Distribution

• Publish only from CI/CD. Do not let humans publish production module versions from workstations.
• Treat the module manifest as a build artifact. Stamp it, validate it, and publish that exact folder.
• Use semantic versions and never overwrite released versions.
• Require valid Authenticode signatures before publishing.
• Give feed readers broad access, but feed publishers limited access.
• Separate development, test, and production feeds if release approvals matter.
• Add JEA endpoints for privileged functions instead of granting broad local administrator rights.

Conclusion

You built a private PowerShell module distribution flow using Azure Artifacts and PSResourceGet. You created a NuGet feed, registered it as a PowerShell repository, published modules through Azure DevOps, stamped versions with GitVersion, signed files with an Azure Key Vault-backed signing workflow, and restricted privileged operations with JEA.

In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed.

For asynchronous, queue-based workloads, backlog is the true scaling signal – not infrastructure utilization.

In this article, you’ll learn:

• How Amazon SQS queue depth can work as an autoscaling metric for event-driven workers
• How to scale workloads on Amazon EKS using KEDA with AWS pod identity
• How KEDA calculates desired replicas from queue depth
• How to tune queueLength, activationQueueLength, cooldowns, and HPA behavior
• How to validate scaling behavior and troubleshoot common issues

Why this matters: In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.

1. Prerequisites

Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:

• An Amazon EKS cluster running a supported Kubernetes version
• KEDA installed in the cluster (for example, via the kedacore Helm chart)
• An AWS authentication method selected (IRSA or EKS Pod Identity)
• An existing Amazon SQS queue
• A worker deployment designed to consume messages from the queue

2. Architecture and Request Flow

This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.

Request flow:

1. Producers send messages to the Amazon SQS queue
2. KEDA polls queue attributes to determine backlog
3. KEDA updates the target metrics in the HPA
4. The HPA scales the worker Deployment
5. Kubernetes schedules additional Pods to process messages
6. As the queue drains, replicas scale back down

This model keeps autoscaling decisions tied directly to outstanding work.

3. Implementation Steps

Install KEDA via Helm

The recommended way to install KEDA is via Helm:

# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

# Install KEDA into a dedicated namespace
helm install keda kedacore/keda \
  --namespace keda \
  --create-namespace

What this does: Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.

Deploy the Worker with AWS Identity Attached

An SQS consumer typically requires permissions such as:

• GetQueueAttributes
• GetQueueUrl
• ReceiveMessage
• DeleteMessage
• ChangeMessageVisibility

Example IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ConsumeFromQueue",
      "Effect": "Allow",
      "Action": [
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:orders-queue"
    }
  ]
}

What this does: Enforces least-privilege access by granting permissions only for the required SQS queue.

Configure KEDA Authentication and ScaledObject

KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using queueURLFromEnv avoids hardcoding the queue URL.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: sqs-processor-auth
  namespace: workers
spec:
  podIdentity:
    provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-processor
  namespace: workers
spec:
  scaleTargetRef:
    name: sqs-processor
  pollingInterval: 10
  cooldownPeriod: 120
  minReplicaCount: 0
  maxReplicaCount: 30
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 60
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: sqs-processor-auth
      metadata:
        queueURLFromEnv: QUEUE_URL
        awsRegion: us-east-1
        queueLength: "10"
        activationQueueLength: "1"

What this does: Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.

4. Replica Calculation Logic

KEDA calculates outstanding work as:

ApproximateNumberOfMessages + ApproximateNumberOfMessagesNotVisible (+ delayed messages, if enabled)

Replica calculation:

desired replicas = ceil(outstanding messages / queueLength)

Example (queueLength: “10”):

• 0 messages → 0 pods
• 8 messages → 1 pod
• 25 messages → 3 pods
• 95 messages → 10 pods

Final replica counts are bounded by minReplicaCount and maxReplicaCount.

5. Verification

After applying the manifests:

1. Send 50 messages to the SQS queue

2. Inspect the ScaledObject:

kubectl get scaledobject sqs-processor -n workers

3. Watch pod scaling:

kubectl get pods -n workers -w

4. Confirm pods scale back to zero after processing completes

6. Troubleshooting Common Issues

No scale-out

• Cause: Incorrect queue URL or IAM permissions
• Check: kubectl describe scaledobject, KEDA operator logs

Too many pods

• Cause: In-flight messages counted
• Check: scaleOnInFlight, SQS visibility timeout

Never scales to zero

• Cause: Delayed or unacknowledged messages
• Check: Queue attributes and application behavior

HPA exists but no scaling

• Cause: Authentication or metric fetch errors
• Check: KEDA logs and fallback status

7. Production Tuning Tips

• Choose queueLength based on throughput, not guesswork
• Tune cooldown and HPA behavior separately – they control different scale-down paths
• Enable fallback replicas for resilience if metrics become unavailable
• Decide whether in-flight messages should count based on visibility timeout behavior

Conclusion

Queue-based workloads benefit most when autoscaling is driven by the amount of work waiting to be processed rather than traditional infrastructure metrics. Backlog-aware scaling enables applications to react more quickly to changing demand while reducing unnecessary resource consumption during quieter periods.

Although this article demonstrated the approach using Amazon SQS, Amazon EKS, and KEDA, the same design pattern applies across many event-driven architectures and messaging platforms. The key is selecting a scaling signal that accurately represents workload demand and tuning the autoscaling behaviour to match the characteristics of the application.

As organisations increasingly adopt asynchronous, event-driven systems, workload-aware autoscaling becomes an important part of building resilient, efficient, and cost-effective Kubernetes deployments.

Why Kaneo?

After years of using bloated, overcomplicated project management platforms that distracted from actual work, we built Kaneo to be different.

The problem with most tools isn't that they lack features—it's that they have too many. Every notification, every unnecessary button, every complex workflow pulls your team away from what matters: building great products.

We believe the best tools are invisible. They should amplify your team's natural workflow, not force you to adapt to theirs. Kaneo is built on the principle that less is more—every feature exists because it solves a real problem, not because it looks impressive in a demo.

What makes it different:

• Clean interface that focuses on your work, not the tool
• Self-hosted so your data stays yours
• Actually fast because we care about performance
• Open source with a permissive MIT license

Learn more about Kaneo's features and capabilities in our documentation.

Sponsors

Kaneo is open source. If you find it useful, consider sponsoring the project to help support ongoing development.

Getting Started

One-Click Deployment with drim

For straightforward deployments, use drim - a CLI tool that handles everything for you:

curl -fsSL https://assets.kaneo.app/install.sh | sh
drim setup

That's it. Your Kaneo instance will be running with automatic HTTPS, database setup, and all services configured.

Perfect for quick deployments and production setups where you want things to just work.

Quick Start with Docker Compose

The fastest way to try Kaneo is with Docker Compose. This sets up Kaneo and PostgreSQL with a single Kaneo container:

services:
  postgres:
    image: postgres:16-alpine
    env_file:
      - .env
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kaneo -d kaneo"]
      interval: 10s
      timeout: 5s
      retries: 5

  kaneo:
    image: ghcr.io/usekaneo/kaneo:latest
    ports:
      - "5173:5173"
    env_file:
      - .env
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

volumes:
  postgres_data:

Save this as compose.yml, copy .env.sample to .env, uncomment KANEO_CLIENT_URL=http://localhost:5173, and set POSTGRES_PASSWORD=<password> and AUTH_SECRET=<output of openssl rand -hex 32>, run docker compose up -d, and open http://localhost:5173.

In Docker Compose, the bundled Kaneo container reaches PostgreSQL at the service hostname postgres. If you run the API on your host instead of inside Compose, use localhost or set DATABASE_URL explicitly.

Important: See our full documentation for detailed setup instructions, environment variable configuration, and troubleshooting guides.

Development Setup

For development, see our Environment Setup Guide for detailed instructions on configuring environment variables and troubleshooting common issues like CORS problems.

Configuration

Kaneo requires several environment variables to be configured. The Docker Compose setup above handles the database automatically, but you'll need to configure environment variables for the API and web services.

For complete configuration instructions, including all required environment variables, database setup for non-Docker deployments, and advanced settings, see the documentation. Advanced deployments can still use the separate ghcr.io/usekaneo/api and ghcr.io/usekaneo/web images.

Kubernetes Deployment

If you're running Kubernetes, we provide a comprehensive Helm chart. Check out the Helm chart documentation for detailed installation instructions, production configuration examples, TLS setup, and more.

Development

Want to hack on Kaneo? See our Environment Setup Guide for detailed instructions on configuring environment variables and troubleshooting common issues like CORS problems.

Quick start:

# Clone and install dependencies
git clone https://github.com/usekaneo/kaneo.git
cd kaneo
pnpm install

# Create a .env file in the root with required environment variables
# See ENVIRONMENT_SETUP.md for detailed instructions

# Start development servers
pnpm dev

For contributing guidelines, code structure, and development best practices, check out our contributing guide and documentation.

Community

• Discord - Chat with users and contributors
• GitHub Issues - Bug reports and feature requests
• Documentation - Detailed guides, API docs, and tutorials

Contributing

We're always looking for help, whether that's:

• Reporting bugs or suggesting features
• Improving documentation
• Contributing code
• Helping other users on Discord

Check out CONTRIBUTING.md for the details on how to get involved.

License

MIT License - see LICENSE for details.

We are announcing the deprecation of the Azure DevOps issuer in workload identity federation (WIF) service connections, with planned retirement on July 1, 2027. The Azure DevOps issuer uses the https://vstoken.dev.azure.com prefix in federated credentials. This change is part of Microsoft’s broader initiative to standardize on the Microsoft Entra issuer across Azure services that implement workload identity federation.

Important This deprecation only applies to service connections in Azure public cloud that use single-tenant Microsoft Entra applications or managed identities. Service connections targeting non-public clouds (for example, Azure Government, Azure China, or Azure Stack) and service connections that use multi-tenant applications (signInAudience: AzureADMultipleOrgs) are explicitly excluded from today’s deprecation announcement. The Azure DevOps issuer will continue to be supported for these scenarios until they’re supported by the Microsoft Entra issuer.

Background: Workload Identity Federation in Azure DevOps

More than two years ago, we introduced workload identity federation support for Azure DevOps, enabling secretless authentication between Azure Pipelines and Azure resources with managed identities or app registrations. This was a significant security improvement over the use of app registrations with secrets.

Workload identity federation has proven to be an invaluable feature for our customers, with strong adoption across organizations seeking to eliminate long-lived credentials from their CI/CD pipelines.

Microsoft is standardizing on the Microsoft Entra issuer (https://login.microsoftonline.com/) for workload identity federation across services. Instead of using an OIDC token minted by Azure DevOps in the OpenID Connect (OIDC) flow underpinning workload identity federation, the flow uses an OIDC token minted by Microsoft Entra. The Microsoft Entra issuer has been used for new service connections since last year, and today more than 50% of all workload identity federation service connections use the Microsoft Entra issuer.

Timeline

Since November 2025, new workload identity federation service connections created in Azure DevOps have been using the Microsoft Entra issuer. Dates of upcoming changes are:

• July 1, 2026: The Azure DevOps issuer (https://vstoken.dev.azure.com) is deprecated. New service connections created in Azure DevOps will continue to use the Microsoft Entra issuer by default.
• July 2026 – June 2027: Existing service connections using the Azure DevOps issuer will show a warning in pipeline runs and the service connection configuration UI.
• July 1, 2027: The Azure DevOps issuer will reach end of life and will no longer be supported.

What You Need to Do

Service connections that use the Azure DevOps issuer (https://vstoken.dev.azure.com) are listed at the top of the service connection list, with a warning indicating they need action.

To convert a service connection to the Microsoft Entra issuer, select the Update button.

If you don’t have access to the underlying identity and can’t create a federated credential for the Microsoft Entra issuer, select the Create federated credential in link to create the federated credential and populate it with the issuer and subject manually.

FAQ

Q: Will my existing pipelines break immediately?
A: No. Service connections that use the Azure DevOps issuer will continue to work until retirement on July 1, 2027. We recommend planning your conversion before then.

Q: How is the Microsoft Entra issuer different?
A: The issuer is an implementation detail that’s hidden during regular use. Pipeline tasks work the same and don’t require changes. The Microsoft Entra issuer provides additional benefits by using Microsoft Entra-minted tokens and an immutable federation subject, so the federated credential is guaranteed to be used by the service connection it was created for.

Q: Can I use the Microsoft Entra issuer today?
A: Yes. All new service connections created today use the Microsoft Entra issuer by default. You can also convert existing connections by following the steps above.

Q: Is there any downtime during the conversion, and how long does it take?
A: During the conversion, the existing Azure DevOps issuer federated credential will continue to be used by pipelines that reference the service connection. After we verify that the new Microsoft Entra issuer federated credential works during conversion, pipeline jobs will start using the Microsoft Entra issuer.

Q: What if I have questions about the conversion?
A: Please review Workload identity federation conversion and Workload identity federation troubleshooting. You can also reach out to Azure DevOps Support or visit the Azure DevOps Developer Community.

Q: I create service connections in automation and need to be able to know the federated credential subject before creating it.
A: See Use scripts to automate workload identity service connections.

Q: My Azure DevOps organization has an organization-wide exception to use multi-tenant apps to prevent the error AADSTS70052: The identity must be a managed identity or a single tenant app. What will happen?
A: We’re working on an experience that will provide a service connection-specific exception for multi-tenant apps instead. Until then, you will see no difference in experience.

DoorDash built a centralized Agent Gateway that gives AI agents governed access to MCP tools, handling identity, authorization, credentials, filtering, observability, and rate limits. It now supports 200+ MCP servers and millions of weekly calls while improving security, reliability, and control.

At Halodoc, Airflow is the heartbeat of our data platform - not because it moves the data, but because it's the central point of orchestration, deciding when everything else runs. We run it as a managed service on Amazon MWAA, and from the start we kept orchestration and processing separate: Airflow decides what runs and when; the heavy lifting happens elsewhere.

The journey starts with raw CDC records from the systems behind our core products - pharmacy orders, teleconsultations, appointment bookings - which we upsert into Apache Hudi via Spark on EMR on EKS. From there we build dim/fact models, then ingest them into Redshift, our analytics layer. All of it batched, all of it on an Airflow schedule.

Raw CDC → Hudi (Spark on EMR on EKS) → dim/fact models → Redshift (analytics layer)

Because everything is batched, timing matters - a downstream model has no business running until every upstream table it depends on has landed. We enforce that with Airflow sensors, one per upstream dependency, so nothing runs on a half-loaded picture. But sensors aren't the only thing holding workers hostage: ingestion into Redshift runs through a custom operator built on synchronous psycopg, so each load also pins a worker for its full duration. Fine individually - but when dozens of schedules fire together, blocked workers pile up, MWAA autoscales to compensate, and once we hit the worker ceiling, everything else simply queues. Most of those "busy" workers were just holding a connection open, doing nothing.

Why We Upgraded

Because our workloads are batched, they don't trickle in evenly - they cluster into burst windows where dozens of schedules fire at roughly the same time. In those windows, two things pile up together: ingestion tasks blocked on psycopg waiting for Redshift, and sensor tasks blocked waiting on upstream data. Neither is a new problem on its own, but at burst scale they compound - enough blocked workers at once, and MWAA autoscales, hits its ceiling, and everything else queues behind them.

Airflow 3.0 gave us a direct way to fix both: Assets that don't need a sensor task polling over and over, and deferrable operators that can release the worker slot instead of holding it while they wait.

• Replace polling sensors with data-aware Assets (where applicable). Take that appointment-revenue model from earlier - two upstream tables meant two separate sensor tasks, each looping on its own "is it ready yet?" check, each holding a worker slot the whole time it waited. Assets flip that: the upstream job produces an Asset, and that event triggers the next step directly - no polling, no slot held just to wait.
• Swap worker-blocking ingestion for the deferrable Redshift Data Operator. Move Redshift loads off synchronous psycopg and onto the Redshift Data API through the RedshiftDataOperator, which can defer: it submits the SQL, frees the worker while Redshift runs it, and picks back up when the statement finishes - instead of pinning a slot for the whole query.

Data Asset

As we described earlier, we leaned heavily on Airflow sensors to make sure a downstream job never began its ingestion until every upstream it depended on had finished processing.

As we mentioned earlier, we set out to try the Data Assets feature on the latest Airflow - starting with our non-critical pipelines - for two reasons:

• Downstream triggers only once every upstream is done. A downstream job fires automatically the moment all of its upstream Assets have been updated - no fixed-schedule guesswork, and no poll loop waiting for data to land.
• Fewer sensor listeners on every downstream DAG. It removes the stack of sensor tasks we'd been attaching to each downstream DAG just to watch upstream state.

With Data Assets, we don't have to give the downstream job a schedule of its own. Instead, we add an outlet to the upstream task and an inlet to the downstream:

• Outlet (upstream). When the upstream task finishes, its outlet emits the Asset - a signal that the upstream table is done and freshly updated.
• Inlet (downstream). The downstream picks up that Asset through the inlet we declare, and runs - no cron, no polling, just a reaction to the upstream being ready.

There was a wrinkle, though. Long before Assets, our pipelines already coordinated these dependencies through cross-job dependency with sensors, and every job's sensor configuration lives in a central control table. So adopting Data Assets was never a clean-slate exercise - we had to make it fit the dependency configuration we already had in that control table, rather than replace it.

So the downstream's inlets simply follow our existing configuration - the same dependencies, now read straight from the control table.

That covers the downstream. On the upstream side, each task needs an outlet - the piece that signals when its process has finished.

Our dependency model already carried an important distinction: each upstream is either a strong dependency the downstream must have, or a weak one it can proceed without - the is_skip_allowed flag from that same config. Sensors handled weak dependencies naturally by skipping them. Assets don't: a downstream only fires when its Assets are emitted, so a weak upstream with no new data would stay silent and the downstream would never trigger. To preserve that behaviour, we emit the outlet even when the upstream has no update or new data - the downstream still triggers, and we push the decision into the downstream task, where we validate whether ingestion is actually needed when one of the tables didn't change.

One deliberate exception: we don't emit asset events during backfills or manual triggers. Without that guard, a single backfilled or manually run upstream would cascade into a domino of downstream runs we never meant to start.

That downstream validation is where it all comes together. Before it ingests, each downstream job checks itself against its own configuration: if every dependency meets the rule, it proceeds with ingestion; if not, the ingestion task is skipped.

We layer one more check on top: how recently each upstream actually ran, bounded by a MAX_LOOKBACK_MINUTES window - set to 360 minutes, or six hours. This makes sure the downstream only runs when the dependency rule is satisfied and the data is genuinely fresh. It matters because Data Assets have no built-in freshness guardrail: Airflow only checks that each required Asset has been emitted at least once since the downstream last ran - not how recently. Without the lookback, an Asset emitted yesterday and another emitted today would both count as satisfied, and the downstream would fire on day-old, stale data.

Redshift Data Operator

For loading into Redshift, we had long relied on a custom ETL operator of our own, built on psycopg. Over time it had grown fairly capable: it handles SCD2 for our dimensions, chunks large loads into smaller batches, and manages its own temp tables for staging before the final write.

In practice, each load walks through the same sequence of steps:

1. Drop the staging table.
2. Create the staging table.
3. Insert-select the source data into the staging table.
4. Delete from the destination table.
5. Insert from the staging table into the destination table.

That capability came at the cost we met back in the introduction. Because psycopg is synchronous, the operator holds its Airflow executor slot for the entire run - and some of our loads run well past fifteen minutes. For that whole stretch, a worker stays pinned to a single job that is mostly just waiting on Redshift to finish.

It bites hardest during ETL bursts, when many of these loads fire at once and a whole cluster of workers ends up pinned together. That's what pushed us to the RedshiftDataOperator, which supports deferrable execution: the task submits its SQL, releases the worker, and only resumes once Redshift reports the statement is done.

It isn't a straight swap, though. The Redshift Data API it runs on comes with a couple of hard limits:

• The maximum query result size is 500 MB (after gzip compression).
• The maximum query statement size is 200 KB.
• The maximum number of active queries (STARTED and SUBMITTED queries) per Amazon Redshift cluster is 500.

None of these turned out to be a hard blocker - though two of them needed a bit of attention to stay safely inside the limits. Starting with the simplest: our loads don't pull data back into Airflow - they insert into a staging table, then run delete-and-insert into the target - so there is no large result set coming back, and the 500 MB ceiling simply never applies.

The active-query ceiling needed a small piece of engineering. The Data API counts every STARTED and SUBMITTED statement against that per-cluster limit of 500, so a large enough burst of deferred loads could, in principle. We cap that with an Airflow pool: every task that goes through the Data API has to claim a slot from a pool we sized at 65, so no matter how many DAGs fire at once, the number of in-flight statements stays comfortably below the cluster's ceiling.

The statement-size limit needed a little more handling. When a generated SQL script runs past 200 KB, we fall back to psycopg automatically - the operator picks the right path on its own, so nothing calling it has to care which one ran.

Statement size isn't the only signal we route on. Deferral only pays off when a query actually takes time to run - deferring a sub-second DROP TABLE or CREATE TABLE would just add overhead for nothing. So the first two steps - dropping and creating the staging table - stay on psycopg, and we reserve the Redshift Data API for the three that genuinely run long: the insert-select, the delete, and the stage-to-destination insert.

The hybrid approach gives us the best of both. The operations that genuinely need to wait - insert-select and the like - release their executor slot and defer, while the quick ones take the fastest path. And because we wanted to be careful, we built in one more safety net: if the RedshiftDataOperator ever runs into trouble, the task automatically falls back to psycopg on Airflow's third retry. Since the operator ultimately talks to an API, that guard means a transient hiccup - a timeout, say - doesn't derail the load; it just finishes the old way.

One caveat, to be straight about it: the final delete and insert on the destination run as two separate Data API statements, not one transaction - so if the insert fails after the delete commits, the table reads short until a retry rebuilds it. We could make it atomic by sending both as a single BatchExecuteStatement - that hardening is on the roadmap.

We Hit the Bugs

With the plan set, we moved our MWAA environment from Airflow 2.10 up to 3.0.6 and got to work. This was the version where we'd build the two things we came for - the deferrable Redshift Data Operator and data-aware Assets - and on paper, 3.0.6 had everything we needed.

Then, almost out of nowhere, we ran into a bug. Airflow was issuing far more database round-trips than it should have - a classic N+1 query pattern, this time buried in the core itself as it serialized and scheduled our hundreds of DAGs. At our scale, those redundant queries piled up fast.

The symptom was impossible to miss once we went looking. In CloudWatch, the writer instance of Airflow's own metadata database was pinned near 100% CPU - and it stayed even when almost nothing was running. This wasn't our data warehouse; it was the database Airflow uses to track its own state. On MWAA that database is fully managed for us, so we couldn't simply tune our way out of it.

The important part: this wasn't something in our DAGs or our operators. It lived in Airflow itself, which meant no config knob on our side could make it disappear. We were stuck on the exact version that had the features we wanted and a bug we couldn't live with - so the fix had to come from a newer Airflow. We set our sights on 3.2.1.

The jump paid off. On 3.2.1 version, the N+1 pattern was simply gone: the metadata database settled back down, its CPU returned to normal, and the scheduler got its breathing room back. The bug that had blocked us on 3.0.6 was fixed upstream, and because it was an upgrade within the same major version, the path itself was supported and low-drama.

3.2.1 did have one surprise of its own, though a far gentler one. As soon as it parsed our DAGs, it began flagging a warning we'd never seen before: This Dag uses runtime-variable values in Dag construction. It causes the Dag version to increase as values change on every Dag parse.

This one traces back to DAG versioning, a new capability in Airflow 3: Airflow now keeps track of a version for each DAG's definition. The catch is that one of our DAGs passed pendulum.today('UTC') into its default_args - a value that resolves to a different timestamp on every parse. Airflow saw that shifting value and concluded the DAG had changed, bumping its version on every single parse. The fix was straightforward: keep runtime-varying values out of the DAG and Task constructors, and pin a static start date instead.

Conclusion

We shipped both changes successfully: Data Assets adoption and the deferrable Redshift Data Operator migration. These are critical parts of our production data flow, so rather than flipping everything over at once, we scoped this first phase deliberately - Data Assets across 30 DAGs, and the deferrable Data Operator across 130 DAGs, and watched them closely. After more than a month of monitoring, we've seen no discrepancies and no data issues: the event-driven scheduling and the deferrable loads have behaved exactly as we hoped.

The impact was clear in the numbers. During peak ETL windows - when ingestion loads and the Asset listeners run together - four utilization metrics came down: base worker CPU fell from 26.1% to 7.71% on average and base worker memory from 49.2% to 30.8% on the Redshift side, while scheduler memory dropped from 53.2% to 43.7% and scheduler CPU from 36.1% to 29.4%. And during ingestion, table-locking errors caused by heartbeat timeouts came down by roughly 38%.

There's still headroom to go further. The most immediate step is making the final delete-and-insert swap atomic by moving it to a single BatchExecuteStatement - the hardening we flagged earlier - which we'll take on in the next iteration. But even in this first phase, the direction is unmistakable: letting data readiness drive scheduling, and letting long-running loads defer, has handed us back a large share of the compute we used to burn just waiting.

TL;DR

Opendata Log maintains millions of individually keyed, ordered logs and scales to tens of thousands of active readers on a single instance. It is MIT-licensed, built directly on Object Storage, and deployed as a single Rust binary.

Create your first logs in 2 minutes with our quickstart: https://www.opendata.dev/docs/log/quickstart

There is No Goldilocks Log

In late 2013, Jay Kreps published one of the most influential blog posts in data engineering. He explained why The Log is the universal building block underneath nearly all data systems. It can replicate a state machine across a network, feed a firehose of data into downstream systems, and serve as efficient storage for real-time processing.

He was correct about the abstraction, but incorrectly assumed that this single abstraction could also be implemented by a single system (called Kafka).

A decade of working on logging systems has taught us that logging, as it turns out, is split into two types:

Both of these patterns require a durable log, but they have quite different nonfunctional characteristics that make a system designed for one a poor fit for the other. Some use cases just need the biggest, cheapest pipe they can get. Others need to scale to the millions of small logs.

Funnels

When Kafka was released in 2011, fifteen years ago, it was built to funnel telemetry (clickstream, page views, metrics) data from all of LinkedIn’s servers to Hadoop and/or an online metrics database.

An intriguing similarity across the initial Kafka use cases at LinkedIn was that messages sent to Kafka were keyless. In fact, the initial Kafka implementation discarded the partitioning key after computing a partition, leaving only the payload on storage (it wasn’t until later that Kafka supported keys and compaction).

In the context of funneling, this isn’t surprising. If what you want is to shuttle data from source to destination then a simple pipe that’s optimized for well-balanced throughput is ideal.

Routers

The other use case for logs is routing, which is often seen in messaging applications, feeds and microservice communication. It takes events from input sources and delivers them to specific, addressed destinations or records them for future replay.

The defining characteristic of routers is that destinations are only interested in a particular subset of keys. If you think of the funnel as a dumb broker, then routers need smart logic to distribute the messages in the log to their respective destinations.

Why Kafka is an Excellent Funnel, But a Bad Router

We like to reason about storage systems in terms of read, write and space amplification. The only way to improve all three in a system is to restrict the usage pattern.

When the engineers at LinkedIn designed Kafka, they restricted the API to make it particularly well suited as a funnel with low write amplification. Since data is written to disk via append-only, immutable files which are only cleaned up wholesale when they fall out of retention, the write amplification α_write is ~1x.

The consequence of this low write-amplification design is that reads are heavily restricted. The only read API provided is to scan the data in the order that it was written, but if that’s what you want then your read amplification α_read is also 1x.

Routers need a different access pattern on the underlying log. Instead of one massive log, routers want to store hundreds of thousands or even millions of individual logs.

Since Kafka partitions are isolated from one another, it’s not feasible to store each routing log in its own partition. This means that to find a specific message by key in a funnel like Kafka, your read amplification is the ratio of the entire partition size divided by the size of the record:

              |partition|
  α_read  =  ─────────────
               |record|

This is nearly worst case read amplification, and makes it impractical to use Kafka as a router.

OpenData Log: The right primitive for Routing

Today we’re announcing OpenData Log, a superior mechanism for the Routing log use cases.

Log is an MIT-Licensed, object-native, key-oriented log built on SlateDB. Breaking that down, in reverse:

1. Key-Oriented: Unlike Kafka, Log organizes data by key instead of topic-partition. You’re encouraged to have hundreds of thousands to millions of keys on a single log node.
2. Object-Native: The only requirement for durability is Object Storage. This makes it strongly consistent and absurdly durable.
3. MIT-Licensed: It’s open source and made for hosting yourself.

Keys, not Topic-Partitions

Log allows you to read ordered records for a specific key, even if the key cardinality is high:

let log = LogDb::open(...);

// Scan user-123's log
let mut iter = log.scan(Bytes::from("user-123"), ..).await?;
while let Some(entry) = iter.next().await? {
	println!("seq={}, value={:?}", entry.sequence, entry.value);
}

Notice that there’s no partition in the parameters for log.scan(), you address the log by the key and get the values in return.

For Routing use cases, partitioning is a poor data model. Instead, the right level of abstraction is a “key” that can support orders of magnitude more values than Kafka can partitions.

To support this, Log is designed as a segmented LSM tree keyed by (key, sequence) and segmented by sequence ranges.

If you’re not familiar with LSM trees, the ten-second version is that they are a data structure that stores recent data as a log and compacts older data into a sorted array.

In practice, this architecture means you can append new data quickly and still access old data efficiently with a binary-search. The result is that Log can support millions of independent keyed logs without increasing the latency for fetching data for a specific log.

Scaling without Partitions

While partitions are a suboptimal data modeling concept, some type of data distribution is a requirement for scaling out beyond a single machine. The first strategy available for scaling Log horizontally is deploying multiple readers. You can spin up as many read replicas as you want since data is pulled directly from S3.

Deploying readers that are scoped to a single target key range has the extra benefit of improving the cache locality of reads, thus reducing read amplification.

Cost

Log inherits some of the excellent properties of object storage in its operational costs. We were able to sustain 50,000 concurrent followers tailing their own keyed logs with an 8GB cache and 1 million keys at <50ms p50 latency.

Tradeoffs

There are a few decisions we made to get Log to the point where it can process high cardinality keys and nothing comes for free. In particular, the poll latencies scale with the rate of ingestion. Increased ingestion rates cause the cache to evict hot data before compaction has the opportunity to collocate keys for efficient querying.

The workaround to this, discussed above, is scoping read replicas to smaller ranges to avoid thrashing the cache with blocks that contain data from other ranges.

In addition Log inherits the limitations of object storage latencies, but in practice you can get p50 and p99 poll latencies of ~30ms and ~300ms respectively with 25 MiB/s of throughput.

Finally, Log does not track consumer offsets directly. This separation of metadata and data allows for more flexible deployment mechanisms, and we recommend you use a Key-Value store like SlateDB for offset tracking on the consumer.

Get Started Today

Log is MIT-licensed and available today as part of OpenData and you can run through the quickstart in a couple of minutes.

Full article content is not available for inline reading.

Read the original article →

Data lessons from inside Meta (Shridhar Iyer)

Shridhar Iyer spent 13 years inside Meta's data organization. He joins Tristan on what big tech takes for granted, why you never delete a column at scale, and what it takes to become AI-native.

Shridhar Iyer (he goes by Sri) spent more than 13 years inside Meta’s data organization, most recently as Senior Tech Lead and Director for AI and the data stack. As of this recording he had just stepped back to take a career break, so Tristan figured the best use of that time off was to relive the past decade and pull out some hard-won lessons for the rest of us.

The conversation takes an unexpected turn early. Tristan spotted a line in Sri’s break announcement on LinkedIn: he plans to take Indic and analytic philosophy seriously, possibly studying it formally. So they start with the hard problem of consciousness, why a Hindu metaphysical tradition has been debating it for thousands of years, and why, in the age of AI, all of this has suddenly become very practical. We are building systems we do not fully understand, and the question of what it would even mean for one of them to be conscious is no longer purely academic.

From there it gets concrete. Sri tells the story of how truncating a single column called Extra saved Meta a few million dollars, and why, despite that, you almost never actually delete a column at Meta scale. He traces how Meta’s data stack evolved from Hadoop and Hive into something schematized, unified, and semantically labeled, why the data team went from centralized to embedded inside product, and what companies inside big tech take for granted today that will leak out to the broader ecosystem over the next decade. They close on AI: the two steps to becoming AI-native, the three archetypes Sri sees emerging, and how removing the middle management layer is changing the way people get managed.

Three ideas from the episode

1. At Meta scale, deleting a column is a company-wide event, so you almost never do it. Sri once truncated a debug column called Extra across a year of a core event-logging table and saved a few million dollars, which earned him a “fix of the week” spot at Mark Zuckerberg’s weekly company Q&A. But he also owned the Facebook user table, which roughly 40 to 50 percent of Meta’s warehouse depended on, held to something like 45 nines of quality. At that blast radius you do not delete or truncate. You create a new version, migrate people over with tooling and program management, and simply stop populating the old column.

2. Meta’s real advantage was building abstractions in layers: schematize, unify, then add meaning. Meta strongly typed its schemas both online and offline, enforcing structure all the way upstream to the log statement, which is what made column-level lineage and privacy work possible. On top of that it unified language and compilation across Spark and Presto, unified the catalog and taxonomy so that every table, dashboard, and sub-column is a strongly typed asset with a URI in a central registry, and layered policy management on top for compliance. Only then did the semantic and knowledge layers for AI become feasible. That layering is what Sri thinks other companies are behind on.

3. Becoming AI-native takes two steps, and most teams skip the first. First, get AI-ready: your people, processes, and systems. In practice that means taking a single workflow with its context, learning to run it well, then extracting reusable primitives and figuring out how to use agents at the lowest cost with the right guardrails. Teams that skip straight to multi-agent orchestration burn tokens and get poor outcomes. Second, reorganize around archetypes. Sri sees three that will stick: the builder who automates workflows, the forward-deployment engineer who makes environments AI-ready, and the domain specialist embedded in a product.

Key takeaways

You wrote that you plan to take analytic philosophy seriously. Where does that come from?

Shridhar Iyer: I always had a philosophical leaning, even growing up, so it has always been in the background. It is not surprising for Hindus in general to be fascinated by the hard problem of consciousness, which the philosopher David Chalmers coined in the 1990s. Is consciousness emergent, or is it fundamental to the ontology of the universe? The Hindu philosophical claim is that consciousness is fundamental. It is a very long tradition, going back thousands of years, and it is rigorous and reasoning-based. Hindus and Buddhists were debating who we are and what consciousness is long before modern Western philosophers, with the language and sophistication to dissect the problem.

Are you drawn back into this because of your professional work, or is it separate?

Shridhar Iyer: Two reasons, and AI is definitely one. Scientists have been at the hard problem for four or five decades and cannot nail a single conscious experience, which has caused a rethinking of the materialist view that consciousness simply emerges. That has given a resurgence to alternatives like panpsychism, where consciousness is as fundamental as matter, and idealism, which is closer to the non-dualistic philosophy of Hinduism, where mind is at the bottom of the universe and everything we see emerges from it. The other reason is that most tech leaders working on AI are materialists, and for them AI is a backdoor: if you can show that enough complexity crosses a threshold where consciousness emerges, you can argue it is all just a particular arrangement of matter. But it is legitimately hard to know. We do not have a good hypothesis for how we would provably know whether a system is having conscious experiences.

How did a J2EE developer end up as one of the first data engineers at Meta?

Shridhar Iyer: I came to the US early in the internet boom, and a lot of the work then was digitizing legacy systems, which is funny because that is what we are doing again now with AI enablement. I was a Java consultant, and I took a full-time job at JB Hunt partly for green card reasons and because I was about to get married. I did about a decade there and got an MBA the company paid for, which opened my eyes to business and product rather than just how to build things. Then I joined a small retail analytics startup that took point-of-sale data from Walmart and built reports for companies like Kraft Foods, PepsiCo, and Coca-Cola. That was the very beginning of data engineering. Then someone who was one of the early data engineers at Meta, and is now at OpenAI, pinged me out of nowhere and asked if I wanted to interview. I did not think I had a chance, but I gave it a shot and joined in 2013 as one of the first data engineers there.

What did Meta’s data stack look like when you arrived?

Shridhar Iyer: It was all Hadoop and Hive, so HiveQL, which predates ANSI SQL becoming popular for big data through Presto around 2014 and 2015. Meta has always run its own data hardware, and even its open source is a very different branch, so Presto there is not the open source Presto. For orchestration there was a config-driven layer and then DataSwarm, the predecessor to Airflow. Max, who went on to build Airflow, was my teammate at Meta, and I am sure DataSwarm influenced it. Reporting ran on Oracle Exadata, and dashboards were MicroStrategy or a simple drag-and-drop browser tool. The problems were all about how you log, instrument telemetry, build pipelines, and report.

If you split your 13 years into eras, what were the big changes?

Shridhar Iyer: One was going from centralized to decentralized. Data engineering became part of the product, one of the four pillars alongside software engineering, product management, and design, with analytics and data science. That happened around 2015 and 2016. It makes sense: when you are building a new function, you centralize early so you can share knowledge and build the craft, and once that craft is built into your people and processes you decentralize so you can influence the business where it happens. The other era was scale. I cannot even fathom it now. We used to have thousands of tables, and now it is tens of millions of tables, hundreds of millions of columns and sub-columns, with very complex types in the schema.

What is a basic thing that gets much harder at that scale?

Shridhar Iyer: Let me tell a funny story first. In 2014 we had a core event-logging table that fed a lot of growth analytics, and it had a column called Extra full of non-essential debug information, just in case you had to replay something. The data infra team had a script to truncate a column to save on storage. I was vacationing in India when they were pushing to finish it to save money, so I truncated that column across a year’s worth of data and saved a few million dollars.

Mark does a weekly company Q&A that starts with a “fix of the week,” and they nominated me for it. So I got up in front of the company and explained that I truncated one column called Extra and saved millions. The executives in the front row fell off their chairs, because logging into a column called Extra that no one fully understands is exactly the kind of thing Meta would do.

So how do you actually delete a column or table at Meta scale?

Shridhar Iyer: I owned one of the most core data sets, the Facebook user table. Pretty much 40 to 50 percent of Meta’s warehouse depended on it, with dependencies orders of magnitude deep in the graph. The bar for quality was very high, up to something like 45 nines, because you could not restate half the warehouse.

Even so, we reached a point where recovering the whole warehouse was infeasible, so for a table like that we would tell people the impact and let them decide whether to recover. If something is not that serious, you generally do not delete tables or columns at all. You create a new version and gracefully migrate people over with a lot of tooling, and once the project is done you deprecate the old column. Even then you do not truncate it, you just stop populating it. There is a lot of tooling around lineage, cutting migration tasks to teams, and program managing it.

What do companies inside big tech know today that will leak out to the rest of us?

Shridhar Iyer: You might not even know, because it is the water you swim in. You take it for granted, and you learn to just focus on the swimming, so abstracting the principles out of it is genuinely hard on the fly. But I would point to a few things Meta did earlier than others, and at scale. One was to schematize, both online and offline, strongly typing schemas and enforcing them all the way upstream to the log statement. That was a big undertaking, and it is what made column-level lineage possible, which was fundamental to the privacy work.

On top of that we unified: one compiler and language across Spark and Presto so you can trace lineage and propagate labels, a unified catalog and taxonomy where every table, dashboard, and sub-column is a strongly typed asset with a URI in a central registry, and a unified policy system on top for compliance. Then on top of that we started building semantic and knowledge layers for AI. Meta built these abstractions out at scale and layered them on top of each other, and I think that is paying off now.

How do you onboard someone into an environment like that?

Shridhar Iyer: The data infra is so good that within a week you can be productive and making commits. The complexity is not in building things, because building and visualization are easy. It is in the new principles a data engineer at Meta has to account for, like how to build privacy-aware data and how to deal with grain management at that scale. The hard problems are product and analytics problems for the domain, not engineering problems. So the engineer onboards quickly, and the real work is solving the analytics problems.

Did that change the kind of person you hired?

Shridhar Iyer: Yes. Recruiting shifted from more technical to more product and business focused. In the early days we did software engineering and design interviews because we wanted builder types. Later we wanted more analytics engineer types, which I think you all pioneered in the industry, so we looked for data modeling, good SQL, and a bit of Python rather than heavy coding. For the most senior engineers, interviews became mostly deep design questions.

What should a data team’s role be in helping a company become AI-native?

Shridhar Iyer: There are two necessary steps. First is becoming AI-ready, and that has to happen across people, processes, and systems. AI readiness really comes down to taking one workflow with its context, learning to do it well, and then extracting the primitives out of that workflow along with how to use agents at the least cost with the right guardrails. People tend to skip this, and when you skip it you burn tokens, you add cost, and you do not get good outcomes. Only once you nail that can you automate it through multi-agent orchestration.

The second step is organizing the company into archetypes. I see three: a builder who uses AI to automate workflows, a forward-deployment engineer who goes into different environments and makes them AI-ready, and a domain specialist who builds AI solutions inside a particular product or business area.

What is your view on companies reorganizing around AI, with fewer management layers?

Shridhar Iyer: It is a given that you have to get more done with fewer people, or the ROI is not there, and that follows from making your systems AI-ready. Different orgs move at different speeds. Meta is founder-led, so it will make a quick, disruptive switch at very large scale. Some companies, like Airbnb, are more cautious and want to watch the industry first. I think this year and next are mostly experimentation, because a lot has to settle: AI enablement, people up-leveling their skills, and new roles emerging or merging. The three roles I keep coming back to are the builder, the AI enabler, and the domain specialist embedded in the product.

How does managing people change in this world?

Shridhar Iyer: Teams are becoming larger because the middle layer is being removed, and a lot of managers have been asked to become individual contributors. That changes the dynamic, because you cannot have weekly one-on-ones with everyone on a large team. So the traditional manager-employee relationship is changing.

At Meta, central teams are being built within the product or across the function and then organized into large “super pods” of 20 or 30 people, with smaller sub-pods, all focused around specific problems. The team I helped build just before I left was building the knowledge and context layer for agents. The consolidation is data scientists, data engineers, and analytics engineers coming together to solve domain problems, with the goal of making systems AI-ready so that agents can do the work autonomously, with guardrails you trust.

Introduction

The efficacy of semantic search relies on the accuracy of the underlying Knowledge Graph (KG). In high-velocity domains like on-demand food delivery or e-commerce, the catalog of entities like dishes, products, and merchants changes rapidly.

Current methods for KG construction and maintenance face three critical challenges:

• Inaccuracy and hallucination from Large Language Models (LLMs): Automated models often infer relationships based on statistical text co-occurrence rather than semantic reality. For instance, an LLM might incorrectly classify “Pho” as a child of “Italian Noodle Soup” due to linguistic similarity, leading to irrelevant search results.
• Scalability limits of manual verification: Traditional verification relies on human annotators or domain experts. This approach is slow, expensive, and unable to keep pace with dynamic catalogs containing millions of entities. For example, daily changes in restaurant menus or grocery stock keeping units (SKUs).
• Error propagation in ranking: Inaccurate graph edges propagate errors downstream. If a parent-child relationship is wrong, query expansion algorithms will retrieve irrelevant items, directly degrading Click-Through Rate (CTR) and user trust.

We introduce a feedback-driven verification engine that operationalizes the search interface as a validation environment. Key contributions include:

• User feedback-driven verification: The system treats unverified graph edges as hypotheses. Instead of accepting them as truth, it tests them against live traffic by injecting them into search suggestions and measuring user engagement.
• Hierarchical relationship refinement: Unlike systems that only validate entities (nodes), this framework validates structural links (edges). It confirms whether entity A is truly a parent, child, or sibling of entity B, ensuring structural integrity.
• Adaptive exploration: The system employs a greedy exploration policy. It intelligently balances exploitation by showing known good results with exploration through injecting unverified candidates to gather data without degrading the user experience.

Background

Automated KG construction using LLMs and unstructured content extraction can scale quickly across large, dynamic catalogs. However, relationships inferred from text co-occurrence or vector similarity do not always reflect semantic reality. Manual verification by domain experts remains accurate but does not scale to millions of entities that change daily.

When inaccurate edges enter the graph, ranking and query expansion systems propagate those errors to users. Incorrect parent-child or sibling links lead to irrelevant search results, reduced CTR, and lower user trust. An additional solution is required that can validate graph structure continuously, at scale, without relying solely on manual curation.

Solution

The proposed framework functions as a closed-loop validation ecosystem. It is composed of four integrated modules designed to continuously cycle data from the KG to the user interface and back, using real-world interactions to separate semantic truth from artificial intelligence (AI) hallucinations.

The verification process follows a continuous, iterative loop that cycles data from the backend graph to the frontend user interface and back. This four-step procedure operationalizes the human-in-the-loop validation mechanism:

1. Hypothesis generation
2. Candidate injection
3. Signal aggregation and scoring
4. Graph update logic

Architecture details

KG core

The central repository acts as the source of truth, storing entities such as dishes, products, or merchants, and the connections between them. To manage the verification process, the system introduces a specialized metadata layer that classifies every connection (or edge) into one of two distinct states:

• Verified edges: These are established relationships that have been validated either by high historical traffic or human confirmation. They represent the safe structure of the graph. For example, “Sushi” is definitely a child of “Japanese Cuisine”, and is used to power standard search results.
• Candidate edges: These are probabilistic, unverified relationships generated by automated LLMs or content scrapers. They are treated as hypotheses waiting to be proven. For example, if an LLM ingests a blog post and predicts that “Pho” is related to “Italian Noodle Soup,” this link is stored as a candidate edge, invisible to the main search algorithm until validated.

Search and injection module

This module sits between the KG and the user, intercepting the query execution pipeline. Unlike standard ranking algorithms, which strictly optimize for relevance by showing only the best results, the injection engine employs a balanced strategy known as exploration vs. exploitation.

• The injection mechanism: When a user performs a search, the system retrieves a list of high-confidence results (exploitation). Simultaneously, it deliberately retrieves a small subset of candidate edges related to the query. It injects these unverified candidates into specific, lower-risk slots within the user interface, such as the third or fourth position in a related searches chip carousel.
• Risk management: To prevent user frustration, the system limits the number of candidates shown per session. This ensures that the user is primarily served helpful, verified content, while still providing enough data points to test new hypotheses.

Behavior tracking module

To accurately measure whether a candidate relationship is valid, the system tracks user micro-interactions with high granularity. It captures not just the final click, but the precise context in which the interaction occurred to determine semantic intent.

• Contextual anchoring: The system logs the specific search term, also known as the anchor, used by the user. A click on “Pho” is only counted as a vote for the relationship if the user was searching for “Noodle Soup” at the time.
• Signal classification: Signals are assessed in aggregate to estimate the relevance of a candidate relationship. Higher-intent engagement contributes stronger positive evidence, lighter exploratory behavior contributes weaker positive evidence, and lack of engagement or explicit negative actions contributes negative evidence.

Verification and refinement engine

This is an offline processing unit that acts as the final judge. It aggregates thousands of individual user signals to update the topology of the KG.

Relevance scoring: Instead of complex formulas, the engine calculates a simple confidence ratio. It looks at the total number of times a candidate was shown versus the number of positive interactions it received.

Graph topology updates:

• Promotion (verify): If the confidence ratio exceeds a verification threshold. For example, if the candidate performs as well as known good items, the edge is upgraded from candidate to verified. It becomes a permanent part of the graph and is shown to all users.
• Demotion (prune): If the candidate consistently fails to garner engagement or receives negative signals, it falls below a pruning threshold. The system automatically deletes this edge, effectively correcting the AI’s hallucination and cleaning the dataset.

Implementation

Hypothesis generation

The process begins by identifying a target subject, referred to as the anchor entity. For example, the specific dish “Pho”. The system queries the KG to retrieve a set of potential relationships. This retrieval includes both verified neighbors, where relationships are already confirmed by experts, and candidate neighbors, where the relationships are predicted by AI models but not yet proven.

Candidate injection

Once a hypothesis is selected, the system exposes it to real users to gather evidence. When a user actively searches for the anchor entity, the system dynamically injects the candidate neighbor into the search results.

• User interface (UI) implementation: The candidate is presented alongside verified items, typically in a related categories carousel or a refine search chip list. This reflects standard relevance experimentation in search, with safeguards to ensure the experience remains controlled and measurable.
• Exposure logging: The system logs an impression event specifically linking the anchor to the candidate. This record serves as the baseline, documenting that the user saw the relationship, which is essential for calculating future engagement rates.

Signal aggregation and scoring

Instead of using a raw count of clicks, the system calculates a sophisticated relationship confidence score by aggregating user interactions over time. This scoring model uses a weighted tier system to distinguish between casual interest and strong intent.

• Weighted interaction logic: The system assigns a higher value to actions that require more effort or commitment. For example, a “Purchase” or “Add-to-Cart” action is weighted significantly heavier than a simple click, as it indicates a strong validation of the relationship. Conversely, scrolling past the item quickly or skipping is treated as a negative signal.
• Normalization: To ensure fairness, the total weighted score is normalized against the total number of times the candidate was shown. This prevents niche items with low total traffic but high accuracy from being unfairly penalized.

Graph update logic

Periodically, the verification engine evaluates the confidence score against predefined benchmarks to update the KG’s topology. This is a binary decision process:

• Validation (cementing the edge): If the accumulated confidence score exceeds a strict validation threshold, the system concludes that the relationship is genuine. The status of the edge is updated from candidate to verified. This permanently adds the relationship to the graph, ensuring it appears in future standard searches without the need for further testing.
• Rejection (pruning the edge): Conversely, if the score falls below a rejection threshold, indicating that users consistently ignore or reject the suggestion, the system concludes the relationship is an AI hallucination. The edge is severed or removed from the graph. This pruning action cleans the dataset, preventing the system from making the same bad recommendation again.

Case study: hierarchical refinement in food delivery

To demonstrate the framework, consider a validation scenario in food delivery taxonomy. An LLM-based ingestion pipeline flags a candidate parent-child link Noodle Soup → Dry Mee Pok and stores it as an unverified candidate edge in the KG, ready for live validation.

User-triggered validation:

When a user searches for “Noodle Soup,” the search module injects the candidate alongside verified results. For example, in a “Refine by Dish” filter carousel, and logs an impression linking the anchor query to the candidate.

Outcome collection:

User interactions like clicks, dwell time, scroll behavior, and conversions are captured and weighted over the validation window. The verification engine aggregates these signals and updates the graph: relationships that meet the validation threshold are promoted to verified status; those that fail are pruned or re-mapped to a more appropriate parent node.

Impact

By injecting unverified candidate edges into live search results and recommendation interfaces via a multi-armed bandit (MAB) exploration strategy, the system leverages implicit user feedback to validate semantic truth. This dynamic, human-in-the-loop mechanism effectively prunes erroneous connections and reinforces accurate taxonomies without the need for manual curation, significantly enhancing search relevance in dynamic domains such as food delivery and retail.

Learnings and conclusion

The feedback-driven verification engine operationalizes the search interface as a validation environment for KG relationships. By classifying edges as verified or candidate, injecting candidates through an exploration vs. exploitation strategy, and aggregating weighted user signals, the system promotes accurate relationships and prunes AI hallucinations at scale.

Unlike approaches that validate only entities, this framework validates structural links, confirming whether entity A is truly a parent, child, or sibling of entity B. The food delivery case study shows how a user-triggered search can initiate validation and outcome collection at scale, without manual intervention.

What’s next

Hierarchical confidence tiers

To safely graduate new connections into the production graph, we are introducing a dual-measurement trust system that requires both volume and variety before a new connection goes live: support mass (product hits, graph depth, recency) and corroboration (unique sessions, anonymous cohorts, and temporal spread). Connections must climb a strict state machine: proposed → shadow eligible → canary eligible → production, advancing only when both metrics meet progressively higher thresholds; if a snapshot causes metrics to fall below a tier’s floor, the connection is automatically demoted.

Adversarial and spam resistance

To prevent bad actors, bots, or highly repetitive users from manipulating the search graph, we are building a multi-layered defense system. We enforce per-merchant rate limits and anti‑abuse controls: hourly caps per session/device, exponential backoff for rapidly repeated actions, and a short (few‑hour) freeze of promotions from any user cohort after declines or “irrelevant” signals. For bot and Sybil attack defense, traffic flagged by abuse systems is excluded from trust calculations (but logged for analysis); votes must come from diverse network subnets or cohort buckets, and each bucket is subject to a daily contribution cap.

PGSimCity

PGSimCity is an independent, non-commercial educational visualization of PostgreSQL internals. It is not affiliated with, sponsored, endorsed, or approved by Electronic Arts Inc. SimCity is a trademark of Electronic Arts Inc.

A working model of the PostgreSQL engine

This city's model and explanations describe PostgreSQL 18.4.

Early, reviewed prototype. Reviews have found and fixed inaccuracies in both the model and explanations.

Postgres Index Types Explained: B-tree, GIN, BRIN, and Operators

In my post about JSON and JSONB I told you to create a GIN index with jsonb_path_ops and moved on, because that post was about a column type and not about indexing. This one pays the debt. What a GIN index actually is, why Postgres ships five other index types.

Every example below runs as written on a stock Postgres 17. The plans and sizes are real output from my machine.

An index is a bet

An index is a separate data structure that trades write speed and disk for read speed. Every insert and update has to maintain every index on the table, and every index occupies real space, which you will see measured at the end of this post. That framing matters because the question is never “should I index this table” but “which reads are worth taxing my writes for”. An index that no query uses is a pure tax.

Indexes accelerate operators, not columns

When the planner considers an index, it does not ask “is there an index on this column”. It asks “is there an index whose type knows how to answer this operator”. A B-tree knows how to answer =, <, <=, >=, > and BETWEEN, because it keeps values in sorted order. It has no idea what to do with the array containment operator @>. A GIN index answers @> natively and cannot help you with <.

The glue between an index type and the operators it serves is called an operator class. Most of the time the default class is what you want and you never type its name. The moment it becomes practical is when one index type offers a choice, which is exactly the jsonb_ops versus jsonb_path_ops decision from the JSONB post: same GIN machinery, different set of supported operators, different size.

Keep that lens for the rest of the post. Each index type below is just a different answer to the question “which operators do you need to be fast”.

The demo table

One table, one million rows, shaped like an event log because that is where indexing decisions get interesting: a user id to look up, a status that is almost always ok, a tags array, and a timestamp that grows with insert order.

CREATE TABLE events (
  id         bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  user_id    int    NOT NULL,
  status     text   NOT NULL,
  tags       text[] NOT NULL,
  created_at timestamptz NOT NULL
);

INSERT INTO events (user_id, status, tags, created_at)
SELECT
  i % 50000,
  CASE WHEN i % 211 = 0 THEN 'failed' ELSE 'ok' END,
  ARRAY['app' || i % 7, (ARRAY['auth', 'billing', 'search', 'export', 'sync'])[1 + i % 5]]
    || CASE WHEN i % 397 = 0 THEN ARRAY['beta'] ELSE '{}' END,
  timestamptz '2026-01-01 00:00:00+00' + i * interval '2 seconds'
FROM generate_series(0, 999999) AS i;

ANALYZE events;

Life without an index

Ask for one user’s events with nothing but the primary key in place:

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM events WHERE user_id = 12345;

Gather (actual rows=20 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on events (actual rows=7 loops=3)
         Filter: (user_id = 12345)
         Rows Removed by Filter: 333327
 Planning Time: 0.083 ms
 Execution Time: 15.696 ms

Postgres read the entire table and threw two parallel workers at it to make that less painful. Twenty matching rows required inspecting a million. A sequential scan is the baseline, and on small tables it is often the genuinely fastest plan. On a million rows, for twenty matches, it is the thing indexes exist to avoid.

B-tree, the default for a reason

CREATE INDEX with no USING clause gives you a B-tree, a balanced tree of sorted values. Sorted order is why it covers the widest set of operators: equality, every comparison, BETWEEN, and it can feed ORDER BY without a sort step.

CREATE INDEX events_user_id_idx ON events (user_id);

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM events WHERE user_id = 12345;

Bitmap Heap Scan on events (actual rows=20 loops=1)
   Recheck Cond: (user_id = 12345)
   Heap Blocks: exact=20
   ->  Bitmap Index Scan on events_user_id_idx (actual rows=20 loops=1)
         Index Cond: (user_id = 12345)
 Planning Time: 0.087 ms
 Execution Time: 0.046 ms

From 15.7 milliseconds to 0.046. Same query, same data, three hundred times faster.

A bitmap scan is a two-phase strategy. The Bitmap Index Scan walks the index and collects the locations of every matching row into a bitmap in memory. The Bitmap Heap Scan then sorts those locations by page and visits each table page exactly once. When matches are scattered across the table, as they are here, this beats jumping back and forth between index and table row by row. You will meet this plan shape with every index type in this post, because GIN and BRIN produce their results as bitmaps by nature.

The same B-tree serves range queries for free:

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM events WHERE user_id BETWEEN 100 AND 199;

Bitmap Heap Scan on events (actual rows=2000 loops=1)
   Recheck Cond: ((user_id >= 100) AND (user_id <= 199))
   Heap Blocks: exact=41
   ->  Bitmap Index Scan on events_user_id_idx (actual rows=2000 loops=1)
         Index Cond: ((user_id >= 100) AND (user_id <= 199))
 Planning Time: 0.048 ms
 Execution Time: 0.137 ms

If your predicate is equality or ordering on a scalar, the B-tree is almost always the answer, which is why it is the default.

GIN, the inverted index

A B-tree stores one entry per row. That model collapses when a single value contains many searchable elements: an array of tags, the keys of a jsonb document, the words of a text document. You do not want an entry per row, you want an entry per element, pointing back at every row containing it. That structure is an inverted index, and in Postgres it is called GIN, for Generalized Inverted Index.

CREATE INDEX events_tags_idx ON events USING gin (tags);

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM events WHERE tags @> ARRAY['beta'];

Bitmap Heap Scan on events (actual rows=2519 loops=1)
   Recheck Cond: (tags @> '{beta}'::text[])
   Heap Blocks: exact=2519
   ->  Bitmap Index Scan on events_tags_idx (actual rows=2519 loops=1)
         Index Cond: (tags @> '{beta}'::text[])
 Planning Time: 0.105 ms
 Execution Time: 1.712 ms

The @> containment operator asks "does this array contain these elements", the GIN index looks up beta in its element catalog, and 2,519 rows come back without touching the other 997,481. Swap the array for a jsonb column and this is exactly the index and operator pair from the JSONB post. Full-text search runs on the same machinery.

The cost side: GIN is the most expensive index here to maintain on writes, because one row insert may add many index entries. It earns that cost only when your queries genuinely ask containment-style questions.

BRIN, the index that is barely there

BRIN, Block Range Index, does not store row locations at all. It stores a summary per range of table pages, by default the minimum and maximum value found in each range of 128 pages. A query for a value range lets Postgres skip every block range whose summary cannot contain a match.

That only works when the physical layout correlates with the values, which is precisely the situation of a timestamp on an append-only table: rows arrive in time order, so each block range covers a narrow slice of time.

CREATE INDEX events_created_brin ON events USING brin (created_at);

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM events
WHERE created_at >= '2026-01-10' AND created_at < '2026-01-11';

Aggregate (actual rows=1 loops=1)
   ->  Bitmap Heap Scan on events (actual rows=43200 loops=1)
         Recheck Cond: ((created_at >= '2026-01-10 00:00:00+00'...)
         Rows Removed by Index Recheck: 13120
         Heap Blocks: lossy=640
         ->  Bitmap Index Scan on events_created_brin (actual rows=6400 loops=1)
               Index Cond: (...)
 Planning Time: 0.093 ms
 Execution Time: 3.755 ms

Note lossy=640 and the recheck removing 13,120 rows. BRIN cannot say "row 5 matches", only "something in these pages might match", so Postgres visits the candidate pages and filters. That imprecision is the price of the headline number:

CREATE INDEX events_created_btree ON events (created_at);

SELECT relname AS index_name, pg_size_pretty(pg_relation_size(oid)) AS size
FROM pg_class
WHERE relname IN ('events_created_brin', 'events_created_btree');

index_name      | size
----------------------+-------
 events_created_brin  | 24 kB
 events_created_btree | 21 MB

Twenty-four kilobytes against twenty-one megabytes for the same column, a factor of nearly a thousand. On logging, metrics, and event tables that only grow, BRIN buys you most of the benefit for a rounding error of disk and near-zero write overhead. On columns with no physical correlation it buys you nothing, which is the trade in one sentence.

Partial and expression indexes

These are not index types but modifiers that apply to any of the above, and they solve extremely common problems.

A partial index carries a WHERE clause and only indexes matching rows. My status column is failed in less than half a percent of rows, and failed rows are the only ones I ever look up by status:

CREATE INDEX events_failed_idx ON events (created_at) WHERE status = 'failed';

EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM events
WHERE status = 'failed' AND created_at >= '2026-01-20';

Index Scan using events_failed_idx on events (actual rows=849 loops=1)
   Index Cond: (created_at >= '2026-01-20 00:00:00+00'...)
 Planning Time: 0.226 ms
 Execution Time: 0.515 ms

The index is 120 kB against 21 MB for its full-table equivalent, and every insert of an ok row skips it entirely. This plan is also a plain Index Scan rather than a bitmap: few enough rows, so Postgres walks the index and fetches rows directly.

An expression index indexes the result of an expression instead of a raw column, which is how you index lower(email), or a single hot key extracted from a jsonb document:

CREATE INDEX chunks_source_type_idx ON chunks ((metadata ->> 'source_type'));

The bill

Everything above, measured. This is the table and every index this post created on it:

relname               |    size
----------------------+------------
 events               | 89 MB
 events_pkey          | 21 MB
 events_created_btree | 21 MB
 events_user_id_idx   | 7600 kB
 events_tags_idx      | 2360 kB
 events_failed_idx    | 120 kB
 events_created_brin  | 24 kB

The indexes together add up to more than half the size of the table itself, and every row written pays maintenance on all of them. This is why “just add an index” is not free advice, and why the sizes span three orders of magnitude for the same job on the same data.

Let the operator choose

After all of this, the decision procedure is short, because the operator in your WHERE clause has already made it. Equality and ranges on scalars want a B-tree. Containment questions against arrays, jsonb, or text search want GIN. Time ranges over huge append-only tables want BRIN. Then partial and expression modifiers narrow whichever one you picked to the rows and expressions you actually query.

When a plan surprises you, read it with the operator lens: EXPLAIN ANALYZE tells you which index answered which condition, and a sequential scan usually means no index on the table speaks the operator you used, or the table is small enough that speaking it does not matter.

If the GIN line in my JSONB post sent you here, you now have the whole picture: the column type decides which operators exist, and the operators decide which index earns its disk.

Full article content is not available for inline reading.

Read the original article →

Design systems drift when Figma components and front-end code fall out of sync, and AI often widens that gap rather than closing it. A proposed fix uses "component contracts" — plain-language specs that become the shared source of truth — paired with harness engineering, deterministic guardrails surrounding an AI model. Testing showed an AI agent scored 100/100 building screens from a contract catalog, versus 69/100 when ungoverned.

Visual website builder and CMS, now open source

Use Ycode Cloud to get started in minutes, or self-host it on your own infrastructure.

Website builders have become incredibly powerful, but they often come with trade-offs. When you build on a closed platform, your website lives inside their ecosystem. Your design, content, and infrastructure depend on their pricing, rules, and roadmap.

We believe building a website shouldn’t mean giving up control. Ycode combines a modern visual builder with the freedom of open source — so you can self-host it on your own infrastructure or use Ycode Cloud if you prefer a fully managed setup.

Run Ycode yourself

Fork the repo, deploy it to Vercel, connect Supabase, and you’re ready to build.

1. Fork the repository

Create your own copy of Ycode on GitHub.

2. Deploy to Vercel

Launch your project in minutes using Vercel.

3. Connect Supabase

Add your Supabase and start building.

Bring your own AI

• Anthropic (Claude): Opus 5, Fable 5, Sonnet 5
• OpenAI (ChatGPT): GPT-5.5, GPT-5 Mini
• Google Gemini: Gemini 3.1 Pro, Gemini 3.5 Flash
• xAI (Grok): Grok 4.5, Grok 4.3

Design with a prompt
Update layouts, spacing, styles, and responsive behavior without editing every element manually.

Manage your CMS
Create collections, add fields, update content, and organize structured data through conversation.

Build reusable components
Create new components or improve existing ones while keeping your website consistent.

Pricing

Self-host: $0 per month

• Open source Ycode platform
• Host on your own infrastructure
• Full code access
• Custom domains
• Unlimited users, pages, CMS items, form entries, and languages
• Community support

Cloud: $25 - $30 per month

• Fully managed Ycode hosting
• Automatic updates and maintenance
• Global CDN and fast hosting
• Custom domains
• Unlimited users, pages, form entries, and languages
• 20,000 CMS items
• Community & Email support

Releases

0.2.0
Adds collection filtering with pagination, dynamic sorting, and improved component overrides. Includes many fixes and UX improvements across publishing, filters, and the rich-text editor.

Build with the community

Connect with designers, creators, and founders building and sharing with Ycode.

"I built already 3 websites for my clients using Ycode. Glad to have had the support of the Ycode team and also the great templates available! I’m a big fan of how simple and straightforward the Ycode UI is."
Laís Lara Vacco, Product Designer

"Thanks a lot to the Ycode team for building such a great tool! PS: We switched from WebFlow to Ycode and what a good decision it was."
Pascal, Founder of Yuppa

"Top of the art tool for designing websites compared to Webflow. Also GDPR compliant in many ways and the team is engaging with the community on new releases."
Mikael Larsen, Designer at BK Aktiweb

"For me, it's the best option between WordPress and Webflow. It fits perfectly in my case."
Fabiano F, Marketing Project Manager

"If you're looking for a powerful and user-friendly Webflow alternative, Ycode is a top contender..."
Piotr Bak, Founder of Digital Agency

"We've been using Ycode for almost 2 years now, and it's been a great experience. We rebuilt our website and migrated from another CMS with minimal involvement from our IT department."
Olha L, Growth Marketing Manager

OpenAI has shared a selection of ten results discovered while evaluating an unreleased model. Each resolves or makes substantial progress on a long-standing open problem. These problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics. All of these problems are of substantial interest to their respective mathematical communities. Several are of broad interest across mathematics as a whole.

Jacob Tsimerman, who recently won the Fields Medal, is starting a position at OpenAI. Tsimerman previously wrote a paper categorizing the ways AI might kill everyone. He appears to be so worried about the dangers of AI that he's pivoting to work on AI safety. The star professor wants to use math to advance the study of AI and ensure that the technology won't lead to our extinction.

When Genius Fails—The Intellectual Arrogance of the AI Labs

Being an expert in one field doesn’t make you an expert in all fields. Leopold Aschenbrenner’s hedge fund, Situational Awareness, provided a $20 billion demonstration this week. His claim to fame was being part of OpenAI’s Superalignment team before being fired over alleged leaks (which he disputes) and then publishing an essay in 2024 about the imminence and importance of AGI that launched a thousand media interviews (of him). And then he was running a $20 billion hedge fund. And then it blew up.

Lots of people will be dancing to the news this week because many found him a bit insufferable. And for me, I really don’t mind no longer being asked, “Should I invest in Situational Awareness?” and needing to be delicate about it.

I want to make a larger point, though. The lack of intellectual humility within the frontier AI lab culture he hails from extends beyond him into many verticals other than money management. That being said, as an ex-hedge fund guy myself who also has an AI background, I do have some unique qualifications to at least talk briefly about this.

Situational Awareness LP’s woes are not unique. Many hedge funds have blown up. In fact, I’d say many, many more hedge funds have blown up than have ever been consistently excellent. It’s something that most laypeople don’t realize.

The canonical example is Long-Term Capital Management, which put two Nobel laureates and Wall Street’s best bond traders in one fund. It was the best and brightest in the field, and they quadrupled investors’ money in four years. Then it blew up so spectacularly in 1998 that the Federal Reserve had to rally Wall Street banks to help bail it out (in a preview of 2008).

There’s a whole book about it, fittingly titled When Genius Failed.

It’s not about your peak returns. After all, someone who goes all-in on red at the roulette table five times in a row and wins by luck will have a 3,100% return. I would hope that no one would think this person is a unique genius or qualified to manage money.

It’s not even about “beating the market.” That’s a red herring. In a bull market—or even better, a bubble—anyone who isn’t fully invested, or more than fully invested (with leverage), in the stock market will “lose.” What you care about from a hedge fund is that they are consistent in bull or bear markets. The point of those expensive fees is that they will always perform, even if they look temporarily “bad” against the stock market.

Which, by the way, is not the only market in the world—there are bonds, commodities… but it gets the attention because retail investors love to gamble in it.

Which brings us back to Leopold Aschenbrenner and his hedge fund.

By all accounts, he (like the Korean retail traders) levered into the AI boom (reportedly running around 4x), with July losses across public stocks like neoclouds, memory names, and datacenter power. He also, reportedly, had short positions in software names—the “SaaSpocolypse” trade—that bounced back against him at the same time.

I have no doubt Aschenbrenner is super smart, but this doesn’t look that different from the Korean retail investors who blew up. One of the first lessons any real investor learns is the market can stay irrational for longer than you can stay solvent… if you don’t have the right risk controls. Using leverage is just the most obvious part of it.

Of course, it’s because of his thesis. From his founding essay:

Because—it’s starting to feel real, very real. A few years ago, at least for me, I took these ideas seriously—but they were abstract, quarantined in models and probability estimates. Now it feels extremely visceral. I can see it. I can see how AGI will be built. It’s no longer about estimates of human brain size and hypotheticals and theoretical extrapolations and all that—I can basically tell you the cluster AGI will be trained on and when it will be built, the rough combination of algorithms we’ll use, the unsolved problems and the path to solving them, the list of people that will matter. I can see it. It is extremely visceral. Sure, going all-in leveraged long Nvidia in early 2023 has been great and all, but the burdens of history are heavy. I would not choose this.

Ah, all-in leveraged long Nvidia. We got a taste of his investing style back then, before he even had a fund.

Beyond Situational Awareness

Sam Altman and Dario Amodei regularly trade off in how apocalyptically they describe the future of the labor market. And while I name those two because they’re CEOs of the two most prominent AI labs, this isn’t really restricted to just them. What I personally find infuriating is how little grounding most of these statements have in either economic history or theory—which is perhaps unsurprising, because most everyone making them is a deep expert in AI, not those fields.

And this goes beyond how unpredictable markets and technology are. It reminds me of Thomas Malthus, who actually was an economist. He predicted in 1798 that we’d inevitably run out of food. He was famously wrong.

Beyond food, we’ve had periodic doomsaying about the labor market in the face of technological change. But the reality is, even when there’s been disruption, the labor market has adapted—and to far more dramatic change than we’re talking about right now.

And even beyond that, I’ve now seen multiple cases of either my or other investors’ portfolio companies being approached by companies affiliated with the leading labs with incredible confidence in their own broad-spectrum intellectual superiority.

Let’s just say a materials science startup is talking with an OpenAI satellite company. The data and expertise of that startup are important to the satellite’s core pursuit. Talks are going well. And then, all of a sudden, someone on the OpenAI satellite team asks, “Why don’t we just do this [super hard deep science problem] with ChatGPT ourselves?” I’ve seen this in multiple cases in similarly deep, difficult areas. Like bioengineering. Or semiconductor design.

And look, I’ve personally played around with models, even with CAD and PCB design, with surprisingly good results. I am not an “AI skeptic.” There is a spectrum, though, between R&D and a rote task in a hard field. Especially given it tends to work best—like in my case—when you have a human with at least some expertise driving.

Many within the AI lab community have fully drunk their own Kool-Aid on total AI supremacy, even for tasks that will need a lot of human creativity and help for the foreseeable future.

It’s much easier to say someone else’s job is going to be fully replaceable by AI when you don’t actually know what they do.

The HuggingFace Hack

Recent news also gave us a great example of why this isn’t just an abstract cultural complaint. In mid-July, during OpenAI’s internal security evaluations, GPT-5.6 Sol and a thus far unreleased (more powerful) model escaped their sandboxed test environment. The agents were hunting for a hard benchmark’s answer, and I’d guess that they were told to do whatever it takes.

In doing so, the agents autonomously breached Hugging Face’s production infrastructure through a chain of vulnerabilities that allowed them to get access to the open internet. As per Ben Thompson at Sharp Tech, this was likely due to sloppiness in OpenAI’s controls. Though, to be fair, just yesterday Anthropic revealed that its models (including Mythos 5) had also accidentally compromised real organizations during testing.

Anyway, yes, these models are powerful—which has been something I’ve said as well and is not really in question.

The Kafkaesque side of this comes from Hugging Face, though. They realized they were under attack and tried to use a frontier model from one of the leading American labs to help defend themselves. It refused as part of its “safety” guardrails—the model couldn’t distinguish an incident responder from an attacker. Which makes total sense—you need to actually probe for weaknesses to, you know, figure out the weakness. Instead, they had to use GLM-5.2, a Chinese open-weight model, to defend them.

It’s kind of funny that US companies are relying on Chinese models—given China has been prominent in (successful) hacking attempts on American infrastructure.

But, of course, this is because of a decision by the AI companies that “know better” on how their models should be used and have a rather patronizing attitude generally about giving access to their models.

No one likes an arrogant prick—especially if they’re wrong

Superintelligence has been “right around the corner” every year. We keep getting, bafflingly, predictions of labor market carnage, which is not happening. All of this is helping make AI extraordinarily unpopular with regular people. And for no good reason, especially when regular people have a lot to gain from AI, if the industry stopped trying to make everyone hate it.

I get it. It feels like the end of history because it’s their own little bubble. And they don’t have the perspective to understand things outside of it. This is especially bad because much of the industry has also made itself an island full of PhDs.

And it’s not that I have something against PhDs. I took PhD seminars myself. Many individuals with PhDs have plenty of intellectual humility, especially when they’ve taken the time to gain broader perspectives outside of their field.

But fundamentally, given how long it takes to get one, you’re usually taking a population from a young age that hasn’t had much exposure to anything outside the academic community. This is also compounded by a lot of self-congratulatory rhetoric from the leaders of those communities.

While, despite my best efforts, it feels like I’m bashing the degree, this is more a tendency created by high expertise (and accomplishment) in narrow areas. Surgeons and electrical engineers may be extremely smart and accomplished, but that doesn’t mean they have the ability to weigh in on fields outside their expertise.

The field’s belief in its own apotheosis, however, is not only annoying but may end up damaging its ability to make an impact—through broad unpopularity—or causing actual damage from simple intellectual arrogance, magnified by how much money the industry has to throw around right now.

When Trump came into office, he took the guardrails that were hindering AI development, and Larry Ellison cashed in by signing several large infrastructure deals. This big bet on AI was built on an astronomical amount of debt in every imaginable form. The hypothesis was that whoever controlled the most computing power would control the AI economy. Investors have started questioning whether this level of spending is sustainable in recent weeks and are concerned about whether the money being pumped into data centers will ever return the promised profits.

Leopold Aschenbrenner had planned a multiday celebration for his wedding, but his fund started unraveling just when guests started to arrive. The 24-year-old investor had built a $45 billion investment powerhouse that primarily bought stocks in the AI trade. The firm borrowed too much money to make these AI bets, leaving it at risk as they faltered. Citadel then swooped in and purchased the vast majority of the firm's public stock portfolio at a discount of more than 10% to the market value of those shares. The firm is left with a portfolio of stocks and stakes in startups valued at more than $10 billion, but it is still alive.

Despite growing calls for the US to develop stronger open-weight models to compete with China, investors are reluctant to back open-weight startups.

For the first time I can remember, I'm not choosing my daily driver models on raw intelligence. I'm choosing them on speed.

Have we reached an intelligence tipping point?

This is probably going to age like spoilt milk, but right now, models around the ~Opus 4.6 level seem to be 'smart enough' for most of my daily tasks - code, pulling together research, designing slide decks and doing analytical tasks against a plethora of databases.

While like most I was hyped to play around with Fable, ironically the US Gov shutdown gave everyone time to get used to Opus again. When Fable came back post-hype with additional guardrails, the first thing I noticed was just how slow it is. So slow, actually, that I switched back to Opus pretty quickly.

I've spent a lot of my career making software fast. It's remarkable how much better software feels to interact with when it's fast. In my experience (and many studies), you can take the most beautiful product, but if it's slow, you won't enjoy using it. Equally, you can take a very basic product that's super fast and it will feel brilliantly utilitarian.

It's clear to me that when only a few, big, slow models cleared the aforementioned (and hypothetical) intelligence bar, it wasn't worth the trade off really to use a slower model. Whatever speed you gain you quickly lose in having to redo it because it was broken.

Is 100tok/s the new 100ms?

I've written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected.

The key fact I remember is that to humans, ~100ms feels 'instant', the gold standard. I reckon 100tok/s output on a model is about as fast as I can keep up with. After that, it comes in faster than I can (skim) read. This isn't an exact bar, because increasingly most of the model time is spent in reasoning, and not actually showing you output tokens. And also it massively varies on your output, prose gets output with far fewer tokens per character than code, so your mileage may (and will) vary.

But roughly, 100-200tok/s to me seems pretty damn fast. Below 50tok/s output feels increasingly slow. Ironically, going past 200tok/s seems almost unnerving. I'm sure this feeling will edge upwards as we get used to it and push our agents to do more complicated work.

Given the plethora of new models that I think are ~clearing the aforementioned bar - such as GLM5.2 and DeepSeek V4 Flash GA - that are open weights and small(er), we now have a wide range of models and speed. If you look at the speed rankings of various providers for GLM5.2 on OpenRouter you can see the enormous range of serving speed - from less than 30tok/s at the bottom to 129tok/s at the top.

This is another huge plus to the open weights ecosystem. While there are great benefits in cost that are obvious, the fact that providers are also incentivised to compete on speed like this is really interesting.

But there are limitations

If you're familiar with Pareto's Principle and Amdahl's Law you'll know what's coming up.

Assuming "good enough" models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them.

Take an agent using a model processing at 50tok/s. Most of the time is spent waiting for inference to come back. Now run the same turn at 250tok/s and you'll see that increasingly you are bottlenecked on tool calls on your "local" machine and your decision making.

Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn't move.

And even worse, making your local machine faster on these tool calls is sort of stalling out, because hardware costs have gone parabolic because of AI. Yet again another weird derivative effect of the AI market.

So I suspect (for now at least) there is a limit to how much demand there will be for speed, past a certain point. No doubt there'll be some examples where huge amounts of reasoning are useful (like mathematics research), and speeding that up is helpful. But I'd expect many agents to start getting bottlenecked on your local/internal hardware, database calls and other bits of latency.

The price war is coming

Interestingly OpenAI reduced the cost of their Luna variant by 80% just before the DeepSeek V4 Flash GA release, making it remarkably affordable for a frontier model. While I haven't had as much luck with getting great output out of it vs GLM5.2, I think it points towards an absolute bloodbath of pricing at this end of the market.

You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We're already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus.

While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra.

As the next generation set of GPUs start being deployed over the next few months - Nvidia's Vera Rubin series and AMD's MI400s, amongst others - the new HBM4 memory in those chips will deliver a 2x+ speedup on output tokens from memory bandwidth alone, plus more on top from additional compute and interlink.

In 2027 it's very possible we'll have very good quality models, at reasonable prices running at 500tok/s+. Staring at "still thinking on xhigh effort" for most of your day may finally become a thing of the past.

What will be interesting to watch for - and I'm not sure where to bet - is if the vast 2-3T+ param models actually do perform dramatically better for everyday tasks. On one hand it feels like we've hit a sweet spot right now, on another having an order of magnitude more intelligence in the model may make that sweet spot look very, very primitive.

1. Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day.

2. A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product.

3. I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market.

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Every day, cameras, generate more data than every other source on the planet combined. Warehouses record every dock, aisle, and forklift. Robots capture every second of every task they attempt. Autonomous vehicles log millions of miles of the physical world in motion.

And then almost all of it sits there, unusable.

Not because it lacks value. Because it lacks structure.

You can store video. You can watch video. You cannot query it.

Think about what organizations can actually do with their visual data today.

They can store it, at enormous and growing cost. They can stream it to a wall of monitors that nobody watches. They can retrieve a clip, if someone already knows the camera, the date, and the timestamp they are looking for.

What they cannot do is ask it a question.

A robotics team with 10,000 hours of deployment footage cannot ask: show me every grasp failure involving reflective objects.

A logistics operator cannot ask: pull every near-miss at dock door 7 this quarter.

A safety lead cannot ask: how often are workers entering the exclusion zone while the arm is live?

To answer questions like these, teams fall back on two options, both bad.

1. Armies of people scrubbing footage manually.
2. Or expensive AI inference, re-run from scratch on the same raw video every single time a new question comes up.

That second option deserves a closer look, because it is quietly becoming the default. Modern vision-language models are remarkable, and pointing one at your footage feels like a solution.

But every question means re-processing raw pixels again. Ask ten questions, pay ten times. Your archive never gets smarter. Your costs scale with your curiosity.

The data is a liability that occasionally, expensively, yields an answer.

Text already solved this. Decades ago.

There was a time when the world's text was in the same condition: piles of documents, readable but not queryable, useful only to whoever had time to go through them by hand.

Then databases happened. Text was given structure, and everything changed. Structured language is why search works, why analytics exist, why every enterprise system you rely on can answer a question in milliseconds.

The entire modern software economy stands on one idea: structure your data once, and you can query it forever.

Video never got that moment.

The models arrived first. In the last few years, machines learned to genuinely understand what they see. That work is real, and it is extraordinary.

But intelligence pointed at unstructured data produces answers, not infrastructure. Understanding a video is not the same as making video understandable.

The bottleneck in visual AI is no longer the model. It is the missing layer underneath.

Structuring an interactive knowledge base

This is the layer we built.

CreativAI's architecture allows you to structure visual data. As footage flows in from cameras, robots, or drones, Creativ ingests it and transforms it into structured, queryable intelligence: what happened, where, when, involving what.

Think of this as the rows and columns being built around your visual data. This allows teams to Index once and query forever.

Once your visual data is structured, it behaves like a knowledge base.

• You ask questions in plain language and get answers in seconds.
• Your agents and applications query it programmatically.
• Your ML team searches the archive for exactly the scenarios their models need next, instead of paying to rediscover them.

Every question after the first one is nearly free. The archive compounds in value instead of compounding in cost.

And because it is infrastructure, it sits underneath whatever you have already built. Your models, your perception stack, your applications — all get better data.

It runs where your operations run: in the cloud, on-prem, and soon on the edge.

What this unlocks

For teams building embodied AI and autonomous systems, Creativ turns deployment footage from a storage bill into a training asset. Failure discovery, scenario mining, and model iteration run against structured data instead of raw video.

For operators of physical infrastructure — like warehouses, logistics networks, and industrial sites — it turns camera archives from evidence-of-last-resort into an operational intelligence layer. Incident investigation, compliance, and SLA disputes get answered in minutes.

With any industry, really, you point our solution to it and you get structured visual intelligence.

Different industries, same shift: visual data stops being something you store and starts being something you use.

The SQL layer for Physical and Visual AI

Every major data type eventually gets its infrastructure layer. Text got databases. Code got GitHub.

Visual data is next, and the timing is not an accident.

The world is entering the era of Physical AI: robots in warehouses, drones in the field, autonomous systems in motion, cameras on everything. All of it sees. Almost none of it remembers in a form anyone can use.

CreativAI is the SQL layer for Physical and Visual AI.

We spent our time in stealth building it, deploying it with design partners across robotics, logistics, and enterprise, and proving that the architecture holds up where it matters: in the field.

Today we are making it public.

If your organization is sitting on visual data it cannot query, we should talk.

Reddit is taking a cue from TikTok to bring viral stories to life through video. On Thursday’s second-quarter earnings call, the company said it’s been working on a new “video Reddit” experience that would not only allow users to watch videos from the platform, but also listen to posts in the background.

CEO Steve Huffman told investors that the company already sees people consuming this type of content on other platforms.

He’s right — on TikTok, for instance, there are 19.6 million posts labeled with the hashtag #reddit and another 9.9 million labeled #redditstories. Many of these feature a text-to-speech narration of a viral story published on Reddit paired with an unrelated video, like gameplay footage or cooking content.

“There is an emerging content type elsewhere on the internet of, basically, podcasts where people read Reddit content,” Huffman explained on the call. “I think this version of, like, listened-to or spoken Reddit can be really engaging, as well,” he said. “So that would be almost a different format entirely.”

It’s not clear how these types of Reddit videos will be incorporated into Reddit’s main app, but Huffman teased that the company will have something available for testing “later this year.”

The news comes as mainstream social and entertainment apps experiment with adding TikTok-like video feeds to their products.

Last week, Facebook announced plans to begin testing later this year a reimagined experience that will put a subset of users into a full-screen video as soon as they open the app. Streaming apps like Netflix, Disney+, Peacock, and HBO Max have also added TikTok-like short video feeds in recent months. Amazon offers a short video feed for shopping inspiration. There’s even a short video feed on LinkedIn, of all places.

It’s the TikTok-ification of the web at scale.

In addition, the company touted the early success of video in comments, which launched in June. Reddit said the new feature already accounts for more than 10% of its video posts.

Your people get AI. Get out of their way.

We are all using the same tools. Companies that win will be the ones willing to trust the people who already understand them.

Here’s a number more people should be talking about: Ninety-five percent of corporate AI projects go nowhere. MIT looked into why, and the answer had nothing to do with the tools being used. It was the companies using them.

I recently had two conversations that explain almost everything about why so many companies are finding it challenging to find success using AI. These two conversations were with people in different industries trying to use the same tools, with wildly polarized outcomes.

The thing that separated them was the size of the companies and everything that comes along with that: layers of bureaucracy, the amount of process for approvals, and the hundred years of history about who gets to make a decision.

The first was a friend who runs UX at a hundred-year-old manufacturer, the kind of place with real factories and a brand your parents would recognize. She’s sharp, and she figured out how to build with AI fast. Unfortunately, her company has not. It only recently even decided digital mattered, and leadership’s entire AI strategy has been to tell everyone to move faster without telling or educating anyone how to do that.

Her product managers are basically opening Claude and building things before anyone’s agreed on what they’re actually building, creating dozens of half-baked products, blooming with no strategy. She’s the head of UX, senior to those PMs, the one person who could point all that energy in the right direction, and she can’t.

She’s stuck waiting on IT to approve access to the same AI tools her team is already using, asking again and again for permissions that never come, while people with a fraction of her judgment ship mediocre products that solve no real problem.

The second conversation was with an engineering lead on a small product team I work with, the kind of team where the founder is still in the room for the calls that matter. We were working out how design gets approved, and on a small team, the default is to let all the stakeholders weigh in and end up with design by committee, which reliably produces the worst version of anything.

To his credit, he shut that idea down before it even started. For the day-to-day, he said, he and the designer would make decisions together and keep moving. Only the genuinely big questions would go to the founder or the full team.

The insight really came when he told the founder his job, at this point, was to be a filter that removes the small stuff, so her attention was saved for the few decisions that actually needed her. Thus removing any need for committee-based decisions.

Both of these companies were approaching their process with the same tools, such as Claude and Cursor. The difference is that one of them is small enough to let a good decision happen with trusted team members, and the other has spent a century building machinery to guarantee that no decision happens without permission.

This makes it look like a big-company problem, and it mostly is, but not entirely. What’s really separating these two companies runs deeper than size: whether the people on top trust the people doing the work enough to get out of their way.

A hundred-year-old manufacturer can choose that as surely as a five-person startup can. Most of them just won’t.

It was never about the technology

In the end, the overall challenge is not AI. Most companies are built, top to bottom, to reward control to the people who own the process, who sign off, who run the departments, and it rewarded them for a good reason, because for a hundred years, making the thing was the hard part, and whoever could marshal the making earned the promotions and praise.

Gary Hamel has spent years making the case that hierarchy was really our answer to scarce information. When the person at the top was the only one who could see the whole picture, routing every decision up to them made sense, and the skill of commanding and coordinating all that work was genuinely rare.

Neither of those things is true anymore.

Everyone can see the forest through the trees now, and the rewards we gave to managing the makers, that we spent a century promoting people for, stopped being the rare, valuable thing it used to be.

As we know, making isn’t the hard part anymore, which means the archaic systems that we built to find and promote and protect the controllers are now the single biggest thing standing between your company and the people who already know what to do.

Your people already cracked it

Ethan Mollick has a name for the people at your company who quietly cracked this: secret cyborgs. They’re using AI to do better work, faster, and they are deliberately not telling you, because the rules your company wrote about AI were written out of fear, and admitting you used it feels like admitting you’re replaceable.

Microsoft and LinkedIn’s Work Trend Index put real numbers on it: roughly three in four knowledge workers already use AI at work, most of them smuggling in their own tools without approval, and more than half won’t cop to using it on anything that matters. McKinsey caught leaders lowballing their own employees’ AI use by a factor of three, and concluded, in the most polite consulting voice imaginable, that the bottleneck isn’t the workforce. It’s leadership.

Think about that. People adapted on their own. They adapted so fast and so quietly that you didn’t even see it, and the reason you still can’t is that they’ve already learned it isn’t safe to show you.

The machinery in the way

Here’s why companies can’t catch up, and it has nothing to do with how smart leadership is or how much time they’re willing to put in.

We’re in the middle of a generational shift, and it’s forcing a reckoning with out-of-date structures, out-of-touch policies, and a plain fear of change.

Steve Blank calls that accumulated mess organizational debt, the structural cousin of technical debt, quietly compounding until it becomes the thing holding you back.

Sixty years ago, Douglas McGregor split management into two theories. Theory X assumes people need to be watched and directed or they’ll coast. Theory Y assumes people, handed something worth doing, will run at it. Most companies say Theory Y in an all-hands and are built, brick by brick, as Theory X. Every gate, every approval, every “loop me in before you do anything” is a small monument to the belief that the safest place for a decision is one level above the person doing the work.

Gary Hamel, again, has spent a career putting a price tag on that belief: the sheer drag of bureaucracy, the layers whose entire output is permission. In the world we just left, that “drag” was survivable, the tax you paid for coordinating a lot of people. In this one it’s fatal, because the thing bureaucracy is worst at protecting is the thing that suddenly matters most: a fast, trusted, human call about what’s worth doing.

Marty Cagan has been yelling a version of this at the product world for years. There are teams you hand a solution and tell to go build it, and teams you trust to find the right problem and solve it. The first kind ships what it’s told. The second kind decides.

You’re gonna be shocked at which one AI just made ten times more valuable, and which one most companies are historically structurally yearning to produce.

What the grip costs you

This is where it gets expensive. Amy Edmondson’s whole body of work on psychological safety says people only bring you their real thinking, their misfires, their weird experiments, when it’s safe to do it. A company that treats AI use as a compliance risk instead of a superpower is quietly instructing its best people to keep their best work in a locked drawer.

Daniel Pink wrote a book on what actually moves knowledge workers, and, guess what, it wasn’t control! It was autonomy, mastery, purpose. Grip the wheel too hard and you don’t just slow your best people down, you bore them.

Bored people with rare, portable skills do not stay bored for long. They leave. Usually for a leader who says yes and gets out of the way.

Nobody gives up status without a fight

So why won’t some companies just loosen up? Because you’re asking the people with the most power to volunteer that the thing which earned them that power counts for less now. The classic executive who climbed the corporate ladder by being the sharpest maker in the building now runs a building where making isn’t the valuable part anymore.

Nobody gives up status without a fight, so instead of loosening the grip, they tighten it.

They add a review, ask to be looped in, invent a new step that plants them in front of the work, because if they can’t be the sharpest maker in the room anymore, they can at least be the one whose sign-off you still need. Das Narayandas dug into why AI rollouts stall and found the culprit hiding in plain sight: the tools threaten the identity and status of the very people who have to approve them. It shows up as one more approval step, one more “let’s align first” before anyone’s allowed to actually get the work done.

That’s a genuinely disorienting place to stand, and the most human response available is to hold on tighter. Add a review. Ask to be looped in. Erik Brynjolfsson has shown that the payoff from a general-purpose technology always lags, sometimes by years, and that the lag is the organization slowly, painfully rebuilding itself around the new reality.

None of this waits for you. The gap between what your people can do and what your company will let them do is widening every day, and you are the one holding it open. Call it governance if it helps, but the reality everyone that matters knows it’s fear.

How it actually changes

Corporations aren’t doomed…yet. Real change doesn’t come from an announcement, though. Not a company-wide email, not an all-hands about how everything’s different now, not a Claude access for every employee and a declaration that “we’re into AI now!” Those are all just the noises companies make when they want credit for change without actually doing it.

They change the way they always have, through one person other people actually trust. Nancy Baym’s research at Microsoft landed on a reason for stalled adoption that nobody expects: the learning stays invisible. The tools work and the training happens, but the people who cracked it did it in the background, so nobody else ever gets to see how.

Remember the secret cyborgs from earlier? What actually moves an organization is taking one of those people, giving them real room and the cover to work out loud, and letting everyone else watch what good looks like now. That’s how it snowballs. Somebody credible makes it impossible to ignore and gives everyone else the permission they need to follow suit.

If you’re reading this as the person being held back rather than the one doing the holding, this isn’t permission to be bitter. No org is going to hand you the keys to the castle by default, so stop waiting for it and stop hiding your best work in a locked drawer. Try doing what you’re doing in the open. Show the results, show your reasoning, make it boring for anyone to argue with.

Because trust has to be earned. Stephen M.R. Covey built a career on a simple version of it: trust grows out of character and competence, proven over and over in ways people can actually see. The person for finally forces that company to change earns that spot the hard way. They get so consistently, visibly good that ignoring them starts to look like the real risk.

That can be you.

I know it’s slower and less fair than it should be, but it’s the move that actually works.

The good news is that you don’t have to go find this person. You don’t have to hire them or run a search or buy anything. They’re already on your team, right now, quietly doing the thing and waiting to find out if it’s safe to show you.

The companies that win the next few years won’t be the ones with the best tools. Whatever your company makes, the tools to make it are about to be the same for you and everyone you compete with. The edge that’s left is the human factor. Figuring out fastest how to trust the people who already know what to do, step aside and give them permission to make the changes that need to be made, the improvements and success should follow.

Judgment just became the most valuable thing a person can bring to the table. The only real question left is whether your company is brave enough to let them use it. The good news is that this is the rare kind of problem you could start fixing by Monday.

References and further reading

On the people who already adapted

• Dan Maccarone, Everyone’s Sharpening Knives for the Wrong Fight: once building stops being the hard part, the whole conversation about how to build is aimed at the wrong problem.
• Ethan Mollick, Detecting the Secret Cyborgs: employees quietly use AI and hide it, so the company can’t benefit from what it can’t see.
• Microsoft & LinkedIn, AI at Work Is Here. Now Comes the Hard Part (2024 Work Trend Index): most knowledge workers already use AI, most bring their own tools, and most leaders admit they lack a plan.
• McKinsey, Superagency in the Workplace: leaders underestimate their employees’ AI use threefold, and the bottleneck to scaling is leadership.

On why the org rewards control

• Douglas McGregor, The Human Side of Enterprise: Theory X (control) versus Theory Y (trust), and how most companies preach one and are built as the other.
• Gary Hamel & Michele Zanini, Humanocracy: hierarchy was built to manage scarce information and impose control, and neither the scarcity nor the coordinating skill it prized is rare anymore.
• Steve Blank, Organizational Debt Is Like Technical Debt, but Worse: the accumulated pile of outdated structures and policies that quietly compounds until it holds a company back.
• Marty Cagan, Product vs. Feature Teams: teams you trust to find the right problem versus teams handed a solution to build.
• Das Narayandas, Why Employees Resist AI, and How Companies Can Win Them Over (HBS Working Knowledge): AI rollouts stall because the tools threaten the identity and status of the managers who have to approve them.

On what control costs you

• Amy Edmondson, The Fearless Organization: people only surface their real work when it’s psychologically safe to do so.
• Daniel Pink, Drive: autonomy, mastery, and purpose, not control, are what move knowledge workers.

On why the lag is the org, and how change spreads

• Erik Brynjolfsson, Daniel Rock & Chad Syverson, The Productivity J-Curve: the gains from a general-purpose technology arrive only after organizations rebuild themselves around it.
• Nancy Baym, Eleanor Dillon & Sonia Jaffe, Peer Influence Can Make or Break Your AI Rollout (HBR): adoption stalls because the learning stays invisible, and it spreads through trusted peers, not mandates.
• Stephen M.R. Covey, The Speed of Trust: trust is a function of character and competence, earned by demonstrating results people can see.

AI Image Enhancement & Upscaling with Full Control

AI image enhancement with control. Upscale, Restore & Modify – all in one editor.

Take your

• old photos
• illustrations
• artworks
• landscape photos
• wedding photos
• designs
• drawings
• portraits
• movie posters
• game graphics

to new, razor-sharp dimensions.

Upscaling without control is a matter of luck.

Faces change. Details are misinterpreted. And if something goes wrong, you have to start over again.

More control. Fewer surprises. You decide what gets changed.

Upscale your image. Refine details selectively. Restore content true to the original or consciously adjust it. And go one step back at any time.

Three tools. One seamless workflow.

The three tools can be used flexibly and combined, so that you can achieve the result you envision step by step. Without detours, without complicated settings, directly in the browser.

Upscale

Images are intelligently upscaled and visibly refined without distorting the image’s appearance. UpSynth automatically analyzes your image, reconstructs missing details, and reduces common disturbances such as artifacts or image noise, while preserving important structures and the character of the original.

Restore

Incorrectly interpreted or distorted image areas are selectively restored. You simply mark the desired area, UpSynth reconstructs it based on the original and reduces creative liberties to a minimum. Ideal for faces, old photos, or sensitive details where accuracy is crucial.

Modify

Modify image content selectively according to your specifications. Mark an area and describe via prompt what should be adjusted. Whether replacing objects, removing elements, or changing visual details: UpSynth interprets your instruction in context and creates multiple variants for you to choose from. Modify is perfectly suited for creative adjustments, corrections, and deliberate interventions, without recalculating the entire image.

UpSynth is not a one-click miracle but a controllable upscaling process.

Old images, scans & retro motifs in print quality.

This classic Frankenstein poster was brought back to life from a blurry, old digital version with UpSynth. Whether you have old movie posters, flyers, scanned photos, or illustrations, with UpSynth you can prepare them in brilliant resolution for print or web.

Find your perfect dose of magic with UpSynth

You decide how creative our AI is allowed to be. The magic value controls how strongly UpSynth “interferes” with the image. Low values stay very close to the original, higher values allow more freedom for imaginative details, fill in blurs, and generate new structures.

At 50 % the AI supplements very conservatively and adheres strictly to the original. 100% is the recommended setting. The AI only adds where it makes sense and remains stylistically close to the original. At 150% it is allowed to become more creative, which is perfect for generative details, but also comes with more risk of interpretation.

Remove image artifacts reliably

Through repeated saving, JPEG images become increasingly compressed. The resulting image artifacts and loss of detail remain even when later saved in other formats. Our AI removes these artifacts and restores natural image details, optionally also without enlarging the image.

When control meets creativity

With UpSynth, you can not only enhance images, but also develop them further in a targeted way. If details are missing, damaged, or are to be deliberately changed, Modify gives you the necessary freedom without losing control.

Restore old and damaged photos

A short prompt in Modify mode is enough to selectively supplement missing or damaged image areas and make old photos recognizable again. Even heavily damaged or very old photos can be restored this way. UpSynth creates multiple variants for you to choose from. If the result is not quite right yet, each step can be further refined or undone.

Restore faces and fine details

Small faces and fine details are often lost or appear distorted during upscaling. Especially with old photos, scans, or heavily downscaled subjects, recognizability quickly suffers. With Restore, you can selectively restore exactly these areas. You mark the face or detail, and UpSynth reconstructs it as true to the original as possible based on the source image. This preserves identity while the overall image blends in harmoniously.

Still unsure whether UpSynth is the right choice for your images?

Test your first image for free, start a one-time trial if needed, or take a look at more results in our showcases. You only decide once you know whether it’s right for you.

Full article content is not available for inline reading.

Read the original article →

Google is enhancing the Gemini desktop app by adding dedicated tabs for image and video generation and introducing a camera attachment feature for capturing photos.

Apple's new Apple Upgrade leasing program hints at the company's next era of growth. Due to memory shortages, hardware prices have increased, and prices are only going to get steeper. Apple appears to be changing the psychology of the purchase. Instead of asking customers to justify spending more than $1,000 on a device, it's asking whether they can afford roughly a dollar a day, which is a much easier decision.

Balaji Srinivasan, a tech founder, said that he believed the US was descending into anarchy, so he decided to build a new city from scratch. He started by creating a tech community in Malaysia, setting up a campus with co-working spaces, a high-end gym, and meals designed by longevity guru Bryan Johnson. Two years in, his vision is now colliding with reality. The project has been shut down over licensing issues. Within hours of the government announcement, Srinivasan signed an agreement to open a campus in Kazakhstan.

While code is easier and faster to write, good software is still hard to get right as it needs care and requires good judgment.

The unicontext is a scenario in which the ways you should act become the same across all different contexts.

Leaked marketing renders from Evan Blass reportedly reveal the unannounced Pixel 11 Pro Fold, alongside new Pixel Buds and a Pixel Watch, ahead of Google's launch event. The foldable appears similar to last year's model, keeping a tall outer screen and 8-inch inner display, but with a redesigned camera array featuring a glowy light teased for the Pixel 11 Pro. Google's Made by Google event is set for August 12th, where these devices are expected to be announced, though prices may rise.

Gunnersbury Museum and Park has introduced a new modular identity by Wiedemann Lampe to solve a long-standing visibility problem: most people knew the park but not the museum. The system uses separate symbols and colors—a green tree for the park and a purple building for the museum—that combine into a yellow identity when representing both together, giving each its own voice while maintaining a cohesive brand. Supported by flexible illustrations, human-focused photography, and intuitive yellow wayfinding, the rebrand is designed to increase awareness, encourage exploration, and help the historic site become a more recognizable cultural destination.

To celebrate its 40th anniversary, Bethesda has introduced a new retro-futuristic logo that replaces its clean, corporate identity with one inspired by the company's original 1980s and 1990s branding. The redesign brings back the "Bethesda Softworks" name, uses an angular cyberpunk-inspired typeface reminiscent of its early years, and aims to better reflect the publisher's long history and personality. Fans have overwhelmingly praised the change, seeing it as a welcome return to Bethesda's roots and a reminder that distinctive branding can be more memorable than generic minimalism.

AI has spawned distinct design idioms—sparkle emojis, streaming text, shimmering "thinking" indicators, tiny icons, and beige/cream palettes with orange accents—some likely to fade, others already spreading into broader software design.

Design that groups information into meaningful units reduces cognitive load and errors.