Full article content is not available for inline reading.

Read the original article →

Kimi K3, Moonshot AI’s latest open-weight model, ranks 1st on our single-shot Frontend Arena with an Elo of 1392. This is 10 positions higher than Kimi K2.6 and 16 positions higher than Kimi K2.7 Code, marking the largest jump we’ve seen in the Moonshot model line.

However, we found that Kimi K3 uses an extreme amount of thinking tokens, using over 12x more reasoning than Claude Opus 4.8 and over double that of Kimi K2.6.

This is a lot of reasoning, and we decided to dig deeper to find out exactly what Kimi K3 is thinking about.

We find that Kimi K3’s performance can be primarily ascribed to its unique chain-of-thought approach where it appears to iterate upon designs much like how a full AI agent would, but inside its chain of thought. This strategy produces intricate, intentional websites with creative component design, as well as improves Kimi K3's ability to integrate with outside dependencies.

How Kimi K3 Simulates an Agent in its Chain of Thought

A brief scan through Kimi K3’s thinking traces is all we need to explain its performance. Kimi K3 has unique multi-staged thinking traces, where it switches from planning to decision making to designing individual parts of the final website. This is exactly the agentic workflow, and it’s a pattern we’ve never seen before. In fact, during the final design phases, Kimi K3 writes out sample code so that in the final generation, it can correctly create the interactions it had painstakingly designed.

Compare this to the previous models in the model line, which have much shorter reasoning traces and rarely write code. Instead, they just decide on high-level details or sections before diving straight into the final output.

If we aggregate across all the generations from the Kimi models, we find that it writes over 10x as much code during reasoning than any other model in its model line, with more reasoning tokens spent coding than actually reasoning.

No other model has this density of code blocks during reasoning, and it’s because Kimi K3 iterates on individual components during the reasoning, “testing mentally” instead of using the normal tests it would have in an agent loop. This also implies that Kimi K3 is effectively thinking in code, expressing its planned specification in the form of code and concrete constraints rather than vibing up an inexact plan.

No other large model does this sort of high-level planning and low-level iteration during the reasoning stage, as this strategy increases the amount of time spent reasoning instead of working. By choosing this strategy for its reasoning traces, Kimi K3 deliberately produces results that take more tokens but perform better, essentially trading tokens for intelligence. This results in much slower generations with higher preference scores, setting a new Pareto frontier on our Preference vs Speed chart.

Kimi K3’s Thinking Weaponizes Unsplash Images

However, Kimi K3 has a secret weapon: phenomenal indexing on its training data. Ordinarily, reasoning is much weaker than an agentic trace because the agent can’t use tools like web search to update its internal thinking. However, for Kimi K3, its learned index of the Internet is so strong that it can simply check its mental index for if the reasoning it just output was up-to-date and correct. Below is an example of Kimi K3 using this for Unsplash, a free images provider that models commonly use to find hero visuals and other filler images. On top is the chain-of-thought reasoning, and below is the Unsplash CDN image that the model thought about when it generated the ID.

Kimi K3 thinks about what image it wants, one-shots an Unsplash ID for that image, then thinks about whether than Unsplash ID is actually valid to see if it wants to use it.

This means that Kimi K3 is astoundingly good at finding images for its websites. Here’s the website that the above chain-of-thought generated.

Other models, like Fable 5, don’t utilize their learned index as much as Kimi K3 does, nor do they reason as much about the images they generate. Below are some example generation of Kimi K3 versus other models, where Kimi K3 didn’t miss a single image decoration while Claude Fable 5 and Kimi K2.7 Code had to rely on alt text and defaults instead.

This thinking also extends to dependency usage, where Kimi K3 creates better scroll animations, charts, and frontend UIs simply because it’s thinking about how to use those dependencies during reasoning.

What this means for model selection

Kimi K3 is a step forward in design improvements and a huge leap forward for open source models as a whole. Launches like this are a reminder of how fast the open source frontier is moving, and how open source models can present new ideas that push everyone ahead. Every release like this gives researchers and developers more powerful foundations to push from.

We will continue monitoring Kimi K3 performance and how it compares to other models. Congratulations to the Moonshot AI team on the launch, and see if you like Kimi K3 more than other models on DesignArena.ai.

DeepSeek founder Liang Wenfeng in His Own Words: 64 Quotes from DeepSeek's Investor Call

Key takeaways:

• Vision-driven: No KPIs, no written vision, even "no organization"—it runs on goodwill toward the world and an obsession with AGI. "We're just a bunch of very ordinary people."
• Restraint as strategy: API pricing aims only to recover hardware costs in ten months. No fighting for consumer traffic, no chasing enterprise fads. "The more restrained you are, the more likely you'll pull it off."
• Open source is the point: Even the strongest model will be open-sourced. Not worried about others competing. "I only worry they won't be able to deploy it."
• AGI roadmap: CoT → Agents → continual learning → "singularity" → embodied intelligence. "After continual learning, it can develop the next version itself."
• Compute reality: About 20,000 "H-equivalent" GPUs; 12–18 months behind the U.S.; chasing with one‑twentieth the compute. For domestic chips: "The ecosystem is fine; capacity is the bottleneck."
• Endgame view: China may match foreign models this year, but "that still isn't AGI." "There are too many base-model companies in China; it will converge."

Vision, motivation, and culture

"We don't have an organization; we're driven by a vision."

1. When we started the company, our original intent wasn't "how much money will I make," or "go to the capital markets," or "IPO," or anything like that. That wasn't the intent. The first few dozen people never thought that way. If they did, they wouldn't have joined.
2. Overall, we're doing this with a lot of goodwill toward the world, and we think it's useful to humanity. It's something beyond money.
3. About twenty years ago, in management, the person I admired most was Jack Welch, former CEO of GE. Looking back now, most of what he said may no longer be right, but he got one thing right: the most important thing for a company is its vision. Managing a big company isn't about rules and procedures—it's about vision. What is vision? Vision isn't a slogan on the wall. Vision is how you do things, not what you say.
4. In fact, we don't really have an organization—we're driven by a vision, organized by a vision. That vision isn't even written down. We've never written anything.
5. We're not especially capable. We don't have more money than others, and it's not that our people are better than everyone else's. We're just a bunch of very ordinary people.
6. Our company is built on consensus. It's not that I decide everything. I seek consensus. My authority inside the company is built on consensus.
7. Our management has two lines: one top-down and one bottom-up. Bottom-up means everyone decides what they want to do and just does it. No one manages them. No KPIs. In general, we hope "formal work" doesn't take more than half of an employee's time. The other half is unallocated—they can do whatever they want.
8. We generally don't work overtime. First, research needs a relaxed environment; if you push too hard, you can't do research. Second, we're very focused, which means we do very few things.
9. CoT doesn't consume resources. For research, it doesn't burn GPUs—you only need a small number of cards. What you need is ideas. The barrier is low; anyone can try it. But who can actually get something out of it—I don't know if that's talent or what.

Open-source strategy

"Even our strongest model will be open source, because I don't see what's good about closed source."

1. Why do we insist on open source? Because the vision itself requires open source. Without that vision, you can't organize people.
2. Zhipu also open-sources, but their open source isn't the same as ours. Their open source has a "being chased" feeling—they don't see it as the point. For us, it is the point.
3. AI is big enough that in the end it may take up 10% of human society's GDP. One person can't monopolize it. You have to share it with others, or you won't survive.
4. I think we will open-source, and our strongest model will probably also be open-sourced, because I don't see what's good about closed source.
5. Even if you open-source the model and tell people everything, the barrier is still very high. It's still very hard for others to actually use it.
6. If we only make, say, a 6× profit, open source doesn't really hurt. But if you want a 100× profit, then open source does affect that.
7. We were open-source from the start. We won't treat you differently just because you're a competitor—that's part of open source. I'm not worried about others deploying our model to compete with us. I want them to be able to deploy it. I only worry they can't—some detail is wrong, results get worse, or costs get high.
8. Are the open-source models we provide the same as what we deploy ourselves? Yes, they're the same. We won't open-source a weaker model and then use a better one internally. Last year, basically everything on the consumer side was open source, and we didn't see a conflict with our consumer service.

Restraint and pricing

"Restraint is a strategy. Ten months to recoup costs is a reasonable profit."

1. If we want to make AI work in our hands, the first thing, I think, is restraint. You can't think "a huge share of human GDP should be mine." The more you think like that, the less you'll succeed.
2. For me, restraint is a strategy. Sometimes you give up something to gain other things.
3. For our API pricing, what we consider a reasonable profit is: buy a batch of equipment from the market, and recoup the cost in ten months. I think that's reasonable. It's not profit-maximizing. If you wanted to maximize profit, you'd price higher. In this price range, demand isn't elastic. Even if I raise the price by 50%, token usage doesn't change much.
4. Let me tell a story. For our DCP (Decoder Context Parallelism) model, at first we worried demand would be too high, so we set the price relatively high. People on the team weren't very happy. Later I lowered it—down to one quarter—and everyone was happy. When we cut the price, people in the company group chat were cheering.
5. Someone just commented that "ten months to break even is too high." True, there's still room to lower prices. There's room for optimization on the model side too, so there's still a lot of room overall.
6. At that cost level—ten months to break even—we can do it. Other companies can't. Alibaba or Tencent, without our optimizations, their costs should be several times higher.
7. If I cut prices further, demand won't increase, or will increase only slightly. At this price, everyone can afford it and is satisfied. Cutting prices won't bring the company more revenue, and it won't add more value to society. Even if it's cheaper, it doesn't increase "happiness" by much.
8. The lower the cost, the bigger the model I can train, and the more I can afford to take on larger models. With limited compute, if my compute efficiency is higher, I can take on larger models.
9. A commercial company has no incentive to pursue model efficiency. If efficiency improves… costs drop… then what do you earn? For us, it's part of the vision. Our teammates are ordinary people; they know using this costs money. They have empathy: others have to pay to use it, so if it's cheaper, it's easier to accept.

Commercialization

"Both C-end and B-end are side products on our road to AGI."

1. We do AGI to do AGI. We just happen to produce something we can commercialize, so we use it. That's different from other companies. Other companies build the model to serve consumer users or enterprise users. For us, both consumer and enterprise are side products on the road to AGI.
2. When we suddenly got popular around last year's Spring Festival, that wasn't in our script. We never expected it. We just wanted to make the technology good.
3. When users suddenly surged last year, we didn't try to keep them for monetization, or fight for those commercial interests. We didn't fight for users, we didn't try to make money—but we worked hard to serve them well.
4. This year it's quite possible our API/AI revenue could reach a few hundred million dollars in ARR, if demand continues expanding and if we can buy more GPUs. If AI revenue can reach a billion dollars, our cash flow can basically turn positive.
5. Over the past three years, at any point, if you came to me to talk about a commercialization roadmap or product lines, it would be a waste of time, because you can't predict the future.
6. If this year we can have a few hundred million dollars of enterprise revenue, and next year enterprise revenue continues, and demand keeps growing, we won't be far from net profit. In the worst case, just selling APIs can support a listed company. If there's no new technical progress later and our technology "freezes" here, then we'll fully focus on selling APIs and doing the services well—that's enough.

AGI tech roadmap

"AI isn't lacking taste or intuition. It's lacking continual learning."

1. You can think of AI progress as climbing steps. Last year's step was CoT. This year's step is Agents. Why steps? Because each next step builds on the previous one. Agents rely on CoT, and CoT relies on the previous step—the language model.
2. After Agents, we think the next problem to solve is continual learning: how to let the model keep learning continuously, rather than just giving it one strong training run. It should be able to learn over a long period like humans do.
3. After continual learning, we may reach a "singularity." It's when a model that can keep learning can already do everything humans can do. It can develop its own versions, do research by itself, and build the next version—build even more advanced AI models.
4. This "singularity" isn't really a point; it's also a gradual process. It may be a long, gradual change—not a sudden jump.
5. After that, I think you get embodied intelligence. Then it enters the real world: it can do housework, take care of the elderly, and so on.
6. We think this roadmap is the easiest. We can follow it without overtime. If the roadmap were reversed—for example, if you try to do embodied intelligence first—you'd suffer. It's hard labor.
7. AI isn't lacking taste or intuition. Its taste and intuition are fine. If you ask it to write an article, I think its taste and intuition are basically fine. What it's lacking is continual learning.
8. Everyone in the world is researching this. From an investor's perspective, what they see most is Agents. But for researchers like us, what we see more is learning, and how to solve the learning problem.
9. Internally we care about this narrative: for the next version of our model, we hope it can help our own development. It can improve DeepSeek's efficiency. Our model's first job is to raise DeepSeek's own work efficiency. First it has to be useful to us. That's the fastest path to AGI.

Compute, chips, and the China–U.S. gap

"We're behind the U.S. mainly on resources. On people, almost no gap."

1. Right now we have about 20,000 H-equivalent compute cards, and most of those arrived recently—within the past one or two months.
2. Our gap with the U.S. is mainly resources. On people, almost no gap, because it's basically the same group of people. Talent isn't the bottleneck; resources are the biggest bottleneck. The talent gap is essentially also a compute gap.
3. For the largest models today, we can't afford to train them. The biggest models have about 800B active parameters; domestically, we're still at tens of billions. If I want to train a model as large as the leading ones, I'd need 50,000 GB300s or Huawei 950s—200,000 cards. That's just training, not even research.
4. Within what we can afford, more cards is always better. At a reasonable price, we buy as many as we can. Ideally, if we can spend the money within six months, that's best. Turning money into Nvidia cards is definitely better than leaving it in the bank.
5. Our gap with the U.S. may be 12 months, maybe 12–18 months, or maybe 6–12 months. Simply put: about two years behind, but doing it with one-twentieth of the U.S. compute. In the future we want to rewrite that narrative: use a fraction of the compute, but shorten the time to 6 months, 3 months—maybe even surpass them in some areas.
6. Nvidia's CUDA moat is being eroded quickly. One reason is that AI can now write code, so I can use AI to build the ecosystem. Another reason is new technologies—for example TileLang (an open-source high-performance AI operator programming language developed by Peking University's School of Computer Science). Using a higher-level language to write CUDA operators, you can quickly rewrite Nvidia's whole ecosystem.
7. There's a historic opportunity for domestic AI chips to replace imports. We believe within the next year we'll see something validated: the ecosystem for domestic chips has no problem. People used to think there were problems—couldn't use them, not usable—but within a year, I think we can change that perception.
8. Our goal in buying Huawei 950 is still to help Huawei improve the ecosystem. Huawei 950 "super nodes" can substitute Nvidia GB200/GB300 in performance and price. The only tradeoff is: four Huawei cards equal one Nvidia card, and you're two years behind in time. Huawei 950 super nodes will ship in Q3 or Q4 this year; Nvidia GB200 shipped in Q3 two years ago.
9. You can depreciate Nvidia cards over five years. Huawei cards, at most three years. Huawei 950 is fine to use this year; next year should still be OK. After that, it may really become too power-hungry.

What DeepSeek won't do

"Video generation, 3D, world models—these don't set the upper bound of intelligence."

1. AI is a broad field. There are many things we think aren't on the main line—for example 3D and video generation. I don't think they're strongly related to the main line of intelligence, so we won't do them.
2. When video generation first came out, it was very hot—like you had to do it, or you weren't an AI company. But you can see what happened: after Sora, everyone did it—big companies, small companies. Then small companies cut it. It doesn't affect the ceiling of intelligence. Commercially, it's a good business, but it's not about intelligence. We won't do something just because it's a good business.
3. World models, I think, also don't have much to do with the upper bound of intelligence right now, so we won't do them. In our view, world models and intelligence aren't the most important thing at this stage.
4. Multimodal matters a lot for products, and for consumer products. But for the ceiling of intelligence, it's a component, not the main line itself. We will definitely do multimodal, and we're doing it. V4 and later versions will support native multimodal.

Data labeling

1. The cost of data labeling in the U.S. isn't really different from China. China doesn't have a cost advantage in labeling, especially for high-end labeling. That makes it hard for us to invest in labeling data the way the U.S. does. This path is hard in China because labeling is just too expensive.
2. Right now, half of the most important people in the company are labeling data.
3. The bottleneck isn't that we can't quickly hire more people. It's not about money, and not about cards either. But we are expanding fast. So we think within a year, it's reasonable to expect that the high-quality data problem in China can be handled much better.

Competition and endgame

"There are too many base-model companies in China. It will converge."

1. Our biggest core interest is keeping the team stable. That's our biggest core interest—maybe the only one. As long as I can keep the team stable, I'll be able to do AGI. Money isn't the problem, resources aren't the problem; everything else is easy to get.
2. We've always been very restrained. We don't want to become an opponent of any big internet company or small company. I'd rather help them do this.
3. Open source, goodwill, helping others—none of that caused us to lose anything. No impact at all. If anything, it may be a plus.
4. There are a bit too many model companies in China right now—still too many. The U.S. maybe has three. China has too many doing base models. Everyone is doing the same thing. Resources get spread thin, and each company gets less. I think it will definitely converge.
5. If my goal is to take 5% of global GDP from AI, I'll be defeated by someone willing to take only 1%. Because that person says: I do it well, but I only need 1% of global GDP. Then if another person comes along and says: I only need 0.1%, they beat the previous one. People who take more get beaten by people who take less. OpenAI at the beginning thought it could monopolize the world, but in reality it will face many challengers.
6. In AI, in China, in one or two years we can get to roughly the same level as abroad, or maybe even substitute foreign models this year. In the current paradigm, it's not that hard. But it's still not AGI.
7. Right now, returns from model scaling are still very obvious. We haven't had a chance to hit the "scaling wall." It's still far away. When Silicon Valley says scaling is over, that's for Silicon Valley. For China, we're still far from that—we haven't scaled to that level at all.

Full article content is not available for inline reading.

Read the original article →

Everyone Should Know SIMD

SIMD has a reputation for being complex. I've met many very good software engineers who dismiss it as something too complex to learn or a niche optimization meant for only the highest-performance software, not useful in everyday programming.

I think that's wrong. SIMD can be simple to understand, and common "process N values at a time" SIMD code to speed up a naive for loop almost always follows the same general shape. Once you learn the basics, writing SIMD is just about as easy as a for loop. And when it's not, it's usually a good sign to skip it for now.

Every developer should know at least that much SIMD.

This post uses Zig for examples but is a general piece that applies to any programming language. Support for SIMD instructions varies by programming language and I hope that more programming languages expose these generic concepts in the future!

I hate that I have to do this for every post now, but I also want to note this was completely hand-written with no AI assistance.

Background: What Is SIMD?

If you already know what SIMD is, skip this section.

SIMD allows a CPU to operate on multiple values in parallel. For example, instead of comparing one byte at a time, a CPU can compare 4, 8, or even more bytes with a single instruction.

If you ever see loops like this in your code:

for (byte in bytes) { /* ... */ }
for (character in string) { /* ... */ }
for (value in array) { /* ... */ }

There is an opportunity to use SIMD. SIMD turns those into this:

for (8 byte chunk in bytes) { /* ... */ }

This results in a localized speedup that directly maps to the parallelism: you process data 4x, 8x, or even faster.

The only real requirement for this to pay off is that you need to be regularly processing a large enough number of bytes. If you're doing these for loops across data that is only ever a handful or dozens of bytes, it's not worth it. But if this is iterating over hundreds, thousands, millions of bytes, the payoff will be huge.

That's the basics. Projects such as simdutf and simdjson take this to an extreme and use SIMD techniques that can be difficult to understand. But you do not need to write algorithms like those to benefit from SIMD. The common case is dramatically simpler.

The Common Shape

The common "process N values at a time" SIMD code follows the same five steps:

1. Broadcast any constants you need and initialize vector accumulators, if any.
2. Loop over input one vector-width chunk at a time.
3. Perform the comparison or arithmetic across all lanes in parallel.
4. Reduce or store the vector result as needed.
5. Handle the remaining elements with a scalar tail. A scalar tail is just your normal loop from before vectorizing, but it only processes the remainder that doesn't fit into a full vector.

As you do this more and more, you'll begin to naturally decompose every for loop into these five steps and writing SIMD becomes nearly as natural as writing a scalar loop.

A Real Example

Let's look at a real example from Ghostty. We'll look at the scalar implementation, the SIMD implementation, and then map it back to the common shape above.

I have a slice of decoded codepoints that I want to consume until I see a value at or below 0xF (a C0 control character). Terminals are mostly plain characters to be printed, so we try to batch all those together. So this loop finds the end of the next printable run as quickly as possible.

The scalar loop is one line:

while (end < cps.len and cps[end] > 0xF) end += 1;

It processes one codepoint at a time. It is easy to understand.

Here is the generic vector version with no CPU-specific intrinsics and no comments. I will explain it in detail later.

if (simd.lanes(u32)) |lanes| {
    const V = @Vector(lanes, u32);
    const threshold: V = @splat(0xF);
    while (end + lanes <= cps.len) : (end += lanes) {
        const values: V = cps[end..][0..lanes].*;
        const greater_than_threshold = values > threshold;
        if (@reduce(.And, greater_than_threshold)) continue;
        const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold);
        end += @ctz(~mask);
        break;
    }
}

while (end < cps.len and cps[end] > 0xF) end += 1;

This can improve the loop's throughput by up to 4x with ARM NEON (including Apple Silicon), 8x with AVX2 (most modern x86 CPUs), and 16x with AVX-512 (some Intel CPUs and AMD Zen 4 and newer).

In real-world end-to-end throughput from terminal program to finalized terminal state on an AVX2 Intel desktop, this was more like a 5x speedup. You always lose some of the ideal speedup due to the other stuff around the SIMD code, but... that's still 5x!

Okay, now I understand that those 12 lines are going to look really alien to someone not familiar with the concepts. So now let's back up and explain it step by step, mapping it directly to the shape previously mentioned.

Step 1: Broadcast Constants

Let's start with the first three lines:

if (simd.lanes(u32)) |lanes| {
    const V = @Vector(lanes, u32);
    const threshold: V = @splat(0xF);

simd.lanes(u32) is a helper in Ghostty that returns the number of u32 values the target CPU can process at once. These individual values are called lanes. On ARM this returns 4, AVX2 returns 8, and AVX-512 returns 16. If the target doesn't have a vector size we want to use, it returns null and we skip all of this code and do zero SIMD work.

@Vector(lanes, u32) creates the vector type. If lanes is 8, then V is a single value containing eight u32 values that the CPU can operate on in parallel. And so on.

Finally, we need to compare every value to 0xF. A vector comparison requires a vector on both sides, so @splat(0xF) copies, or broadcasts, 0xF into every lane. The result is a vector that looks like this:

{ 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF, 0xF }

This is step 1: prepare the vector type and broadcast any constants. Some algorithms also initialize a vector accumulator here, but this algorithm doesn't need one.

Step 2: Loop One Vector at a Time

Next, we loop over one complete vector at a time:

while (end + lanes <= cps.len) : (end += lanes) {
    const values: V = cps[end..][0..lanes].*;

If lanes is 8, we only enter the loop when at least eight values remain. Inside the loop, we load those eight values into the vector values. At the end of every loop, end += lanes moves forward by eight values instead of one.

The requirement for a complete vector is important. If only five values remain, we can't load an eight-lane vector. There are various tricks to handle this, but we do the easy thing and handle them via our scalar tail, which I'll explain later in step 5.

This is step 2: load and loop over the input one vector-width chunk at a time. You can see the lane-count speedup here!

Step 3: Perform the SIMD Operation

Now we perform the comparison:

const greater_than_threshold = values > threshold;

Both values and threshold are vectors, so this maps to a vector operation (a literal vector CPU instruction). The one > compares every lane in values to every corresponding lane in threshold. If there are eight lanes, this is equivalent to performing the scalar comparison cps[end] > 0xF eight times, but it does it in one CPU instruction instead.

The result is another vector with one boolean per lane. Conceptually, it looks something like this:

values:                 { 0x41, 0x42, 0x43, 0x0A, 0x44, 0x45, 0x46, 0x47 }
threshold:              {  0xF,  0xF,  0xF,  0xF,  0xF,  0xF,  0xF,  0xF }
greater_than_threshold: { true, true, true, false, true, true, true, true }

This is the actual SIMD operation. There is no explicit inner loop. The > operator applies to every lane in parallel.

Step 4: Reduce the Vector Result

We now have a vector of booleans, but the original loop needs to know the location of the first value at or below 0xF.

First, let's handle the common case where every value is above 0xF:

if (@reduce(.And, greater_than_threshold)) continue;

@reduce(.And, ...) combines every boolean using and and returns a single boolean. If every lane is true, we continue and process the next vector. In our example, lane 3 is false, so @reduce returns false and we fall through to find exactly which lane failed.

If any lane is false, then we need to find exactly which lane failed:

const mask: std.meta.Int(.unsigned, lanes) = @bitCast(greater_than_threshold);
end += @ctz(~mask);
break;

@bitCast turns the vector of booleans into an integer with one bit per lane. A 1 bit means the value was greater than 0xF and a 0 means it wasn't. We invert the mask so failed comparisons are 1, and then @ctz counts the number of zero bits before the first failure. That count is the index of the first failing lane.

We add that index to end and break because we found the control character.

Step 5: Finish with the Scalar Tail

After the vector loop, we run the exact scalar loop we started with:

while (end < cps.len and cps[end] > 0xF) end += 1;

If the input length isn't an exact multiple of the vector width, this processes the remaining values. For example, an eight-lane vector loop leaves anywhere from zero to seven values for this loop. This is called the scalar tail.

This loop also handles CPUs where simd.lanes(u32) returns null. In that case we skip all of the SIMD code and the scalar loop processes the entire input. The original implementation remains both the fallback and the tail.

Recap: The Common Shape

1. @splat(0xF) broadcasts the comparison value into every lane.
2. The while loop loads lanes values at a time.
3. values > threshold compares every lane in parallel.
4. @reduce, @bitCast, and @ctz find the first failed comparison.
5. The original scalar loop handles the remainder and unsupported CPUs.

Why Can't the Compiler Do This?

Sometimes it can! Compilers can auto-vectorize simple loops, particularly regular arithmetic loops without complex control flow. You should always compile the scalar version with optimizations and see what your compiler produces before manually writing SIMD.

But compilers are severely limited in what they can auto-vectorize and are in general very poor at it. Auto-vectorization has been an active area of compiler research for decades, and recent research still begins from the observation that production compilers regularly miss vectorization opportunities.

Everyone Should Know SIMD

Every developer should be able to recognize the opportunity and, most importantly, should not be scared of SIMD. If you see a hot loop scanning, comparing, counting, or transforming a large amount of contiguous data, you should be able to imagine processing it a vector-width chunk at a time.

This post demonstrates that these common cases follow a very regular pattern that you quickly get used to. And with good language support, you don't need to know any assembly or CPU-specific quirks to get easy improvements.

Footnotes

1. Very impressive projects like simdutf and simdjson use extremely complex SIMD tricks to achieve their goals. But this isn't what I'd consider "everyday SIMD."

2. C0 controls extend beyond 0xF. This is the cutoff Ghostty uses for this specific code path; ESC and other control-sequence handling happens elsewhere.

3. Generic vectors remove the CPU-specific syntax, not CPU-specific code generation. Zig still lowers these operations to the instruction set enabled for the target. Ghostty falls back to scalar code when it can't choose a supported vector width.

4. The comparison itself is one vector operation. Loading the vector, reducing the result, and locating the failed lane require additional instructions. The important part is that we're doing multiple comparisons at once.

5. This post was based on a Lobsters comment I wrote.

Full article content is not available for inline reading.

Read the original article →

Prompt Caching In Agents

Large language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it.

A coding agent sends the model its system prompt, tool definitions, project instructions, conversation history, tool calls, and tool results. On the next turn it sends almost all of that again, plus a small amount of new material. Once a session has grown to tens or hundreds of thousands of tokens, recomputing the whole prompt for every turn is slow and expensive.

Prompt caching is what makes this somewhat economic, but it is also quite fragile. A changed tool definition, a model switch or a provider routing decision can turn what one would expect to be a cheap incremental request into a full replay of the context.

For coding agents, cache behavior is therefore not just an implementation detail or optimization. It affects latency, cost, tool design, session design, and even which product features should be made available.

What a KV Cache Contains

A transformer processes a prompt in two broad phases. During prefill, it reads the input tokens and computes attention state for them. During decode, it produces new tokens one at a time.

At each attention layer, every processed token produces a key and a value. These are not quite like key-value lookups in a hash table: both are arrays of numbers, usually floats or lower-precision quantized values. When processing a new token, the model compares that token's query with the earlier keys to determine how relevant each earlier token is. It then uses those relevance scores to form a weighted mixture of the corresponding values. In that sense, a key is what the model matches against, while a value is the information it retrieves (but the lookup is fuzzy rather than "returning a single exact match" like a dictionary lookup.)

Those keys and values are retained so that the next generated token can attend to everything that came before without recomputing the earlier tokens. This retained state is the KV cache.

Conceptually, a request looks like this:

request 1:

[system][tools][user][assistant][tool result][user]
<--------------------- prefill -------------------->
                       |
                       K and V tensors per token and layer

request 2:

[system][tools][user][assistant][tool result][user][new]
<---------------- reusable prefix ----------------><--->
                                                    |
                                                    new work

The real representations are more complicated, model-specific, and "quite" large. The important property is that they correspond to a particular token prefix. Two prompts that mean the same thing but tokenize differently do not share a KV cache. If a token changes in the middle, everything after that token is a different continuation.

Prompt caching extends the lifetime of this state beyond one generation. When the next API request from the coding agent begins with the same tokens, the inference system can reuse the stored work for the matching prefix and prefill only the new suffix. So far, the theory.

Where the Cache Lives

In order for a cache to work it needs to be stored somewhere, and it needs to be addressable. There are two broad ways inference systems make KV caches available to a later request.

The simpler approach is session affinity. It works by keeping the KV cache on or near the GPU that computed it, and routing the next request back to the same worker. A session ID or prompt-cache key becomes a trivial routing hint and so you can potentially even deal with this problem on the HTTP load balancer level without having to look into the payload.

request(session-42) --> router --> worker 7 --> GPU 7 KV cache
next(session-42)    --> router --> worker 7 --> GPU 7 KV cache

This avoids moving a very large cache over the network. It is fast when it works, but it constrains scheduling. The selected worker can become overloaded, restart, or evict the entry. A router may also decide that balancing the fleet is more important than preserving one session's cache. It is however a very attractive solution because it works with little extra deployed infrastructure and hardware.

The other approach is to distribute the cache. KV blocks can be stored in another memory tier or made available across workers, so a request is not tied as tightly to one GPU.

                         +--------------------+
request --> scheduler -->| worker 3 / GPU 3   |
             |           +--------------------+
             |
             +----------> distributed KV blocks
             |
             +----------> worker 9 / GPU 9

That improves scheduling flexibility and recovery, but moving, indexing, and retaining KV blocks is itself a systems problem. Implementations mix GPU memory, host memory, local storage, remote storage, prefix-aware routing, and eviction policies in different ways.

To put KV caches into perspective: they can be large but they are in some ways smaller than one would assume. With various tricks, the size of KV caches can be reduced to a handful of gigabytes, even for long conversations.

Caches and Prefixes

Pi sessions are trees, not lists. /tree can move the active conversation back to an earlier point and continue along another branch. A rewind can discard the active suffix without deleting it from the session file. A new branch can share most of the old context, a little of it, or effectively none of it. This design is not unique to Pi, quite a few coding agents have something at least conceptually similar. Even if you do not represent the session as a tree, it's not uncommon for agents to have some form of rewinding.

                             +-- E -- F  another branch
                             |
session S: root -- A -- B -- C -- D  current branch
                   |
                   +-- Z  branch near the start

All three branches can have the same Pi session ID. From the router's perspective they are one session. From the prompt cache's perspective they are three token sequences with only partial prefix overlap.

If the cache keeps reusable prefix blocks, jumping from D to F may still reuse root -> C. If it only retains the hottest continuation, if the shared blocks were evicted, or if the request is routed elsewhere, the hit can be much smaller. Jumping to Z may preserve only the system prompt and initial tool definitions even though it starts from A. The precise cache management behavior here depends greatly on the providers.

The reverse can also happen. /fork or a new session can produce a new session ID while carrying over a large amount of identical context. A routing system that isolates caches by session key may fail to notice that useful overlap.

The reusable prefix determines what work can be cached. Session identity merely helps the infrastructure find likely content. On some systems the routing key is crucial to manage caches, on others it's merely an optimization.

Explicit vs Automatic Prefix Caching

Provider APIs expose caching in two main styles.

Anthropic's traditional interface uses explicit cache_control points. The client marks boundaries after stable parts of the request, such as the system prompt, tool definitions, or the latest cacheable conversation content. The server can then write or look up the prefix ending at those points. The boundary is explicit, but reuse still requires the content before it to match. Not only are the cache points explicit, so is the pricing. You pay for cache writes, and you get to choose for how long which comes at different price points.

Other APIs use automatic prefix caching. The client sends the request normally, and the provider finds a reusable prefix without client-placed breakpoints. A prompt-cache key or session header may improve routing or grouping, but it does not make different prefixes equal.

Why Tool Loadouts Trash Caches

Tool definitions usually appear before the conversation and they are "folded" into the system prompt internally. Their names, descriptions, and JSON schemas are model input just like any other text. Adding one tool, removing one, changing its schema, or even serializing the tools in a different order can move the first mismatch close to the start of the prompt.

turn 1: [system][read][write][bash][conversation...........]
turn 2: [system][read][write][bash][deploy][conversation...]
                                   |
                                   old conversation is now
                                   after a mismatch

This is a common surprise with plugin systems and MCP-style tool catalogs. Loading a tool only when it becomes relevant sounds efficient because fewer schemas are sent initially. On most models, however, the newly expanded loadout invalidates the cached conversation that follows it. Saving a few tool-schema tokens can cause tens of thousands of conversation tokens to be processed again.

Some newer model APIs support additive tool loading. A tool can become available at a specific tool result inside the transcript instead of being inserted into the original tool list. The old prefix remains unchanged:

[system][initial tools][conversation][new tool][next turn]
<--------- cached prefix ----------->

Pi nowadays supports this for models with native deferred-tool mechanisms. When an extension makes a purely additive change with setActiveTools(), Pi records the added names on the tool result. For supported Anthropic models it uses deferred definitions and a tool_reference and for supported OpenAI models it emits the corresponding tool-search items. Other models get a safe fallback: Pi sends the complete active tool list on the next request, which works functionally but may wipe the prompt cache.

The word additive matters as removing tools, replacing one loadout with another, or changing prompt snippets still changes earlier input. An extension that rebuilds the system prompt, shuffles tool order, injects timestamps, or changes active tools every turn can accidentally defeat caching for the entire session.

Extensibility means Pi cannot guarantee cache stability on behalf of every extension. We can provide cache-friendly mechanisms; extensions still have to use them and from what we have seen, for many extensions cache efficiency is an afterthought. This is, in part, because when you pay on a fixed subscription the associated cost with cache misses is not quite as obvious.

Interruptions and TTLs

Some important prompt caches have short default lifetimes. Anthropic's default five-minute cache is particularly important because it is shorter than many normal coding activities. If you go to sip a coffee when using Fable, and you come back 10 minutes later, a single "say hi" message will cost you more money than you expect.

That's because while the user may think of a coding session as continuously active, the inference provider sees a sequence of isolated requests:

model request --> run tests for 7 minutes --> model request
                  no cache traffic here

A long build, a test suite, lunch, a meeting, or simply stopping to review a diff can outlive the cache. The next request contains the same prompt, but the stored KV state is gone and the prefix is billed again as input.

Since Pi is currently not a permitted harness on Anthropic's subscription we're following the 5 minute default that Anthropic recommends for API users. However from looking at Claude Code's codebase we know that for their own subscription users, they are increasing that cache timeout to one hour. The increased cost of this however often is not worth it, when you need to pay API token prices.

But you can opt into this. Some providers such as Anthropic expose longer retention controls. For supported direct APIs, Pi users can set PI_CACHE_RETENTION=long to request them. That is still only a request: Pi cannot force a gateway to retain an entry, prevent eviction under memory pressure, or keep a cache alive while no model request is being made.

The Price of a Miss

Providers usually price uncached input, cache writes, and cache reads differently. Cache reads are commonly discounted because the expensive prefill work has already happened. Cache writes can carry a premium because the provider is promising to retain state for later use.

Imagine a coding session with 100,000 tokens of history followed by a short new request as the Fable example above. When the cache works, almost all of that history is charged at the lower cache-read price. Only the small amount of new material needs to be processed at the regular input price and potentially written to the cache.

When the cache misses, the provider has to process the entire 100,000-token history again at the regular input price. It may also charge to write that history back into the cache. This is why a short request such as continue can be surprisingly expensive after a cache expires. In a long coding session, re-reading the old input can cost much more than generating the next answer.

Caching also has a chance to create non-obvious incentives.

The user should want high hit rates because they reduce latency and price. An inference operator that owns the GPUs should want them too: less prefill work means more requests served with the same hardware. A well-designed cached-token discount can align both sides while leaving the operator with better margins.

A gateway or reseller can have a different incentive. If it earns revenue from input tokens billed at the uncached rate, a cache miss can produce a larger customer invoice than a hit. Whether that also produces more profit depends on its upstream costs, contracts, and who operates the cache. In a badly aligned stack, the party responsible for routing may not bear the full cost of misses, while the party billing the user earns more revenue when they happen.

That does not mean providers sabotage caches but it means cache performance should be observable. Users should not have to infer that only from a surprisingly large bill. Understanding if something odd is going on with caches can be an important insight.

Strict cache adherence also means less flexibility for a gateway to route you to the best option in-between turns. You might want to take a cache hit to continue with a different model which from that point onwards might be more economical, or it might be the case that you might be better off load balancing to another provider.

Why Pi Does Not Prune Aggressively

Now that you've made it this far, you probably have an idea why Pi does not prune tool calls. It is tempting to control cost by continuously deleting old tool results or rewriting history and sometimes that is necessary, especially near the context-window limit. But as we have learned, pruning has a cache cost of its own.

Deleting content from the middle changes the prefix at the deletion point. All surviving conversation after it may need to be processed again. The immediate cost of rewriting a long cached context can exceed the future savings from removing a small number of cheaply cached tokens.

A rough break-even comparison is:

one-time rewrite cost
    ~= surviving tokens after the edit * (uncached price - cache-read price)

future savings per turn
    ~= pruned tokens * cache-read price

This is not only an accounting question as old tool results often contain the evidence the model used to make later decisions. Removing them can degrade behavior even if a summary preserves the gist.

Pi therefore prefers a stable, append-oriented transcript and does not treat every old token as waste. Compaction is available when context pressure justifies a lossy rewrite. Because compaction deliberately creates new context rather than accidentally re-billing an unchanged prompt, Pi treats it as a cache reset rather than a cache failure in its session statistics.

The goal is not the smallest possible prompt but the best trade-off among model context, cache reuse, latency, and price.

Simultaneously there can be a case for pruning too. If you are working with providers that do not discount you for good cache usage, or it's for whatever reason not possible to get high cache rates, it might be preferable to prune. It definitely improves the opportunity for the router to balance between different backends as caches are not transferable.

What Pi Can and Cannot Do

Pi works to keep stable inputs stable. It passes a consistent session ID and provider-specific cache hints, places explicit cache points for APIs that require them, records cache-read and cache-write usage, and supports message-anchored additive tool loading where models allow it. Its default transcript behavior also avoids gratuitously rewriting old context.

Pi cannot control every layer after the request leaves the machine. It cannot choose a provider's eviction policy, extend a cache beyond what the API permits, keep a particular GPU alive, or guarantee that a gateway honors affinity. It also cannot preserve a prefix that an extension changes.

What it can do is make cache health visible.

The interactive footer shows cumulative cache reads and writes as R and W, plus CH for the latest request's cache-hit rate. The /session command gives a fuller view: total cached and uncached input, cumulative hit rate, cost, and an estimate of tokens and dollars re-billed by significant cache misses.

Messages
Total: 178
User: 6
Assistant: 58
Tools: 114 calls, 114 results

Tokens
Input: 7,129,883
  Cached: 6,776,832 (95.0%)
  Uncached: 353,051
Output: 30,013
Total: 7,159,896

Cost
Total: $6.054
Cache Re-billed: $0.728 (161,744 tokens, 2 misses)

Users who want misses called out as they happen can enable Show cache miss notices in /settings, corresponding to showCacheMissNotices in settings.json. Pi then inserts a warning after a significant miss, including the estimated re-billed tokens and cost. When it can observe a model switch or an idle gap beyond the usual short TTL, it says so. For other misses it reports the fact without pretending to know what happened inside the provider.

Common Reasons for Worse Cache Performance

When a session's cache-hit rate looks wrong, the usual causes are:

1. Idling. A command, review, or conversation pause exceeds the provider's retention window.
2. Model or provider switches. KV state is model-specific and generally does not move across providers.
3. Branch navigation. /tree, rewinds, forks, and alternate branches can change the active token sequence even when the session ID remains the same.
4. Compaction or manual history rewriting. These intentionally replace part of the prompt and establish a new prefix.
5. Tool and reasoning level changes. Adding, removing, reordering, or editing tool definitions changes an early part of the request unless the model supports message-anchored loading and the change is purely additive. Reasoning level changes usually have the same effect.
6. Dynamic system prompts. Timestamps, random values, changing project context, and extension-provided prompt snippets can invalidate everything after them.
7. Extension context transforms. An extension that modifies old messages or provider payloads can make an apparently stable Pi transcript unstable on the wire.
8. Provider routing and eviction. The prompt can be identical and still miss because the relevant KV blocks are no longer available where the request lands.

Full article content is not available for inline reading.

Read the original article →

Announcing Fugu-Ultra v1.1 🐡

We’ve been thrilled by the reception to the Fugu model family. Thanks to everyone who tried it, shared feedback, and trusted Fugu with real work.

Today, we’re releasing Fugu-Ultra v1.1 → sakana.ai/fugu

Upgraded to incorporate the latest frontier models, resulting in stronger performance across every benchmark shown, including gains of up to 7.9 points over v1.0, with particularly strong results on ProgramBench and Terminal Bench 2.1.

Fugu-Ultra v1.1 is more capable across coding, agentic tasks, and advanced reasoning, and available at the same price as Fugu-Ultra v1.0.

The frontier keeps moving, and Fugu keeps getting better.

Introducing Sakana Fugu

Introducing Sakana Fugu: A full multi-agent orchestration system accessible via a single model API. Our ‘Fugu Ultra’ model matches the performance of Fable and Mythos, delivering frontier capability without the risk of export controls.

Try it: 🐡 sakana.ai/fugu. Fugu stands shoulder-to-shoulder with leading models like Fable and Mythos across the industry's most rigorous engineering, scientific, and reasoning benchmarks.

Read the full blog: sakana.ai/fugu-release

Beyond Bigger Models: Why are Orchestration Models the Next Frontier

Progress in AI has been driven largely by giant, monolithic models. But the most powerful systems of the future will be collaborative ecosystems. Today, this orchestration is no longer just a technical optimization. It has become a geopolitical and operational imperative.

For an organization or a nation, relying on a single company's model for critical infrastructure, finance, or governance is a material vulnerability. This risk is no longer a hypothetical possibility, but a reality. As we have seen with recent export controls imposed on models like Fable and Mythos, access can disappear overnight.

Collective intelligence is the practical hedge against this concentration of power. Because Fugu orchestrates an underlying pool of swappable agents, it simply routes around vendor restrictions. By orchestrating the world’s models, we are delivering the resilient blueprint required for true AI sovereignty.

How does it work?

Sakana Fugu is itself an LLM, trained to call various LLMs in an agent pool, including instances of itself recursively. Fugu dynamically orchestrates the world's best models to tackle complex, multi-step tasks. Fugu is a multi-agent system that behaves like a single model. You send a request to one endpoint, and Fugu decides how to handle it internally.

Fugu manages model selection, delegation, verification, and synthesis automatically. It solves tasks directly when that is enough, or coordinates a team of expert models when a problem calls for more. The complexity of a multi-agent system never reaches your code.

At launch, Sakana Fugu comes in two models accessed via a single OpenAI-compatible API:

• Fugu: balances strong performance with low latency for everyday work. It fits naturally into tools like Codex for coding, as well as chatbots and interactive services. You can also opt specific agents out of its pool for data compliance.
• Fugu Ultra: our flagship model tuned for maximum answer quality on hard, multi-step problems. It coordinates a deeper pool of expert agents for demanding work like AI research, cybersecurity analysis, and patent investigations.

Introducing AB-MCTS

We’re excited to introduce AB-MCTS! Our new inference-time scaling algorithm enables collective intelligence for AI by allowing multiple frontier models (like Gemini 2.5 Pro, o4-mini, DeepSeek-R1-0528) to cooperate.

Blog: sakana.ai/ab-mcts
Paper: arxiv.org/abs/2503.04412

Inspired by the power of human collective intelligence, where the greatest achievements arise from the collaboration of diverse minds, we believe the same principle applies to AI. Individual frontier models like ChatGPT, Gemini, and DeepSeek are remarkably advanced, each possessing unique strengths and biases stemming from their training, which we view as valuable resources for collective problem-solving.

AB-MCTS (Adaptive Branching Monte Carlo Tree Search) harnesses these individualities, allowing multiple models to cooperate and engage in effective trial-and-error, solving challenging problems for any single AI. Our initial results on the ARC-AGI-2 benchmark are promising, with AB-MCTS combining o4-mini + Gemini-2.5-Pro + R1-0528, current frontier AI models, significantly outperforming individual models by a substantial margin.

This research builds on our 2024 work on evolutionary model merging, shifting focus from “mixing to create” to “mixing to use” existing, powerful AIs. At Sakana AI, we remain committed to pioneering novel AI systems by applying nature-inspired principles such as evolution and collective intelligence.

Algorithm (TreeQuest): github.com/SakanaAI/treequest
ARC-AGI Experiments: github.com/SakanaAI/ab-mcts-arc2

Many ARC-AGI-2 examples that were unsolvable by any single LLM were solved by combining multiple LLMs. In some cases, an initially incorrect attempt by o4-mini is used by R1-0528 and Gemini-2.5-Pro as a hint to get to the correct solution.

OpenWorker

Beta - OpenWorker is in open beta: fully usable, updates itself, and we're actively polishing rough edges. Issues welcome.

AI that gets your everyday tasks done. OpenWorker is an open-source AI coworker that lives on your desktop and delivers finished work, not just chat: a polished document, a Slack reply with the numbers, an updated calendar, a triaged inbox.

It runs on your machine and doesn't lock you into any model: bring your own API key for OpenAI, Anthropic, Google, or an open-weight provider, or run fully local with Ollama. Your data leaves your machine only through the model and integrations you choose.

Download

macOS (Apple Silicon) macOS 12+ · signed & notarized · auto-updates

Windows 10/11 (x64) builds are not yet code-signed, so SmartScreen will warn; signing is in progress

Open the app, add a model key (or point it at Ollama), and ask for something real.

How it works

1. Tell OpenWorker the outcome you want - "prepare a customer brief," "untangle my calendar," "draft a report," "check where the release stands across Jira and GitHub."
2. It breaks the task into steps and works across your desktop, files, and connected apps.
3. Before anything consequential - sending a message, changing a calendar, running a command - it checks in and you approve or redirect.
4. You get the finished deliverable, not a to-do list.

Under the hood:

┌────────────────────────────────────────────────┐
│              OpenWorker desktop app            │  native shell + GUI
├────────────────────────────────────────────────┤
│           local agent server (Python)          │  engine · tools · connectors - built on aisuite
├───────────────┬────────────────┬───────────────┤
│  your files   │   your tools   │  your model   │  everything runs with your keys,
│  & terminal   │ 25+ connectors │  any provider │  on your machine
└───────────────┴────────────────┴───────────────┘

What it can do

• Produce real deliverables - documents, spreadsheets, reports, and web pages land as files you can open and share.
• Work from Slack - mention @OpenWorker in a channel; a session opens on your desktop, the work happens with your tools, and the answer comes back as a thread reply.
• Use your everyday tools - 25+ integrations including GitHub, Slack, Jira, Notion, Linear, HubSpot, Outlook, monday.com, Gmail, and Google Calendar, plus your terminal and local files. Any tool reachable over MCP plugs in too, with per-tool control.
• Run on a schedule - automations for recurring work: a morning brief, a weekly report, a standing watch over a channel. Runs land in the app with full transcripts.
• Ask before acting - writes, sends, and shell commands are approval-gated. Unattended runs park their asks in an inbox instead of acting on their own.

Bring your own model

Model access is yours: pick a provider, paste your key, switch anytime. Supported out of the box:

OpenAI · Anthropic · Google Gemini · Inkling (Thinking Machines) · GLM (Z.ai) · DeepSeek · Kimi (Moonshot) · Qwen · MiniMax · Mistral · Grok (xAI) - plus open-weight models via Together and Fireworks, and fully local models via Ollama.

Privacy

OpenWorker is local-first. Everything lives on your machine: the agent loop, your conversations, connector tokens, and model keys - all in the app's local secret store. The only cloud piece is a small service that brokers OAuth handshakes for connectors. You can always use the App without signing-in - use the connectors via manually-created credentials/API-keys.

Run from source

Prerequisites: Python 3.10+, Node 20+, and (for the desktop shell) the Rust toolchain via rustup.

git clone https://github.com/andrewyng/openworker
cd openworker

# 1. One-time bootstrap - creates the Python venv at .venv
#    (on Windows, run from Git Bash or WSL)
bash packaging/setup_dev_env.sh

# 2. Start the local agent server
.venv/bin/openworker-server --cwd ~/some/project --port 8765
#    (Windows: .venv\Scripts\openworker-server.exe)

# 3. In a second terminal, start the UI
cd surfaces/gui
npm install
npm run dev        # browser UI on the Vite dev port

The standalone server creates a per-launch token at <state-dir>/sidecar-8765.token; Vite reads that user-only file when it starts. For direct API calls, send its value in the X-OpenWorker-Token header. The desktop app uses an in-memory launch token instead and never writes it to disk.

Repository layout

Built on aisuite

OpenWorker's engine is built on aisuite, a lightweight Python library providing a unified chat-completions API across LLM providers and an agents layer with tools, toolkits, and MCP support. If you want to build your own agent harness rather than use ours, start there; this repo is a working reference for what aisuite can carry.

Contributing

Contributions and bug reports are welcome - open an issue or a pull request. The app updates itself, so fixes reach installs quickly. For any PR, please attach screenshots of what was broken and how it is fixed now.

License

MIT

FLUX 3 - Real World Models: Towards Multimodal Flow Models as the Backbone of Visual Intelligence.

FLUX 3 is now available in Early Access.

FLUX 3 is our new multimodal foundation model. It jointly learns from images, videos, and audio within a unified architecture, because what it needs to learn is not any one of these elements in isolation. Instead, a model must learn a representation of the world: how objects hold together, how things move, and how events sound.

No single modality provides a complete description. Each is a projection of the same underlying reality, captured by different sensors, each of which loses some information in the process. Images capture spatial structures and relationships at a specific point in time. Videos restore the dimension of time and reveal temporal dynamics and physical laws. Audio reveals causal relationships between mechanical phenomena and acoustics that vision alone cannot detect. Language links these perceptions to goals, abstractions, and instructions.

Learn from one and you get a good model of that projection. Learn from all of them at once and their mutual constraints tell you more: the sound has to match the impact, the motion has to obey the mass, the future has to follow from the past. The modalities stop being separate and start being evidence about one underlying reality.

FLUX 3 is our first model built entirely on that principle, and a checkpoint on our mission to develop real-world visual intelligence: models that perceive, predict, and act across physical and digital environments. Early results in content creation and physical AI suggest it is the right path.

FLUX 3: One model, multiple capabilities.

FLUX 3 builds on Self-Flow, our approach for efficiently aligning multimodal generation and understanding within the same underlying architecture. Based on this approach, we significantly scaled up compute and data resources to train FLUX 3 across video, images, and audio at the same time.

Self-Flow vs. Flow Matching (FM). Left: generation error (Fréchet distance) per modality, each normalized to FM = 100 (lower is better). Right: success rate on manipulation tasks averaged over four task groups through finetuning (higher is better).

Capabilities & Early Evaluations

As a result, FLUX 3 is capable of mixing modalities and generating images and video+audio jointly; both from pure text prompts as well as when providing input references such as images and video. We are highlighting a few of the model’s key capabilities below.

Video

FLUX 3 can create highly diverse videos with audio up to 20 seconds in length in a single generation.

Its core capabilities include the following (all outputs come with native audio generation):

• Text-to-video generation.
• Image-to-video generation, either continuing from a starting frame (“animation”) or using images as visual references.
• Video-to-video generation from a reference clip, carrying central elements of a source video - for instance the same character - into a new scene or context.
• Generative video-audio continuation from input video and audio.
• Keyframe-to-video generation for controlled transitions between defined moments.
• Multilingual dialogue.
• A broad range of visual styles and aspect ratios, extending far beyond conventional cinematic output.
• Agentic chaining of individual clips into longer, multi-shot sequences.
• High style diversity -- FLUX 3 Video easily handles ranges of styles from candid camcorder footage to animation and cinematics.
• Strong typography generation and animated designs.

For the preliminary analysis below, we generated 10-second text-to-video clips in 720p with audio.

Evaluations are early and we expect further improvements

As the model and the harness around it are still in development, these results are preliminary, and we expect further improvements during the early access phase. Across early evaluations, FLUX 3 was preferred over Grok Imagine Video in up to 69% of comparisons, Kling v3 Pro in 60%, Happy Horse v1 in 59%, Happy Horse 1.1 in 57%, Seedance 2.0 and Gemini Omni Flash in 52%. FLUX 3 was preferred over Runway Gen-4.5 in 77% of comparisons and over Luma Ray 3.2 in 93% of comparisons.

While still in development, FLUX 3 Video is already particularly strong in capturing human facial expressions, associating sounds with physical events, and multilingual capabilities. Furthermore, these capabilities can be combined to create sequences lasting several minutes, where visual references help ensure that the characters remain consistent across all scenes.

Image

FLUX 3 can synthesize and edit images in a wide variety of styles, aspect ratios, and resolutions. In preliminary evaluations conducted during midtraining, FLUX 3 already shows a significant improvement over earlier versions of FLUX: its ability to handle complex prompts and text generation has improved significantly. The model produces a wide range of output styles (see the following samples), and is able to render high-accuracy text in multiple languages.

As with video evaluations, these are preliminary results, and we expect further improvements before release. We will open up an early access phase for FLUX 3 Image in the following weeks.

Action

FLUX 3's world understanding extends to action prediction. We have taken two routes to it: integrating native action prediction into FLUX 3 directly, scaling up our initial work in Self-Flow; and using the pretrained video backbone as a dynamics-aware foundation that specialized action models can be finetuned from with limited task-specific data.

For the second, mimic robotics was one of the first partners to gain early access to FLUX 3. Together we developed FLUX-mimic, a video-action model combining the FLUX 3 backbone with mimic's expertise in robot learning for dexterous manipulation and production deployment.

Launch Plan

Over the next few weeks and months, we will make the following capabilities available, each after an early access phase for ensuring smooth rollout, collecting feedback and rigorous safety-testing. All capabilities are built from the same underlying multimodal flow matching model. These capabilities and models include:

• Video and audio generation and editing through APIs and private weight access. (“FLUX 3 Video”)
• Action prediction through selected research and commercial partners, beginning with mimic robotics (“FLUX-mimic and FLUX 3 Action”)
• Image synthesis and editing through APIs and private weight access. (“FLUX 3 Image”)
• Open-weight access to a multimodal backbone, for content creation (video, audio and image) and action prediction. (“FLUX 3 Dev”)

We will also release more technical details on the underlying approach.

What’s next?

We are only beginning to scratch the surface of versatile, capable, unified multimodal models, and what they will enable. From interactive image & video editing, simulation to computer use and physical AI, the frontier is wide open. While we gradually roll out these new capabilities, we are already working on the next generation models. Our goal is to unify perceptual, action and language prediction in the same unified model.

If you are interested in exploring and building with FLUX 3, get in touch here. If you are interested in contributing to our mission, join us!

Why it matters

The deal shows that established tech operators are consolidating the nascent long‑horizon AI‑agent market, giving Sierra immediate access to a platform that can power autonomous, multi‑step applications across enterprises.

On July 23rd, 2026, former Salesforce CTO and tech veteran Bret Taylor posted on X that Sierra is acquiring TakeOff, which he described as “the leader in long-horizon AI agents.” The tweet also highlighted the involvement of former Google executive Clay Bavor and TakeOff founder Aakash Thumaty, noting that the TakeOff team would join Sierra.

“I’m incredibly excited to share that Sierra is acquiring TakeOff, the leader in long-horizon AI agents. @claybavor and I couldn't be more excited to welcome @letsleverup and the entire TakeOff team. In less than a year, they've built something truly special: an agent platform” – Bret Taylor, X post, July 23rd, 2026

The announcement provides the only public confirmation of the transaction. No financial terms, valuation, or closing date were disclosed. Neither Sierra nor TakeOff issued a separate press release, and no investors were named in the post. The X thread includes a link to TakeOff’s own status, which simply echoes the acquisition news without additional detail.

Who is behind the deal?

Bret Taylor has a long track record of building and scaling internet‑scale products. After co‑founding FriendFeed, he joined Facebook, where he helped launch the Open Graph platform, and later served as CTO of Salesforce, overseeing the integration of Slack. His current role at Sierra is not detailed in the announcement, but his involvement signals that the acquiring company sees strategic value in AI‑agent technology.

Clay Bavor is best known for leading Google’s AR/VR initiatives, including the development of Daydream and Project Starline. His presence on the announcement thread suggests that Sierra may be tapping expertise in immersive and interactive technologies, which aligns with the “long‑horizon” capabilities of TakeOff’s agents.

Aakash Thumaty (Twitter handle @letsleverup) represents the founder‑team of TakeOff. The X post by Thumaty (linked in the thread) did not add new information beyond the acquisition announcement.

What is a “long‑horizon” AI agent?

In current AI research, agents that can plan and execute over extended sequences – sometimes spanning minutes, hours, or even days – are still rare. Most commercial offerings focus on short‑term interactions, such as chat assistants that handle single‑turn queries. Long‑horizon agents aim to maintain state, adapt to evolving goals, and coordinate across multiple tools or APIs without continuous human supervision.

Industry players such as DeepMind’s Gato, OpenAI’s recent multi‑step reasoning models, and Anthropic’s Claude‑3 series have demonstrated fragments of this capability, but a turnkey platform that lets developers deploy such agents at scale remains elusive. TakeOff’s claim of building a “leader” platform in less than a year indicates rapid progress, though the public technical details of the platform are not disclosed.

Why the acquisition matters now

The AI landscape in 2026 is shifting from hype around foundation models to concrete productization of autonomous systems. Enterprises are experimenting with agents that can manage supply‑chain logistics, orchestrate cloud resources, or automate complex customer‑service workflows. By acquiring TakeOff, Sierra positions itself to offer an end‑to‑end solution that combines a proven long‑horizon agent stack with its own (undisclosed) infrastructure or market reach.

Analysts have noted a wave of consolidation in the AI‑agent space, with larger cloud providers and platform companies acquiring niche specialists to fill gaps in their AI roadmaps. The timing—less than a year after TakeOff’s platform debut—suggests that Sierra moved quickly to secure technology before competitors could replicate it.

What remains unclear

The announcement provides no insight into the financial terms, the post‑acquisition integration plan, or how TakeOff’s existing customers (if any) will be affected. Sierra’s broader strategy—whether it plans to embed the agent platform into a larger suite of products, sell it as a standalone service, or use it internally—remains opaque. Likewise, the size of the TakeOff team and its engineering depth are not disclosed.

Future reporting will need to track any follow‑up statements from Sierra, regulatory filings that might reveal the purchase price, and product roadmaps that demonstrate how the acquired technology will be brought to market. Until then, the deal stands as a clear signal that established tech leaders are betting heavily on autonomous AI agents capable of sustained, multi‑step operation.

AMD and Cerebras Announce Industry-Leading Ultra-Low-Latency and High Throughput AI Inference Solution

News Highlights

• AMD and Cerebras are collaborating to advance a workload-optimized approach to ultra-low-latency AI inference infrastructure.
• AMD Helios™ and the Cerebras Wafer-Scale Engine will operate as a single disaggregated inference workflow, combining ultra-high-throughput from AMD Instinct™ GPUs, with ultra-fast token generation of Cerebras Wafer-Scale Engine.
• Cerebras plans to deploy AMD Helios in its data centers, with the joint solution expected to be available first through Cerebras Cloud in the second half of 2026.

AMD (NASDAQ: AMD) and Cerebras Systems (NASDAQ: CBRS) announced a technical partnership to deliver a new disaggregated AI inference solution that combines AMD Helios™ rackscale solutions with the Cerebras Wafer-Scale Engine. Unveiled at Advancing AI 2026, the solution is designed to deliver the ultra-low latency required for the most advanced AI applications while dramatically increasing the throughput and efficiency.

The joint AMD and Cerebras solution will deploy AMD Helios alongside Cerebras Wafer-Scale Engine technology integrated in a single inference workflow for maximum performance and efficiency. AMD Helios will provide a high-performance, scalable throughput engine. Cerebras Wafer-Scale Engine technology will provide ultra-fast, ultra-low latency decode and token generation. Together, the two compute engines are expected to deliver up to 5x higher tokens per second per watt (T/s/W).

AI inference workloads increasingly have different requirements across latency, throughput, token capacity, cost and scale. High-volume workloads prioritize maximizing token generation, while coding, real-time copilots, live agents and agentic workflows demand faster response times. These differences are driving demand for heterogeneous infrastructure that matches compute technologies to specific workload requirements.

The AMD and Cerebras solution addresses this challenge through disaggregated inference, optimizing the two primary stages of the workflow independently. AMD Helios provides ultra-high throughput, processing prompts and large context windows. The Cerebras Wafer-Scale Engine accelerates the memory-bandwidth-intensive token generation, with ultra-low latency. By connecting these best-in-class engines through one integrated workflow, the companies are creating a differentiated platform for ultra-low-latency inference without sacrificing throughput or scale.

“AI inference is becoming one of the largest infrastructure opportunities in AI, and its growing diversity requires a more flexible approach,” said Dr. Lisa Su, chair and CEO, AMD. “AMD Helios delivers leadership performance and scale for the broadest range of inference workloads. Together with Cerebras, we are extending that leadership into the most latency-sensitive applications and creating a powerful new platform for real-time agentic AI.”

“The demand for ultra-fast inference is growing at an unprecedented pace. Cerebras delivers the world’s fastest, ultra-low-latency inference,” said Andrew Feldman, CEO and co-founder, Cerebras. “Partnering with AMD gives us an incredible opportunity to bring that performance to even more customers.”

Fast token generation is becoming increasingly important as AI moves into software development, autonomous agents, robotics, scientific discovery and other applications where response time directly shapes the user experience and the usefulness of the system. The joint solution brings together complementary architectures purpose-built for these demands.

AMD Helios provides the high-throughput prompt engine, rack-scale efficiency and deployment scale required to process large numbers of complex requests. Cerebras Wafer-Scale Engine technology provides the ultra-low-latency and decode performance needed to return tokens in real time. The result is a solution designed specifically for the ultra-low-latency segment of the inference market, with AMD Helios as the foundation for high-throughput and balanced inference workloads across the data center.

Cerebras plans to deploy AMD Helios systems in its data centers, with the joint solution expected to become available initially through Cerebras Cloud in the second half of 2026.

Based on modelling by AMD Performance Labs and Cerebras in July 2026 to determine tokens per second per kilowatt (TPS/kW) at a comparable interactivity point with Kimi 2.6 1T Model comparing an AMD Helios rackscale solution with Cerebras WSE to a Cerebras WSE-only configuration. System manufacturers may vary configurations, yielding different results. MI400-021

OpenAI's naturalistic GPT-Live audio AI model is now integrated with the ChatGPT desktop application.

We’ve raised $300M at a $10.3B valuation, led by Sequoia Capital alongside Andreessen Horowitz, Jane Street, Diffusion, Argo, and SK Hynix. We’re grateful for their support on our journey to Gigawatt-scale.

Our mission is to run the world’s inference. We believe the world’s most powerful models require an entirely new form of computing. Frontier models deserve frontier inference clusters: co-designed chips, packages, PCBs, cold plates, interconnects, and more that push the entire pareto curve on the most demanding workloads.

Today, under 1% of the global population has access to the most advanced AI models. Distributing access will require better technology to deliver more tokens / watt and many Gigawatts worth of compute. Our mission has never been more urgent.

We’ve kicked off fabrication of hundreds of millions of dollars worth of inference clusters. We’ve built a new 10-Megawatt lab fifteen minutes from our office to enable continuous deployment of our first-gen hardware and rapid prototyping of future generations. Production is the product.

We’re excited about bringing our first product to life, and we’re even more excited about what’s to come. If you’d like to work on thousand-chip scale-up domains, in-house SMT lines, and new RL environments for recursive kernel generation, you should join us in accelerating inference.

A Huawei-led consortium has released a technical report that documents full-parameter post-training of DeepSeek's V4 family on Huawei Ascend chips, measuring the run at 34.22% model FLOPs utilization.

OpenAI said today it is making ChatGPT Health, a feature that helps users with health-related queries, available to all U.S.-based users over 18 across all plans. The announcement comes a day after a Florida-based pastor sued the company for giving a near-fatal suggestion not to consult a doctor.

The company started testing the feature through a dedicated hub earlier in January, allowing users to integrate data from other services and their personal information from services such as Apple Health, Function, and MyFitnessPal.

At that time, OpenAI said users were asking 230 million health-related queries each week. That number has now gone up to 300 million.

Users can also integrate their medical records from hospital systems like Epic and Oracle Health, and health platforms like One Medical and Function Health.

OpenAI said earlier that users needed to use the health hub for health-related queries. Now they can choose to draw insights from connected information in the health section in all queries. The company said during the testing it observed that 70% of health-related queries took place outside the dedicated hub. Through this new feature, users can use their health information to get information on food or allergies in the general chat.

The company noted that its models have made progress in answering health-related queries. The company noted that the smallest model from its latest release, GPT 5.6-Luna, outperforms GPT 5.5 on HealthBench evaluation, an open source benchmark developed by the company to evaluate large language models (LLMs) on health queries. OpenAI said that it doesn’t use any user data to train its model, and it works with physicians to improve its models on health queries.

Despite these performance gains, the company maintains in its terms that its services are “not intended for use in the diagnosis or treatment of any health condition.” The company cited these clauses in response to the above lawsuit, and also told The New York Times that it is working on making health- or medicine-related answers safer. With the latest roll out, OpenAI said it wants people to verify information and take medical decisions based on professional advice.

Several studies have highlighted that AI bots are not reliable for medical advice. However, this has not deterred companies like Anthropic and Google from launching health-related AI features.

Health in ChatGPT is rolling out to logged-in U.S. users with free, Go, Plus, and Pro plans on the web and iOS this week.

Spectators typically leave a UFC fight with ringing ears and a sheepish resolve to get back to the gym. Wang Xingxing went straight to his laboratory.

“I thought I could make a bigger robot for boxing competitions,” says Wang, the founder and CEO of Hangzhou-based humanoid-robotics firm Unitree. “Then I thought, Why not do an even bigger one that can actually carry a human inside?”

The upshot of Wang’s creative fervor is the GD01, which Unitree bills as the world’s first mass-produced, transformable mecha robot. Standing 9 ft. tall and weighing half a ton, it can transport a pilot in its torso-mounted cockpit, walk on either two legs or four, and wield fists powerful enough to knock down walls. Heavy-duty mechanical hinges connect titanium-alloy limbs to a protective casing of carbon fiber, enabling smooth, deliberate strides on brutish red legs.

Wang showed off the GD01 exclusively to TIME as part of his first international media interview, revealing that beyond mixed martial arts, he took a dash of inspiration from James Cameron: “We actually drew some references from a similar robot featured in Avatar.”

Wang has done more than arguably anyone else to bring this science fiction closer to reality. A native of China’s coastal city of Ningbo, he founded Unitree in 2016 when he was just 26 years old and has moved from robot dogs to humanoid robots and now, with GD01, human mobility machines. But while the GD01 might inspire awe, humbler humanoids like Unitree’s flagship G1 are the ones set to become the workhorses of a global robotics transformation already gathering pace in China, the U.S., and beyond.

This is the next great tech frontier, one that has previously been the purview of Star Wars and Blade Runner: AI that walks, sees, hears and physically interacts with the real world. Morgan Stanley predicts that as many as 13 million humanoids could walk among us by 2035—equivalent to the population of Bolivia. That number is forecast to rise to 1 billion by 2050, when the industry could be worth $5 trillion.

Unitree is hardly alone staking a claim in this space. In January, Tesla announced the third generation of its Optimus humanoid robot with a goal to eventually produce 10 million units per year at factories in Fremont and Texas. In late June, California-based Figure dispatched its latest 03 humanoid to BMW’s Spartanburg plant in South Carolina, where its earlier 02 model helped assemble 30,000 cars last year. Oregon-based Agility, meanwhile, has secured deployments everywhere from Toyota to Latin American e-commerce giant Mercado Libre. But Unitree is the world leader in terms of volume, dispatching more than 5,500 units in 2025, comprising over a quarter of a global market that is only just waking. In June, Nvidia announced that its bleeding-edge Jetson Thor chip would be fitted into Unitree’s H2 humanoid for sale to research institutions around the globe.

We could be, Wang says, anything from two to 10 years away from a “ChatGPT moment” for humanoids, whereby a robot achieves sufficient “generalization” to be able to execute 80% of requested voice commands in an 80% unfamiliar environment. “Once the technology surpasses this ... inflection point, robots can gradually be rolled out for large-scale commercial use,” he says.

“We don’t see any significant competitors outside of China.”

If the timeline for that breakthrough is a matter of debate, its potential impact is not. Nonunion robots (the only kind) could operate around the clock on assembly lines, replace some nurses in hospitals, and pick fruit under the hot sun.

The positive outlook is that wide-scale humanoid adoption will herald an unprecedented era of global abundance, automating mundane and dangerous labor while ushering in post-scarcity societies. There is a less rosy divination, however. Amid a mass labor displacement, robots concentrate even more wealth into fewer human hands. With production severed from wages, people could face an identity crisis as robots begin to dominate manual labor and service industries, while governments contend with strained social safety nets and rising competition between the U.S. and China—leaders in humanoids as they are in AI—over raw material and technology. On June 3, three U.S. lawmakers introduced the bipartisan Guard Act, which seeks to ban Chinese robots deemed to be a national-security threat. Unitree was singled out by John Moolenaar, a bill co-sponsor and chairman of the House Select Committee on China, who accused Wang’s firm of benefiting from “generous state subsidies” and threatening to bankrupt American competitors.

With AI, we’ve already seen how the ability of large language models (LLM) to parse huge reams of data also makes them susceptible to hallucinations and unintended consequences, like designing biological weapons. But while an LLM can suggest harm, a robot can enact it, auguring a terrifying new cybersecurity peril in the age of embodied AI. Online scams already siphon off staggering sums, but a hacked humanoid could move the threat from screens into homes, streets, nurseries, and factory floors.

The tension between these visions can be reduced to a choice of myth: Are Wang and his GD01, and indeed their competitors, akin to Luke Skywalker and the faithful, guileless C-3PO—or more like Frankenstein and his monster?

Wang, unsurprisingly, veers toward the positive, predicting that in 10 years every household may have a small robot that can do your laundry, grill your steak, and even provide basic care if you fall ill. They can also take on tasks most humans would rather avoid, from steep mountain farming to sewer maintenance.

But if that sounds like a future arriving faster than society can absorb it, Wang insists the transition will be gradual. “It will evolve step-by-step,” he says. “If society finds that robots don’t perform well in a certain sector, or if people simply don’t like them there, we can easily pivot and re-direct their application to other fields. So, everything remains well within controllable boundaries.”

Wang is an unlikely prophet of the AI age. Slight and bespectacled, he possesses none of the performative bravado associated with many technology founders. Instead, he speaks about robotics with quiet, unwavering conviction.

The possibilities and promises of a humanoid revolution have occupied him since he was 10 years old and first watched early televised robotics experiments performed at MIT by Marc Raibert, founder of Boston Dynamics. He spent his free time building model aircraft, conducting experiments, and, before long, assembling miniature turbojet engines.

In 2009, Wang enrolled at Zhejiang Sci-Tech University in Hangzhou, majoring in mechatronics engineering. As a first-year student, he says, he built his first small bipedal robot, capable of walking a few steps, on less than $30. That same price-conscious approach would underpin Unitree’s flourishing. While pursuing a master’s degree at Shanghai University, he developed a prototype quadruped robot, XDog, which still occupies pride of place in the Unitree headquarters. Departing from the hydraulic-powered designs popularized by his idols at Boston Dynamics, Wang opted for a cost-saving electric-motor model, which helped XDog win second prize in a national entrepreneurship competition.

After graduation and a stint at Shenzhen-based drone trailblazer DJI, Wang launched Unitree with 2 million RMB ($300,000) in capital from an angel investor. It has since become a cultural phenomenon; an estimated 600 million people watched videos of Unitree’s H2 and G1 humanoid robots swinging nunchucks and executing flips as part of February’s Spring Festival Gala—China’s equivalent to the Super Bowl halftime show. It was the second year that Unitree humanoids provided the headline act, and all eyes are on what Wang has planned next time. “I have no idea,” he confesses with a laugh.

In March, Unitree filed for an IPO that would value the firm at $6 billion, sparking a buying frenzy in mainland Chinese markets for companies that would benefit along with it, such as upstream suppliers.

But high expectations are both a blessing and a curse. While Unitree’s first-quarter revenue surged over 68% year-on-year to $62 million, adjusted net profit halved over the same period to just $6 million, which the company blamed on a significant spike in R&D and sales costs. Wang talks about “a flip side” to viral internet fame. “Because the hype gets so intense, it places a fair amount of pressure on our company and the industry as a whole,” he says. “Everyone starts expecting your technology to leap forward at a breakneck pace. But the truth is, deep tech breakthroughs require a certain amount of time.”

Unitree’s blossoming came just as the Chinese government began leaning into AI and robotics, which President Xi Jinping has highlighted as key examples of the “new growth drivers.” China’s Ministry of Industry and Information Technology in turn declared humanoids a “disruptive innovation” on the same level as smartphones and EVs—breakthroughs destined to transform human production and lifestyle, making leadership a matter of economic and national security. It’s a policy push backed by hard cash; since late 2024, Chinese cities have put together investment funds worth over $26 billion, according to Morgan Stanley. Unitree hasn’t disclosed the exact level of state support it receives, though as one of Hangzhou’s “Six Little Dragons”—elite local startups—it benefits from the city’s $14 billion Sci-Tech Fund. “China is an ass kicker, next level,” Elon Musk told a Tesla earnings call in January. “To the best of our knowledge, we don’t see any significant [humanoid robot] competitors outside of China.”Wang’s role leading that charge was evident when he appeared as the youngest member of a high-profile business symposium hosted by Xi last spring. More than 140 Chinese companies now make humanoid robots, yet the perception inside China is that Unitree is not just another robotics startup, but a hardware platform destined to replicate what BYD achieved for EVs and DJI did for drones. Wang’s strategy is simple but hard to beat: first, control the most difficult core components; leverage scale to relentlessly drive down costs; and use each product generation to unlock new markets.

Unitree has slashed the pretax price of its flagship G1 humanoid from $16,000 to $13,500 in just 18 months. (Still, with components that cost under $9,000, according to recent estimates by SemiAnalysis, that gives Unitree a handy 33% gross margin.) The R1 model now retails for under $5,000, and over six years the price of their “robot dogs” has dropped from $45,000 to under $2,000. Wang insists Unitree wins through manufacturing cost efficiency—for example, it produces its own actuators, the “muscles” responsible for movement, which account for 50% to 70% of a humanoid’s cost.

While the G1 might not boast the peak performance of some Western competitors in terms of load bearing and durability, it has crossed the “good enough” threshold for real-world deployment. With the ability to carry 5-kg payloads for 10 to 15 minutes continuously without overheating, it is moving out of R&D labs and into pilot deployments for warehouse logistics and material handling.

“Humanoid robot technology is still in its relatively early stages.”

That said, to date only a fraction of humanoids sold are employed in actual useful work. The reason is very simple: today’s robots operate at about 30% to 50% of human efficiency for general factory and warehouse tasks, such as stacking and carrying boxes. Only 9% of Unitree’s sales go directly into industrial applications, with 17% for commercial venues such as tourist sites, exhibitions, and retail stores. That leaves 74% of Unitree’s humanoid sales to universities, research institutions, and individual developers looking to hone their own AI and motion-control capabilities. And the company is far from alone. The entire modern humanoid industry is essentially selling robots to people trying to build better robots. “Humanoid robots are just not ready for the mass market,” says Grace Shao, who publishes the AI Proem newsletter.

Wang is unconcerned, however, and compares this trend to the early days of the personal computer, when they were primarily purchased by those in the business: “As the software becomes increasingly mature, with more developers joining each year, the proportion of genuine industrial deployments is growing at an incredibly fast pace.”

For now, Unitree is still wrestling with the same white whale as its legion of competitors: the generalization gap, which Wang calls “the biggest headache for the entire global scientific community.” While today’s back-flipping machines are engineering marvels, they remain stubbornly task-specific. If you want a robot to pick up a pen and move it across a desk, you must program every granular motion. But to make AI truly useful in the physical world, robots must transition from executing rigid scripts to understanding novel commands—like being told to simply “go to the grocery store.”

Achieving this milestone will require overcoming two self-reinforcing problems: the scarcity of 3D data and the diversity of deployment environments. Unlike LLMs like ChatGPT, which can parse the virtually boundless internet, robots need manipulation data collected in the real physical world, which is costly and difficult to scale. It must typically be collected manually by repetitively training machines how to perform certain tasks, such as folding shirts, stocking refrigerators, or shaking a mai tai. Plus, explains Jiang Zheyuan, CEO of Beijing-based humanoid firm Noetix, “even if a well-performing manipulation strategy is trained for a specific task, the success rate often drops sharply when the environment changes slightly—for example, object displacement, different lighting, or different tabletop materials.”

Thanks to firms like Unitree, the hardware for humanoids is advancing quickly. The real battleground is over the robot brains, or vision-language-action (VLA) models, which are designed to turn human inputs into embodied AI performing intuitively and accurately. The VLA space alone will be worth $40.50 billion by 2035, predicts Kaiso Research, a market intelligence firm, and many of tech’s biggest players are aggressively pursuing their own models, including Nvidia’s GR00T N1.7, Google DeepMind’s RT-2-X, and Alibaba’s Qwen Robot Suite.

That battle is increasingly about one thing: collecting data. In New York City, for example, Micro AGI, which counts Unitree as a partner, is sending camera-wearing staff to clean apartments free of charge simply to generate 3D data to train its physical AI model. But as a dedicated robotics company, Unitree is arguably in a weaker position than Tesla, Xiaomi, or XPeng, which have diversified revenue streams and established supply chains. So solving this generalization gap is key to maintaining its status as today’s market champion. After all, its absolute sales are by any measure minuscule, meaning whoever closes the gap first could easily surge ahead.

There may be a game changer on the horizon: revolutionary new 6G connectivity would enable AI-native networks on which collective learning happens seamlessly across millions of connected devices. If one humanoid in a distant warehouse wrestles with a novel tool and finally masters it, that newly generated data can be instantly uploaded and shared across the entire global fleet. 6G will empower us “to make a digital twin of the world,” Cristiano Amon, CEO of $200 billion semiconductor firm Qualcomm, tells TIME.

Boosted connectivity would solve the robotics bottlenecks of weight and battery life. Navigating our world requires simultaneously processing 4K video, spatial lidar, audio, and delicate tactile or proprioceptive feedback. But carrying heavy computational hardware and the means to power it severely limits both robot agility and operating time. 6G could potentially combine cloud processing with network lag of a near instantaneous 0.1 to 1 millisecond. Suddenly, lightweight physical robots could be animated by a cloud-based AI brain operating in real time. When a robot fumbles, 6G provides a safety net via remote human teleoperation.

As these machines come online, a significant challenge will be securing any centralized network. Last year, cybersecurity analysts discovered a “backdoor” in Unitree’s Go1 robot dogs, whereby third parties could log onto a portal to see the robot’s location, view live camera feeds, and even control it remotely. In response, Unitree shut down the cloud service that enabled the breach. Wang insists his “robot products operate completely disconnected from the network. If the client does not permit it, the robot will not save any environmental information whatsoever.” Besides, he adds, the data stored on a robot is “entirely negligible” when compared with that kept on a smartphone.

Then there is the fear, familiar from fiction, that machines built to act on our behalf might one day be made to act against us. Even today’s robots have shown physical AI carries physical risk: viral clips capture Unitree machines swinging an arm or a leg into unsuspecting children who stray into the path of their choreography. As humanoids graduate to generalization, whether they’ll become more cautious or more dangerous is a key question facing the industry. Wang plays down the potential for renegade robots to wreak havoc. “That won’t happen,” he says. “These robots are engineered with rigorous hardware constraints and safety protections.”

The need for safety will hit home if human-shaped machines step off assembly lines and into living rooms. On June 30, Shenzhen-based UBtech launched its lifelike U1 humanoids specifically for companionship, featuring silicone skin and biomimetic facial expressions, and powered by what the company claims is world’s first empathic LLM “capable of recognizing more than 20 fine-grained emotional states.”

But the greatest obstacle to the household humanoid may not be technical. First proposed by Japanese roboticist Masahiro Mori in 1970, the “uncanny valley” hypothesis posits that as an object becomes more humanlike, our affection for it grows. Once it crosses a critical threshold—getting too close to human—our response plunges into revulsion. As firms experiment with anthropomorphic robots with real hair, it remains to be seen whether the valley can be bridged.

At Unitree’s headquarters, the GD01 was displayed hanging from a reinforced steel joist. Wang was unwilling to operate it or even climb inside. “It might not be safe,” he shrugs of his creation, which carries a retail price of $650,000. When TIME’s photographer asked for a nearby promotional background to be relocated, Unitree’s technicians protested that shifting it would interfere with the lidar of the nearby phalanx of G1s practicing that now iconic Spring Festival Gala acrobatic martial-arts display. When the performance finished, those technicians rushed to spray their now sizzling lidar with coolant following that four-minute exertion.

For all Unitree’s lead in the humanoid market, today’s machines remain strangely theatrical: bodies dressed as intelligence, rehearsing a role they have not yet learned to live. “Humanoid robot technology is still in its relatively early stages,” cautions Wang. “It will likely take some time before we see genuine, large-scale deployment. It won’t happen overnight.” Nearby, GD01—part prototype, part prophecy—stands sentinel, as if quietly rebuking his master’s prudence.

SpaceX has started to turn away satellite operators seeking dedicated rides on its Falcon 9 rocket beyond 2028. The company has also stopped building some non-reusable components for the Falcon family. SpaceX has publicly stated that it intends to offramp its Falcon 9 rocket for Starship, but has not disclosed a timeline. The company is likely to still use the Falcon 9 for launches for the Department of Defense and NASA.

It's been a busy six months, that's for sure. I'm not going to bury the lede here; the short TLDR is I'm joining the folks over at https://antithesis.com/.

If I wind back time to November 2024, it was apparent to me back then that our profession would change. To be frank, in the last six months our profession has changed more than it has in the last 30 years. Software authoring has been commoditised. Everyone is now a software developer, but being a software developer does not mean that they're an engineer.

The word engineer is used trivially in our profession, but in other industries the word engineer means failures are unacceptable.

Now I'm a little bit old and crusty these days; I turn 44 next week, and I've seen some absolutely horrible codebases, but if I'm to be honest, I can remember some of the first code that I wrote, and I have regrets. If you caught my talk at the AI Engineer World Fair, one of the things I shared was some deep concerns that we are entering into another Eternal September. You see, we didn't create enough software engineers after the 2000s dot-com implosion to properly support an apprenticeship model of learning in our industry.

It's now twenty-six years since that event, and now that everyone can create software, we've got some hard questions to solve. But whilst many things have changed, the job of software engineers is to produce experiences without defects.

So I've been pondering how we are going to fix this predicament.

Here's my hypothesis:

1. The discipline/techniques of formal verification and deterministic system testing are about to cross the chasm. There's a whole lot of brownfield software out there that's been written over the last 30 years that is being affected by the infinite software crisis (“how do we do code review now?” / “the volume of code/change is too high”) all at once.
2. Not enough skilled practitioners in the discipline/techniques of formal verification and deterministic system testing exist.
3. Whilst the costs of building a simulator for a project (new or retrofitting an existing project) are significantly cheaper now, there are entire categories and classes of problems that will not surface unless you emulate a deterministic computer (which provides a forceful way to make anything deterministic).
4. Antithesis, when used in conjunction with adversarial code reviews by an LLM and with language analyzers driven by pre-commit hooks, will be a key component in software factories that enables people/agents to deliver reliable software without the burden of having to learn this specialized knowledge.

Creation is now near-free. Verification/understanding is not, yet. It's time to engineer away the slop.

What makes it different

Most of the library is built on the experimental html-in-canvas API, which lets WebGL effects read and redraw your live DOM. Text stays selectable, links stay clickable, and the whole page becomes a texture that fire, fluid, and glass can distort in real time.

Where html-in-canvas is not supported, components degrade gracefully to pure WebGL overlays, so every visitor gets a working page.

• 25 components and counting: Liquid, Blaze, Glass, Shatter, VHS, Particle Reveal, and more
• Framework agnostic: React, Solid, Preact, Vue, Svelte, and vanilla JS builds for every component
• Copy, do not install: components ship as source through a shadcn-compatible registry
• Zero config: each component is self-contained with sensible defaults and typed props
• MCP ready: point the shadcn MCP server at the registry and let your AI assistant install components

Quick start

Add a component with the shadcn CLI (run npx shadcn@latest init first if you have not used it before):

npx shadcn@latest add @canvas-ui/liquid-react

Swap liquid for any component and react for solid, preact, vue, svelte, or vanilla. The source lands in components/canvasui/ in your project, yours to edit.

Browser support

The full html-in-canvas experience currently requires Chrome or Edge 140+ with the chrome://flags/#canvas-draw-element flag enabled. Everywhere else, components automatically fall back to WebGL overlay rendering.

Development

This repo contains the library source (src/lib), the documentation site (Next.js 16, Tailwind v4, deployed to Cloudflare Workers), and the registry build.

npm install
npm run dev        # builds the registry, then starts next dev
npm run build      # production build
npm run deploy     # build and deploy to Cloudflare

Contributing

Issues and pull requests are welcome.

License

MIT + Commons Clause. Free to use in your own projects, commercial or not. The Commons Clause only restricts selling the library itself.

Getting locked out of an account is no fun. Google has a few ways to help you regain access if you happen to forget your password or lose an authenticator, including recovery contacts and backup codes. Now, Google has a completely new option: your face. You can now give Google a video record of your face and sign into your account with a selfie, which sounds like something people are going to just love.

You will have to set this feature up ahead of time if you want the option of regaining account access with a selfie later on. To get started, verify your account type is supported. You won’t be able to configure selfie sign-ins for Workspace accounts, child accounts, or any account enrolled in Google’s Advanced Protection Program.

Configuring selfie sign-in requires you to record a video, which Google will store on its servers. Google’s selfie sign-in landing page includes the typical disclaimers about privacy and data access, promising that the company will keep the video encrypted and won’t use it for any other purposes unless you opt in.

Selfie videos can be used for login purposes, to verify your age for accessing certain account features, and to create an AI avatar. Currently, Google does have an optional toggle in the setup flow that allows the company to use your selfie video to improve its facial recognition tech. A Google spokesperson confirms this is not required to use the new account recovery feature, and the company will not use the video for anything else if you leave the box unchecked.

Assuming you’re okay with all that, you will use the camera on your phone or computer to record yourself looking around as instructed. This allows Google’s models to accurately map your face so the system can compare that to selfies you take in the future to verify your identity. Google notes it may ask users to update their selfie videos on occasion.

When you use a selfie to access your account, Google will have you move your head around to ensure you’re a living human. It matches this data against the original video, and boom, you’re logged in. However, in the age of AI, it’s not very hard to make deepfake videos of someone. It’s even possible to do that in near real-time, as would be required for Google account recovery.

Google claims it has multiple layers of security measures to detect deepfakes, but this method may not be as secure as other options. After all, Google won’t allow those using Advanced Protection to use selfie sign-ins. Advanced Protection also requires you to use a security key, restricts third-party apps from accessing your account, and conducts more Gmail scans to detect phishing.

If you don’t mind giving Google your face, you can record a selfie video in just a few minutes. Google notes that you can choose to delete the video from its servers at any time from your account settings.

About Confidential Containers

Confidential Containers addresses the need to protect data in use within cloud native environments. While data at rest and in transit are commonly encrypted, processing data in memory often leaves it vulnerable. Confidential Containers leverages hardware-based Trusted Execution Environments (TEEs) to encrypt data during computation. This allows organizations to run sensitive workloads on third-party infrastructure with the assurance that the infrastructure operator cannot access the data being processed. While most security solutions focus on data at rest or in transit, this project focuses on protecting data in use.

The project began as an open source effort in 2021 through a collaboration between Red Hat, Intel, IBM, and other industry partners to bring confidential computing to cloud native applications. By integrating hardware security features into the Kubernetes ecosystem through the Kata Containers runtime, Confidential Containers allows organizations to run sensitive workloads on public clouds or private infrastructure with increased assurance. These advancements allow developers to deploy confidential workloads using familiar Kubernetes workflows, reducing the complexity of adopting TEEs.

Confidential Containers Key Milestones and Ecosystem Development

Since joining the CNCF Sandbox, Confidential Containers has achieved significant growth, supported by major technology organizations and cloud service providers, including Microsoft Azure, Intel, AMD, IBM, NVIDIA, Alibaba, and Red Hat.

The project’s ecosystem now reflects robust engagement, with over 1,000 combined GitHub stars across foundational core organization repositories, over 1,200 merged pull requests across 26 repository groups, and more than 1,500 historical and active tracking issues. This activity is driven by a vibrant community of over 150 active contributors from hardware vendors, hyperscale cloud providers, and security researchers, all supported by more than 15 cumulative engineering lifecycle releases.

Confidential Containers continues to mature through seamless integration with Kubernetes and infrastructure automation via tools like Kyverno. It is also being integrated into KServe for supporting confidential AI use cases.

“The Confidential Containers community has successfully transformed TEE-backed security from a niche research interest into a robust, production-ready foundation for sensitive Kubernetes workloads. By joining CNCF, we secure the neutral, open governance required to set the industry standard for privacy-preserving infrastructure.” — Ariel Adam, Maintainer, Confidential Containers

“Our community has effectively abstracted the complexity of heterogeneous hardware-level isolation into standard, accessible Kubernetes patterns, ensuring security scales with the stack. Confidential Container’s transition to incubation allows us to align ‘data-in-use’ protection with the broader cloud native ecosystem, similar to how we’ve standardized critical infrastructure before. We are now delivering the secure, native capability required to deploy use cases such as confidential AI inference at production scale.” — Mikko Ylinen, Maintainer, Confidential Containers

The CNCF TOC provides technical leadership to the cloud native community. It defines and maintains the foundation’s technical vision, approves new projects, and stewards them across maturity levels. The TOC also aligns projects within the overall ecosystem, sets cross-cutting standards and best practices, and works with end users to ensure long-term sustainability. As part of its charter, the TOC evaluates and supports projects as they meet the requirements for incubation and continue progressing toward graduation.

“Confidential Containers has demonstrated clear maturity in integrating TEE-backed isolation within standard Kubernetes workflows, earning its place as an incubating project. This transition validates the community’s effort to standardize data-in-use protection, ensuring that security is baked into the infrastructure rather than bolted on as an afterthought.” — Faseela K, TOC Sponsor

“Achieving incubation is the direct result of the project’s robust development history, evidenced by its successful integration across heterogeneous TEE environments. By abstracting complex hardware security into standard Kubernetes patterns, the project allows developers to treat confidential computing as a native capability of their application stack. We’ve done this before with critical infrastructure standards, and this approach will now serve as the secure backbone for the next era of high-intensity, distributed AI inference.” — Kevin Wang, TOC Sponsor

Confidential Containers Main Components

Confidential Containers is composed of several components:

• Confidential Containers Pods: Enables deployment of unmodified containers across TEEs by abstracting complex hardware security into standard Kubernetes Pod resources.
• Trustee: A specialized, scalable attestation service that verifies the integrity of TEE-enabled nodes and workloads before granting secrets or keys.
• Helm Charts & Controllers: Provides lifecycle management, automating the provisioning and orchestration of confidential workloads across heterogeneous TEE environments.
• Hardware Abstraction Layer: Seamlessly integrates with multiple TEE technologies, including Intel® TDX and AMD SEV-SNP, allowing users to run sensitive applications without being locked into a single cloud provider’s hardware.

Confidential Containers Roadmap

The Confidential Containers project roadmap distinguishes between a short-term focus (2-6 months) and a mid-to-long-term outlook (6-18 months).

The short-term roadmap aims to deliver an end-to-end, stable, and easy-to-deploy solution, with development tracked across two GitHub projects: the release project for version-specific development and the Trustee project for ongoing implementation requirements.

In contrast, the mid- and long-term roadmap prioritizes use-case-driven development. Rather than focusing solely on specific features, this approach emphasizes delivering the real-world use cases that customers require, supported by robust CI/CD processes to ensure consistent end-to-end functionality.

To view the full project roadmap, visit: https://github.com/confidential-containers/confidential-containers/blob/main/roadmap.md

As a CNCF-hosted project, Confidential Containers is part of a neutral foundation aligned with its technical interests, as well as the larger Linux Foundation, which provides governance, marketing support, and community outreach. Confidential Containers joins incubating technologies: Backstage, Buildpacks, Chaos Mesh, Container Network Interface (CNI), Contour, Cortex, CubeFS, Emissary-Ingress, gRPC, HAMi, in-toto, Keptn, Keycloak, KubeEdge, Kubeflow, KubeVela, KubeVirt, Litmus, Longhorn, NATS, Notary, OpenFeature, OpenKruise, OpenMetrics, Operator Framework, Thanos, and Volcano. For more information on maturity requirements for each level, please visit the CNCF Graduation Criteria.

To learn more about Confidential Containers, visit confidentialcontainers.org, explore the GitHub repository, or join the community on the #confidential-containers Slack channel.

Building a Custom Metrics Exporter for Kubernetes

Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap.

This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it.

What a metrics exporter actually does

An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules.

In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code.

The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes.

Choosing what to measure

Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types:

• Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down.
• Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges.
• Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages.

Once you know which type fits your signal, choose a name that follows the convention <namespace>_<name>_<unit> in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later.

Setting up the project

The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency:

mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

Registering metrics

Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded:

package main

import (
    "log"
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    jobsProcessed = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "worker_jobs_processed_total",
            Help: "Total number of jobs processed, partitioned by status.",
        },
        []string{"status"},
    )

    queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "worker_queue_depth",
        Help: "Current number of jobs waiting in the queue.",
    })

    jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name:    "worker_job_duration_seconds",
        Help:    "Time spent processing a single job.",
        Buckets: prometheus.DefBuckets,
    })
)

func init() {
    prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself.

Collecting real values

With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker:

import (
    "math/rand"
    "time"
)

func collectMetrics() {
    for {
        // Replace these with real reads from your application.
        depth := float64(rand.Intn(50))
        queueDepth.Set(depth)

        start := time.Now()
        time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
        jobDuration.Observe(time.Since(start).Seconds())
        jobsProcessed.WithLabelValues("success").Inc()

        time.Sleep(5 * time.Second)
    }
}

The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom.

Exposing the endpoint

Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route:

func main() {
    go collectMetrics()

    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    log.Println("Listening on :8080")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("server error: %v", err)
    }
}

Verify the output locally before building the image:

go run .
curl http://localhost:8080/metrics | grep worker_

You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter is working correctly and is ready to be containerized.

Build a container image

A multi-stage build keeps the final image small and avoids shipping a Go toolchain to production. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base. The example below uses Docker, but the same pattern works with any OCI-compatible build tool such as Buildah or Podman:

FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

distroless/static:nonroot contains no shell, no package manager, and runs as a non-root user by default, which satisfies most cluster security policies without extra configuration.

Deploying to the cluster

Two manifests are enough to run the exporter: a Deployment that manages the pod lifecycle, and a Service that gives Prometheus a stable address to scrape. The Deployment sets conservative resource limits appropriate for a lightweight sidecar-style process, and uses the /healthz route for its liveness probe:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-exporter
    spec:
      containers:
      - name: exporter
        image: <registry>/my-exporter:v1.0.0
        ports:
        - name: metrics
          containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            cpu: 50m
            memory: 32Mi
          limits:
            cpu: 100m
            memory: 64Mi

The Service names the port metrics, which the ServiceMonitor will reference:

apiVersion: v1
kind: Service
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: my-exporter
spec:
  selector:
    app.kubernetes.io/name: my-exporter
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics

Telling Prometheus where to look

If you installed Prometheus using the Prometheus Operator or the kube-prometheus-stack Helm chart, use a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

Verifying the scrape

Port-forward to the Prometheus service and open the targets page to confirm the exporter has been discovered:

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Navigate to http://localhost:9090/targets. The my-exporter target should appear with state UP. Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:

rate(worker_jobs_processed_total{status="success"}[2m])

What comes next

A working exporter is the foundation, not the destination. The natural next step is surfacing these metrics to the HorizontalPodAutoscaler so that your workload scales on the signals that actually drive load, not just CPU. That requires a metrics adapter — the Prometheus Adapter is the most widely deployed option — which registers your custom metrics with the Kubernetes Custom Metrics API.

Introduction

Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a multi-cluster MongoDB deployment on Kubernetes that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG.

If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on.

Single Point of Failure: The risk of a single-cluster setup.

Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path.

Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios:

• Disaster Recovery (DR): If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically.
• Live Migrations: Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. Note that coordinating a clean cutover still requires careful application-level planning around connection strings and write consistency.
• Maintenance without Stopping: You can drain and upgrade one cluster entirely while the database continues accepting writes through the nodes in the remaining cluster, with no scheduled downtime.

Architecture Overview

To manage the complexity of multi-cluster deployments, the architecture divides clusters into two roles, ensuring that Kubernetes-level operations (handled by the Operator) and database-level operations (handled by MongoDB) do not interfere with each other. All nodes, regardless of which cluster they run in, belong to a single MongoDB replica set, which is what enables cross-cluster voting and leader elections.

1. Defining Site Roles

Each cluster is assigned to one role to prevent conflicts between independent Kubernetes control planes:

• Main Site: The primary cluster, fully managed by the Operator. It holds the Primary node and handles application writes. The Operator here is responsible for provisioning TLS certificates, user credentials, and replica set configuration.
• Replica Site: The Operator runs with unmanaged mode to true, which means it does not generate certificates or user credentials, and does not attempt to initialize a new replica set. Instead, the user needs to copy TLS secrets and credentials from the Main Site, allowing the Replica Site nodes to authenticate and join the existing replica set. This prevents the problem where two independent operators attempt to control the same database simultaneously.

2. Connecting Clusters with the MCS API

Even though each Kubernetes cluster runs an independent control plane, the MongoDB replica set spans across all of them. The Kubernetes Multi-Cluster Services API (MCS API) makes this possible. When multiCluster.enabled: true is set, the Operator creates ServiceExport and ServiceImport resources so nodes in different clusters can discover and communicate with each other via a shared DNS zone (svc.clusterset.local).

Note that the MCS API is not included in a standard Kubernetes installation and requires a separate implementation such as Submariner, Cilium ClusterMesh, or a managed provider solution like GKE’s native MCS.

To enable multi-cluster service discovery with the Percona Operator for MongoDB, set multiCluster.enabled: true in your cr.yaml:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: main-cluster
spec:
  crVersion: 1.23.0
  image: perconalab/percona-server-mongodb-operator:main-mongod8.0
  imagePullPolicy: Always
  updateStrategy: SmartUpdate
    multiCluster:
      enabled: true  # <---  Enable multi-cluster service discovery
      DNSSuffix: svc.clusterset.local  # <--- The Global DNS standard

After applying the configuration, verify that the Operator created the ServiceExport resources. Note that it takes approximately five minutes for resources to sync across the fleet. This is how it looks:

kubectl get serviceexport
NAME                 AGE
main-cluster-rs0    6m
main-cluster-rs0-0  6m
main-cluster-rs0-1  5m

The Main Site (Cluster A) runs the Primary node and is fully managed by the Operator. The Replica Site (Cluster B) holds Secondary nodes in unmanaged mode. ServiceExport and ServiceImport resources bridge the two clusters via a shared svc.clusterset.local DNS zone, enabling cross-cluster replica set membership.

How to Design for High Availability

Before choosing how to distribute clusters, it helps to understand how MongoDB itself decides which node is in charge.

A MongoDB replica set requires a strict majority of its voting members to elect a Primary and accept writes. With four voting members spread equally across two locations, neither side can reach a majority if the network between them is severed, and both sides stop accepting writes, even if every server is healthy.

The solution is a fifth member, placed in a third location. With 5 total votes, one side reaches 3 votes; a majority, and elects a new Primary. This is the 2+2+1 pattern.

Have an odd number of MongoDB voting members, placed in different locations, so that if one location or network link fails, one side can still have enough votes to keep/elect a Primary.

With the Percona Operator for MongoDB, the 2+2+1 pattern is configured directly in cr.yaml. Start by defining the total number of data-bearing replica set members across both sites:

replsets:
 - name: rs0
   size: 2       # 2 nodes in Main Site + 2 nodes in Replica Site
   expose:
     enabled: true
     type: ClusterIP

Then add the extra member as a separate entry under the same externalNodes section:

externalNodes:
    - host: third-site-rs0-4.psmdb.svc.clusterset.local
      votes: 1
      priority: 1

Deployment Workflow

The following steps deploy the 2+2+1 pattern across two Kubernetes clusters using the Main and Replica site roles described above.

1. Connect the Networks: Set up cross-cluster network connectivity using Cilium ClusterMesh, Submariner, or your cloud provider’s native solution (such as GKE Fleet). This is a prerequisite; the MCS API requires an implementation to be installed before the Operator can create functional ServiceExport and ServiceImport resources.
2. Deploy the Main Site: Deploy the Operator and apply your cr.yaml on the primary cluster. Ensure replica set pods are exposed with type: ClusterIP, this is required by the MCS API since nodes communicate internally across clusters.
3. Copy the Secrets: The Main and Replica sites must share the same TLS certificates and user credentials to communicate. Run kubectl get secrets on the Main Site to confirm the exact names in your deployment.
4. Deploy the Replica Sites: Start the remote clusters, but remember to set their operators to “unmanaged” mode. This tells the nodes to join the existing database instead of trying to create a new one.

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: replica-cluster
spec:
  # ----------------------------------------------------
  # This tells the Operator NOT to start a new database, 
  # but to join the existing one to prevent split-brain.
  # ----------------------------------------------------
  unmanaged: true
  multiCluster:
    enabled: true
    DNSSuffix: svc.clusterset.local

1. Finalize the Replica Set: Update cr-main.yaml on the Main Site to add the Replica Site nodes as externalNodes. Two nodes are added as voting members with lower priority, and the third as a non-voting member, this prevents split-brain if the Replica Site loses connectivity.

replsets:
 - name: rs0
   size: 2
   externalNodes:
   - host: replica-cluster-rs0-0.psmdb.svc.clusterset.local
     votes: 1
     priority: 1
   - host: replica-cluster-rs0-1.psmdb.svc.clusterset.local
     votes: 0
     priority: 0

Once these steps are complete, the replica set spans both clusters and is ready for the failover behavior described in the next section.

Failover Behavior: The Election Process

MongoDB’s replica set protocol includes built-in failure detection and automatic leader election. If the Primary node goes offline, whether a single pod crashes or an entire region is down, the remaining nodes automatically elect a new Primary without any operator or human intervention.

Let’s look back at our 2+2+1 setup. To stay online, the database needs a strict majority (3 out of 5 votes). If Cluster A completely fails, the two nodes in Cluster B plus the node at Cluster C give us exactly the 3 votes we need to choose a new Primary.

Under the default configuration, the median time before a cluster elects a new primary does not typically exceed 12 seconds; this includes the time to detect the failure and complete the election. In a multi-region setup, cross-cluster network latency may extend this window, so your application connection logic should include retryable writes to handle the brief period during which no Primary is available. The electionTimeoutMillis setting can be tuned if your latency requirements demand faster detection.

Architecture After Failover

After a regional failure, your database is still working, but it enters a “degraded” state. This means it is functioning, but with fewer resources than normal.

• The New Primary: A node in Cluster B is promoted to be the new leader and starts accepting new data.
• Less Redundancy: Your database is now running with the exact minimum number of nodes needed for a majority. In our 2+2+1 setup, you are surviving on just those 3 remaining members.
• Reduced Fault Tolerance: Even though your application survived the crash of Cluster A, your safety margin is completely gone. If you lose just one more node in Cluster B or the node at Cluster C, the database will lose its majority and lock into “read-only” mode.

The replica set detected the failure, held an election, and promoted a new Primary, automatically, without human intervention. That is the 2+2+1 pattern working as designed.

How MVCC and Transactions Work in RocksDB

RocksDB is built on an LSM-tree, which never modifies data in-place - every write creates a new version of the key. That's half of what you need to implement Multi-Version Concurrency Control (MVCC).

Real-world databases serve many clients reading and writing the same data at the same time. The simplest way to make concurrent access safe is locking (think of a hash map protected with a mutex). This works from the correctness perspective, but makes readers and writers block each other, which can quickly become a performance bottleneck. MVCC solves this problem, allowing readers and writers to proceed concurrently. The idea is that:

1. Writers never modify or delete keys in-place, instead they create new versions of the keys
2. Readers see a consistent view of the data as it existed when the read started - they access older versions of the keys, while writers keep creating new versions independently

MVCC is used in many DBs including PostgreSQL, CockroachDB, FoundationDB, MongoDB WiredTiger, and LMDB. Today we'll see how RocksDB implements the other half of MVCC - snapshots that give readers that consistent view, atomic updates and transactions built on top.

Versioning Keys

All inserts, updates and deletes go through a write buffer (memtable) and then get flushed to disk into an immutable SST (Static Sorted Table) file - you naturally have multiple versions of the same key. Compaction is the garbage collection process that merges SSTs and removes older versions of keys.

Let's pretend our database is an insert-only list ordered by key. If we add these key-value pairs dog,a, chipmunk,a, cat,a, raccoon,a, we'll end up with a list that looks like this.

It's a toy database, so point lookup queries run by scanning the entire list. Now a query starts, trying to look up dog. While the query is running, a writer proceeds more quickly and overwrites dog with b before the reader reaches it. The reader is still at chipmunk, but the writer already finished inserting dog,b.

Should the reader see dog,b since it's the latest value inserted, or see dog,a because that was the value when the reader started the query? "Readers see a consistent view" clearly answers this - the reader should see dog,a.

Earlier I said the LSM-tree already does half of what's needed to implement MVCC, half because having immutable memtables and SSTs alone is not enough. You also need a way to provide readers a snapshot of the database, and the GC process needs some way of knowing which tables are being accessed by in-flight queries.

Internally, RocksDB assigns each inserted key a monotonically increasing number, which it calls a sequence number. When you insert a key into RocksDB, what actually gets into the memtable and SST files on disk is a triplet of (user_key, sequence_number, value_type), stored together with the value. The user key is the key you provided, the sequence number is a counter incremented on every write, and finally the value type is whether it's an insert or a delete (or other operations, which are irrelevant for this blog post).

RocksDB tracks two sequence numbers - the last allocated sequence (the last number assigned on write), and the published sequence (the number visible to readers, advanced after a write completes).

In our example, the last allocated sequence number is 5 - the number that was assigned to dog,b. The published sequence when the reader R started was 4, and it will be 5 after the write completes.

When a read query starts, RocksDB captures the published sequence number. This sequence is then used for querying the memtables and the SSTs on disk. The reader R, when it sees dog,b with the sequence number 5, will skip over it, as it is allowed to see only keys with a sequence number of 4 or below.

The memtable implementation in RocksDB is not that far from our toy example. As in the example we used, it's based on a linked list, specifically on a skip list. The data structure adds layers of links that allow fast search and insertion in sorted order.

Layer 0 contains every entry - like the linked list from the example before. Each next layer contains a subset of the previous layer's entries. The search starts at the sparsest layer, descending to the full linked list. This structure gives O(log n) average time complexity for search.

When comparing internal keys, RocksDB first orders by the user-provided key ascending and then by the sequence number descending, so that when two keys share the same user key, the one with the larger sequence number comes first. A read query that captured the published sequence number 4 translates into a skip list seek to dog 4. The seek starts with cat 3 at layer 1, skips to dog 5 on layer 1, descends to dog 5 on layer 0 and finally lands on dog 1. The search can stop there since in the internal key order dog 4 is larger than dog 5, but smaller than dog 1. In Rust this could be expressed with the Ord trait this way:

impl Ord for InternalKey {
    fn cmp(&self, other: &Self) -> Ordering {
        if self.user_key == other.user_key {
            // Reversed - the larger sequence number comes first.
            other.sequence_number.cmp(&self.sequence_number)
        } else {
            self.user_key.cmp(&other.user_key)
        }
    }
}

The skip list in RocksDB is concurrent, meaning it doesn't require locking, so reads and writes can truly happen concurrently. The implementation uses atomic operations on pointers. Garbage collection in concurrent algorithms in non-GC languages is a hard problem, which RocksDB solves elegantly by relying on an arena allocator - individual entries are never freed, deletes are inserts with a DELETE value type and the memory for the entire memtable is freed after the memtable is no longer in use.

Atomic Updates

The write batch API provides atomicity - keys written in the same batch become visible together:

let mut wb = WriteBatch::new();
wb.put("a", "1");
wb.put("b", "2");
wb.put("c", "3");
db.write(wb)?;

Atomicity becomes straightforward to implement with the key versioning and sequencing mechanism we discussed above. When the batch is applied to the database (the db.write call), sequence numbers for all keys are allocated atomically by incrementing the sequence number by 3 (the number of keys in the batch) and then publishing that number only after all three keys are inserted into the memtable. This ensures no reader can see a partial batch - they either see all three keys, or none of them.

Snapshots

Every get implicitly captures the latest published sequence number. The snapshot API pins it explicitly, so multiple reads can see the database at the same point in time:

db.put("a", "1")?;
let snapshot = db.snapshot();
db.put("a", "2")?;
snapshot.get("a")?; // Returns "1" - the value at the time the snapshot was taken.
db.get("a")?; // Returns "2" - the latest value.

A snapshot is just the published sequence number recorded when db.snapshot() is called - reads through the snapshot skip keys with higher sequence numbers. RocksDB also registers the snapshot in a list of active snapshots. Compaction checks the list and drops an older version of a key only if no live snapshot can still see it.

Versioning Tables

Going back to GC, older key versions are removed by compactions, but how does the database know when an SST file or a memtable is safe to delete? What if there is an in-flight reader still accessing an already flushed memtable, or reading an SST that compaction is supposed to remove?

RocksDB maintains a structure called SuperVersion, which is a reference-counted view of everything needed for reads:

struct SuperVersion {
    active_memtable: ReferenceCounter<Memtable>,          // memtable accepting writes
    immutable_memtables: Vec<ReferenceCounter<Memtable>>, // memtables ready to be flushed or just flushed
    ssts: ReferenceCounter<SstLevels>,                    // on-disk SSTs, each reference-counted individually
}

When a read starts, it acquires a reference to the structure. The reference is released only after the read is done. When the number of references to SuperVersion drops to zero, it releases its child fields, decrementing their reference counts. This way neither the structure itself nor any of its children can be freed mid-read. The cleanup - freeing the memtable memory or scheduling files for deletion is performed only when a child's own reference count reaches zero.

The Wikipedia definition of MVCC says it's a "non-locking concurrency control method". In the database literature there is a distinction between locks and latches. Locks are logical - over keys, key ranges, or rows. When a lock is held on a key, nobody else can modify it until the lock is released. Latches are physical, e.g. mutexes that protect the database structures from data races during concurrent access.

By this definition, the core of RocksDB doesn't take logical key locks, and the read and write paths are designed to not hold latches in the common hot cases. Some less frequently exercised code paths, however, do require latches - e.g. there is a coarse-grained per-DB mutex held briefly after a flush or compaction to replace the current SuperVersion.

Transactions

Let's see how we can implement a classic balance transfer example with RocksDB, where Alice sends $10 to Bob:

// For simplicity, balances are stored as strings parsed to integers, e.g. "30" for $30.
let alice_balance = parse_balance(db.get("alice.balance")?); // $30
if alice_balance < 10 {
    return Err("not enough money".into());
}

let bob_balance = parse_balance(db.get("bob.balance")?); // $0

// Another writer sets Alice's balance to $40

db.put("alice.balance", (alice_balance - 10).to_string())?; // Set to $20
db.put("bob.balance", (bob_balance + 10).to_string())?; // Set to $10

Never do anything like this! The code snippet has major correctness issues when there is more than a single writer. Alice's balance should have been $30 (40 - 10), but ended up being set to $20 (30 - 10). $10 is lost because another concurrent writer modified Alice's balance between get and put calls.

Transactions exist to solve exactly this kind of problem by providing a way to execute a sequence of operations in isolation, detecting conflicts between concurrent writers. RocksDB builds a transactional layer on top of versioning. Writes are staged in the transaction buffer and get applied to the database only after the transaction commits. Reading from the uncommitted transaction merges the transaction buffer with the rest of the tree. This way transactions can read their own writes.

Transactions in RocksDB come in optimistic and pessimistic flavors. The pessimistic flavor checks for conflicts between transactions when updating a key - calling put or delete. In the optimistic flavor, conflict checks are moved to commit time. The API for both flavors looks the same - the flavor is picked when opening the database as either TransactionDB or OptimisticTransactionDB:

let t1 = db.transaction();
t1.put("a", "1")?;
t1.get("a")?; // Returns "1" - the transaction reads its own uncommitted write.
t1.put("b", "1")?;
t1.put("c", "1")?;
t1.commit()?; // Updates to keys "a", "b" and "c" are applied to the memtable.

Pessimistic Transactions

When using pessimistic transactions, keys involved in write operations are locked to perform conflict detection - the lock manager puts each updated key in a per-DB map, associating it with the transaction ID.

key_locks = {
    "a": T1,
    "b": T1,
    "c": T1,
}

If another transaction tries to update one of the locked keys concurrently, the conflicting transaction T2 blocks until T1 commits or rolls back - if that doesn't happen before the lock timeout, the operation returns an error:

let t2 = db.transaction();
// T1 has not committed yet.
t2.put("c", "2")?; // T2 is blocked waiting until it times out, or until T1 commits.
t2.put("d", "2")?;
// T1 committed.
t2.commit()?;

Locks are released when a transaction commits or rolls back. A third transaction, running at the same time as T1, but modifying a different set of keys, could proceed without being blocked.

By default, the conflict checks are done right when the operation is performed, which doesn't guarantee that no one else has updated the key since the transaction started:

let t1 = db.transaction();
{
    let t2 = db.transaction();
    t2.put("a", "2")?;
    t2.commit()?;
}
t1.put("a", "1")?; // Succeeds since T2 committed before this call.
t1.commit()?;

Setting a snapshot at the beginning of the transaction makes conflict detection catch updates made since the transaction started:

let mut opts = TransactionOptions::default();
// Setting a snapshot is the only difference compared to the previous example.
opts.set_snapshot(true);
let t1 = db.transaction_opt(&WriteOptions::default(), &opts);
{
    let t2 = db.transaction();
    t2.put("a", "2")?;
    t2.commit()?;
}
t1.put("a", "1")?; // Fails because T2 modified key "a" after the snapshot was set for T1.
t1.commit()?;

After T1 acquires a lock on key a, it checks whether the key was updated after the snapshot was set - whether there is a version of the key in the database with a sequence number higher than the snapshot sequence number. If there is, the write returns an error.

Optimistic Transactions

Optimistic transactions don't check for conflicts on write operations, so put or delete never block or fail. Conflict checking happens on commits.

At a high level, commit-time validation is similar to how it's done in pessimistic transactions, but the implementation is more lightweight. Instead of tracking individual locked keys in a shared map, the lock manager hashes keys into 1M pre-allocated mutexes. Each operation on a key records the published sequence number at the time of the call.

While the mutexes are held for all keys involved in the commit, the transaction checks the memtable. If it contains a key with a sequence number larger than the recorded one, it means another transaction already committed the same key - the commit fails. The mutexes are released after all keys are validated and all updates are applied to the memtable.

Optimistic transactions limit validation to memtables (pessimistic transactions check the whole tree). If the memtable is flushed mid-transaction, there is nothing left to validate against, so the commit fails and the transaction has to be retried.

Mapping to Isolation Levels

Isolation levels in databases define semantics of concurrent access - what happens when two concurrent transactions access the same data. I'll cover the most common levels used in modern databases. In SQL databases you can set the isolation level explicitly per transaction. There is no similar high-level API in RocksDB, but you can build different levels yourself with lower-level APIs.

Read uncommitted - the lowest isolation level, where the only guarantee is that concurrent transactions won't physically corrupt the database. This permits dirty reads - a transaction seeing uncommitted data from another transaction. Impossible in RocksDB because transaction writes sit in a buffer that gets applied to the database atomically on commit.

Read committed - a transaction sees only committed data, including data committed by other transactions after it started. You get this with a plain db.get, or when transactions don't use snapshots for reads. Two consecutive gets can see different values mid-transaction:

db.put("a", "1")?;
let t1 = db.transaction();
t1.get("a")?; // Returns "1"
{
    let t2 = db.transaction();
    t2.put("a", "2")?;
    t2.commit()?;
}
t1.get("a")?; // Returns "2"

Snapshot isolation - all reads in the transaction see a consistent snapshot of the database, and the transaction succeeds only if no updates it made conflict with updates made by other transactions since the snapshot was created. Can be achieved by enabling snapshots for both reads and write conflict detection - set_snapshot covers only conflict detection, and reads have to go through the transaction's snapshot(). You avoid non-repeatable reads, but the write skew anomaly is possible.

The snippet below illustrates the anomaly that happens when two transactions read the same data, then each updates a different key based on what it read. Alice has two bank accounts, $100 in each. The bank allows a single account to go negative, as long as the total across both accounts stays non-negative. Two concurrent transactions each withdraw $200 from a different account:

db.put("alice.checking", "100")?;
db.put("alice.savings", "100")?;

let mut opts = TransactionOptions::default();
opts.set_snapshot(true);

// T1 withdraws $200 from checking. T2 concurrently runs the same code,
// withdrawing $200 from savings.
let t1 = db.transaction_opt(&WriteOptions::default(), &opts);
let t2 = db.transaction_opt(&WriteOptions::default(), &opts);

let checking = parse_balance(t1.snapshot().get("alice.checking")?); // $100
let savings = parse_balance(t1.snapshot().get("alice.savings")?); // $100
if checking + savings - 200 >= 0 { // $100 + $100 - $200 = $0, the check passes.
    t1.put("alice.checking", (checking - 200).to_string())?; // Set to -$100.
}

// T2 passes the same check against its own snapshot and sets alice.savings to -$100.

// The transactions updated different keys, so conflict validation passes for both.
t1.commit()?;
t2.commit()?;

// Both accounts are now at -$100, the total is -$200.

Serializable - concurrent transactions run as if they were executed in serial order. You can get something close if you use get_for_update with snapshot isolation when transactions involve only point reads - you avoid dirty and non-repeatable reads and write skew. get_for_update is a special version of get used in transactions to avoid read-write conflicts e.g. when updating a key depends on the result of a read. get_for_update triggers conflict checks the same way as put and delete do. Here is the write skew example fixed by reading the balances with get_for_update, the rest of the code stays the same:

// Unlike snapshot reads, get_for_update reads participate in conflict detection.
let checking = parse_balance(t1.get_for_update("alice.checking", true)?); // $100
let savings = parse_balance(t1.get_for_update("alice.savings", true)?); // $100

// With pessimistic transactions, get_for_update blocks until it times out, then
// fails - the key is locked by T1. With optimistic transactions, the call
// succeeds and T2 aborts at commit.
t2.get_for_update("alice.checking", true)?;

The phantom reads anomaly is possible for transactions running range scans:

// Alice has a single account alice.checking.
db.put("alice.checking", "100")?;

let mut opts = TransactionOptions::default();
opts.set_snapshot(true);
let t1 = db.transaction_opt(&WriteOptions::default(), &opts);

// T1 computes the total across Alice's accounts, locking every key the scan returns.
let mut total = 0;
for item in t1.snapshot().iterator(IteratorMode::From(b"alice.", Direction::Forward)) {
    let (key, _) = item?;
    total += parse_balance(t1.get_for_update(key, true)?);
}
// total = $100.

// T2 opens a new account with a $50 deposit.
// The put doesn't conflict with T1 - T1 never read or locked alice.savings.
let t2 = db.transaction();
t2.put("alice.savings", "50")?;
t2.commit()?; // Succeeds.

t1.commit()?; // Succeeds.

T1 commits believing the total is $100, while Alice's accounts now hold $150. The new account is a phantom - get_for_update locked the keys the scan returned, but alice.savings didn't exist yet, so there was nothing to lock.

RocksDB has a range lock manager that locks key ranges instead of individual keys to prevent phantom reads. It was contributed in 2020, but it's not documented anywhere on the RocksDB wiki and not exposed through the C or Java APIs, so the feature is usable only from C++.

Fixing the Transfer

With all we learned we can finally fix the $10 transfer bug between Alice and Bob. Transactions (either pessimistic or optimistic) with a snapshot set, using get_for_update instead of plain get, are what we need - another transaction can't change a balance between the read and the write without triggering a conflict:

let mut opts = TransactionOptions::default();
opts.set_snapshot(true);
let tx = db.transaction_opt(&WriteOptions::default(), &opts);

let Ok(alice_balance) = tx.get_for_update("alice.balance", true) else {
    // Can happen with pessimistic transactions.
    return Err("another transaction updated Alice's balance".into());
};
let alice_balance = parse_balance(alice_balance);
if alice_balance < 10 {
    return Err("not enough money".into());
};

let Ok(bob_balance) = tx.get_for_update("bob.balance", true) else {
    // Can happen with pessimistic transactions.
    return Err("another transaction updated Bob's balance".into());
};
let bob_balance = parse_balance(bob_balance);

tx.put("alice.balance", (alice_balance - 10).to_string())?;
tx.put("bob.balance", (bob_balance + 10).to_string())?;

if tx.commit().is_err() {
    // Can happen with optimistic transactions.
    return Err("another transaction updated Bob's or Alice's balance or the memtable was flushed".into());
}

Optimistic vs Pessimistic Performance

From the user perspective, the only difference between pessimistic and optimistic transactions is where a transaction can abort on a conflict. Pessimistic transactions acquire locks on every write operation and abort early on conflicts (after exceeding the lock timeout). Optimistic transactions require no per-key locks on writes - coarse-grained locks are taken for the duration of a commit. Optimistic transactions typically provide higher throughput on workloads where conflicts between transactions are rare and where retrying the entire transaction on a conflict is cheap.

I benchmarked two workloads - one where pessimistic transactions win and one where optimistic transactions win.

In the "mostly read" workload, every transaction reads a shared key with get_for_update and writes a unique key - roughly one transaction in 1024 replaces the shared key instead. Starting from 2 threads, the throughput of optimistic transactions stays flat, while pessimistic transactions queue on the shared key's lock.

In the "contended key" workload, every transaction reads the same key with get_for_update, does CPU-bound work and writes the key back. Pessimistic transactions wait on the lock without consuming CPU, so the work runs once per commit. Optimistic transactions discover the conflict only at commit, after the work is done, redoing the expensive work on retry.

Who Uses RocksDB-native Transactions

MyRocks - the MySQL storage engine built on RocksDB, and ArangoDB are the only large projects I'm aware of that rely on RocksDB transactions. Facebook's MyRocks repository was archived on GitHub on Mar 1, 2026, but the engine is still available as a part of MariaDB or Percona Server.

TiDB/TiKV and YugabyteDB, which run on RocksDB, don't use RocksDB-native transactions. CockroachDB's RocksDB-inspired key-value store Pebble doesn't support transactions at all. TiDB, YugabyteDB and CockroachDB implement their own distributed transactional layer above the key-value stores. The reason for this is that RocksDB-native transactions are node-local, i.e. the locks are held per key-value store instance running on a single node, and with a distributed database you have to coordinate transactions across multiple nodes.

Conclusion

While LSM-trees give half of MVCC for free, there is a lot of practical complexity in implementing the other half. Concurrent algorithms are hard to get right, and making them efficient requires low-level techniques like using atomic operations on pointers. If you are implementing a database from scratch (I hope this blog post convinces you not to do that!), you are on your own with these techniques - in Rust, code built on raw pointers gets no protection from the borrow checker, and even memory-safe concurrent code paths covered entirely by the borrow checker are not free from logical races. The hard part is the concurrency, not the MVCC itself. Snapshots, atomic batches and conflict detection in both transaction flavors all come down to the same primitive - comparing sequence numbers.

Full article content is not available for inline reading.

Read the original article →

Architecture as a code

Visualize, collaborate on, and evolve your software architecture with always up-to-date, live diagrams generated from your code.

docs - playground - demo

What is LikeC4? Why "like"?

LikeC4 is a modeling language for describing software architecture and tools to generate diagrams from the model.

LikeC4 is inspired by C4 Model and Structurizr DSL, but provides some flexibility. You customize or define your own notation, element types, and any number of nested levels in architecture model. Perfectly tailored to your needs.

What does LikeC4 look like?

LikeC4 source:

Run CLI to preview:

npx likec4 start

And result:

Template repository - likec4/template
Deployed - https://template.likec4.dev

Check Tutorial - for a quick overview of LikeC4.

Getting help

We are always happy to help you get started:

• Join Discord community – it is the easiest way to get help
• GitHub Discussions – ask anything about the project or give feedback

Contributors

Support development

LikeC4 is a MIT-licensed open source project with its ongoing development made possible entirely by your support. If you like the project, please consider contributing financially to help grow and improve it. You can support us via OpenCollective or GitHub Sponsors.

License

This project is released under the MIT License

Security engineers using Cloud SIEM spend their day-to-day investigating signals, tuning detection rules, managing suppressions, running historical jobs across interconnected workflows, and more. Agents are becoming a practical way to navigate that complexity, and we built a set of security tools for the Datadog MCP Server to support them.

Cloud SIEM is only one part of a broader cloud security ecosystem, so the Security MCP toolset has to grow across many teams and products. That growth comes with a constraint. The tools share a single context window, so every new tool consumes more of that window. Add enough tools and the agent can lose track of which one fits the task in front of it.

We built these tools under that constraint, and we’ll cover what we learned in this post. We’ll show how we scoped tools around what users already do and managed the shared context window with progressive disclosure. We’ll also cover how we tested non-deterministic agent behavior with a custom eval framework and governed a growing multi-team toolset.

Scope MCP tools around what users already do

Deciding what to build first is challenging for a product as broad as Cloud SIEM. Assumptions about what an agent should do can send you in the wrong direction and waste time and resources on developing tools that may prove unhelpful.

Before building any tools, we analyzed API call patterns and Real User Monitoring (RUM) data to find where users already spent time and where they hit friction. We also reviewed Bits Chat logs to see what people asked for directly.

A few patterns stood out. Detection rule authoring was among the most common. Of the sample pool we analyzed, 44% of messages involved authoring and editing detection rules, and users often asked the agent to assess a rule and even name it:

Can we read the security signal I have open? Tell me: What exactly fired and why (rule name, detection method, what was new/anomalous) with Who/what triggered it (user, account, IP, user agent, geo)?

Signal investigation was a distinct pattern. Users regularly pasted raw signal JSON into the chat to understand what a signal meant and what to do about it. That told us an agent needs direct access to a signal’s full contents, not just a summary, so we built a tool that returns a signal’s full payload.

Another pattern was bulk triage. Users wanted the agent to take triage actions for them, such as assigning a signal, closing it with a verdict like benign or malicious, and leaving a note for whoever looks next:

Assign the signal to me, and close it as a benign true positive.

They also wanted to do this at scale:

Mark all those signals as archived against these IPs.

The Cloud SIEM UI lets you manage up to 50 signals at a time, and we found users working through signals in consecutive batches of 50 to get past that ceiling. About 25% of customers triaged signals in bulk, and 14% pushed right up against the 50-signal cap.

One session showed how far past the cap the demand could go. Over a 9-minute window, a single customer fired seven bulk triage calls back to back. Each of these seven came back at exactly 52,366 bytes, roughly 50 signals each.

A response that size meant that the user was maxing out the 50-signal cap repeatedly, triaging more than 350 signals in one sitting. Users wanted to act on more than 50 signals at once, and an agent was a natural way to achieve that.

Keep the shared context window lean

Scoping the work still left the harder problem of managing shared context. The Security MCP toolset started with tools for querying security findings. As it grew to cover Cloud SIEM, Code Security, App and API Protection, and more, the shared context window grew with it. A larger context window raises the risk of confusing the agent. We didn’t want an agent investigating App and API Protection signals when a user asked about Cloud SIEM.

We built more than a dozen security tools in all, and two of them best illustrate the context problem. The first is a schema tool that teaches the agent how to author a detection rule, giving it the fields and grammar a valid rule needs. The second is a triage tool for acting on signals in bulk.

The principle behind both is the same. Give the agent only what a task needs, and keep everything else out of the context window. The schema tool applies it through progressive disclosure, and the triage tool resolves signal IDs from a query instead of carrying them in context.

Return only the schema a task needs

Detection rule creation made the problem concrete. The schema tool carries rule schemas for several products, including Cloud SIEM, Workload Protection, and App and API Protection, each with its own fields, options, and permission models. Serving everything at once fills the context window with detail most tasks never use.

We designed a schema tool that uses progressive disclosure. Instead of serving every product’s schema upfront, the tool returns what a task actually needs.

The tool breaks the schema into named sections, such as top-level fields, query syntax, options, and examples. It also accepts filters for rule type and detection method, and it can drop any sections that aren’t needed. Leave the filters out and the tool returns everything. Pass them and it returns only the matching slice.

Take a request to build a brute-force detection rule:

I want to create a Cloud SIEM detection rule for brute-force attempts. Trigger it when the same source IP makes an unusual number of failed login attempts to our login or auth API endpoints within 5 minutes. The traffic comes from our nginx API gateway logs in production.

The agent doesn’t pull the full catalog. It leaves the section list unset, so the tool returns every section, but the rule type and detection methods it specifies filter the contents down to what the task needs.

{
  "telemetry": {
    "intent": "Fetch detection rule schema/grammar before authoring a log_detection brute-force rule for nginx auth endpoints"
  },
  "rule_types": [
    "log_detection"
  ],
  "detection_methods": [
    "threshold",
    "anomaly_detection"
  ]
}

The savings depend on how narrowly the agent can scope its request. Each security product owns its own rule types, so filtering to a single rule type cut token usage by about 15% in our testing. Filtering further by detection method, as in the request above, brought the reduction to 41–47%.

Triage hundreds of signals from a single query

Bulk triage needed its own tool. Our first version accepted a list of signal IDs and called the batch triage endpoint in a loop until every signal was updated. The approach worked, but two problems surfaced as we tested it.

Signal IDs are long strings, and every ID counted toward the tool’s input tokens. Passing hundreds of them meant loading a large block of text into context on every call. Including each ID was also slow for the agent, which added latency on top of the token cost.

We redesigned the tool to accept a search query instead of a list of IDs. The tool runs the query, fetches the matching signal IDs itself, and triages them in batches until it finishes. The IDs never enter the agent’s context, so input token cost stays flat no matter how many signals the query matches. The tool now handles up to 500 signals in a single call.

Before the redesign, updating 50 signals took at least 2 minutes. Afterwards, updating hundreds of signals took a little over a minute.

Test non-deterministic agent behavior with a custom eval framework

Careful scoping improved tool selection, but it didn’t prove the tools worked. Agent behavior is non-deterministic, and the technology is young enough that no checklist can sign off a tool as production-ready. Traditional tests assume the same input produces the same output, and an agent doesn’t offer that guarantee. We needed a way to verify tools worked reliably and didn’t break existing ones.

We built a custom eval runner and integrated it with the AI Platform team’s continuous integration (CI) eval runner. When anyone modifies the toolset, the runner automatically runs evals across all security tools. Any regression in the baseline eval score greater than 5% is flagged. The runner also flags any tool that ships without eval coverage, which became one of our first governance mechanisms. Contributors can’t quietly add tools that go untested.

Make the evals meaningful and cost effective to run

An eval is only as useful as the behavior it measures. We check whether the agent called the tools it should, avoided the ones it shouldn’t, used a sensible number of tools, passed correct arguments, and followed a reasonable trajectory to the answer. A weighted score rolls these checks up, and we give tool selection accuracy the most weight, because the agent picking the right tool from a large toolset is our top priority.

We also run each scenario several times and average the results, which reduces the noise that non-deterministic behavior introduces. Each eval generates an experiment in Agent Observability that shows where a failure occurred, so a drop in the score comes with the detail needed to fix it. We targeted a baseline success rate of about 80%, which leaves room for the occasional hallucination without masking real regressions.

Running scenarios repeatedly gets expensive, so the runner mocks tool responses. Instead of calling a real tool and pulling back a long response, the eval uses stubbed data. Mocking cuts output token cost, and the savings compound every time we rerun a scenario to average out noise.

Govern a growing multi-team toolset

Automated evals caught regressions in a single tool, but quality across the whole toolset became a coordination problem as more teams contributed. Without shared standards, a new tool could degrade the ones already serving customers.

We set up a lightweight governance committee designed around a self-service model. The reason is development velocity. Unlike deterministic features, MCP tools improve mainly through rapid iteration and heavy dogfooding against real use cases, so a heavy approval process would slow the exact feedback loop that makes them better. Every tool stays self-service through the stages before general availability (GA), so contributors ship and iterate without waiting on review. Contributors are responsible for meeting the established standards, such as a minimum eval score, before a tool reaches customers.

To make those standards clear and easy to understand, the committee maintains shared infrastructure and develops best practices that any team can reuse. The committee’s main concern is whether a new tool degrades the toolset already in production.

Key takeaways for building MCP tools in complex domains

Building MCP tools for Cloud SIEM showed us that real user behavior, not assumptions, should decide what you build first. Real agent behavior should decide what you refine next. Progressive disclosure keeps a shared context window manageable, custom evals keep non-deterministic tools reliable, and light governance keeps a multi-team toolset coherent without slowing anyone down.

Another lesson is that no checklist marks an MCP tool as done. Building production-quality tools in a complex domain is iterative, and the most reliable signal is watching how agents and users behave together in practice. To learn more about what workflows our MCP Server supports, see our Cloud SIEM documentation.

It's 2:47 AM. Your phone lights up. Three alerts fire simultaneously: elevated latency on your payment service, a spike in database query time, and a drop in checkout completion rate. You open your dashboards. All red. But which one is the cause and which ones are the symptoms?

If you've been there, this post is for you. Intelligent Workloads is our answer to the, "Where do I even start?" problem that plagues on-call engineers in complex, distributed systems. This walkthrough shows you—concretely—how to set one up, what you'll see, and how to use it to cut your time to causal analysis.

What Is an Intelligent Workload?

An Intelligent Workload is a persistent, monitorable entity that represents a complete business transaction. It groups every component involved in a critical flow (APM services, infrastructure hosts, databases, serverless functions, browser apps, message queues) into a single unified health view, and keeps that grouping accurate automatically as your architecture evolves.

The key word is transaction-centric. Most monitoring tools are service-centric: you watch Node A, Node B, Node C. Intelligent Workloads watches "the checkout flow," the end-to-end chain that spans Node A through Node N, and tells you which link in that chain is the problem.

Step 1: Find the Transaction That’s Powering Your Business

Before you create a workload, you need to identify a business transaction you want to focus on. Intelligent Workloads and Transaction 360 call this a“focal transaction,” the central transaction that serves as the heartbeat of your business process and will provide the scope for your workload.

In New Relic, navigate to your APM summary page to locate the Transactions table. When a critical path like a checkout endpoint begins to degrade, it instantly bubbles up in the table as a performance outlier. Selecting the “View details” button drops you directly into Transaction 360, an end-to-end, cross-stack view that automatically correlates all entities and telemetry related to that focal transaction.

For example, during a World Cup ticket sale, Transaction 360 provides instant forensic clarity when checkouts stall. To see if a recent deployment is related to the checkout lag, the overview section gathers the necessary context by pooling your system dependencies and live alerts. The Dynamic Flow Map renders the network path, color-coding the seat-reservation database that is choking under the traffic volume. The Participating transactions table uncovers performance shifts across the execution chain, isolating whether the lag stems from the gateway itself or a downstream payment service. Transaction 360 puts these metrics and more at your fingertips, giving you everything needed to troubleshoot one of your most critical business paths.

Step 2: Create Your First Workload (One Click)

In order to proactively monitor a critical transaction such as the check out service mentioned in step 1, you’ll want to create a workload.

1. From the Transaction 360 view, select the “Save as workload” button.
2. Type in the name of your new workload.
3. The new workload will use the transaction trace data to identify every service, database, and infrastructure components relevant to the focal transaction.
4. You have the option to filter out upstream or downstream dependencies to scope the workload strictly to the components your team owns and manages.
5. Review the membership. You'll see your APM services, the databases they query, the infrastructure hosts they run on, and any downstream services or queues.
6. Save the workload.

That's it. You now have a persistent workload that updates its membership dynamically, so for example when you deploy a new service into the transaction path, it appears. When you decommission one, it drops out. No manual tag updates required.

Step 3: Configure Workload Health

Here's where Intelligent Workloads starts saving you from false alarms. Not every entity in your workload should be able to trigger a health alert. If a non-critical async analytics tracker starts throwing errors, you don't want your checkout workload to go red and page your on-call team at 3 AM.

1. In the settings section of workloads, select “Configure health status”
2. You have three options for configuration:
   1. Static: used for a maintenance window.
   2. Automatic: A rules-based configuration to determine the status of each entity.
   3. Alert policy and conditions: Define custom alert rules to determine the workload health status.

The workload will still show all the entities involved in your business transaction and their health state and you can still see if they're degraded. They just won't count against the overall workload health unless you configure your health definition to do so. This is the difference between alert noise and actual signal.

Step 4: Add Business KPIs

Once you map a critical user journey, the new Business KPIs feature becomes the key to tying strategic business insights directly to your live observability data. This step transforms your Intelligent Workload from a technical triage tool into a business intelligence layer.

When a workload is created, two default KPIs are created as well:

1. Workload P99 Latency: The maximum time it took for the fastest 99% of executions of the focal transaction to complete.
2. Workload Success Rate: The percentage of successful executions of the focal translation.

Out of the box, your workload shows standard golden signals: throughput, error rate, latency. But the questions that really matter in an incident are typically different and more business oriented. How many users are experiencing this? Is revenue being affected? Is the cart abandonment rate spiking? You can add custom KPIs to help you answer these questions.

You can add existing KPIs from other workloads or create a custom KPI by following the steps below:

1. Open the summary page of the intelligent workload
2. Click on Add new KPI
3. Choose a NRQL-based metric, which can be any data in your New Relic account or a custom attribute. For example:
   • SELECT sum(cart_value) FROM Transaction WHERE appName = 'storefront' AND name = 'checkout_complete' → rolling sum of storefront revenue
   • SELECT uniqueCount(case_id) FROM CRMScraperSnapshot WHERE status = 'open' → current number of open support cases reported by an external system
4. Set thresholds for "degraded" and "critical" states.
5. Name it something a VP could understand: "Checkout Completion Rate," "Revenue per Minute," or "P95 Checkout Duration."

Note: The second example in step 3 (CRMScraperSnapshot) uses a custom event generated by an external scraping script to model a business process by an intelligent workload.

Now when something goes wrong, you can provide business-oriented answers like, "The checkout latency spike has caused our rolling storefront revenue to drop by $40k this hour, and open support cases in our external CRM have already jumped to 52.” That's the message that gets the right people focused on the right problem, fast.

Step 5: Use the AI Summary During an Incident

When your workload goes unhealthy, the AI Summary is your first stop — not your dashboards. The AI Summary automatically analyzes the current state of all entities in your workload and provides:

• Isolation of "Golden Signals" — surfaces the specific metrics that are anomalous across the workload, not just a flat list of everything that's red.
• A prioritized investigation path — "Start with the payment-service database queries because query time increased 340% in the last 15 minutes, which correlates with the latency spike in the checkout service."
• Health propagation — if the root issue is three layers deep (e.g., a slow database query is degrading a service, which in turn is degrading the checkout flow), the AI traces downs through that chain and shows you where to look.

This is the, "Where do I even start?" answer. Instead of opening twelve dashboards, you open one view and get a directed investigation path.

A Real Scenario: Tracing a Checkout Degradation

Here's how these pieces come together in a real incident.

T+0:00 — Your checkout workload health turns yellow, alerting your team of a subtle performance slip. You review your Business KPIs and notice early signs of trouble: rolling storefront revenue is starting to dip below its baseline, and open support cases have bumped up slightly.

T+0:02 — You select the AI Summary button and it surfaces a bottleneck hidden two hops downstream from your focal transaction. It highlights “a downstream dependency, payment-service, is experiencing severe latency driven by a 340% spike in query duration on payment-db.”

T+0:04 —The investigation steps of the AI Summary suggest: “1) Check Performance Risks for a recent deployment on payment-service to investigate the query duration spike.” You click Performance Risks directly in the workload sidebar menu and instantly confirm the culprit: a newly introduced N+1 query pattern.

T+0:07 — You identify the deployment that introduced the regression, escalate to the backend engineer responsible, and initiate a rollback.

T+0:22 — Checkout completion rate returns to 91% and the workload turns green. Without Intelligent Workloads, this investigation would have required navigating manually between APM, infrastructure, and database monitoring — and potentially 45–60 minutes of correlation work before pinpointing the cause.

What You Need to Get Started

A few requirements to be aware of before you dive in:

• Distributed tracing must be enabled for your services. Transaction 360 and Intelligent Workloads depend on trace data to map the dependency chain automatically.
• The more services in your transaction path that are instrumented, the more complete the map. Uninstrumented services create gaps in the dependency view.
• The AI Summary feature requires access to NRAI.

If you're not sure where your distributed tracing coverage stands, take a look at the Transaction 360 view for your focal transaction first, any gaps in instrumented entities will be shown at the top of the page.

Getting Started

1. Enable distributed tracing if you haven't already → Distributed Tracing docs
2. Create your first Intelligent Workload → Create an Intelligent Workload
3. Set up Business KPIs → Business KPIs docs
4. Explore the UI → Intelligent Workloads UI overview

Questions? Reach out in the New Relic Community Forum

Migrating application secrets to the GitOps paradigm is one of the hottest topics for teams adopting Argo CD. Until recently, the Argo CD documentation for secrets management was just a list of secret-related solutions without any clear guidance on which to pick.

With the release of Argo CD v3.x the project has finally taken a clear stance on secret management. Today, the documentation page clearly recommends external secret controllers over manifest generation plugins. At the time of writing, the following solutions are recommended:

• Sealed secrets
• External Secret Operator
• Kubernetes Secret Store CSI driver

We’ve already covered the first two solutions in our previous guides. Today, we’ll complete the trilogy by showing how you can secure your Argo CD secrets with the Secret Store CSI driver.

Using an external secret solution

Before we dive into the secret store driver, it’s important to understand the concepts behind GitOps Secrets. Even though in theory there are several different secret solutions, in reality most of them fall under two categories:

1. Secret solutions that are implemented by external controllers without any coupling to Argo CD
2. Secret solutions that are tied to Argo CD and the manifest generation process

The first category is what the new Argo CD documentation recommends. This is the case where two different tools handle secret management and application deployments with completely different lifecycles.

In this setup, Argo CD doesn’t even know what the real secret values are. It deploys the application with several secret references/pointers. An external secret operator is then responsible for converting secret references into real secrets. Each tool does what it does best and more importantly, any secret rotation process doesn’t need application re-deployments or sync operations in Argo CD.

On the other hand, we have secret solutions that are tied to Argo CD, either in the form of plugins or any other tools that modify manifests in place when a sync operation takes place. These are not recommended by the Argo CD team.

Having a secret solution tightly coupled with Argo CD forces Argo CD into the realm of secret management. Not only does it make Argo CD aware of the actual secret values, but any kind of rotation needs to pass through Argo CD and its synchronization process.

The secret store CSI driver is a solution that follows the first approach and thus enjoys the implicit approval from the Argo CD team.

Avoiding Kubernetes secrets by mounting files directly on pods

If you have read our previous guide or you already know how the external secret operator works, you might wonder what’s the advantage of using the Secret Store CSI driver. At first glance, both approaches work in a similar way

• Both solutions are decoupled from the Argo CD sync process
• Both fetch secrets from an external secret source and bring the secrets into the Kubernetes cluster
• Both use a secret pointer/reference CRD
• Both support mounting secrets as files on the workload pods
• Both can automatically refresh the files when a secret changes

The biggest difference between them is that the CSI driver does not use Kubernetes secrets.

The External Secret Operator always uses Kubernetes secrets, which are then mounted as files using the standard Kubernetes volume mechanism. The CSI driver completely bypasses this step and mounts the secrets directly as files without any Kubernetes secrets on the cluster.

Whether this feature is important to you depends on your security policy. Several companies have different security constraints and requirements that affect how their workloads run in a Kubernetes cluster. If part of the security policy is to never have Kubernetes secrets anywhere, then the CSI driver is a better option for you.

The end result in both cases is the same, the secrets reach your Argo CD application as files and are automatically rotated if the secret provider changes them. In all cases, we assume that your application is loading its secrets from files and not environment variables.

So if you already have a working solution with the External Secret Operator and your security team is happy with it, there’s no real technical reason to adopt the CSI driver.

The example secret application

You can find the source code and Kubernetes manifests for our application in the argocd-csi-secret-store-example repository on GitHub.

It’s a very simple application that reads a database connection credential from /secrets. There’s no real database of course. The application just prints its own secrets so that you can see the flow of information from the secret store to the Kubernetes pods

Even though this is a demo application, it has several important characteristics that help with secret management.

1. It loads secret information from files and not environment variables.
2. It shows where it loads secrets from. This makes debugging very easy.
3. It automatically reloads secrets if they change. We’ll see this later in the secret rotation scenario.

Installing HashiCorp Vault and the CSI secret store driver in your Kubernetes cluster

Let’s start by deploying our “infrastructure” applications first. You can install Vault using the public Helm chart.

argocd app create vault \
--project default \
--repo https://helm.releases.hashicorp.com \
--helm-chart vault \
--revision 0.28.0 \
--sync-policy auto \
--sync-option CreateNamespace=true \
--parameter server.dev.enabled=true \
--parameter injector.enabled=false \
--parameter csi.enabled=true \
--dest-namespace vault \
--dest-server https://kubernetes.default.svc

Just for this demo we install Vault with server.dev.enabled so that we don’t deal with sealing/unsealing. The default admin token is “root” and we’ll use it later to log in to the web UI. We also pass csi.enabled=true for CSI integration.

In a production environment, Vault should be handled by your security team and have proper credentials. We create applications using the Argo CD CLI for simplicity here. In a production setup, all applications should be stored in Git. See anti-pattern 2 in our comprehensive Argo CD guide.

The next step is to create the credentials that the application is using. In this contrived example, it’s database credentials for an imaginary MySQL instance.

You can do this from the vault CLI or from the Web interface:

This concludes the “demo” installation for HashiCorp Vault. As a reminder, Vault is just one of the many providers supported by the CSI driver.

Fetching external secrets from HashiCorp Vault

With the vault installation ready, we can now install the CSI driver. This step depends on how you create your Kubernetes cluster and on any Infrastructure-as-Code tool you already use.

See also the official installation instructions.

The last piece of the puzzle is to actually set up the integration between the CSI driver and vault. This happens via a dedicated SecretProviderClass. See the example in our application repository.

You can easily deploy it like any other Argo CD application:

argocd app create vault-secret-store \
--project default \
--repo https://github.com/kostis-codefresh/argocd-csi-secret-store-example.git \
--path "./manifests/vault-integration" \
--sync-policy auto \
--dest-namespace default \
--dest-server https://kubernetes.default.svc

Notice that, like the external secret operator example, Vault is set up to trust the Kubernetes cluster it is running on. There’s no other token or secret stored anywhere (either in Git or the application itself)

Everything is ready now regarding secret management. Vault is running, and the CSI driver connects to it to fetch secrets.

Passing secrets as normal files without Argo CD involvement

Finally, let’s deploy our application. It’s very similar to the one we used in the External Secrets Operator post. It’s a very simple Web application that prints a “secret”. Again, you can do this with Argo CD and verify that everything is synced and healthy:

Now if you visit the Web UI of the application (using port forwarding or any other networking method), you’ll see the full result. An application reading secrets from Vault without any hardcoded token anywhere.

This is how the whole process works:

1. Vault is running inside Kubernetes and also has a trust relationship with the same cluster.
2. The CSI driver is also active on the same cluster and can read secrets from Vault.
3. The secrets are mounted directly as files under /secrets.
4. The application is just reading files without any knowledge of how/where these secrets are stored.

The main advantage of this setup is security and simplicity:

• There are no Kubernetes secrets in the cluster
• There are no application tokens that are used for vault access
• The source code of the application just reads files at /secrets
• The communication between the vault and the CSI driver is transparent to the application

We’ve now explained the initial deployment of the application. But what happens when a secret needs to be rotated?

Refreshing secrets without restarts and any Argo CD sync operations

The Secret Store CSI driver has a similar capability to the external Secret Operator for automatically refreshing secrets without application restarts. This means that you can change a secret value, and if the application is correctly wired it can auto-reload the secret on its own.

We can test this scenario easily by changing the secret values in Vault (using the CLI or the Web interface).

After the secret changes in the vault, the Secret store driver detects it (the refresh period is configurable) and can automatically update the secret’s file contents in the mounted folder.

For the application, everything is transparent. The source code just sees a file change on the filesystem. Our example application is already configured correctly to automatically reload secrets on the fly.

Secret rotation works with zero effort and without any involvement from Argo CD. No sync operation is required, or killing pods manually. One of the biggest challenges in a big organization when it comes to secret rotation is not only understanding which applications use a specific secret, but also how exactly to make the applications “see” a secret change.

With the CSI Secret store driver, secret rotation is straightforward. There’s no need to hunt down individual applications for restarts or missing an application that was never part of the rotation process.

If your application can’t refresh secrets on its own, you can use any external controllers such as Reloader to achieve the same result. The CSI Secret store driver can be configured to use Kubernetes Secrets as a configuration option. Secret rotation can then follow the traditional process where you must restart applications in order for the new secret to take effect.

Conclusion

In this guide, we’ve explained how to use the CSI Secret driver for secret management of Argo CD applications. We’ve seen how you can rotate secret values without any sync operations or application restarts.

We’ve now covered the full trilogy of how to handle secret management with Argo CD applications. The updated Argo CD documentation now has a clear recommendation, and there’s a comprehensive guide for each one of them

• Secret Store CSI driver (this post)
• External Secret operator (previous post)
• Sealed Secrets controller (previous post)

If you need help in your Argo CD onboarding process or just need advice on how to improve your Argo CD promotion workflows, check out our Enterprise Argo support offering.

Happy deployments!

Your agent can now create Webflow components from a screenshot, update over 50 design tokens in one conversation, and query your site analytics in plain language. Here's what shipped in MCP 2.0.

Reworking design tokens while building out new components can be a solid day of work because there are so many things to consider. But let me tell you, my Cursor agent just did both in a couple of conversations.

Webflow has always been built for the people who design and build on the web. With the new MCP 2.0 release, your AI agent can finally step into your shoes. It can build components with props and variants, wire them to your CMS, and your Webflow site can now serve conventions that are tailored for your new AI-first workflows. Forms, assets, SEO schemas, custom code, even your traffic analytics are now accessible in your agent conversations.

Webflow MCP 2.0 is a pretty big release! Let me share my highlights with you in this post.

And if you'd rather watch than read, head over to YouTube.

But first: no more bridge app!

Quick note before we jump into my favorite new features.

If you used the MCP server earlier this year, you might remember that you had to keep a browser tab with the running bridge app open so your agent could reach and interact with your site. With the new MCP server release, that requirement is gone for most actions, and we're working on making every tool and workflow work without the bridge app.

Only when your agent needs access to the current canvas, or you task it with something like "update the page I'm on," will it let you know about the bridge app and hand you instructions on how to enable it.

With that out of the way, let's get to the good stuff!

Hand off your design work

To scale your Webflow sites you should invest time in a properly defined design system. The design system's foundation is based on CSS variables. And you might have guessed it, MCP 2.0 gives your agent full access to them. It can read your variable collections, create and reorder variables, and add new variable modes. That means an agent can reason about the actual tokens driving your site instead of guessing hardcoded values.

In my example, I set up a new Webflow site. Initially, I was quite happy with its look and feel but after sleeping on it, the colors felt out of place and simply "too much". Adjusting all the colors would have been quite a bit of work. I also wasn't sure what colors would be a better fit.

So, instead of editing tokens one at a time to find the right look, I handed the problem to the Cursor agent.

Please update my Webflow site. Initially I liked the bright and strong colors, but it's too much now. Please check the design system and propose different color palettes. Offer me a way to preview the alternatives.

With the established MCP connection, Cursor agent accessed my Webflow site, read the variable collections driving the design, and even built a quick prototype to preview calmer color palette alternatives. Instead of trying things out on my site, I could see these new colors in action.

After deciding on a new color palette, the agent got to work and updated over 50 design tokens in one go while I was watching it. It even set up a new variable mode for the different color palettes so that I could switch between them easily.

I must admit, I'm still blown away by the quick turnaround here.

Build a component from a screenshot

Tweaking existing components and styles is one thing. MCP 2.0 also gives your agent full control over Webflow components: it can create them, add props, variants, and slots, and set metadata, all built from the existing elements and styles in your site.

One workflow that I discovered way too late on my AI journey is that you can feed AI agents screenshots to provide additional context. In my case, I had a rough component sketch. I screenshotted it and handed it to Cursor.

Can you create a new "manifest" component for my Webflow site? It should look more or less like this. Make all fields configurable.

With all this context, Cursor can analyze the screenshot, inspect the site's existing structure, and build new components using the existing elements and styles. It can even screenshot its progress to verify that the result aligns with the provided screenshot.

It still feels a bit unreal that I could "build" this component without any manual effort.

Teach your agent your preferences with skills and rules

To get the most out of your agent conversations, context is everything. Ideally, you don't want to rely on your model's training data; you want to provide all the information at the right time. But how do you provide context when your site lives in Webflow? How can your markdown instructions enter all these agent conversations?

To share context your Webflow site now has a new "Instructions" section accessible from the sidebar.

We divide your AI context into rules and skills.

Rules are your always-on guardrails. They apply to every agent interaction, no matter the task. Things like "use our design tokens", "never hardcode colors", or "match our naming" belong into global rules.

Skills are on-demand playbooks. The agent loads one only when a task calls for it. A "create a component" skill can capture exactly how your team builds components: structure, props, variables, naming. We recommend creating custom skills for every custom workflow.

You can then author your guardrails and workflows right in Webflow. And to make the context native, you can even reference Webflow primitives like variables, components, and styles in your AI context. This way your agent has access to actual data instead of working off outdated information.

And do you remember the component I mentioned above? I have a small confession to make: it heavily relied on a pre-defined skill. The agent didn't need to improvise or figure things out. A create-component skill told it exactly how a site's component should be structured, so that it nailed building it on the first try.

With rules and skills, your agent stays in its lane without you course-correcting every single step.

But there's a lot more

These are my favorite new features, but MCP 2.0 has plenty more to offer:

• Read your forms, list submissions, and update or delete them.
• Build on page branches and enforce roles and permissions.
• Manage assets and folders, and compress images to webp or avif without leaving the conversation.
• Read and write head and footer code, at the page level and the site level.
• Access your site's history (on Enterprise plans) to understand who changed what, and which changes came from an agent.
• Ask about your traffic in plain language: top pages, channels and sources, engagement, and time on site.

Try it yourself

Webflow MCP 2.0 is live. Connect it to your agent, teach it your conventions with skills and rules, and let it work across your whole site.

The setup instructions are on developers.webflow.com. Go build something, and if there's something you can't do with your agent yet, let us know.

Full article content is not available for inline reading.

Read the original article →

AI can fake your portfolio. It can’t fake the second question.

The artifact just became the cheapest thing in hiring. The room test is the live follow-up you can’t generate, and it’s where interviews actually get decided in 2026.

Seven or eight years ago I made it to the final round at a large international company. Four interview stages, maybe five. Portfolio review, case studies, team conversations, the whole gauntlet. I passed every one. The last step was a conversation with the VP of design, and everyone around the process treated it as the easy one. The handshake round.

She asked me how design thinking had changed the way I work.

The label threw me. Design thinking was just becoming a term of its own back then, a thing with a capital D that people put on slides and certifications. I practiced it every day: framing the problem before touching solutions, testing assumptions on real users, iterating in the open. But where I worked, nobody called it design thinking. We called it solving the problem. So I asked her, honestly, what do you mean by design thinking? The room went silent.

She explained, briefly and politely, what she meant. And while she talked, I watched her make up her mind in real time. I could see the verdict assembling behind her eyes before I said another word. I spent the rest of the interview demonstrating that I did everything the term described, that I had been doing it for years without the label. It did not matter. Every answer after my question arrived at a door that had already closed.

For years I filed that story under bad luck. Wrong vocabulary, wrong year, wrong VP. It took sitting on the other side of the table, running these conversations weekly as a manager, to understand what actually happened. I did not fail a knowledge test. The work was real and the practice behind it was real. I failed something else: the live, unprepared moment where you translate your own judgment for a stranger, under pressure, with no artifact to hide behind.

I call that moment the room test. And I think it is about to become the entire interview.

The room I run now

Not long ago I sat on the other side of that silence. I have blurred the details; the two questions are not.

The strongest case study I had seen in months. A banking onboarding flow, which I can judge closely because I run one. Clean problem framing, research artifacts, a decision log, before and after numbers. The candidate presented it fluently. If the interview had ended at the presentation, we would have been talking about an offer.

Then a colleague asked the first question: you sequenced identity verification before showing the user any value, most teams do the opposite, what made you choose that order? A pause. The candidate walked us through the screen we were already looking at, again.

I asked the second question: what almost shipped instead? Nothing came back. Not a weak answer. No answer. There was no other version. There was no almost.

I want to be careful with this story, because it is not about catching anyone, and this article will not become one of those. I do not know how that case study was made, and I did not need to know. What the two questions established was narrower: the judgment we were hiring for was not in the room that day. Whether it existed somewhere else, in a different form, on a better afternoon, the room could not say. Rooms are not truth machines. They are pressure tests. Mine failed one years ago while carrying eight years of real practice.

But the two stories share one structure, mine and the candidate’s. The artifact survived and the conversation did not. And the reason this matters more now than it did in 2019 is simple: the artifact used to be expensive to make, it is not anymore.

The inversion

For twenty years the design career ran on a quiet division of labor. The artifact carried the proof: the portfolio, the case study, the deck. The interview existed to verify the artifact, to confirm that the person in front of you made the thing behind them. Expensive object, cheap conversation.

AI inverted that economics in roughly two years. A polished case study, complete with research narrative, decision log, and before and after metrics, can now be assembled in an afternoon by someone who was never near the project. So can the deck. So can the take home exercise, which is why hiring managers are quietly abandoning it. The visual signal that used to mean I can produce work at this level now means I have a subscription.

Pawel Klasa, a product designer writing in Bootcamp, made the case this spring for walking away from the portfolio entirely: “AI can fake a portfolio in an afternoon.” His answer is a public catalogue of continuous work, writing, talks, shipped experiments, a body of evidence too long and too messy to counterfeit. It is a good answer, and it is also a decade of homework. Most designers do not have a decade.

What everyone has, whether they want it or not, is the conversation. The artifact is now the cheapest thing in the hiring pipeline, and the conversation is the expensive one. The proof did not disappear, it moved.

One thing you can do with that this week: inventory your own proof. List everything you would show someone to establish your level, then mark each item as artifact or live. If everything on the list can be attached to an email, the list is weaker than it was two years ago, through no fault of yours.

Writers felt it first

Design is not the first craft to hit this wall. Writers got there a year earlier, because text was the first thing AI made free.

Matt Lillywhite, an essayist who publishes in The Daily Draft, wrote a piece in May about the folklore of spotting AI writing. He spends most of it dismantling the popular tells: the punctuation theories, the suspicion of similes, the idea that a qualifier is a confession. Then he lands somewhere more interesting: the giveaways worth taking seriously live outside the text entirely, in the author. Publish something you barely understand, and eventually a human being asks you to expand on a point, or challenge a statistic you never verified, and the gap between the writing and the writer opens up in front of a live audience, “while they gradually realize you don’t fully understand your own argument.”

Replace author with designer, article with case study, live event with portfolio review. Nothing else in the sentence needs to change.

Design hiring is already moving in this direction. Tom Scott, a recruiter who runs design and design leadership searches and writes the Verified Insider newsletter, described the shift in February: “Live, in-progress work can outperform a polished portfolio.” Interviews built around messy Figma files that are days old, concepts that have not been validated, decisions that have not been defended before. He reports seeing these formats far more often now, and his read on who struggles is precise: candidates whose articulation outruns their execution once the conversation leaves the script.

Now the honest caveat, before this argument gets too pleased with itself. The live conversation is not unfakeable either. Shraddha Sunil and Mudit Saraf, writing in Harvard Business Review in June after interviewing 120 talent acquisition leaders and analyzing more than 6,000 screening sessions, found that candidates can perform convincingly in remote interviews with real time AI assistance: “the ability to perform well in interviews is becoming infinitely scalable and practically free.” One asterisk you should hold while reading that: both authors cofounded an interview screening company, so treat the finding as directional evidence from people with a stake, not neutral research.

But notice what real time assistance can and cannot carry. It can carry the first question, the one every candidate saw coming. It struggles with the second one: the follow up that departs from every script, that asks for the almost, the tradeoff, the regret, the specific Tuesday when the decision got made. Recall can be delegated to a hidden window. The texture of having been there cannot. Which is why rooms are decided at the second question, and why the useful response to all of this is not paranoia about other people.

It is asking whether your own defense of your own work is script deep. Before your next interview, volunteer the messy file. Walk someone through work that is two weeks old and unresolved. If that idea frightens you, that fear is information.

What the room actually measures

“The hand can never consistently produce better than the eye can discern.” — Julie Zhuo

Seniority is two independent measurements, not one ladder. Level is the scope you own, granted by the organization. Stage is your maturity as a practitioner, accumulated rep by rep, and no reorg can hand it to you or take it away.

Level shows up in artifacts. Look at what someone owns and shipped, and you can read their scope off the deliverables. But artifacts are exactly what became generatable. Stage never lived in artifacts. Stage shows up in real time, under questioning, in the space between a hard question and your first honest sentence.

The room test is the Stage axis made visible. Specifically, a room reads you on four axes.

• Whether a hostile question lands as information or as threat. The mature practitioner hears an attack on the work as data about the work. The early one hears it as data about themselves, and you can watch the difference in the first three seconds.
• Whether your patterns transfer under pressure. Anyone can apply their process to the project they prepared. The room asks you to apply it to a variant you have never seen, live, which is the only condition under which a pattern proves it is a pattern and not a memory.
• Can you defend a tradeoff you made six months ago, including the part where you would now decide differently? The best answer to what almost shipped is never nothing. It is a specific option, a specific reason, and usually a small regret.
• Whether you know where your own work is weak before the room finds it. The hiring side sees the same pattern from the opposite chair: the candidates who name their own gaps before being asked are the ones who read as credible.

Notice what is missing from that list. Polish, which is now generatable. Recall, which is now delegable. And confidence, which was always performable. Confidence theater can survive a presentation. It rarely survives a follow up, because theater has no almosts.

One more thing the list refuses, and I refuse it deliberately. Failing a room does not make you a fraud. I failed one carrying nearly a decade of real practice, because one specific skill was missing: translating private judgment into public defense, in someone else’s vocabulary, at conversation speed. The old model treats that skill as theater, as interviewing well, as something beneath the real work. That framing is wrong on the facts, not merely the feelings. When the judgment is real, the defense is the last mile of the work.

My own data points the same direction, with the caveats it always carries. The assessment I built asks scenario questions instead of collecting self ratings, which makes it, in effect, a private room: nobody is watching, but you cannot pattern match your way to the senior sounding answer. Around one hundred designers have taken it, a small and self selected sample, so read these as instrument observations, not population statistics. When I rewrote the question bank so that scope signals and maturity signals could separate, four in ten profiles came off the diagonal: their Level and their Stage told different stories. And nearly half of the people who start it call themselves senior, while the assessed results center a full level lower, a comparison of unmatched groups, so a shape, not a verdict. The narrow claim both numbers support: most of us have never had our self assessment tested, and the first test moves it.

Reps, not talent

Here is where this article leaves the AI panic genre, because the panic genre ends at naming the problem and this is where the useful part starts. The room test has a property that polish never had: it is trainable, in public, for free, starting Monday. I watch it being trained every week, because I run these rooms from the manager’s chair: calibrations, promotion cases, portfolio reviews, stakeholder challenges. The designers who pass them are not the most talented ones I manage. They are the ones with reps. Four reps, specifically.

• Narrate your decisions before anyone forces you to. In crits, in standups, in the Figma comment, say the why out loud while the decision is fresh. This was exactly the rep missing from my VP interview: eight years of practice and zero repetitions of explaining that practice in anyone else’s vocabulary. The judgment existed. The translation did not, because I had never once rehearsed it.
• Close the loop on every project. Go back after shipping and name what worked, what did not, and what you would change. The question what almost shipped instead has an answer only if you kept your almosts. Most designers throw them away the day the final version wins.
• Invite the hostile question early, in rooms where it costs nothing. Ask a peer to attack your strongest case study for ten minutes. Ask your lead to play the skeptical stakeholder. The first time someone questions your work face to face should not be the time it matters.
• Sit in on reviews you are not required to attend and watch how someone a stage beyond you takes a hard question: how long they pause, what they concede, when they push back. Judgment under pressure is learnable by observation long before it is learnable by experience.

None of these require talent, a budget, or permission. They require a tolerance for small early embarrassment, which is precisely the price that most people optimizing their artifacts are trying to avoid paying. Pay it in the cheap rooms.

Stop optimizing the artifact

I will not pretend the room deserves the weight it is about to carry.

Rooms misfire. Mine did, seven years ago, rejecting real judgment over a missing label, and I have no doubt rooms I run have misfired since, in ways I did not catch. Rooms can be gamed, and the HBR data says the gaming is industrializing. And remote work spent five years shrinking the number of rooms we sit in at all, which means fewer cheap chances to practice and higher stakes when a room finally arrives. The industry has not solved any of that, and I cannot solve it here.

The room gets the weight anyway. Not because it is fair, but because it is the last assessment standing that you must take in person, as yourself, with no artifact in front of you. Everything else in the pipeline, the portfolio, the case study, the take home, the polished answer to the expected question, can now be produced without you. The room is the only place left where your presence is the evidence.

So the practical conclusion is a ratio. Most designers spend something like twenty hours polishing the artifact for every hour rehearsing its defense. That ratio was rational when the artifact was the proof. The proof moved. Stop optimizing the artifact. Start rehearsing the defense.

The VP asked me one question, and I lost the room. It took me years to understand that the question was never really about design thinking. It was about whether I could stand next to my own work without the work speaking for me. I could not, then. I can now, and not because I became smarter or more senior, because I got reps.

A case study can be generated. A witness cannot. In every room that matters from here on, you are not there to present the work.

You are there to prove you were present when it happened.

Full article content is not available for inline reading.

Read the original article →

OpenAI launched Health in ChatGPT for US adults, allowing users to connect Apple Health and supported medical records for more contextual health conversations. The company said connected health data and related chats would not be used to train foundation models or target ads.

Introducing Runway Media Router. The first preference-optimized router for generative media. Instead of hand-picking a model for every request, you define what "best" means once, for cost, quality, or latency, and the router selects the right video, image, or audio model automatically. Live now in Runway Dev.

Claude's voice mode just got smarter

And you can connect it to apps like Gmail and Slack.

Since last year, Anthropic has offered a voice mode through Claude, allowing you to speak to its chatbot instead of writing out your prompts. If I had to guess, most people probably don't know Claude has voice input. However, for those that have used it, the consensus has been that it could use work. One issue is that before today Anthropic routed voice mode queries through Haiku, its smallest model, to reduce latency. That meant voice mode worked well enough for simple questions, but could struggle with more complicated requests. Today, Anthropic is releasing an update to address that complaint.

Now when you use voice mode, it can turn to the company's Sonnet and Opus models for help. Provided you pay for Claude access, the tool will default to the last system you used for text chat. You can also switch between Haiku, Sonnet or Opus mid-conversation through the model picker. "Voice mode uses the fastest version of whichever model you've selected, so the conversation runs smoothly," Anthropic notes. Additionally, voice mode can now pull context from connected apps such as Gmail and Slack, as long as you grant Claude permission to do so.

An Anthropic spokesperson told Engadget voice mode uses a turn-based architecture, so all interactions will see Claude listen to you, pause to think and then respond. It's not fully duplex like OpenAI's new GPT-Live system, which can simultaneously process speech and generate an output. In practice, that should make talking to Claude feel less natural than ChatGPT. Another limitation of Claude's voice mode is that it can't automatically detect the language you're speaking in if you decide to switch languages mid-conversation. You need to either tell it out loud you're about to switch or select the language you're about to speak in from the voice settings menu. However, Anthropic has added support for additional languages, including Indonesian.

"This release is focused on intelligence and tool access," Anthropic told Engadget. "We're continuing to invest in voice and we'll have more to share later this year."

Anthropic is rolling out the new voice mode in beta to all users across its desktop and mobile apps, as well as web client. If you're using Claude through a free account, Anthropic will limit you to a single connection and your prompts will all go through Haiku, though you can speak to Claude in all of the languages voice mode now supports.

Compute pricing coverage runs on headline rates, but many workloads don't consume them that way. Training, fine-tuning, and high-throughput inference need multiple identical GPUs in the same machine or cluster. A company that buys H100 futures to hedge a future four- or eight-GPU requirement carries real basis risk. The futures will hedge a rise in the broad H100 price and do nothing about the lack of co-located capacity.

Key Points

• Intel's earnings report lands as the chipmaker tries to find its place in the AI boom.
• Revenue jumped 25%, the most robust growth for any period since the third quarter of 2011.
• The stock is up over 170% this year, though it's slumped of late.

Intel reported better-than-expected second-quarter results on Thursday, notching its fastest revenue growth rate for any quarter since 2011 and issuing guidance that topped expectations.

Here's how the chipmaker did versus LSEG consensus estimates

• Earnings per share: 42 cents, adjusted, versus 21 cents expected
• Revenue: $16.1 billion, versus $14.42 billion expected

Shares of the chipmaker climbed immediately following the report, but sank during trading on Friday.

Intel shares are up over 170% so far in 2026 as of Thursday's close after soaring 84% last year, when the U.S. government took a 10% stake in the company as part of an effort to support U.S. chip manufacturing. However, the stock has been in a slump more recently, dropping 28% in July.

Despite the recent downturn, the company is getting a boost from the artificial intelligence infrastructure boom, which is helping sales of its server processors. Intel's 25% revenue growth was the fastest for any period in almost 15 years.

"AI is driving unprecedented demand for compute," CEO Lip-Bu Tan said in the statement. "As we continue to execute, Intel is well-positioned to capture sustainable growth across our CPU franchise."

For the current quarter, Intel said it expects adjusted earnings per share of 38 cents on revenue between $15.8 billion and $16.8 billion. Analysts were expecting revenue of $15.1 billion and EPS of 27 cents, according to LSEG.

Intel also said it's starting to craft long-term agreements with customers for its server CPUs, some with pricing locked in and others focused on chip volume.

It's a move that's becoming common, particularly in memory, as vendors try to preserve current high pricing and market power in case the AI market turns. Intel said it had reached 10 long-term agreements, and CFO David Zinsner said the company is supply constrained, with data center customers demanding more than it can produce.

"Customers continue to signal a strong and sustainable spending environment," Zinsner said on an earnings call with analysts.

Revenue in the company's client computing group, which makes chips for PCs, rose 13% to $8.9 billion. It's still Intel's biggest unit, but the robust growth is coming from its data center business, where revenue rose 59% to $6.3 billion. Intel said it expects flat PC sales in the third quarter because of the memory shortage.

Intel is boosting its capital expenditures, targeting a "meaningful increase" next year, as it aggressively tries to morph into a manufacturer of chips for other companies. Zinsner told CNBC's Kristina Partsinevelos that the company's latest manufacturing process, called 14A, is ahead of where older technologies were at the same point in the cycle. Intel said its foundry reported $5.8 billion in sales, up 31% on an annual basis.

"I did want to give investors at least a line of sight to expect that the number will be up," Zinsner said on the earnings call. He said most of the spending would be for factory tooling.

Still, Intel did not reveal a major customer for its foundry, as investors and potential customers keep waiting. It primarily manufactures its own chips. Intel's foundry landed Fortinet as its first named customer under Tan earlier this week, but it's using an older manufacturing technology to make security chips.

Intel's gross margin also recovered to 42%, up from 2.5% in the year-ago period, which the company attributed to benefits of scale with more revenue, as well as selling chips with higher margins and pricing.

Today, we’re welcoming The Interaction Company of California, the makers of Poke, to Cognition.

If you’ve used Poke, you know why we’re so excited. Poke is a personal agent that lives in your texts. It messages you first, follows up with you, and feels less like software and more like a friend. People have exchanged more than 100 million messages with it in just the last three months, and it’s the only AI agent approved to text natively on Apple Messages.

The Interaction team has built an agent that people love: it’s proactive, it knows you, and it’s fun to talk to. That’s exactly how working with Devin should feel, and now we get to build it together.

We’ve known and admired Marvin and his team for years. In fact, Walden and I were early angels in Interaction. They are exceptional people who move extremely fast and aim for only the most ambitious products. And from the beginning, our teams have been making the same bet: always-on cloud agents.

Today, Poke users can continue using the product just as before. Soon, Cognition’s models and infrastructure will make Poke even faster and more reliable.

Devin is used by some of the world’s top companies for software engineering, while Poke is beloved by hundreds of thousands of people as a partner in everyday life. Together, we’re excited to shape the way humans and AI collaborate for the future.

Stripe is in talks to acquire OpenRouter, a startup that helps developers choose between AI models. The talks are still ongoing and could fall apart. Industry experts estimate that the business could fetch about $10 billion in a sale. Stripe is separately pursuing a deal for PayPal.

Optimizing the cost-to-outcome frontier in real world context means using the right model for each task, and optimizing the context, skills, tools, and agent harness around it. It's possible to take saturated frontier capabilities and deliver them at scale and at lower cost through models optimized for high-usage products, while continuing to use frontier models for frontier needs. The model is only one part of the hill-climbing system. Harness, memory, context, tools, skills, and user interactions all shape the evals and performance of these agentic systems.

Huawei is leading the way in reducing China's dependence on foreign AI chips. The company claims it has figured out workarounds for making near state-of-the-art silicon without leading-edge machinery. Huawei's deputy chairman attributes the company's progress to the restrictions the US has imposed on hardware exports. China still lags far behind the US, and chip-production capacity problems are preventing Chinese companies from deploying AI as quickly as they would like.

How AI Is Changing Open Source

AI entered software development at full speed this year, and it is significantly impacting open-source projects as well. In this article, I discuss several trends I have recently observed in open source in connection with AI, and how these trends are changing the world of open-source software.

This article was originally published on my Czech blog, but it received such an overhelming response that I decided to translate it into English and publish it here as well.

Project Inflation

One of the trends that AI brings in general is an explosion of content. Search results are filled with generated websites, and social networks are inundated with generated images and videos. Source code is no exception. Today, GitHub is drowning in an ever-increasing number of repositories.

However, it is not as if a larger number of high-quality projects are being created. On the contrary, these are projects where you have no idea whether you can rely on them or not. In the past, if you stumbled upon a more extensive project with thousands of lines of code, there was a certain assumption that if someone went to the trouble of creating something like that, they would have some knowledge of the problem, a personal connection to their creation, and some willingness to maintain it going forward.

You can no longer rely on this at all. Today, you can generate a project with several thousand lines of code in a matter of moments. It could be complete nonsense or even something dangerous; it could be something functional that someone generated for their own immediate needs and posted to GitHub, but with no interest in turning it into an open-source project. Because a repository with code doesn’t make an open-source project. The difference between a piece of code on GitHub and an open-source project is that an open-source project solves problems and use cases for its users, not just the author’s one-off need. And most authors of such quick-and-dirty code simply aren’t interested in doing that.

This is clearly visible in projects like MeshCore, for instance. There are dozens of forks of everything imaginable. Missing a feature in the official MeshCore firmware? You just fork it, vibe-code the missing piece, and dump it on GitHub as MeshCore-UltimateEdition. The problem is that it was created with minimal effort, the author usually has no relationship to it, gets bored after a month, and it becomes abandonware before it even has a chance to age.

About ten years ago, people started saying that the concept of Linux repositories had run its course. In the 2000s, they were practically the only source of Linux software. If a project didn’t make it into distribution repositories, it had a problem. But then the number of open-source projects grew at such a rate that distributions couldn’t keep up. Users had to start getting their software elsewhere, and software authors learned to do without distributions. Just a few years ago, the “everything I need, I find in Debian” approach seemed definitively dead.

However, it is possible that curated software sources – like Linux distribution repositories – will make a comeback. The open-source software world is becoming so chaotic that users will once again start appreciating sources containing curated software that someone has vetted for them and that they can rely on six months down the road.

Review Overwhelm

Another trend that AI has triggered in open source is ‘review overwhelm’. Previously, writing code acted as a natural filter because it required a non-trivial amount of effort and time investment. That is now gone, making code creation fast and easy. But someone still has to review this code before it goes into serious production. The review processes that worked in open-source projects for years are now at their capacity limits.

In GNOME 50, support for Google Drive was removed because nobody had been maintaining it for a long time. Users were naturally unhappy about it, and eventually, one user stepped up, re-added the support, and submitted it upstream to the gvfs project.

A colleague responsible for maintaining that project lamented that it was a change involving 4,000 lines of code. Even though it seems to work at a basic level, it was clearly generated using AI. He will still have to go through it line by line to verify that it actually works as intended and meets the code quality standards required to commit to maintaining it long-term.

Most of the effort has thus shifted from code creation to code review, which is typical for AI. The problem in open source, however, is that developers experienced enough to review and merge code were already a bottleneck before AI. Now, the problem has deepened significantly. And in the example above, my colleague can count himself lucky that the contributor is responsive and has shown long-term interest in the issue.

Today, that is more of a rare exception. Common contributions consist of someone wildly vibe-coding something without any deeper interest or understanding of the subject, and throwing it over the wall to the maintainers.

I have a fairly recent experience with this in Meshy. Someone submitted a pull request with 9,000 lines of changed code, which was supposed to add support for macOS. I spent an hour one evening doing a very quick review, and even during that short time, I ran into numerous issues: the code was blatantly AI-generated, several thousand lines were just completely useless replacements of single quotes with double quotes, parts of the code unrelated to the problem were modified, and it overwrote all the changes I had made in the main branch over the last few weeks.

The author never responded to my comments and I never heard from him again. My takeaway was that even that one hour was too big of a time investment for contributions like that, and next time I will reject them much faster.

Some projects are responding to this situation by tightening basic contribution requirements. For example, Flathub’s decision to reject AI-generated apps caused quite a stir. Many people criticized it as shooting themselves in the foot, but you have to look at their reality.

Flathub currently hosts several thousand apps, with more added every day. Only three people handle the reviews. Although their review process is highly automated, they do it very thoroughly, and a lot of manual input is still required. It’s clear their goal isn’t just to spot the worst slop, but to maintain a relatively high standard of code hygiene. In the last six months, I submitted two apps to Flathub, and the review process ultimately contributed to improving the quality of the apps themselves.

However, this has now clashed with the reality of people submitting completely vibe-coded apps without a shred of personal effort. The ticket template for requesting inclusion asks a few questions, including a requirement to upload a short video showing how the app works. It really isn’t demanding, and anyone can put it together in 15 minutes. Yet even that is too much effort for creators of AI slop.

Instead of fulfilling these minimal requirements, some labeled it an attack on Linux’s freedom and immediately vibe-coded an alternative to Flathub that was supposed to be open to everyone. Unsurprisingly, it barely lasted a month.

Not only do open-source maintainers lack the capacity to satisfy this demand for code review, but they are also losing the motivation to do it. Often, it would be faster for them to write the feature themselves, but the review process was historically how they cultivated new long-term contributors and potential successors. When someone sends you a vibe-coded contribution that cost them zero effort and which they likely don’t even understand, how do you expect to mentor them into a contributor who will help the project in the long run?

Open-source software was never just about the end result; it was also about the process – where contributors build a relationship with the project and grow into someone who will eventually pass that on to others. This stands in sharp contrast to the world of AI, where it’s all about the result. As fast as possible, with as little effort as possible.

Declining Motivation to Publish Code

In the 1990s, Francis Fukuyama declared democracy and liberal economics to be the ultimate victors in the arrangement of the world order. Today, as democracy erodes globally and the existing economic order crumbles, that looks like a prematurely bold statement to say the least. Similarly, just a few years ago, impressed by the developments of the last few decades, some hailed open source as the ultimate winner among software development models. Are we about to face a sobering reality check similar to Fukuyama’s thesis?

Lately, I’ve been observing a subtle, yet present trend of stepping back from open-source development. One argument against open development I hear concerns the aforementioned review overload. For some projects, the costs associated with being overwhelmed by AI slop can outweigh the benefits of useful community contributions. They might still publish the source code for transparency’s sake, but they transform from an open-development project into an open-source, closed-development project. And those who don’t care as much about transparency may close off the source code entirely.

Another argument against making source code public is the fear of license circumvention. Today’s LLMs train on source code regardless of its license and can then easily generate a similar solution that you can publish under whatever license you choose.

This isn’t an issue for permissive licenses, as the author has already accepted that anyone can do practically whatever they want with the code. However, AI poses a direct threat to copyleft licenses like the GNU GPL. Authors usually choose these to ensure their work remains open forever and that anyone who uses it shares their improvements back with the community. If an LLM trains on a project you’ve worked on for years and then generates a very similar solution published under a proprietary license, it effectively bypasses this principle.

Take MeshCore again as an example: the protocol itself and the firmware are open-source, but the clients are closed. Recently, it came to light in the community that a core team member secretly applied for the MeshCore trademark and started vibe-coding his own closed-source solutions based on the available code. MeshCore founder Scott Powell cited this as something that reaffirmed his decision to keep the client source code private. Specifically, he wrote:

So, I see open source, in the age of AI, as offering up your blood, sweat and tears for others to rip-off, but in innumerable ways.

We may disagree with Powell’s perspective, but it represents a legitimate stance that I see more and more often around me. I see lifelong open-source advocates – people who used to publish every last helper script because they wanted to share – who now keep those things to themselves, offering them to others only upon request. They have reasons similar to Powell’s.

Open source also grew out of the need to share. Writing code was hard; maintaining it was even harder. Why should everyone implement the same thing independently? Let’s join forces in an open-source project, write a shared library, and everyone can benefit from the results. The infrastructure powering the Internet today was built on this foundation. But AI is suppressing this need.

For instance, I encounter opinions that WordPress is dead because “I can just easily generate my own CMS.” In my view, that severely underestimates what an open-source project actually provides. It is so much more than just writing code, and this strategy of swapping a dependency on an open-source project for a dependency on an LLM might not pay off in the long run.

Nevertheless, the reliance on shared open-source components has indeed decreased to some extent. AI might not replace everything, but why depend on a large external library when you don’t even need 10% of its functionality, if AI can quickly rip off that 10% for you after learning from the original library? And once you have your own implementation, why would you contribute improvements back to a shared open-source project?

The final argument against publishing source code that I’ve been hearing lately is security. Granted, I’ve heard this argument throughout the two decades I’ve been involved in open source, but it has never been this loud. For years, critics have claimed that open source is insecure because it allows attackers to study the code and hunt for vulnerabilities. In response, open-source advocates argue that security through obscurity is not real security and that open-source software is safer because “given enough eyeballs, all bugs are shallow.”

Today, however, open-source projects are literally flooded with security vulnerability reports generated by AI. The volume is so unprecedented that it is genuinely easy to fall into the trap of believing closed code is safer. It’s interesting to note that while news headlines cover how many bugs AI has found, they rarely mention how many security bugs AI has fixed. Fixing them still requires a deep understanding of the codebase and is still done by human programmers. And just like reviewing pull requests, it is overwhelming their capacity.

In this case, though, I believe it’s just a temporary trend. Open-source projects will eventually wade through these security reports, the general security of maintained open-source software will improve, and the ecosystem will benefit in the end. As for the other trends mentioned in this article, it’s hard to say. I’m not quite as unconditionally optimistic there.

Designers can turn competitors' negative reviews into persuasive evidence, since scattered complaint patterns reveal gaps stakeholders can't dismiss as opinion. Mining sources like Reddit, G2, and Trustpilot for recurring 1-2 star complaints—time-boxed and counted, not curated—surfaces real user language and frustration clusters. Charting these clusters' frequency converts qualitative gripes into data-backed arguments that move stakeholder conversations from opinion to evidence.

Studio Chapeaux argues that typography has become the most important element of packaging design, with bold, distinctive typefaces outperforming logos and illustrations by remaining legible, expressive, and impactful on crowded shelves. To demonstrate this approach, the studio created SC Ostrea, a custom typeface designed specifically for its oyster and beer brand, showing how typography can become an integral part of a product's identity rather than a supporting element. The project also marks the beginning of a type foundry focused on packaging, addressing what the studio sees as a lack of typefaces optimized for high-impact commercial products.

What if design could make science irresistible to kids?

Inside a growing number of Italian primary school classrooms, science is something children build, shake and observe with their own hands. This is Scintillab, an educational kit promoted by Fondazione Agnelli together with Ferrari, developed with the scientific contribution of CNR – Public Relations and Integrated Communication Unit and co-designed by Fondazione OpenDot, bringing STEM disciplines into the classroom through direct experimentation. Cylinders to guess by sound, shadow theatres, ramps for rolling objects: each tool turns an abstract concept into something children can touch and get wrong on their way to understanding it.

Since launching in 2025/2026, the project has already reached almost five thousand students across six Italian provinces. We spoke with Laura Dellamotta, president of Fondazione OpenDot, about co-design, inquiry-based learning, and putting the body back at the centre of how children learn.

To begin with, could you tell us what Scintillab is and how the project came about?

Laura Dellamotta: Scintillab is an innovative educational kit co-designed by OpenDot in collaboration with CNR, created to bring STEM disciplines into primary school classrooms through hands-on experimentation, inviting students and teachers alike to look at the world with curiosity and scientific rigor. It’s an initiative promoted by Fondazione Agnelli together with Ferrari, with the scientific contribution of CNR’s Public Relations and Integrated Communication Unit, and it’s offered to schools completely free of charge.

The project was initiated by Fondazione Agnelli, long active in STEM education through initiatives like Matabì and HOP, which brought in CNR for the scientific design of the kit’s content, with Ferrari stepping in as funder. OpenDot was then selected to co-design and develop the kit itself, both conceptually and physically. The choice made sense given our background: we started as a fab lab and grew into an innovation hub centered on research applied also to education, giving us both the experience and the concrete capability to design and produce, from early prototypes through to finished objects ready for classrooms.

At the heart of the initiative is an educational kit designed for primary schools. What does the box contain, and how is it used by teachers and students?

Laura Dellamotta: Scintillab is what we like to call a “box full of science”, designed for classes from third grade up, meant to turn the classroom into a laboratory of discovery. Inside are simple, open, versatile tools organized around different areas: cylinders you shake to guess what’s inside from the sound; a light-and-shadow area where children build a kind of theatre and observe how shadows change; a gravity-and-movement area with ramps and objects to roll down them; a math-and-probability area with coins and dice; and a balance area for building improvised scales.

Activity sheets guide the whole experimental journey. It follows Inquiry-Based Learning, so experience always comes before theory. Teachers act as guides rather than dispensers of answers and go through a free training day, accredited on the SOFIA platform, before using the kit. Students work in small groups, encouraged to experiment and get things wrong, since here a mistake isn’t a failure but a step toward discovery. The project was designed with particular care for schools with fewer resources, to show that doing science doesn’t require complex technology, just attentive eyes and curious minds.

Co-design is a key element of OpenDot’s approach. How did this methodology shape the development of Scintillab?

Laura Dellamotta: Scintillab is the product of a complementary network including Ferrari, Fondazione Agnelli, CNR and ourselves, where collaboration between very different skill sets generates real value. OpenDot acted as a kind of third space, connecting the scientific rigor of CNR with the educational needs of the school and the culture of making typical of design. Co-design is what transformed Scintillab from a simple supply of materials into a genuine educational method, one that doesn’t teach content so much as build the capacity to ask questions about the world.

One of our core principles in education is putting the body back at the center of learning, and co-design let us translate abstract concepts like probability or gravity into gestures, physical exploration and concrete prototypes, embracing error as part of the experience throughout.

Scintillab is based on inquiry-based learning. How does it work, and what are its main benefits?

Laura Dellamotta: Inquiry-Based Learning turns the classroom into a laboratory of discovery, moving away from lecture-based teaching. The teacher stops being a dispenser of answers and becomes a guide, there to spark curiosity rather than give immediate solutions. Activities are built to raise questions – where does my shadow go when I turn off the light, why do things always fall down – that naturally lead to further investigation. Children work in small groups, observing and testing their own ideas, and mistakes are treated as a normal, necessary part of the process.

The benefits are wide-ranging: curiosity paired with scientific rigor, the ability to reach a goal through different routes, abstract concepts made physical and tactile, and collaboration built through group work. And because it relies on simple materials rather than expensive technology, it works just as well in schools with very limited resources, showing that scientific discovery is accessible to everyone.

Where has Scintillab been implemented so far, and how many schools have been involved?

Laura Dellamotta: In its first school year, 2025/2026, Scintillab has already reached 59 school institutes, 238 teachers and 4,700 students across the provinces of Cuneo, Savona, Modena, Parma, Ancona and Matera. Over the next two years, the goal is to consolidate and extend the initiative nationally, reaching 800 teachers and 15,000 students, expanding into Brindisi, Salerno and Catania.

Feedback has been very encouraging: among 125 teachers who responded to a follow-up survey, 81% had already used the box in class, and all 15 activities had been tested in at least one classroom. Teachers reported greater student participation, stronger observation skills and increased interest in science, with particular engagement from students usually less active during lessons. Overall, 88% described their experience as “very positive,” with the remaining 12% “fairly positive”.

Why is it important to introduce children to STEM subjects from an early age? And what role can design play in this process?

Laura Dellamotta: Primary school is when children build their basic sensibilities, languages, and cognitive and relational skills, so introducing STEM early means working on prevention rather than emergency. When mediated by the body and direct experience, this counters a kind of poverty of experience – without physical exploration, children risk losing the ability to grasp nuance – and keeps learning whole, so intelligence never gets separated from perception and movement.

Design plays a key role as the discipline that builds relationships between spaces, people, tools and bodies, translating abstract concepts into physical experiences and tangible prototypes children can touch and manipulate. It creates spaces of possibility that allow freedom and student protagonism rather than imposing one right way of doing things, treating technology as an extension of the body rather than an end in itself. In the end, design applied to STEM education shifts the focus from teaching passive content to enabling active learning, where science becomes an adventure to explore with the whole body.

What does it mean to design a laptop in an era when the very concept of a laptop may be obsolete within a decade? As artificial intelligence rewires the relationship between humans and their devices, the designers tasked with building tomorrow's hardware face a challenge unlike any before: creating objects for experiences that do not yet exist.

In a wide-ranging conversation, Brian Leonard, Vice President of Design at Lenovo, spoke candidly about the seismic shifts reshaping his field, from foldable screens and modular hardware to the quiet death of the purely software-driven interface.

From thin and light to... something entirely different

For years, the guiding ambition of laptop design was deceptively simple: make it thinner, lighter, and more affordable. That era, Brian suggests, is firmly behind us.

"If we look back a couple of years ago, we were focused on how do we make it thinner, how do we make it lighter," he says. "And now it's completely different, because we have the opportunity to change the way we use computers every day."

The shift, he argues, is not merely aesthetic. The arrival of capable AI has thrown open a door that hardware designers had not previously needed to consider: the possibility that the fundamental act of computing — strings of letters forming words, words forming sentences, manual inputs yielding predictable outputs — may itself be replaced.

"I think in the future, our relationship with the devices that we use can dramatically change. We can shift people from that manual process of being doers to now directing a process. Giving your intent. And AI helps understand who you are so that it delivers an outcome that's more personalised to you."

The hardware-software-AI triangle

Brian, who describes himself as a trained hardware designer responsible for somewhere between 700 and 800 notebook designs over the course of his career, is candid about how the scope of his work has expanded.

"When we look at what the future is, it's not just about hardware. It's about this relationship between hardware, software, and AI. So it's no longer just about what it looks like and what it did, but what are those possibilities?"

This triangular relationship, he says, is central to Lenovo's new AI strategy, which includes a personalisation platform called Qira. The goal is not to deliver generic AI responses, but to use control of hardware, software, and AI in combination to create something more tailored.

"It starts to become a more personalised experience versus a generic experience."

Foldables, rollables, and the expanding screen

One area where Brian's enthusiasm is unmistakable is in form factors that depart from the traditional clamshell design. Foldable and dual-screen devices, he argues, are not novelties but logical responses to where computing is heading.

"Foldables, rollables, dual displays... these types of things have a huge role in the future of computing, because even software flows are going to be different."

The Yoga Book, one of Lenovo's more experimental dual-screen products, was directly inspired by observational research; designers watching people in coffee shops arrive with notebooks, external keyboards, and stands, cobbling together makeshift workstations on café tables.

"That is actually what led into the original concept work for Yoga Book. We started building models with two displays; if you're going to do that and have an accessory keyboard in front, why wouldn't you have two displays?"

Brian sees these devices as challenging the primacy of the physical keyboard, not by eliminating it, but by making it optional, turning it into a modular element of the device.

"I can use a physical keyboard when I really have some intense writing to do, but I can also use a digital keyboard if it's just a little bit of editing. It starts to let me make the choice."

The ecosystem imperative

Individual devices, however innovative, are increasingly insufficient on their own. Brian describes a design philosophy centred on building products that function as part of a coherent system rather than isolated objects.

"We've been working on a piece of software called Smart Connect, which connects my phone, my tablet, and my PC together so I can use the value of each one of those devices independently. But the next step is: I need a picture, or I need to make a sketch on my tablet for the presentation I'm working on, and instantly I sketch it, and it's available to me."

The aspiration, he says, goes beyond visual consistency or shared branding. It is about genuine functional integration; designing so that the whole exceeds the sum of its parts.

"I don't want to take a picture and email it to myself and then open it up in email and offload it and then put it into a presentation. That should be seamless."

Modularity, repairability, and the return of the physical button

Our conversation also touched on a quieter but growing movement in consumer electronics: the demand for devices that can be repaired, upgraded, and personalised at a component level. Brands such as Framework and Fairphone have built followings around this principle, and Brian sees it as a natural direction for Lenovo too, one with deep roots in the company's own history.

"I can remember opening up the palm rest on a historical ThinkPad, and inside there are blue tabs to take out the hard drive, the optical drive, the memory. At that point, it was table stakes for how we designed computers. Somewhere along the way, the world focused on thin and light."

That focus, he suggests, came at a cost, and the industry is now correcting course.

"The world is now focused on different things: the responsibility of the products, which may be about sustainability, accessibility, as well as repairability. That's the true value of some of the modular things; making these things repairable so that it's easy to upgrade as well as get service parts."

On the broader question of tactility and physical controls, Brian is equally direct. Asked about design trends he is finished with, he pointed immediately at the wholesale migration of controls to touchscreens.

"I really hope we're done with: all we have to do is give you a screen, and you press and do things on the screen. I love the trend in automotive where they put the buttons back. I hope we get away from things being completely software-driven and bring back some of those natural, multimodal things. The tactility."

This, he notes, is not mere nostalgia. The haptic and physical quality of an interface communicates something about the value and craft of an object. He traces Lenovo's own keyboard heritage directly to the IBM Correcting Selectric typewriter, through the ThinkPad's acquisition from IBM, a lineage of considered physical design that stretches back decades.

Designing a dream workspace

Asked what he would build if freed from commercial constraints entirely, Brian's answer was refreshingly telling. Rather than describing a single device, he described an environment.

"I would really be interested in designing a workspace that encompassed an ecosystem of devices that were about my person, focused on it more like the cockpit of a car versus a desk and a chair. To get out of the scope of the devices and design an environment that supported the devices."

It is a vision that captures the broader direction of his thinking: away from the object, and towards the experience. Away from the isolated screen, and towards the connected, personalised, multimodal workspace of the near future.

What emerges from our conversation is a picture of a design discipline in genuine flux; not anxious about that flux, but energised by it. The questions that preoccupied hardware designers for the previous two decades (how thin, how light, how cheap) have given way to something far more open-ended: how should a device understand you, adapt to you, and work alongside all the other devices in your life?

Brian's answers suggest that the most interesting design decisions ahead are not about millimetres or materials, but about the intelligence woven into the relationship between human and machine. Whether through foldable screens, physical controls making a comeback, or AI that learns the contours of individual intent, the future of personal computing appears to be one that is, finally, genuinely personal.

Substack quietly crossed 50 million active subscriptions this year, with 5 million of those paid. That number matters less than what it represents. Not only authors and content creators, but also designers, illustrators, photographers, and independent editors are rebuilding their careers around a platform that does not run on algorithmic approval. WE AND THE COLOR built its own Substack channel around that same principle, and it has become one of the clearest community hubs we have for design-minded readers who want direct access to our reporting instead of whatever a feed decides to show them. For creative professionals tired of chasing reach on Instagram or LinkedIn, this shift is not a trend. It is a structural correction.

In this article, I will explain why Substack works for creative fields specifically, not just for journalists and political writers. The article also introduces a few frameworks worth borrowing when you plan your own move.

What Makes Substack Different From Algorithm-Driven Platforms?

Every social platform sits between you and your audience. An algorithm decides who sees your post, when, and how often. Substack removes that layer. When someone subscribes to your publication, your writing lands in their inbox. No feed competition. No shadow limits. No guessing why last week’s post reached ten thousand people and this week’s reached three hundred.

I call this gap the Recognition Debt: the cost creators pay when a platform demands constant performance just to stay visible. On Instagram, you post today, and the algorithm forgets you by tomorrow unless you post again. That debt never gets paid off. It only resets. Substack does not eliminate effort, but it removes the debt structure entirely. A subscriber stays a subscriber whether you publish daily or once a month.

The Direct Reach Ratio

Here is a framework worth tracking: the Direct Reach Ratio, or the percentage of your audience that actually receives your content versus the percentage that merely follows you. On Instagram, that ratio often sits below 10% for accounts without heavy engagement. On Substack, assuming reasonable email deliverability, it approaches 100%. That difference changes how you plan every single post.

The Numbers Behind Substack’s Growth

Substack’s own growth data explains why creative professionals are paying attention. Gross writer revenue reached $450 million in 2025, and the top ten earners on the platform now collectively pull in more than $40 million a year, up from $25 million in 2022. The company reached a $1.1 billion valuation in 2025, roughly seventy percent higher than its 2021 mark.

What stands out more than the money is where growth actually comes from. Substack reported that 32 million new subscribers arrived through the app itself over just three months in 2025, not from outside links. Monthly active app usage grew 139% year over year. Readers are opening Substack the way they once opened Instagram: as a destination, not a forwarded link.

Why Creative Professionals Are Migrating From Instagram and TikTok

Designers and illustrators built entire careers on Instagram between 2014 and roughly 2022. Then the platform pivoted toward video and started favoring accounts that post constantly. Many creatives never wanted to become full-time video editors. They wanted to make things and show them to people who cared.

Fashion and lifestyle creators have started the same move, and their reasons echo what designers report. Several influencers who shifted revenue efforts toward Substack cited an unpredictable algorithm that kept changing which content it favored, regardless of quality. Meanwhile, LinkedIn added ads, and organic reach kept dropping for personal accounts, pushing creative professionals away from that platform too.

Editorial Sovereignty

Substack gives creators what I call Editorial Sovereignty: full control over format, pacing, and presentation, without a platform quietly reshaping what gets seen. You choose the layout. You choose whether a post is five hundred words or five thousand. You choose whether to include a mood board, a process breakdown, or a full case study. No format gets penalized because it does not match this week’s trending style.

The Compounding Audience Effect

Social platforms reset your reach constantly. Every post starts from zero. Substack works differently. Each new subscriber adds to a durable, owned list that does not decay when the platform changes its rules. I call this the Compounding Audience Effect, and it is the single biggest reason creative professionals should think of Substack as an asset, not a channel.

Consider the difference in practice. A designer with 5,000 Instagram followers might reach 400 people per post. A designer with 5,000 Substack subscribers reaches close to 5,000 people every time because email delivery does not depend on engagement scores. Over a year, that gap compounds into a completely different relationship with an audience.

A Necessary Caution: Substack Is Not Immune to Change

Substack deserves a fair, critical look too. In late 2025, the company changed how its Notes feed works. It now surfaces content from creators you do not follow, not just people already in your network. That shift helps new voices get discovered, but it also means your existing subscribers may see less of what you post on Notes. There are also persistent rumors that Substack may introduce advertising at some point in 2026. No platform stays static forever, and creative professionals should build their core audience around the email list itself, not around any single discovery feature.

How WE AND THE COLOR Applies These Principles

At WE AND THE COLOR, we treat our Substack channel as an extension of our editorial mission rather than a marketing add-on. It gives readers who care about typography, branding, and visual culture a direct line to our reporting, without an algorithm deciding whether they see it. If you want design and creative industry insights delivered straight to your inbox instead of filtered through a feed, our Substack is worth a look.

How to Start a Substack as a Designer or Artist

Getting started takes less setup than most creative professionals expect. A few practical steps make the difference between a Substack that grows and one that stalls.

• Pick one clear theme. A publication about “everything design” attracts fewer loyal subscribers than one about a specific niche, such as type foundry reviews or architectural photography.
• Publish process, not just polish. Readers on Substack respond to moodboards, drafts, and behind-the-scenes thinking. Finished work alone rarely sparks conversation.
• Use Notes consistently. Notes function as Substack’s internal discovery engine. Twenty minutes a day of thoughtful Notes activity often outperforms hours spent on other platforms.
• Price a paid tier early, even if low. A modest paid tier signals value and gives your most engaged readers a way to support your work directly.
• Cross-link sparingly. Mention your Substack on other platforms, but treat it as the destination, not the afterthought.

What This Means for the Future of Creative Publishing

Substack is on pace to cross 10 million paid subscriptions before the end of the year, according to industry analysts tracking its growth curve. Video newsletters are becoming a mainstream format, not a novelty, as creators tired of algorithm-driven video platforms look for a steadier alternative.

For creative professionals, this points toward a wider pattern I call the Attention Ownership Curve: the long-term shift of creative careers away from rented attention on social platforms and toward owned relationships with an audience. Designers who build on Substack now are positioning themselves ahead of that curve, not behind it.

Frequently Asked Questions About Substack for Creative Professionals

Is Substack only for writers and journalists?

No. Substack supports text, images, audio, and video, which makes it just as useful for designers, illustrators, and photographers as it is for writers.

How is Substack different from Instagram for designers and artists?

Substack delivers content directly to subscribers through email, without algorithmic filtering. Instagram decides who sees your posts based on engagement signals you cannot fully control.

Do I need a large following to start a Substack?

No. Many successful creative publications started with a few hundred subscribers and grew through consistent, focused content and Notes activity.

Can Substack replace a personal portfolio website?

Substack works well alongside a portfolio site rather than as a full replacement. Use it to build a direct relationship with your audience, and use your website for a polished project archive.

Is Substack free to use?

Yes. Starting a publication is free. Substack only takes a percentage, currently 10%, when you enable paid subscriptions.

Microsoft introduced MAI-Image-2.5-Pro for high-fidelity image generation and editing, alongside MAI-Voice-2-Flash for faster, lower-cost voice applications. Both models entered public preview and joined Microsoft's broader production model lineup.

Google Research discusses the AI economy's rapid growth, focusing on innovations and impacts across industries. The piece highlights AI's transformative effects on sectors like healthcare and finance. It emphasizes the need for strategic investment and regulation to harness AI's potential responsibly.

The 'frontier' moniker justifies capital spending to investors, who believe that the spending is the moat.

Google's research points to a future where the demand for highly skilled workers is accentuated rather than diminished.

Download a dictation app and start talking.

Google Photos introduced a toggle letting users switch instantly between Gemini-powered Ask Photos and classic keyword search, without navigating through settings. The addition responds to user complaints about irrelevant AI results and the previous absence of a quick way back to classic search, following a mid-2025 rollout pause. Users can now choose their preferred search mode per query, though Google still applies its own judgment about which system fits, and non-adopters must complete onboarding before accessing the AI option.

Apple is expected to unveil its first foldable iPhone, likely called the iPhone Ultra, in September after waiting for foldable display technology to mature and for Samsung to eliminate the screen crease. Samsung's new Galaxy Z Fold 8 Ultra demonstrates a nearly crease-free display that is believed to closely match the panel Apple will use, marking a significant improvement over previous generations.

Loova brings together top AI models like Seedance 2.0, Nano Banana Pro, Sora 2, VEO 3.1, and Kling O1 in one powerful platform.

Rams System Icons reimagines skeuomorphic design with Braun-inspired aesthetics, combining tactile hardware details, clean typography, and a strict grid system to create a modern, highly functional icon set.

Toon Tone is a free cartoon color guessing game where your memory for animated characters gets tested using hue, saturation, and lightness controls.