Muse Spark Open is available today and brings leading agentic capabilities to your desktop for the first time on Apple M5 Ultra Mac Studios.

Today, we’re releasing the weights for Muse Spark Open, the most capable US open-weight model to date. Muse Spark Open is a natively multimodal model with 816B total parameters (42B active) and a 1M-token context window. Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.

We believe the future is for everyone, and that includes the freedom to run advanced models wherever, including at home. Muse Spark Open is the first model in its class to run on your desktop using two Apple M5 Ultra Mac Studios with the latest MLX framework.

Muse Spark Open is available today through API platforms at Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter and Together AI. Model weights are available for download on Amazon SageMaker AI, Google Cloud Model Garden, Hugging Face, and Microsoft Foundry Models.

It runs on leading inference engines like SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp, and Unsloth with optimizations for AMD, Apple, Intel, and NVIDIA silicon. Developers can easily further customize the model using the PyTorch framework.

Key Benchmarks

For more details about our evaluations, see the evaluation methodology report.

Optimized for Apple Silicon

Leveraging the latest version of MLX, Muse Spark Open was optimized to run on Mac Studio with M5 Ultra, and together they’re the best way to run a powerful model on your own desktop.

Apple’s unified memory architecture lets the M5 Ultra's integrated GPU use up to 512 GB per Mac Studio, so two machines can hold an entire model. And the chip’s 1.2 TB/s of memory bandwidth is fast enough to read the small slice each token needs. The result is two Mac Studios behave like one supercomputer with a terabyte of unified memory.

We tested a range of agentic workflows running entirely on-device, using Muse Code powered by Muse Spark Open, including shipping real iOS changes to Ice Cubes (an open-source Mastodon client) — bug fixes, new features, full build — and driving DaVinci Resolve end to end, with the model reviewing footage frame by frame to find the strongest moments and crop them vertically.

Available Now

We’re sharing this work because we want everyone to have access to the best models. The open-weight version of Muse Spark ships with custom, auto-tunable Helion GPU kernels in SGLang, delivering 1.03-3.95× kernel-level speedups on NVIDIA and AMD GPUs, and a 3% end-to-end speedup on AMD MI350.

Safety and Preparedness

Releasing open-weight models is part of how we bring the benefits of personal superintelligence to everyone, and part of how safety work advances across the field.

Open weights also carry their own considerations, and Muse Spark Open has additional safety post-training for that reason. The results demonstrate strong safeguards across risk categories measured.

We evaluated the model in line with our Meta Superintelligence Scaling Framework, covering the most severe risks, and it met the thresholds our framework requires for release. Detailed results are in the Muse Spark Open model card, with a full evaluation report available soon.

License

We’re releasing the weights under the Muse Model License with limited commercial terms, which allow many developers to use the model for free.

Get Started

You can customize Muse Spark Open for your use case by leveraging PyTorch’s TorchTitan to further tune the model. To run it on Apple silicon, use MLX, which pulls the weights directly from Hugging Face.

Muse Spark Open is available now through Hugging Face: https://huggingface.co/collections/meta-models/muse-spark-open

MLX Adapter for running Muse Spark Open on Mac Studio.

Hosted Inference Partners: Together AI, Fireworks, Baseten, Modal, Groq, Nebius, and OpenRouter.

Built or published something with Muse Spark Open? Share with us on the Hugging Face Forums — we’d love to see what you create.

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Since launching Computer in February, Perplexity has been an early pioneer in personal agents. We’re committed to the hard problems of agent governance, detecting unsafe behaviors and preventing rogue agent activity. In July, we replaced our commercially available sandbox provider with a more robust and resilient in-house solution we named SPACE.

SPACE provides high security for advanced agentic systems. In fact, a recent barrage of security testing proved SPACE resilient against network-bypass techniques like those used in the OpenAI/Hugging Face incident.

Today, we're open-sourcing SPACE to give developers a foundation for building safer agent systems and contributing to stronger protections across the industry. Developers can access the repository here.

Agent governance is an engineering problem, and open source is essential to solving it. When developers have systems they can inspect, adapt, and build on, they find more weaknesses and develop better tools. Open-sourcing SPACE gives developers a production-tested foundation for agent runtimes, with isolation, state recovery, and credential and network controls built in.

Creating a foundation for other builders

Sandboxes keep agents secure, letting them access the tools and files they need to complete these tasks, while keeping sensitive information protected.

Open-sourcing SPACE gives developers a starting point for the environments where their agents run code and work with files. Teams can see how the SPACE sandbox platform works and build on its implementation. They can also adapt it to their needs rather than recreating the underlying infrastructure.

In Computer, SPACE supports isolated code execution, pause and resume, recovery, and session branching. Those capabilities provide the foundation for assistants that could:

• Analyze data: Clean a spreadsheet, run calculations, and create charts in a separate workspace.
• Resume work: Pause for someone to review a draft or supply missing information, then resume with their files and running state preserved.
• Explore alternatives: Test two ways to clean a dataset in separate copies of their workspace, or restore an earlier file after a mistake.

These examples illustrate how developers could use SPACE's documented isolation and session controls. Platform and security engineers can also review the isolation model and security examples before integrating SPACE into their own systems.

Increasing security for autonomous work

When we give agents more room to solve problems, we need clear limits on how they do it. We know that agents pursuing legitimate tasks can still cross security boundaries when they attempt to work around missing files, failed requests, or permission denials.

We can put model safeguards into place to reduce the likelihood of unsafe behavior. Adding independent controls in the surrounding system helps contain the damage if those safeguards fail. They limit what an agent can access regardless of what it decides to do.

SPACE puts those controls around a separate workspace for agent-generated code and files. We designed it to combine isolation, recovery, and efficient execution. This lets agents perform useful work over long sessions without relying on the model alone to respect security boundaries.

Running inside its own operating system

Each SPACE sandbox runs inside a small virtual machine with its own operating system, creating an isolation boundary between the agent’s work and the host system. A central control plane coordinates when these workspaces are created, paused, resumed, or removed.

SPACE treats the sandbox as untrusted and manages network access and platform-held credentials outside the sandbox. Agents can connect to authorized services without those credentials being stored in their working environment.

Frequent file snapshots and checkpoints of the full running environment preserve progress across interruptions. To keep these operations efficient, SPACE stores changes rather than repeatedly copying entire environments. This reduces the work needed to create, restore, or branch a sandbox.

Improving production performance

Since July, SPACE has powered all Computer sessions and has handled millions of sandbox creations per day. In a comparison on the same production traffic, sandbox creation was three to five times faster than with our previous provider.

We also tested SPACE against agents deliberately trying to breach its boundaries. Our published security evaluation found no escape from the virtual machine to the host in 108 attempts, but some found ways around network restrictions. We fixed those weaknesses, and neither identified attack method succeeded in follow-up testing.

We also disclosed related weaknesses we found in other sandbox platforms to their vendors, extending that work beyond our own infrastructure.

Contributing to safer agentic AI

SPACE is part of a broader effort to strengthen the systems people rely on as AI takes on more work. Through the Secure Intelligence Institute, we conduct security research and collaborate with the wider community, translating findings into practical defenses.

We believe open-sourcing security tools gives developers more ways to test defenses, find weaknesses, and build safer AI systems. We've shared tools that address different parts of that work:

• BrowseSafe, a model and benchmark for detecting malicious instructions hidden in web pages.
• Numbat, a security toolkit, open-sourced in July, that helps detect, investigate, and prevent dangerous agent behavior on computers
• Bumblebee, a scanner open-sourced in May, that checks developer machines for risky software packages, extensions, and AI tool configurations

Making these tools available lets other teams inspect our work, adapt it to their needs, and contribute improvements. With SPACE, we're extending that approach to the environments where agents execute code and work with files.

Deploying for agent systems

Akamai has tested and launched SPACE on its cloud platform and is working with Perplexity to bring it to its customers. Crusoe is also partnering with Perplexity to add SPACE to its Managed AI Services portfolio. CoreWeave has published setup guidance for developers who want to deploy SPACE for evaluation on CKS.

Developers can start with the SPACE’s setup instructions and security guidance to deploy it for their own agent systems. The repository and documentation define the supported configurations and included components.

Muse Spark Open is an 816-billion-parameter open-weight model that can run on relatively affordable consumer hardware. Meta's benchmark tests used a pair of M5 Ultra Mac Studios with an unknown memory configuration. The model uses a mixture-of-experts design and activates 42 billion parameters for each token. It supports a context of up to 1 million tokens and accepts text, images, video, and audio, but only outputs text.

Bringing the magic: Google breaks open a bottleneck in quantum computing

We've shown that magic state cultivation works on Google Quantum AI hardware.

Google Quantum AI has demonstratively opened up a key bottleneck in quantum computing: we’ve shown that “magic state cultivation” works in practice on our Willow processor, reducing errors about 40x compared to without it. This breakthrough, published today in Science, is a crucial step towards scaling up quantum computing to practical use.

The "magic state" bottleneck

Since 2004, long before quantum computing was a practical engineering pursuit, researchers have foreseen a particularly tricky bottleneck.

To achieve “universal” quantum computing and execute useful programs, you need to be able to run (and error-correct) a specific set of quantum operations. You can do this by first creating what are called "magic states," so dubbed because they can help you create the operations you need for going beyond classical computing.

But there’s a catch: creating high-quality, error-corrected magic states is incredibly resource-heavy. Some estimates have shown that up to 95% of a quantum computer’s entire capacity might need to be dedicated to making them — leaving little room for actual computing. An extra challenge is that once you use a magic state, you can’t re-use it, making magic states a scarce resource.

So for 20 years, researchers have been chipping away at this cost of magic, bringing it lower and lower.

How we made this breakthrough

In 2024, Google Quantum AI researchers Craig Gidney, Noah Shutty, and Cody Jones harnessed an idea from the literature, refined it, and developed what we now call “magic state cultivation.” Their approach promised an especially large reduction. The idea was that cultivation would act like a filter. You would inject a noisy state into a small region on a quantum chip, and then run a series of highly coordinated measurements to check for errors and remove any flawed attempts. Craig and team showed how that would let you grow, or “cultivate,” an exceptionally pure, low-error quantum state that you could simultaneously adapt into a larger, useful encoding.

On paper, it looked great. If we could make this idea actually work on our quantum hardware, it could substantially reduce the size of the quantum computer needed for practical tasks.

In practice — and on par with developing many new technologies — it required a whole series of technical innovations and engineering.

As just one example, when first testing the cultivation protocol, Google Quantum AI research scientist Emma Rosenfeld was seeing data that seemed to defy the laws of physics. After months of confusion, simulations, discussions with colleagues, and consistency checks, she found that it arose from an incorrect assumption in the team’s go-to measurement technique. Specifically, a particular kind of noise in the lab caused by electrical equipment drift and temperature variation was interfering with the benchmark’s validity. The solution required a pivot in both our theoretical and engineering approaches. The team added extra control to the system to reduce the impact of that noise, and invented a new benchmarking technique to solve the issue more fundamentally, by using cultivation as a tool to measure itself.

With this and many more algorithmic innovations and engineering solutions, we established a new state-of-the-art in quantum metrology. On Google Quantum AI's Willow processor, our protocol achieved an extraordinary 40-fold reduction in errors relative to un-cultivated states. And we got T-states with a record logical fidelity of 99.99%.

Now published in Science, this work is the first demonstration of a practical solution to one of quantum computing's most severe resource bottlenecks. It demonstrates, on real hardware, that we can efficiently create these crucial magic states.

Our findings and the future

We also took the first steps in demonstrating that Craig and others’ theories are compatible with our quantum computing architecture. We learned how to successfully switch between “color codes” (one newer type of quantum error correction, ideal for cultivation) and “surface codes” (another, more established method), grafting states from a distance-3 color code to a distance-5 surface code. Originally developed in the 2024 theory paper, this grafting protocol showed that in practice, purified states can survive and transfer smoothly inside our hardware.

Of course, we still need to demonstrate magic state cultivation at a larger scale to make it realistic for practical quantum computing, and plenty of technical challenges remain. But this achievement takes a key theoretical idea and turns it into an experimentally demonstrated reality. It brings us one step closer to our goal at Google Quantum AI: building a useful, error-corrected quantum computer to solve otherwise unsolvable problems.

The problem

64% of product pages tested had at least one critical accessibility issue.

This year, 39,976 Shopify and 19,424 WooCommerce product pages from stores ranked among the world’s top one million websites were scanned. On 37,874 of them, at least one issue in the highest potential-impact category was discovered.

On a product page, that could mean a customer cannot read important information, use a form or select a buying option.

Critical findings are only part of the picture though. 96% of product pages failed at least one automated accessibility check.

These are not just technical faults. They can stop customers using a product page. In the UK alone, an estimated £17 billion in online spending does not happen each year because people find websites difficult or impossible to use. A customer may use a screen reader, enlarge the page or navigate with a keyboard. And when your product page breaks, they generally don't complain or email you. They just leave and buy from somewhere they can actually use (your accessible competitor, perhaps?).

Understanding the headline

What a critical finding can mean for your customers

Critical is the highest potential-impact rating.

For your customer, a critical finding could make important product information, a form field or option impossible to use. The report identifies the barriers with the highest potential impact and shows how common they are across product pages.

64% of the product pages tested contained at least one critical finding.

Shopify and WooCommerce have both invested significantly in accessibility, particularly in checkout and the parts of the buying journey they directly control.

However, your product page is built from platform code, a theme, apps and plugins, agency or developer work and the store's own content.

Any of those layers can introduce a barrier.

That's why accessibility needs an owner inside the business. Someone needs to find where a problem sits, assign it to the right team or supplier and make sure it gets fixed.

By popularity

More traffic did not mean better accessibility

Product pages from the most popular stores were (marginally) worse-performing.

Among stores ranked between 500,000 and one million, 63% of product pages had a critical accessibility failure. Among stores in the top 100,000, that rose to 69%.

A high-traffic store puts every unresolved accessibility barrier in front of a larger audience. Accessibility needs the same ongoing attention as conversion, performance and security.

What fails most

The accessibility problems found most often

Poor colour contrast was the standout problem. It appeared on 80% of the product pages tested.

Rule-level results

Across the product pages tested, 52 axe-core rules identified at least one failure. The table below shows the 14 that appeared most often.

Overlays

Accessibility overlays fix... nothing.

Accessibility overlays are often sold as a quick way to make a website accessible. The appeal is obvious: one product promises to take a complicated problem off your hands. However, 62% of the 1,691 product pages where an accessibility overlay widget was detected still had at least one critical accessibility failure.

Installing an overlay is not proof that the accessibility work is done.

What to do next

Start with these six checks on your own product pages

1. Make text easier to read

   Increase the contrast between text and its background.

2. Give every link a useful name

   Use link text that explains where the link goes or what it does.

3. Keep headings in a logical order

   Build a clear page structure without skipping heading levels.

4. Make tap targets easier to select

   Give buttons and links enough space to be selected accurately on a touchscreen.

5. Use the same name on screen and behind the scenes

   Use the same wording visually and in the name exposed to assistive technology.

6. Add useful alternative text to images

   Describe an image’s relevant content or purpose when it cannot be seen.

Bonus: AI and shopping

The same page structure that helps your customers shop also helps search engines and machines understand your products too. Clear headings, useful image descriptions and properly built, labelled form fields help your customers find product information, navigate by keyboard or shop with a screen reader. Those same elements also tell search crawlers and AI assistants what the product is, which options are available and what each button, link and field does.

Method

The research covers 59,400 product pages, all tested using one consistent process. One product page was selected per Shopify or WooCommerce store in HTTP Archive's global top-million website dataset. Every page included in the report received the same mobile Lighthouse accessibility audit through Google PageSpeed Insights.

Who built this report

I’m Dave. I’m a Web Accessibility Specialist, and I built this report to show store owners which accessibility problems appear most often on product pages.

Microsoft detailed the training of MAI-Base-2, a 2.8T-parameter mixture-of-experts model with 80B active parameters that it said surpassed every open-weight model in its held-out pre-training evaluations. The deep dive covers its data strategy, compute efficiency, full-stack infrastructure, and Microsoft's plans to turn internally trained frontier models into enterprise offerings through Foundry.

Today, we release d1-ultra, our new multimodal decision model with state-of-the-art performance. Most decision models work well on easy and common tasks. We built d1-ultra for harder, messier real-world use cases, like spotting a defect on a circuit board against its spec or checking a damage claim against the customer’s photo.

d1-ultra is available today on the Liquid API at $0.042 per million input tokens.

What d1-ultra does

Decision models answer questions about a situation with a probability for each possible answer. d1-ultra takes unstructured data (text, images, or both) and one or more questions as input. It reads this input in one forward pass and returns the probabilities, without generating any tokens.

d1-ultra answers the same three question types as d1:

• noul: a yes/no question, answered with a probability from 0 to 1.
• Choice: one label from many, with a probability per label.
• Score: a position on a scale.

Results

Compared to d1, we based d1-ultra on a larger open-weight model with a series of Liquid-specific heavy customizations. It provides significant gains when combined with our training pipeline, especially on knowledge-related tasks.

We compared it to our previous model d1 and Jev 1.14 across 112 unique evaluations.

Accuracy is not enough: decision models also output probabilities that must be properly calibrated. We realized that most decision models are overconfident in their responses. That’s why we specifically trained d1-ultra to produce more reliable probabilities.

d1-ultra outperforms Jev 1.14 across all tasks, delivering better-calibrated answers and higher accuracy at the same price. To see how it works in practice, you can find more demos running on d1-ultra in the d1 Playground.

d1-ultra also outperforms all other decision models Artificial Analysis Decision Index by at least 8 points.

Availability and Pricing

d1-ultra is available on the Liquid AI API. Create your API key and start building today. d1-ultra will soon be available through Vercel and OpenRouter.

The model reads text, images, or both at the same time, like in this example:

import base64, os, requests

image = base64.b64encode(open("board.jpg", "rb").read()).decode()

response = requests.post(
    "https://api.liquid.ai/decisions/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['LIQUID_API_KEY']}"},
    json={
        "model": "d1-ultra",
        "images": [f"data:image/jpeg;base64,{image}"],
        "state": "Camera image of a circuit board on the production line.",
        "questions": 
            {"defect": {
                "type": "noul", 
                "instructions": "Does this circuit board have a defect?"
             }
         },
    },
)

print(response.json()["answers"]["defect"]["noul"])

The full API reference is in the decision models documentation.

d1-ultra is billed on input tokens only, with no output tokens. Images are counted as input tokens at the same rate as text: 1.5 tokens per 32×32-pixel patch, so a 1024×1024 image costs 1,536 tokens. Each question is billed as its own prompt, including its text and all images.

d1-ultra is our most capable decision model so far. We’re committed to continuing our work on decision models across a range of sizes, with new features, higher decision quality, and lower latency, and we plan to provide regular updates.

Citation

For citations, please use the following reference or BibTeX:

Liquid AI, "d1-ultra: State-of-the-Art Decision Model", Liquid AI Blog, Oct 2026.

@article{liquidAI2026d1-ultra,
  author = {Liquid AI},
  title = {d1-ultra: State-of-the-Art Decision Model},
  journal = {Liquid AI Blog},
  year = {2026},
  note = {www.liquid.ai/blog/d1-ultra},
}

The Gradient Is a Covariance

Why Post-Training Is Sparse, Part 1: the score, the Fisher, and a bound that holds for any reward

Why Post-Training Is Sparse · Part 1 of 3. Part 2 finds where the sparsity comes from. Part 3 asks what policy gradient can reach.

• Reinforcement learning (RL), on-policy distillation and supervised fine-tuning (SFT) on near-policy data change only a small fraction of parameters. Small and sparse are different claims.
• On the sampler’s own samples the score has zero mean. So the expected gradient is a covariance, the Fisher is a variance, and the Fisher caps the gradient for any reward.
• At one position the Fisher is the variance of logit sensitivities: whatever moves every candidate together counts for nothing, so it is small when the sampler is confident or its likely candidates react alike.
• None of this needs on-policy sampling. It needs a score centered under the sampling distribution.

The figures step through like slides. Use → or ▶, and step back with ←.

1. Small is not sparse

Why the update is small, or carries little, does not explain why it lands on a few coordinates.

RL fine-tuning changes only 5–30% of a model’s parameters, across seven algorithms and ten models (Mukherjee et al., 2025). On-policy distillation and SFT on near-policy data behave the same way (Zhang, Mukherjee & Peng, 2026). Two common explanations are that RL only reweights answers the model can already produce, and that a scalar advantage with $B$ possible values carries at most $\log_2 B$ bits per episode (Information Bandwidth). The first says the update is small, the second that it carries little. Neither makes it sparse: a hundred million parameters each moving a little is small and dense, and can carry those few bits just as well. The question is which coordinates move, and why the same few can be found before any reward is seen.

2. The gradient is a covariance, the Fisher a variance

On the model’s own samples the score averages to zero, so the expected gradient is a covariance and the Fisher is a variance.

$$ \mathrm{score}_i=\frac{\partial\log\pi_\theta(y\mid x)}{\partial\theta_i}, $$ $$ \mathbb{E}_{y\sim\pi_\theta}\big[\mathrm{score}_i\big]=\sum_y\partial_i\pi_\theta(y\mid x)=\partial_i 1=0, $$ $$ G_i=\mathbb{E}\big[A\cdot\mathrm{score}_i\big]=\operatorname{Cov}(A,\mathrm{score}_i), $$ $$ F_i=\mathbb{E}\big[\mathrm{score}_i^2\big]=\operatorname{Var}(\mathrm{score}_i), $$ $$ |G_i|\le\operatorname{std}(A)\sqrt{F_i}. $$

Probabilities sum to one, so whatever a parameter does, the responses it raises are paid for by others, and weighted by probability the changes cancel. With the score’s mean at zero, the policy gradient $\mathbb{E}[A\cdot\mathrm{score}_i]$ is a covariance and the Fisher is a variance; Cauchy–Schwarz gives the bound. A baseline that depends only on the prompt changes nothing in $G_i$, since $\operatorname{Cov}(A-b,\mathrm{score}_i)=\operatorname{Cov}(A,\mathrm{score}_i)$. It changes only $\operatorname{std}(A)$ and the noise of the estimate, and the Optimal Token Baseline is the choice that minimizes that noise.

A large Fisher is necessary for a large gradient, not sufficient. After “She drank a cup of”, the output rows for tea and coffee have a large Fisher; if the reward does not care which one is chosen, their gradient is zero. Measured before RL training, the top 1% of coordinates by Fisher hold 47.40% of the top 1% by gradient magnitude, against 1% at random (Zhang, Mukherjee & Peng, 2026). The Fisher says where learning can happen; the reward decides where it does.

The response space is finite (finite vocabulary, bounded length), so every exchange of a derivative and a sum below is exact. Probabilities are positive, which softmax sampling at temperature 1 without top-k or top-p guarantees. The reward $r(x,y)$ and the prompt distribution do not depend on $\theta$.

$$ \mathbb{E}_{y\sim\pi_\theta}[\mathrm{score}_i]=\sum_y\pi_\theta\,\frac{\partial_i\pi_\theta}{\pi_\theta}=\partial_i\sum_y\pi_\theta=0, $$ $$ G_i=\partial_i\,\mathbb{E}_x\Big[\sum_y\pi_\theta\,r\Big] $$ $$ =\mathbb{E}_x\Big[\sum_y\pi_\theta\,\mathrm{score}_i\,r\Big]=\mathbb{E}[r\cdot\mathrm{score}_i]. $$

Subtracting $\mathbb{E}[r]\,\mathbb{E}[\mathrm{score}_i]=0$ turns the product into a covariance, and the same step turns $\mathbb{E}[\mathrm{score}_i^2]$ into a variance. Both hold per prompt, so $G_i=\mathbb{E}_x[\operatorname{Cov}(r,\mathrm{score}_i\mid x)]$, which is why a baseline $b(x)$ drops out.

The full Fisher $F_{ij}=\mathbb{E}[\mathrm{score}_i\,\mathrm{score}_j]$ is also the expected curvature: $\partial_j\mathrm{score}_i=\partial_i\partial_j\pi_\theta/\pi_\theta-\mathrm{score}_i\,\mathrm{score}_j$, and the first term has mean $\sum_y\partial_i\partial_j\pi_\theta=0$, so $F_{ij}=-\mathbb{E}[\partial_i\partial_j\log\pi_\theta]$. It must be estimated on responses the model samples. On data labels the same formula gives the empirical Fisher, a different quantity.

3. One position: the Fisher is a spread of sensitivities

At one prefix, a parameter’s token Fisher is the variance of how strongly it moves each candidate’s logit.

$$ \frac{\partial\log\pmodel(a\mid s)}{\partial\theta_i}=\kappa_{i,a}-\bar\kappa^{p}_i, $$ $$ \bar\kappa^{p}_i=\sum_a\pmodel(a\mid s)\,\kappa_{i,a}, $$ $$ f_i(s)=\operatorname{Var}_{a\sim\pmodel}\big[\kappa_{i,a}\big] $$ $$ =\tfrac12\sum_{a,b}\pmodel(a)\,\pmodel(b)\,\big(\kappa_{i,a}-\kappa_{i,b}\big)^2 . $$

The softmax subtracts a mean: the token score is how strongly $\theta_i$ moves the drawn token’s logit, minus how strongly it moves the average candidate. Without that subtraction the second moment would be $f_i(s)+(\bar\kappa^{p}_i)^2$. The removed part is the common component, which raises every logit together and changes no probability. The pairwise form says the same thing: only differences between likely candidates count. So a parameter has a small token Fisher exactly when it acts about equally on the candidates the model finds likely, either because the model is confident or because those candidates react alike.

4. Along the response: the Fisher adds up

Across positions the cross terms average to zero, so the Fisher is a sum of token Fishers.

$$ \mathrm{score}_i=\sum_t\mathrm{score}_i^{(t)}, $$ $$ \mathbb{E}\big[\mathrm{score}_i^{(u)}\,\mathrm{score}_i^{(t)}\big] $$ $$ =\mathbb{E}\Big[\mathrm{score}_i^{(u)}\;\underbrace{\mathbb{E}\big[\mathrm{score}_i^{(t)}\mid s_t\big]}_{=0}\Big]=0\quad(u\lt t), $$ $$ F_i=\mathbb{E}\Big[\sum_t f_i(s_t)\Big]. $$

Here $\mathrm{score}_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{p}_i(s_t)$ is the token score at position $t$. Given the prefix $s_t$, the next token is drawn from $\pmodel(\cdot\mid s_t)$, so its score averages to zero. For $u\lt t$ the earlier token is part of that prefix: its score is already fixed and comes out of the inner expectation. The cancellation holds in expectation, not sample by sample. It makes the Fisher grow with length but treats every parameter alike, so it sets the scale, not which coordinates are large.

5. What this needs: a centered score

None of this needs on-policy sampling. It needs the score centered under the distribution that drew the samples.

$$ c_i^{(t)}=\kappa_{i,y_t}-\bar\kappa^{\mu}_i(s_t),\qquad c_i=\sum_t c_i^{(t)}, $$ $$ \bar\kappa^{\mu}_i=\sum_a\smu(a\mid s)\,\kappa_{i,a}, $$ $$ G_i=\mathbb{E}_{\smu}\big[A\,c_i\big]=\operatorname{Cov}_{\smu}(A,c_i), $$ $$ |G_i|\le\operatorname{std}_{\smu}(A)\sqrt{F^{\mu}_i}, $$ $$ F^{\mu}_i=\mathbb{E}_{\smu}\Big[\sum_t\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}(s_t)\big]\Big]. $$

Off-policy, the token was drawn by an inference engine or a stale model, and under that sampler the softmax score averages to $\bar\kappa^{\mu}_i-\bar\kappa^{p}_i$, not to zero. Subtracting it at every prefix is score centering. Sections 2–4 then go through word for word with $\smu$ in place of $\pmodel$, and on-policy the centering comes free. This is the gradient of the covariance loss: the drawn token is raised and every token is lowered by how often the sampler draws it. It needs the sampler’s full next-token distribution at every prefix, which engine log-probabilities and stale models provide and human demonstrations do not.

For any sampler $\smu$ whose full next-token distribution is known at every prefix:

1. Zero conditional mean, by construction: $\mathbb{E}_{a\sim\smu}\big[\kappa_{i,a}-\bar\kappa^{\mu}_i\big]=0$.
2. Cross terms vanish: for $u\lt t$, $c_i^{(u)}$ is fixed by $s_t$, so $\mathbb{E}_{\smu}\big[c_i^{(u)}c_i^{(t)}\big]=\mathbb{E}_{\smu}\big[c_i^{(u)}\,\mathbb{E}_{\smu}[c_i^{(t)}\mid s_t]\big]=0$.
3. The Fisher is a variance: by 2 only diagonal terms survive, and by 1 each is $\operatorname{Var}_{a\sim\smu}[\kappa_{i,a}(s_t)]$. By 1 and the tower property $\mathbb{E}_{\smu}[c_i]=0$, so the second moment is the variance.
4. The update is a covariance: $\operatorname{Cov}_{\smu}(A,c_i)=\mathbb{E}_{\smu}[A\,c_i]=G_i$, and Cauchy–Schwarz gives the bound.

On-policy, $c_i=\mathrm{score}_i$ and $G_i$ is the gradient. Off-policy, $G_i$ is the expected update of the centered estimator: the gradient of the covariance loss, not in general of the expected reward.

References

• Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
• S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
• M. Marek, M. Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. arXiv, 2026.
• Y. Li, J. Xu, et al. The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL. ICML 2026.
• Y. Li. Information Bandwidth in Reinforcement Learning. Blog, 2025.

Citation

@online{li-2026-gradient-is-covariance,
  title  = {The Gradient Is a Covariance},
  author = {Li, Yingru},
  year   = {2026},
  month  = oct,
  url    = {https://richardli.xyz/post/gradient-is-covariance/}
}

Krea 3: Frontier Image Editing Model

Krea 3: 10× Faster, 3× Cheaper

Krea 3 is an image editing model built for production. It changes only what you ask, finishes an edit in under 1s, and costs $0.06 per image.

Price per image

• Krea 3 $0.00
• Grok Imagine 2.0 $0.00
• Nano Banana 2 $0.00
• Seedream 5.0 Pro $0.00
• Nano Banana Pro $0.00
• GPT Image 2 $0.00
• Ideogram 4.5 $0.00

Seconds per edit

• Krea 3 <0s
• Grok Imagine 2.0 0s
• Nano Banana 2 0s
• Seedream 5.0 Pro 0s
• Nano Banana Pro 0s
• GPT Image 2 0s
• Ideogram 4.5 0s

We distribute through the world’s leading organizations, helping them advance their industries.

One model, Every brief

Edit one region, Keep everything else

Draw a box and say what changes. Krea 3 redraws inside it and leaves the rest of the image as it was.

Prompt in #FFC12E, Get that exact color

Name a hex code and Krea 3 matches it on the product, and only on the product.

Up to eight references, One image

Point at people, products, places and styles by number. Krea 3 puts them in one picture.

A new scene, The same character

Change the styling, setting, and mood while keeping the face familiar. Explore character consistency across a sequence of edits.

Native 2K

Every detail holds

Krea 3 renders 2K natively.

Made for designers: Posters, packs, brands, covers

Swap the product, the photo or the brand and keep the layout and type.

Send a photo, then a voice note. Krea 3 answers with the edit before you finish your next thought.

Just say it, Edits as fast as you talk

Only what you asked, Not one pixel more

“Change only her coat to cobalt blue.” Orange marks every pixel each model changed.

Krea 3, inside your tools: Photoshop to Blender

Frequently asked questions

What is Krea 3?

Krea 3 is Krea's new image editing model. It is built for production: product-consistent scene changes, object removal and variants at volume.

Is Krea 3 just a smaller model?

No. Krea 3 is a new model trained for editing. It is not a distilled copy of a larger model. It spends its compute on the region you change, not on the whole image.

How is Krea 3 so fast and inexpensive?

Most edits change a small part of the image. Krea 3 finds that region first and renders only what changes.

Are these prices temporary or subsidized?

No. The launch price is the list price. We expect it to go down, not up.

What is Krea 3 good at?

Best at product scene changes, object removal and photo cleanup.

Can I self-host Krea 3?

Yes, on enterprise plans. Krea 3 runs on your own GPUs in your cloud. Talk to us about license terms and hardware sizing.

How do I test it on my own images?

Send us one workload. We run it next to what you use today and share the outputs, acceptance rate and cost.

How do I get started?

Call Krea 3 through the Krea API, or try it in the Krea app today. For volume pricing, evaluate your workload with our team.

AI Gateway Production Index — October 2026

Every month, AI Gateway routes tens of trillions of tokens between production applications and AI labs. That traffic gives us a view of what AI usage actually looks like in today's enterprise, and we publish it here monthly.

October 2026 summary

• Open-weight models now process more than seven in ten gateway tokens and account for nearly one in four dollars. Their share of spend has more than quadrupled since June.
• September’s gateway spend was about 13 times December’s, while the average price per token was 47% lower. Almost one in three teams changed their most-used lab in September, an index record.
• Nearly 70% of September’s gateway spend on Chinese models went through US-based providers, up from less than 10% in December.
• Jev was the fastest-adopted model in AI Gateway history, reaching 35% of paid teams in its first two weeks, but it accounted for just 0.27% of gateway spend over that same period. Opus 5.5 took more than one in ten gateway dollars in its own first two weeks, the largest spend share of any September model launch.

Frontier spend up 900%, spend share down 22%

In August, open-weight models processed the majority of AI Gateway token volume for the first time. The money is now following. Open weight’s share of spend has more than quadrupled over the last three months, from one in twenty gateway dollars in June to nearly one in four in September.

Open weight's growth is coming from both ends of its price range, and it mirrors the wider gateway. The cheapest models drive token growth, while the expensive models drive spend.

Led by Kimi K3, models priced at least four times the open-weight average cost per token accounted for about two-thirds of the increase in open-weight spend.

In contrast, DeepSeek-v4.1-Flash alone processed nearly half of all September tokens but took just 2.6% of spend. Its price per token was about a fifth of the open-weight average.

Customers are increasing budgets on both sides of the market. Since December, spend on frontier models has grown tenfold; spend on open-weight models has grown more than seventyfold.

In September, 68% of teams that put more money into open-weight models did the same for closed-weight models. And more than two in five teams already paying for frontier models spent more on them than in August, the highest share this year.

Spending on the frontier is still growing, but spending on alternatives is growing faster. A company can spend more with a frontier lab while relying on it for fewer workloads, so the lab gets more money, but a smaller part of the customer’s business.

AI spend increases 13x while the cost per token halves

Market share is changing hands inside a market that is getting much bigger and cheaper.

September's gateway spend was about 13 times December's, an increase of 1,180%, while the average price per token fell 47%. The median paid customer active in both December and September more than doubled its spend.

Teams are spending more, but labs have to keep winning their business. In September, nearly one in two teams changed their most-used model. And almost one in three changed their most-used lab, an index record.

Even teams with lower token volumes use a wide mix of models. Since December, teams that processed at least 10 million tokens over that period used an average of 15 different models. The average rose to 26 for teams processing at least 100 million tokens, and 42 for those processing at least a billion.

The model is becoming a replaceable part of a much larger investment. Companies can continuously improve their AI products and workflows while changing which models power them. They can spend more on AI without becoming more dependent on any one lab.

Chinese models are big business for American providers

More than two-thirds of September’s gateway spend on Chinese-origin models went to US-based providers, up from less than a tenth in December. The Chinese lab builds the model; the American provider runs it and sells access.

DeepSeek’s models processed 58% of all gateway tokens for 6.6% of spend. More than four in five of those tokens were served by DeepSeek itself, yet two in five dollars spent on its models went through US-based providers.

Moonshot AI, the Beijing-based lab behind Kimi, accounted for 3.6% of gateway tokens and 13.2% of spend in September. US-based providers served nine in ten of those tokens.

Z.ai, also based in Beijing and the creator of GLM, accounted for 7.2% of all tokens and 4.5% of spend. US-based providers served seven in ten.

Customers can use Chinese models through access sold by American providers. For teams that put more money into Chinese models in September, an average of 53% of their spending on those models went through US-based providers and 41% through China-based ones.

The same Chinese model can compete with US-based models while generating revenue for US-based providers.

September model launch scorecard: Jev, Opus 5.5, and GPT-6

Whether a model launch is successful depends on what you measure: how broadly it’s tried, how much usage and spend it receives, or what share of its initial users return. We ranked every model launch in September with at least two weeks of data by these metrics.

Jev is the most-adopted and most-called model

In its first two weeks, Jev reached more than a third of paid teams, nearly double the reach of Sonnet 5.5 in second place. This is the fastest adoption of any model in AI Gateway history.

Jev also received the most requests of any model in September, even though it launched halfway through the month. Of the teams that tried Jev in week one, 55.6% returned in week two, the fifth-highest retention rate.

Jev is a decision model. Compared with typical language-model requests, a Jev request is faster, cheaper and uses a fraction of the tokens. So although its adoption was fast and broad, it accounted for just 1% of the month’s tokens and 0.15% of spend.

Opus 5.5 takes more than one in ten gateway dollars

The launch that received the most dollars was Opus 5.5, with more than twice the share of any other September launch, followed by GPT-6 Astra and DeepSeek-v4.1-Flash.

Opus 5.5 caught on quickly, with almost nine in ten dollars coming from teams already using Opus 5. Existing customers were trying the new release while the wider Opus family was losing market share.

Across all versions, Opus accounted for almost a third of gateway dollars in September, down from one in two in August. Anthropic’s overall share fell below half of total gateway spend for the first time, dropping from 63.7% to 43.9%.

Luna brought users back; Astra brought spend

GPT-6 Luna reached nearly three out of every twenty paid teams in its first two weeks, in third place behind Jev and Sonnet 5.5. But its return rate was the highest of the month; almost two-thirds of the teams that tried it in its first week came back in the second.

GPT-6 Astra reached a smaller audience but took more budget. In its first two weeks, it accounted for about one in twenty gateway dollars; Luna accounted for less than one in a hundred. Luna reached almost three times as many teams while Astra captured more than five times the share of spend.

OpenAI is attracting more customers with Luna, but capturing far more money with Astra, even though fewer teams use it.

About this report

This report uses anonymized, aggregate traffic routed through Vercel AI Gateway through September 2026.

• Token volume includes input, output, reasoning, cached-input and cache-creation tokens.
• Spending is estimated at published list prices; actual bills may differ. Average price per token is estimated spending divided by token volume.
• Open-weight classifications follow the current AI Gateway model list, a broader definition than in earlier reports.
• Switching rates compare paid teams with at least 10 million paid tokens in both months. “Most-used” is measured by paid tokens.
• Provider location means company headquarters.
• Launch metrics cover external paid teams during each model’s first 14 days. Reach uses paid teams active from the start of the launch month through the data cutoff. Spend share uses total paid gateway spending during the model’s window. Return rate is the share of first-week teams that used it again in week two.
• Model counts cover December–September, with each version counted separately. Team-size thresholds use total input and output tokens over that period.
• All figures use the most recent data and protocol available; prior months may be revised as methodology is updated.

FLUX 3 Image Klein

Today we are releasing FLUX 3 Image Klein, a 5.6B parameter open weights model for text-to-image generation, image editing and multi-reference generation with up to 10 reference images. It shares core capabilities with the full FLUX 3 Image model: native 4K generation, pixel-perfect editing, and bounding boxes that control where elements are placed or confine edits to a defined area.

FLUX 3 Image Klein builds on FLUX 3's shared learnings across images, video and audio. Images contribute spatial detail, video adds motion and physical dynamics, and audio provides information about events and interactions. Together, these signals help the model generate images that feel grounded in the real world.

The release includes a base model for fine-tuning and a step distilled model for low-latency generation, along with FP8, INT8 and NVFP4 builds developed with NVIDIA support, under the FLUX Kommunity License.

Leading open weights quality with fewer parameters

In internal evaluations, FLUX 3 Image Klein ranks first in text-to-image, multi-reference generation and image editing, ahead of larger open weights models. It also rates above the closed models in the comparison, Ideogram V4.5 and FLUX.2 [pro]. At 5.6B parameters, it is the smallest open model in the comparison, against 7B for Qwen-Image 2.1 and 12.9B for Krea 2. Compared with FLUX.2 [klein] 9B, it gains 91 Elo in text-to-image and 70 in image editing, with 38% fewer parameters.

FLUX 3 Image Klein was also optimized for consumer GPUs. Fewer parameters make it practical to run locally and cheaper to adapt, and teams with data sovereignty or local deployment requirements can evaluate and adapt the model on their own infrastructure. This is also key for on-device image editing, allowing consumers to edit images directly on their favorite laptops. The generator weights require 11.2 GB in BF16 and 5.6 GB in FP8. With the INT8 build, FLUX 3 Image Klein runs on NVIDIA GeForce RTX GPUs with as little as 8GB of VRAM, through ComfyUI or Diffusers, extending what FLUX.2 [klein] started to even smaller hardware. With the distilled model and the INT8 build, it generates a 1MP image in about 4 seconds on an NVIDIA GeForce RTX 5090 and under 9 seconds on an RTX 4070 through ComfyUI.

The base checkpoint is built for fine-tuning: Diffusers and Ostris's AI Toolkit support LoRA training from day one, and NVIDIA's NeMo AutoModel ships recipes that work directly from the released checkpoints.

For even faster production workloads that need lower latency, we also work with teams on specific customizations that get you there.

On the speed–quality frontier

FLUX 3 Image Klein also extends the speed–quality Pareto frontier. Through the BFL API, it delivers lower latency than Qwen-Image 2.1, FLUX.2 [pro], Ideogram V4.5 and Krea 2, and the highest rating at its latency and above. In the charts below, the violet line traces the highest rating available at each latency and the gray line shows the previous frontier.

Latency is measured end to end, from request submission to the finished 1 MP image, including queueing. FLUX 3 Image Klein was timed on our servers. Other models were timed from the client, which includes polling and network time. Krea 2 was timed on its Medium tier; its Large tier took 31.1 seconds for text-to-image.

Get started

FLUX 3 Image Klein is available today as open weights and through the BFL API.

Try it

• BFL API: the distilled model is available as flux-3-image-klein, with prompt upsampling and pixel restoration, which preserves unedited regions of an image exactly. Pricing starts at $0.009 per 1K image.
• Playground or Hugging Face.

Build with it

• Weights on Hugging Face: base and distilled.
• Reference inference code, including a Gradio demo for bounding-box control.
• Try it out on Hugging Face Spaces.
• Day-one support in ComfyUI, Hugging Face Diffusers, TensorRT-LLM, SGLang Diffusion and vLLM-Omni.
• LoRA training with Ostris's AI Toolkit and recipes in NVIDIA NeMo AutoModel.
• Trained on NVIDIA GB200 NVL72 and optimized for NVIDIA's accelerated computing platform, serving through NVIDIA TensorRT LLM, vLLM-Omni and SGLang Diffusion.
• On NVIDIA GB300, the distilled model generates a 1MP image in under 0.6 seconds in NVFP4. Serving a LoRA adds around 5% to generation time.

License

The weights are available under the FLUX Kommunity License, which allows free commercial use for individuals and companies with annual revenue under $5M, alongside research and non-commercial use. Larger organizations can deploy commercially through our self-hosted licensing options or API.

Safety

In an effort to make AI technology both open and safe, we implement multiple safeguards throughout the model development lifecycle to combat misuse. These include dataset filters during pre-training to limit exposure to harmful content, and targeted fine-tuning during post-training to suppress harmful behaviors, making it progressively harder to elicit harmful content. Ahead of the release, the FLUX 3 Image Klein checkpoints were rigorously red teamed internally and evaluated by our external safety partner Cinder, demonstrating substantially fewer vulnerabilities than other popular open weights models. More on our safety methodology is available here.

FLUX 3 Image Klein is the second open weights release from the FLUX 3 family, following FLUX 3 Action. More will follow.

Learn more

• FLUX 3: Real World Models, the research behind the FLUX 3 family
• FLUX 3 Action, our open weights 7B model for robot control

Citation

If you use FLUX 3 Image Klein in your work, please cite:

@misc{bfl2026flux3imageklein,
  author       = {{Black Forest Labs}},
  title        = {{FLUX} 3 Image Klein},
  year         = {2026},
  url          = {https://bfl.ai/blog/flux-3-image-klein},
}

My dear friends, I’d like to show you my latest toy project Introducing Handwired: turn your regular JavaScript code into a neural net! - no training - exact outputs vs the JS program - fully differentiable All runnable in a browser, without GPU

Full article content is not available for inline reading.

Read the original article →

After my mother passed away I ended up with a big box full of old film negatives containing photos from childhood family trips and similar. So I decided to buy a negative photo scanner and found that the Plustek 8200i was supported under Linux so I bought that. Only to discover that the chipset in the scanner had changed to one that wasn’t supported under Linux anymore. So I had some success making Linux configuration tools for some of my hardware using Claude code, like my Dell Camera controller app so I thought, maybe Claude can help me get a working SANE driver. I basically got stuck and got nowhere for the longest time and even basically gave up on the project for a few Months.

Then I saw a post on X from Tom Crider aka Glorious Eggroll talking about how he had managed to fix a mouse driver using AI tooling. He basically described how he set it up so that the AI agent had full control of a Windows instance to capture data needed to fix the driver. So I pasted his tweet into Claude and asked it to help me set up a windows virtual machine to do the same things Tom as doing for my scanner. I think the combination of having the ability to run scans under windows and capture the USB data while doing so worked wonders, probably combined with improvements to Claude code made a huge difference. Within a few hours I had my first ever working scan under Linux, sure with some quality issues, but 100% recognizable as the image in the negative. Some days later I had a near perfect driver, including support for Infrared scanning ready to go, only to discover that in the time between me giving up and then trying again someone had committed code to SANE to support the scanner already…..it happens sometimes. I also used Claude to write an application for doing negative scanning, which I named Negative Vibes, thought that was a cute name.

Anyway, with the success of my scanner effort I thought, I wonder if I can use this to finally fix the last issues with MIPI camera support on my laptop. So I told claude to re-use the harness I had created and try to fix the MIPI camera on my laptop by updating the kernel driver and libcamera as needed. I started the job in the morning and by the end of the day I had a much improved situation with patches for both the kernel and libcamera ready.

So I thought that this might be worthwhile sharing with the world, and while anyone could start from scratch I did ask Claude to make the harness generic and document it to make the pathway for anyone else who wants to try this on their hardware a little easier. Writing a driver for undocumented hardware has always been tedious. You capture bus traffic, stare at hex dumps, and form hypotheses about what each register or control transfer does. You then test them one at a time, usually for weeks. None of it is conceptually hard, but it takes a lot of patience and skills.

That kind of work suits an AI model well. It can read thousands of lines of USB capture, notice that a sequence repeats with one byte changing, and propose what that byte means. It can write the test to confirm it, and it doesn’t get bored. The human still decides what to build and checks that it works on real hardware, but the slog shrinks a lot.

hw-harness

Be aware that I did not use hw-harness in this form for my projects as this was basically bespoke code at that time, so this project is an attempt to make the harness more generic and shareable from the scripts I was using while doing the work. What hw-harness is meant to help your Claude or Codex session with:

1. Record a reference session. Run the vendor’s Windows driver in a KVM virtual machine, with the real device passed through to the guest.
2. Capture from outside the VM. Record the traffic on the Linux host with usbmon. It sits below the passthrough layer, so the guest can’t see it.
3. Repeat the same operation on Linux with your own driver, and capture that the same way.
4. Diff the two traces. Both come from the same instrument, so they compare directly. The differences are your list of driver bugs.

For devices with no bus to tap, such as an internal MIPI camera, the harness uses Windows-side inspection instead. That means INF files, registry data, ACPI tables and device properties.

The approach should be fine in the sense it doesn’t rely on copying any code instead it observes what the hardware does and only extract configuration data from the Windows drivers, so every constant in the new driver traces back to something recorded.

So for those interested I have 3 short descriptions of the projects I did and what was needed.

Project 1: a SANE backend for the Plustek OpticFilm 8200i

The first project is a SANE backend for the newer Plustek OpticFilm 8200i. It uses the Genesys GL128 controller (USB 07b3:1825) and had no Linux support. The older model with the same name uses a GL845 and is already handled by SANE.

The backend supports:

• Color, gray, infrared and color+infrared modes
• Resolutions from 300 to 7200 dpi, at 8 or 16 bits per channel
• Dust and scratch removal from the infrared channel
• An exposure multiplier for dense negatives

It’s packaged as a Fedora RPM, built automatically on every push. I mostly tested it with my own Negative Vibes application as there are not that many options under Linux for negative scanning.

Upstream SANE’s genesys backend also gained support for this scanner in September 2026, so I compared the two. Both resolve the same fine detail at 3600 dpi. Mine is faster at full resolution (34.6 s against 54.4 s) and supports infrared dust removal. Upstream offers more resolution steps. The webpage has an interactive side-by-side comparison you can check out. These are single-unit, single-frame results, so treat them as one data point. Check out the interactive page I made showing the current state of the two drivers, hopefully the discoveries I made can be used by the SANE project to fix some smaller items in the upstream driver. Anyway, I put this project up more as a proof of concept due to that since I expect everyone including myself will use the upstream driver once it reaches parity with this one.

Project 2: Negative Vibes

A driver alone isn’t much use if scanning negatives is still painful, so I built Negative Vibes. It’s a GNOME app, written in Rust, for scanning colour and black-and-white negatives and slides on any film-capable SANE scanner.

The scan is kept exactly as the scanner delivered it: 16-bit, linear, with the infrared plane. Everything else is a stack of non-destructive settings on top, so you can change the look and export again. It includes:

• Density-space negative inversion with film base and dye-crosstalk correction
• Roll-wide settings with per-frame overrides
• Infrared and software dust removal
• IT8 calibration through ArgyllCMS
• Export to TIFF, JPEG, PNG and DNG, with ICC profiles and metadata

It ships as a Flatpak that includes the scanner backend. It’s tested on the 8200i so far. The goal is to support any film scanner SANE supports. For that, SANE needs a standard way to expose infrared channels, which it doesn’t have yet.

Project 3: Fixing the MIPI camera on my ThinkPad

The camera in my Lenovo ThinkPad X1 Carbon Gen 14 (Aura Edition) uses a Sony IMX471 sensor behind an Intel IPU7. On Linux it produced dark images with heavy vignetting and a strong yellow-green cast. mipi-fixes-x1-aura collects the fixes. It’s only relevant to this laptop, which is why it isn’t one of the bigger packaged projects, but I have already submitted the kernel patches to Kate Hsuan and the libcamera patches is being reviewed by Wim Taymans for submission to Libcamera.

Kernel (IMX471 driver):

• A crop-selection bug returned an error before storing the crop rectangle, so libcamera assumed the full pixel array.
• The test-pattern register was declared with the wrong width.
• The analogue gain cap was too low, which cost about 1.8 stops of sensitivity.

libcamera software ISP:

• Lens shading correction was missing entirely, leaving the corners about 2.7 stops darker.
• White-balance statistics were summed at the sensor’s native bit depth against an 8-bit black level. That skewed white balance, and the fix applies to any RAW10/RAW12 sensor.

In a controlled low-light test, frame brightness rose from 31.1 to 90.9. The red/green and blue/green ratios moved from 0.890 and 0.672 to 1.047 and 0.823, close to a USB webcam reference. Blue is still about 17% short in low light, which I put down to grey-world white balance. Fedora 44 RPMs are on the releases page.

What I learned

• Reference captures are a big step. Having the AI compare a known-good trace against a failing one turns a vague “it doesn’t work” into a concrete list of differences. For example for the MIPI camera having a USB camera also connected so Claude could compare the two captures was critical.
• The AI is good at the tedium. That covers decoding captures, writing probes and cross-checking constants. I still have to be the one holding the physical hardware.
• Real hardware is the only judge. Every claim above was checked on the actual scanner or laptop.

If you have hardware with no Linux driver, try hw-harness. Thanks again to Glorious Eggroll for the inspiration.

NOTE: I am not claiming that my scans or MIPI camera is perfect now, but the scanner gives me decent looking scans and the MIPI camera looks a lot better than before. So over time I am sure we be able to tweak and polish further, but I am happy that it allowed me to move the needle despite lacking the in-depth knowledge for this type of work.

Redis is an in-memory database. Salvatore Sanfilippo started it in 2009, and it is now one of the most widely deployed pieces of server software. It is often used as a cache in front of a slower database.

You talk to it over the network with short text commands such as SET user:42 "Alice" and GET user:42. Each key maps to a value, and the value can be more than a string: Redis has lists, hashes, sets, sorted sets, streams and more, each with its own commands.

We often represent sets of integers as bitmaps (or bitsets). The simplest bimtap is a single machine word of, say, 8 bits. It can be used to represent all sets of integers smaller than 8. By adding more words, you can represent sets of integers smaller than 16, or any value you like. You can then do fast operations over bitmaps such as intersections and unions. They translate into a few bit-wise operations such as AND, OR, XOR.

Redis has supported bitmaps for a long time. You call SETBIT key offset 1, and Redis stores a plain bitmap inside an ordinary Redis string: bit i of the bitmap is bit i of the string. It is simple and fast. But the string takes one byte per eight bit positions, up to the largest offset you ever set, however many bits are actually set. Setting a single bit at offset four billion allocates 512 MB, and every BITCOUNT or BITOP has to scan all of it.

A compressed bitmap avoids this cost: it stores a set of integers in a form that takes little room when the set is sparse, and it computes counts, intersections and unions without decompressing. Roaring is probably the most widely used compressed bitmap format. Rather than compressing a long string of bits, Roaring adapts its storage to the data: sparse regions are stored as short lists of integers, dense regions as plain bitsets, and long stretches of consecutive values as runs. It is often both smaller and faster than an uncompressed bitset. Roaring is used by many systems you already know: Elasticsearch, Apache Spark, ClickHouse …

Antonio Viggiano has brought Roaring into Redis itself. It should ship in Redis 8.12. There is a new type: TYPE key returns bitmap and OBJECT ENCODING key returns bitmap-roaring. It is opt-in: CONFIG SET bitmap-default-roaring yes.

I built Redis at the merge commit and ran the server on one core of an Intel Xeon Gold 6548N. For each workload, I create two bitmaps, a and b, once as strings and once as Roaring bitmaps, with the same content, and I time the commands. My workloads:

• 0.1% random: 100,000 random ids out of 100 million (think: the users who used a given feature today);
• 50% random: 50 million random ids out of 100 million;
• runs: ten runs of one million consecutive ids in a range of 100 million (think: cohorts of users who signed up in the same week);
• hashed 32-bit ids: 100,000 random 32-bit values, so the string is 512 MB.

You can see that Roaring can be much faster than a standard uncompressed bitmap in some cases. The hashed ids are even more compelling. The string must span all four billion bit positions, and each BITOP scans 512 MB.

Compressed bitmaps like Roaring are not always better. There are many tradeoffs. But turning them on will soon take one configuration setting. Check out the Roaring bitmap website for more information.

The social-proof machine for AI techniques and tools is circular, and it’s getting absurd. A rant.

The social proof machine

On the way to the office the other day I was catching up on AI coding videos, and I heard something I’d heard before, but this time it was the last straw. A popular agentic coding influencer said: “this skill is absolutely blowing up on social media!”.

OK… so?

We’re in this strange era where a lot of people are talking about AI coding in ways that aren’t backed up by anything except social proof. Other people are talking about it, other people are doing it, so it must be good.

In particular, there’s a flood of new frameworks, skill repositories, plugins, and whole agentic apps that are supposed to replace your entire office with a bunch of virtual employees. They have tens of thousands of GitHub stars. People get on YouTube and say, “Look, this thing has tens of thousands of GitHub stars, so this is the new hotness.” Then you start to dig into it, and you realize there’s no there there. There’s zero evidence anyone is using these tools for anything other than making breathless videos about how they tried it for a week, and it replaced a team that didn’t exist in the first place, and they felt like they were getting so much done.

The tools that do nothing (except accumulate stars)

What I see a lot of lately is tools that promise to make agentic coding more effective, or to hold down your token count, launched with very excited AI-generated copy about how well they work. Then somebody does a little digging, and it turns out they have no benefit whatsoever.

There was one tool for reducing token count and making agents more efficient. The folks at JetBrains did a rigorous analysis and found that at best it was zero gain, and from there it went downhill: it actually increased token counts and cost. Then somebody else did their own research and found the exact same thing. I’ve seen a few of these now. It’s not just that the claims are overblown; there’s no benefit at all. And yet the repositories sprang up overnight like mushrooms, collected tens of thousands of stars, and people treat those stars as a measure of legitimacy.

Stars used to mean something

They used to. A development tool could only pile up that many GitHub stars if it was genuinely in widespread use across a lot of working shops over a long period of time. Now plenty of the people promoting these things on social media and YouTube aren’t working engineers with a deep background. AI tools have opened the field to a lot of people who weren’t previously involved in software development, or only tangentially. Don’t get me wrong, I think that’s great. I’m not interested in putting walls around the ivory tower, and I love it when something brings a whole influx of new people into software.

The downside is that when a technology lets anyone look legitimate… then whoever has the best social media skills, the best production values, and the best on-camera charisma can make something look real when it isn’t.

It’s compounded by the fact that these tools are very good at making you feel busy and effective. The agents are happy to tell you how smart and correct you are and how much work they’ve done. You can have a little virtual office full of virtual employees, all looking very busy. And the agents can write voluminous, professional-looking copy about how great the tool is that you published overnight on a GitHub page. And even build you a professional-looking website for it. It’s not at all obvious that there’s no “there” there.

OK, so what?

If you’re someone using the ramp of agentic coding to get into software development, what do you do with this information? I could say you should follow and subscribe to old graybeards like me. And you should; some of us actually do have useful things to contribute. Also, I have a family to feed, so check out Graceful.Dev.

But beyond that, the key is metrics. Sit down, step back, and figure out what defines actual effectiveness and actual delivery. Hard metrics for whether you’re accomplishing things, whether what you’re trying is making the process better, whether you’re pushing your goals forward. Which metrics depends on your position:

• In a shop that already has products, it’s probably internal metrics: Is the bug count going down? Are we making our delivery goals? Are new features getting in front of customers, and are they buggy? Is the feedback good or bad? Are we gaining customers or losing them?
• If you’re building your own thing: Where are you on the line to market? Do you have a hard goal? Do you know what needs to be in place before you get there?

There are many ways to define metrics, and plenty of books about it. I think we’re in an era where anyone involved in software development has to get a lot more literate about these business-facing measurements, because we have minions to do the grunt work we didn’t have before. Now you’re the one responsible for asking whether you’re actually progressing, and whether you’re getting better at progressing.

So when a new tool sounds exciting, figure out your evaluation criteria for deciding whether it actually helps. Agents are good at helping with this. Tell one you want to set up an eval. I did this recently for a work process that could be split across many lighter agents or a few larger ones. I had an agent construct an evaluation protocol to work out which approach performs better with current models, came to some hard conclusions, and rolled them into my standing guidance.

That’s what you have to do in an era of everything from well-meaning people who aren’t as rigorous as they could be, to total hucksters with 10,000 stars on their GitHub repositories. Become responsible for your metrics, your measurements, your hard data: what’s showing you that this is helping or not helping you reach your goals? That means knowing what your goals are, and that’s more essential than ever.

Spacelift Announced as the First Infrastructure Automation Platform to Integrate with Cursor Origin

Your engineers already push half their code through an AI agent before a human ever looks at it. That’s fine when the change is a function signature. It’s a different conversation when the agent is proposing a change to the load balancer sitting in front of production, or more importantly, your IAM policy. Someone still has to answer for what gets applied, and right now that someone is you.

The fix isn’t reviewing faster. It’s putting guardrails inside the workflow where agents actually propose changes, so a policy check and an approval step stand between any pull request and production, no matter who or what opened it. Spacelift already works this way for GitHub, GitLab, and Bitbucket. The next place that needs to happen is wherever AI agents do their work.

This is where Cursor Origin fits in. Origin is Cursor’s new version control platform for teams building with AI agents. Spacelift is proud to be the first infrastructure orchestration platform to integrate with Origin as a launch partner.

Meet the repo built for AI agents

Origin is Cursor’s alternative to GitHub: a place to host repositories, open and review pull requests, and keep code, pull requests, and AI agents in the same interface. It can mirror existing GitHub repositories at the click of a button, so teams don’t have to migrate everything manually on day one. This makes migration easy, or allows teams to use both so they can continue working even when providers are having issues.

Same guardrails, new source

Spacelift already runs on top of GitHub, GitLab, and Bitbucket. Policies trigger on push, plans run before anyone applies anything, and every deployment carries the guardrails you set up around it. The Origin integration extends that same model to infrastructure changes that start life inside Cursor.

Spacelift connects to Origin as a registered app authenticated with its own signing key rather than a person’s credentials. The app is scoped to source code and API access, not authentication, and it shows up right where a GitHub App would: in Origin’s settings and on the pull request view. The integration is available across all of Spacelift’s regions.

Whether a deployment change originates from an AI agent or a human operator, Spacelift enforces the exact same governance rules. Your OPA policies and your approval flow apply the same way to a change opened in Origin as one opened anywhere else. The agent can write the plan, but doesn’t get to skip the review.

Origin gets the same trigger model Spacelift already uses for GitHub: a push updates a tracked run, and opening a pull request kicks off a proposed run, etc. Every framework Spacelift supports today, Terraform (including Terragrunt), OpenTofu, Pulumi, CloudFormation, Ansible, and Kubernetes, works the same way through Origin. Nothing changes about what you can run. The only thing that changes is the VCS where it is run from.

The agent doesn't get to skip review

AI agents are already writing production infrastructure changes faster than most platform teams can review them manually.

In Spacelift’s 2026 Infrastructure Automation Report, 76% of infrastructure teams say they would apply AI-generated Terraform and HCL to production with little or no review, and 93% have already had at least one AI-caused infrastructure incident. Tools like Origin put those agents directly inside the pull request flow developers already trust. Letting agent-generated changes slip through a review process built for a slower era is exactly how teams get hurt, and among early adopters of agentic AI, the incident rate is already running at 33%.

Being first here means Spacelift customers don’t have to choose between adopting AI-assisted development and keeping production guardrails in place: The same plan, policy check, and approval step that stands between a person’s pull request and your infrastructure now stands between an agent’s pull request and your infrastructure from day one, rather than after the first incident forces the question.

“We built the governance model for Spacelift to travel with the pull request, not the platform it lives on. Origin was a natural extension: the same policies, the same approval flow, the same audit trail, now supported by an agentic first source rather than a human one as our software development undergoes a similar shift,” said Spacelift SVP Product and Design, Jonah Kowall

Enable governance for Origin

Connect Origin from Integrations in your Spacelift account, the same place you’d connect GitHub, GitLab, or Bitbucket. Every pull request Origin opens gets the same plan, policy check, and approval step your other repositories already have.

As you can see, Spacelift is dedicated to bringing you integrations with as many tools as possible, allowing you to have the flexibility to use whatever makes the most sense for your team and workflow. For more info on Cursor Origin, check out their launch announcement here and documentation here. You can connect Spacelift to your Origin repositories in the Integrations section of the platform. Take it for a spin, and let us know what you think.

Solve your infrastructure challenges

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation, and controls to simplify and accelerate the provisioning of cloud-based infrastructures.

Logpush delivers logs in batches as quickly as possible, enabling Cloudflare to provide information almost in real time. During our recent Birthday Week, we made Logpush available on all self-serve plans and Transformers generally available.

Those changes were among eight major updates to Cloudflare Observability and reflect our commitment to make every Cloudflare feature available to everyone. This post takes a deeper look at how Transformers work inside Logpush's export pipeline. Observability teams use Logpush to export Cloudflare logs to security information and event management systems (SIEMs), analytics platforms, and object storage. Before those logs reach their destination, teams may need to remove health checks, omit sensitive fields, or reshape records for dashboards and investigations. Today, this often requires a separate processing service that teams must deploy, scale, monitor, and maintain.

For example, a team that collects logs from multiple CDNs can add a static provider field to normalize records before they reach its analytics platform. Teams also pay to export, ingest, and store records they discard as soon as they arrive, so filtering them before delivery can reduce downstream data volume and cost.

With Transformers, that processing moves into the Logpush export workflow.Teams define a SQL query to filter records, redact sensitive data, and reshape the output by adding, renaming, removing, or nesting fields. Logpush applies the query before export, so each destination receives data in the form it needs. We made this workflow simple for customers by embedding a SQL transformation engine in Logpush's high-throughput export pipeline while keeping performance and resource use predictable.

How Logpush Transformers work

A Transformer is a SQL query written for a specific Logpush dataset. The query references fields from that dataset and defines which records and fields Logpush should export. Logpush applies the query to each batch before export. Records that do not match the WHERE clause are removed. For the remaining records, the output of the SELECT statement becomes the exported record. Transformers can change the structure and contents of each record, and the transformed results are exported in NDJSON format.

Customers can create, preview, and manage Transformers in the Cloudflare dashboard through Transformer Studio or programmatically through the API. After saving a Transformer, they can attach it to a compatible NDJSON Logpush job that uses the same dataset. Each update creates a new version, so customers can review how a Transformer has changed over time.

The following example reshapes a Cloudflare HTTP request log to match a team-defined schema used across CDN providers. It does not convert between serialization formats: both the input and output use NDJSON.

{
  "ClientIP": "203.0.113.42",
  "ClientRequestHost": "example.com",
  "ClientRequestMethod": "POST",
  "ClientRequestPath": "/api/checkout",
  "ClientRequestUserAgent": "Mozilla/5.0",
  "EdgeResponseStatus": 502,
  "EdgeStartTimestamp": "2026-05-21T15:00:00Z",
  "RayID": "8e2a1c60ef9e1c9a"
}

The Transformer renames and nests fields, redacts part of the client IP address, adds provider and error information, removes unused fields, and filters out internal and health-check traffic:

SELECT
  extract(epoch FROM to_timestamp(EdgeStartTimestamp)) AS unix_ts,
  regexp_replace(ClientIP, '\.\d+\.\d+$', '.XXX.XXX') AS redacted_ip,
  UPPER(ClientRequestMethod) AS method,
  EdgeResponseStatus AS status,
  RayID AS ray_id,
  'cloudflare' AS provider,
  named_struct(
    'host', ClientRequestHost,
    'path', ClientRequestPath
  ) AS request,
  EdgeResponseStatus >= 500 AS is_server_error
FROM http_requests
WHERE ClientRequestHost NOT IN (
  'internal.example.com',
  'health.example.com'
)
  AND ClientRequestPath NOT LIKE '/healthz%'
  AND ClientRequestPath NOT LIKE '/metrics%'

The resulting NDJSON record uses the normalized schema:

{
  "unix_ts": 1779375600,
  "redacted_ip": "203.0.XXX.XXX",
  "method": "POST",
  "status": 502,
  "ray_id": "8e2a1c60ef9e1c9a",
  "provider": "cloudflare",
  "request": {
    "host": "example.com",
    "path": "/api/checkout"
  },
  "is_server_error": true
}

How Transformers executes SQL with Apache DataFusion and Arrow

Cloudflare's data pipeline ingests more than one billion events per second. Its Logpush delivery service, which we call Bufstream, delivers more than 110 million Logpush records per second to customer destinations. It runs across hundreds of Kubernetes pods, each claiming a time slice of data from Buftee, Cloudflare's distributed log-based in-house message broker. This model distributes work horizontally across Logpush jobs. Job volume follows a Pareto-like distribution: most jobs operate at lower volumes, while a small number of high-volume jobs process millions of logs each second. Transformers needed to meet the same operational requirements as the rest of Logpush: reliable delivery, high throughput, and predictable resource use without introducing additional failure scenarios into the export hot path.

Our initial prototype ran customer-defined WebAssembly Transformers in Cloudflare Workers, which provides globally distributed, isolated compute. However, invoking a Worker would insert a network call into the Logpush export path. That remote dependency could introduce transient timeouts and retries, reducing delivery throughput and increasing delivery latency.

We decided to keep Transformer execution in-process within our delivery service to prioritize reliability and delivery throughput. We evaluated several SQL execution engines and selected Apache DataFusion, which already powers Cloudflare Basin.

Apache DataFusion is a Rust query engine built on Apache Arrow, with a SQL planner and vectorized, columnar execution engine. In a benchmark, executing a prepared Transformer against a synthetic 5,000-row http_requests batch from a single Go goroutine took about 3 ms, equivalent to roughly 1.6 million rows per second.

Attaching a Transformer to a job adds CPU work to its processing path. Bufstream uses per-job transform parallelism so high-volume jobs can execute transform batches concurrently. Embedding DataFusion in Bufstream lets those batches use the service's existing delivery infrastructure, including its scheduling, batching, and destination controls, without introducing a separate transformation tier or network hop.

Choosing DataFusion meant connecting Bufstream, which is written in Go, to an engine written in Rust. The next challenge was moving typed batches efficiently across that boundary.

The final design of the in-process Transformer pipeline is shown in the diagram below.

From records to columns

Bufstream receives compressed Logpush batches containing Cap'n Proto records from Buftee. For a job with a Transformer attached, Bufstream decompresses the batch, parses the eligible logs, and builds a typed Apache Arrow RecordBatch before executing the transform.

Apache Arrow is a language-independent, columnar in-memory format designed for efficient analytics on modern CPUs and GPUs. It represents flat and nested data as typed, contiguous columns. Arrow IPC gives Bufstream and DataFusion a standard way to exchange batches across the foreign function interface (FFI) boundary between Go and Rust, exposed through a C application binary interface (ABI) and called from Go using cgo, Go's mechanism for calling C code.

The Go and Rust runtimes do not share ownership of transform data. Go passes a pointer to its input Arrow IPC buffer to Rust only for the duration of the cgo call, and Rust does not retain that pointer after returning. Rust owns the transformed output buffer; Go copies it into Go-managed memory and then explicitly releases the Rust allocation. This makes ownership and lifetimes explicit across the language boundary, avoiding cross-runtime memory-management bugs.

Using Arrow IPC means the Rust DataFusion wrapper can use an Arrow StreamReader to read typed RecordBatch values, rather than parse individual rows from an intermediate format such as JSON and rebuild typed Arrow columns for execution. Field types, nullability, and nested values are represented consistently on both sides of the boundary, so DataFusion receives data in its native columnar model.

Crossing the Go/Rust boundary is not free: batches still need to be serialized, decoded, and managed in memory. Arrow IPC keeps that work batch-oriented and avoids per-record parsing and conversion. This makes the boundary a better fit for Logpush's high-throughput delivery pipeline while retaining schema validation against the Logpush dataset definition.

Every Arrow batch needs a schema describing its fields and types. Rather than infer that schema from the logs in a particular batch, Bufstream supplies a generated schema from each Logpush dataset's specification. This gives DataFusion a stable view of the available fields and types, enabling query validation. The table named in a Transformer's FROM clause selects which dataset schema the query uses. For example, this Transformer queries the http_requests dataset:

SELECT ClientIP, EdgeResponseStatus, RayID
FROM http_requests
WHERE EdgeResponseStatus >= 500

Executing the transform

When a customer saves a Transformer through the API or Transformer Studio, Cloudflare stores its SQL query in R2. Bufstream retrieves the query through a multi-layer cache and translates it into a reusable DataFusion logical plan that defines the scan, filters, projections, and expressions. Bufstream prepares this plan once per time slice of a job's data and reuses it for every Arrow batch in that slice. For every new Arrow batch, the Rust wrapper binds the batch as the plan’s table input, executes the query, and returns Arrow batches containing the selected records and fields.

Transformers are designed to fail closed. If a transform cannot be executed, Bufstream neither advances the job's delivery offset nor sends the untransformed batch. Delivering raw data in place of transformed data could violate the schema, filtering, or redaction assumptions of downstream pipelines, so Bufstream opts to re-attempt failed transforms instead.

To prevent a persistently failing job from consuming capacity needed by healthy jobs, Bufstream progressively delays retries and reports errors on the job. Jobs that remain in a failing state for an extended period are automatically disabled. Customers can monitor successful and failed transformations, data volume before and after transformation, and transformation duration in the Transformer Health view of the Health tab in Logpush.

Returning to the export pipeline

After execution, Bufstream converts the Arrow output into NDJSON logs. It batches, gzip-compresses, and uploads those logs to the configured destination. Destinations accept data at different rates and can temporarily slow down. Bufstream adjusts the number of upload goroutines while respecting destination concurrency limits.

By embedding SQL execution in Bufstream, Transformers use Logpush’s existing distributed infrastructure, batching, and destination controls. Cloudflare’s data pipeline handles that complexity, while customers create and apply a Transformer with only a few steps.

Get started

Open Transformer Studio from the Logpush page in the Cloudflare dashboard and follow these steps:

1. Choose the Logpush dataset you want to transform.
2. Write your SQL query and preview it against a sample record.
3. Save the Transformer.
4. Attach it to a matching Logpush job.

Transformers is available on Free, Pro, Business, and Enterprise plans. See the Transformers documentation for setup details and the Logpush pricing documentation for current pricing information.

Sequences and other volatiles in Neki

Hi there! My name is Mats, and I have been working on database systems for a bit over ten years. My focus has shifted, but a topic that has stuck with me is the query language; its syntax, semantics, and implementation. I have contributed to ISO standardization efforts, researched and developed query language features from scratch, and implemented features in both runtime layers and query planners.

For the past eight months, that work has been the Neki query planner and its runtime engine. In this blog, I will discuss how Neki, PlanetScale's sharded Postgres, implements sequences in our sharded system to match native Postgres. I will focus on two aspects that need special attention: volatile expressions and cross-shard synchronous evaluation.

Implementing Postgres sequences in Neki

In PostgreSQL, sequences are ubiquitous. A sequence is abstractly a unique number generator, which can be used to implement a counter, an identity, or a surrogate key.

A key property of sequences is their guarantee to never generate the same value more than one time. (That guarantee has a caveat of its own: a sequence can be reset or cycled, which clearly would repeat values, but these aspects are out of scope for this post.) Interestingly, a sequence does not guarantee that its output numbers are always consecutive; when transactions roll back, for example, consumed sequence numbers are not reclaimed.

Can Postgres handle sequences for us?

Each Neki shard is its own complete database system with its own query planner and an extensive SQL language support, so the Neki query planner can offload a lot of work to it. In fact, we view it as a virtue to do so: if we can relay the entire query to the shard, that is the best possible case. But generally, that is not possible to do.

The most obvious reason is that no shard has all the data, they only have their slice. Detecting these best cases requires a careful and complicated analysis of the SQL statement and the Neki data topology, with a strong understanding of the semantics of the SQL language and its PostgreSQL dialect with its various unique features.

That's where we come in.

But this post is not about these best cases and how we detect them. Instead, our focus is on some of the pathological cases where we cannot relay all the work to one of the shards. As I will explain, sequences fall into this category where we must assume centralized control. But before we go into sequences, we must understand some details of Neki query planning.

The general case of Neki query planning

In the most general case, each shard only needs to provide raw data. Everything else has to be possible in the router itself. For a simple query like Query 1 below, each shard hosting the (sharded) table animals returns its rows with the requested columns, and the router concatenates those rowsets into the full result and returns it to the client.

Query 1: SELECT...FROM

If we extend this query with some simple expressions over the columns, seen in Query 2, we now have a choice: evaluate the expressions in the router, or on the shard?

Query 2: add simple expressions

Both choices will give the correct result, so the question is one of optimization, which as it turns out is nontrivial:

• Is the router or the shard more resource-constrained?
• Which of the two is actually fastest to evaluate?
• Is it more efficient to network-transfer the evaluated expression, or the raw data from the column?
• What about nested expressions; shall we split them and evaluate one part here and one part there?

For the purpose of simplicity, we will here assume that Neki is using a simple heuristic whereby we always push the evaluation to the shard if we can.

Router-only expressions

But some expressions don’t give us that choice. Neki defines this class of expressions as router-only: they must be evaluated on the router and we must never push them down to the shard.

One example of such an expression is the now() function. It returns the wall clock time recorded at the start of the transaction.

If we let each shard evaluate it, the natural clock drift and the different arrival times of the request would produce different values on different shards, which violates the semantics of the expression. So we need to evaluate it in the router.

Consider Query 3, which adds now() to the mix.

Query 3: adding now()

A valid approach here is to run Query 2 on the shards, concatenate the results, and then evaluate now() in the router and append the value to each row. We don’t know how many rows the shards will return, so we have to make sure evaluating now() repeatedly in the router always gives the same value. There are a couple of ways to guarantee that.

We could evaluate it once into a single-row, single-column table and join that with the rows from the shards. Or we could cache the value per statement and read from the cache on repeated evaluations. Either way, every row gets the same wall clock time.

And this summarizes the Neki query planner's general model:

• Detect what parts of the query cannot run on the shards
• Compose a plan wherein those parts instead run in the router
• Send most other things into the shards

How Postgres classifies expressions

So far, so good, but what about sequences? In order to fully understand how to deal with sequences, we need to understand PostgreSQL's expression classification of immutable, stable, and volatile. During semantic analysis, one of the phases of query planning, every expression in the statement gets one of these classifications based on its own definition and its inputs.

The immutable class

In Query 2, all the expressions are immutable. This means that they will produce the same output each time they operate on the same inputs, regardless of how many times they are evaluated and even between statements. Such expressions are the majority, and they are simple enough to not warrant additional discussion here.

The stable class

A stable expression produces the same output within a statement. The now() function from Query 3 is one example. to_char() is another, and a useful one since it takes inputs.

For Neki, this means we cannot blindly push stable expressions to the shards. We must also make sure their dependencies are the same on every shard. Otherwise we might get 'Wednesday' back on one row and 'Onsdag' on another.

This can happen when a user includes a SET lc_time = ...; in their session. In this case, we have to push that statement to any shard that needs it for evaluation. The set_config() function is treated the same way. now() as I explained above is router-only for clock-synchronization reasons, but to_char() is not.

Enter volatility

A stable expression returns the same value every time it's evaluated within a statement. A volatile expression makes no such promise. It may produce a different output on every single invocation. It doesn't always, but it can.

As an example, consider the nextval() function. It takes the name of a sequence and returns a new, unique value each time it's evaluated. This is very useful, and can be used to implement a counter in a query or a table.

In Neki, these sequence tables exist on all shards, but they all start with an empty sequence. Since it is possible to reference any sequence in any query (assuming no ACL rules), evaluations on sequences require centralized governance to avoid duplicate values across shards.

Thus, sequence operations are all router-only. The authoritative shard is the physical host for all sequences, which the router uses to get new sequence values to distribute. This is how Neki achieves cross-shard synchronization for sequence allocation.

Table expressions

The driving force to evaluate expressions in the query execution model is the row cardinality. That is, each expression must be evaluated once per row to obtain the output needed for that row. That seems simple enough, but the complexity lies in "once per row" — as it turns out, determining that is also non-trivial!

A SQL statement is made up of clauses that form an abstract data model. We will not go into all details of that here, but I will define what a table expression is.

The composition of FROM ... JOIN ... WHERE ... (and GROUP BY, HAVING, and more) form a logical unit known as a table expression. The table expression forms the basis for the evaluation of all expressions in the SELECT clause's output list. It is the rows of the table expression that determine the "once per row" condition of the volatile expressions.

It is this table expression that forms the basis of evaluation for our SELECT clause expressions. So, we would compute 8 + 1, lower('BISMUTH'), nextval('tag_seq'), and then we are good to return results! We know that we will only evaluate the volatile nextval('tag_seq') expression one time, and thus we will only consume one sequence value.

The Neki case: a distributed query planner

Now, I must also mention that it is entirely valid to swap the order of evaluation between the JOIN and the WHERE. That is, it would be fine for us to first filter down feedings to its two 'morning' rows, and then join that subset table with animals. Which order to do it is a general optimization problem, but it is often beneficial to filter first and join afterwards, since evaluating the predicate is often cheap.

In Neki, a very large cost is the network-transport time between router and shard. In the case of a cross-shard join, which we are focusing on here, we have to go to some shards to fetch data for animals and some (same or different) shards to fetch data for feedings. Only when we have both concatenated sets of data are we able to perform our join. We are unable to do it on any given shard, since it does not have all of the data either. Thus, we have a great interest in minimizing the traffic going between router and shard.

Putting it all together

We note that the table expression can be evaluated shard-locally, and thus we can push down to shard-local evaluation on all the immutable expressions (first two columns). But we still have to evaluate both now() and nextval('tag_seq') on the router to avoid the uncoordinated shards from generating conflicting values.

now() is stable, so we evaluate it once and use the same value on all the outgoing rows. nextval('tag_seq') is volatile, so we must generate a new value for each row. And since we don't know how many rows each shard will return, we cannot generate the sequence values in advance and then draw from a cache. We must interpose the evaluations on top of the dynamic stream.

This is a DML (Data Modifying Language) query. It updates each row of the animals table by setting the tag_id column to the next sequence value from the sequence 'tag_seq'. In an UPDATE query, the table expression is expressed in a different syntactic form. You can build a complex one with WHERE and FROM clauses, but here we'll stick with the trivial case: the table expression is just the relation named after UPDATE, which is animals.

This is a challenge for Neki. The cardinality of animals is unknown, and its rows are spread out across the shards in an unknown distribution. We cannot send nextval('tag_seq') to the shard, as we discussed above, and we don't know how many values to produce. So what query do we send to the shards?

The answer is not obvious, and there is more than one solution. What Neki does today is first establish the table expression in the router using Query 8, interpose evaluation of nextval('tag_seq') on top, and run the actual update, feeding it with parameterized sequence values.

I hope this deep dive into Neki query planning brought you insights into our complex system, and how the classical problems of query planning map to the distributed system that is Neki.

Happy hacking!

Mats

TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads.

Core components:

• Scheduler: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time.
• Kernels: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest MLA (Multi-head Latent Attention) implementations on Blackwell for agentic workloads.
• Entrypoint: SMG-integrated AsyncLLM for low-overhead CPU-side request handling.

Differentiation

TokenSpeed takes a fundamentally different architectural approach from existing LLM inference engines. The PyTorch Foundation highlighted these architectural differentiators when welcoming TokenSpeed to the PyTorch Ecosystem Landscape:

TokenSpeed is an open source LLM inference engine and the first to separate the control plane from the execution plane. The control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management, including request lifecycles and KV cache state, at compile time rather than runtime. The execution plane is implemented in Python, enabling fast iteration and lowering the cognitive load for researchers and engineers. This architecture combines strong correctness guarantees in the core scheduling system with the development velocity of a high-level execution layer.

TokenSpeed also treats kernels as a first-class, modular subsystem, separating them from the core engine through a portable public API, centralized registry and selection model, and an extensible plugin mechanism for heterogeneous accelerators.

More importantly, TokenSpeed is the Switzerland of open-source LLM inference engines: neutral because it's backed by the non-profit LightSeek Foundation rather than a commercial company, with serious hardware and engineering support from vendors like NVIDIA and AMD.

News

• [2026/10] 🔥 TokenSpeed MLA 2 for NVIDIA Vera Rubin.
• [2026/10] 🔥 685 tokens/s per user: serving Qwen3.8-Flash-Next on NVIDIA GB300 with TokenSpeed.
• [2026/10] 🔥 Advancing Kimi K3 inference on AMD Instinct MI355X GPUs with TokenSpeed.
• [2026/09] Kimi K3 Optimization on GB300 — Part I.
• [2026/08] Qwen3.8 Flash Next at Day 0 and GLM 5.3 Flash at Day 0.
• [2026/08] Qwen3.8 at Day 0: 2.4T-Scale Inference with TokenSpeed.
• [2026/08] TokenSpeed joins the PyTorch Ecosystem.
• [2026/07] Kimi K3 at Day 0: Frontier Model Enablement on Leading Platforms with TokenSpeed.
• [2026/07] TML Inkling at Day 0: FP4 Inference on NVIDIA and AMD with TokenSpeed.
• [2026/06] Deep dive into the design and optimization of TokenSpeed-Kernel.
• [2026/05] 🚀 TokenSpeed hits 580 TPS on Qwen3.5-397B-A17B for agentic workloads.
• [2026/05] TokenSpeed announced — a speed-of-light LLM inference engine for agentic workloads.

Blogs and talks

For technical blogs, conference talks, and engineering articles from LightSeek Foundation, visit the LightSeek Blog.

Sponsors and partners

LightSeek's work is advanced by the support of sponsors and partners across the AI ecosystem.

Performance comparison

TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workloads (Kimi K2.5, B200)

Documentation

Start here:

• Docs Index
• Getting Started
• Launching a Server
• Model Recipes
• Server Parameters
• Compatible Parameters
• Parallelism

Citation

@misc{tokenspeed2026,
  author       = {{TokenSpeed Team}},
  title        = {{TokenSpeed}: A Speed-of-Light {LLM} Inference Engine},
  year         = {2026},
  howpublished = {\url{https://github.com/lightseekorg/tokenspeed}}
}

Full article content is not available for inline reading.

Read the original article →

It's mid-afternoon and you're halfway through a feature branch when the alert lands: The error rate’s climbing on the payment service. You know that code and wrote most of it. What you don't know is whether this started with the morning's deployment, whether it's one region or everyone, or whether the real break sits three services downstream from anything you touched. So you leave your editor. You hunt for a dashboard, don't find the right one, write a New Relic Query Language (NRQL) query from memory instead, then open traces in one tab and logs in another. Fifteen minutes later, you've lost the thread on the feature you were building, and the actual debugging hasn't started.

This context gap is what New Relic Ground Truth closes. Your AI agent has your code, your branch, and your recent commits; what it has no idea about is how any of that behaves once it's running. New Relic Ground Truth supplies that piece—telemetry, logs, traces, deployments, and alerts, exposed through the open Model Context Protocol (MCP) so your AI agent can reason about the running system the same way it reasons about the source. No custom integration work required. You stay in the editor, ask in plain language, and the context comes to you. The examples that follow—full list here— are built across different integrations, but follow the same template: Once your AI agent can see what’s in production, hopping over to another dashboard stops. Less context switching and faster root cause analysis means shorter mean time to resolution (MTTR) and more of your day spent shipping instead of hunting for the right dashboard.

Note: All environments support both OAuth (recommended) and API key authentication.

What you can ask your AI agent

Here are some real, natural language prompts that work across any supported environment. These responses are powered by trusted operational context from New Relic Ground Truth, enabling AI agents to retrieve production insights without requiring developers to manually navigate dashboards or write NRQL:

• "What are the top errors for my payment service in the last hour?"
• "Did my last deployment cause a performance regression?"
• "Give me p99 latency broken down by host for the last 30 minutes"

New Relic Ground Truth automatically translates natural language into NRQL and executes the appropriate query through its MCP capabilities. Developers simply describe what they want, eliminating the need to learn NRQL syntax while enabling engineers who don’t use New Relic every day.

Developer use cases

Each example below starts with a common developer job and shows how New Relic Ground Truth MCP Server helps finish it. We demo each one in a specific coding agent, but the same workflows run in any supported environment.

Deployment health analysis with Claude Code

Every deployment carries risk. Even well-tested releases can introduce regressions that only show up under real production load. There’s now a faster way to answer the question: "Did this deployment break anything?”

With New Relic MCP Server connected to Claude Code, a developer can ask a single question right from their terminal:

"How did my last deployment perform?"

New Relic MCP Server pulls together a comprehensive picture from across our Intelligent Observability Platform—metrics, logs, traces, error rates, alert history, and more—and returns a full deployment health report. If something went wrong, Claude Code not only tells you what broke; it also has enough context to start working on a fix right in the same session.

Using Claude Code? Install the New Relic Claude plugin
Our New Relic Claude plugin is now live on the Anthropic marketplace. You can install it directly from the marketplace or access our source on GitHub and install it using the command below:

claude plugin install newrelic@claude-plugins-official

Infrastructure optimization with Kiro

Managing Infrastructure as Code (IaC) involves an annoying disconnect: You write Terraform or CloudFormation in your editor, but the performance data proving whether you over-provisioned or under-resourced lives in a separate dashboard.

To close this gap, we launched a New Relic Kiro Power. This transforms Kiro into an infrastructure engine driven by real production data. It pulls live telemetry—like CPU bottlenecks and memory spikes—straight from New Relic into your Integrated Development Environment (IDE). If a service is struggling under peak load, Kiro doesn't just show you the metric; it also analyzes the New Relic data and instantly generates the exact IaC code changes needed to right-size your environment. New Relic provides the production ground truth, and your IDE writes the fix.

Closing observability gaps with GitHub Copilot

One of the most common problems teams face when shipping new services is incomplete observability coverage. You deploy something and realize weeks later that you have no application performance monitoring (APM), no browser monitoring, and/or no change tracking. Finding and fixing those gaps manually is tedious and easy to deprioritize.

Our New Relic MCP Server integration with GitHub Copilot enables developers to identify observability gaps, understand production behavior, and accelerate remediation without leaving their coding environment.

How it works:

1. A GitHub Action tracks code releases for a repository and triggers an observability gap analysis
2. GitHub Copilot chat connects to the New Relic MCP Server, which in turn communicates securely with GitHub's MCP Server through an associated GitHub App
3. The analysis runs and surfaces specific gaps
   1. For example: Missing APM, browser monitoring, or change tracking
4. GitHub Issues are automatically created for each gap, complete with an implementation checklist, detailed instructions, and deep links to relevant New Relic documentation
5. GitHub Copilot coding agent is assigned to each issue and gets to work, analyzing the requirements, implementing the necessary code changes, and opening a pull request

The power is in the full loop: New Relic detects the gap, GitHub Copilot fixes it, and the PR contains everything needed to close that gap—tested, documented, and ready for review. Future checks will extend to infrastructure monitoring, synthetics, logging, and alerts.

Orchestrating automated observability with GitHub custom agents

Finding the critical information with GitHub custom agents and MCP Servers takes knowledge and time to engineer prompts that make it easy for the AI agent to use the right tools and format a meaningful response. Using a custom agent from GitHub finely tuned to New Relic MCP Server allows for simpler one-line inputs that generate robust responses.

Take the example of the incident response custom agent: You can check in on an incident within your organization by running a simple prompt like this:

“Check on the latest incident for this application”

The custom agent can then create a full report and kick off a multi-step process to analyze any major incidents and break down what has impacted any critical data with the incident.

The bigger picture: Why New Relic Ground Truth matters for developer productivity

MCP integrations help remove silos. Your coding agent becomes aware of production, not just code.

The New Relic 2026 AI Impact Report found a 5x improvement in teams' ability to ship features and updates to production. Not because the AI got smarter about code, but because it finally has context about what the code is doing once it's running. Less context switching, faster debugging, shorter time from alert to fix.

AI agents are only as effective as the context they receive. Ground Truth provides trusted context drawn from New Relic telemetry and makes it available through the open MCP, enabling AI development environments to investigate issues, understand production behavior, and resolve problems faster without leaving the IDE.

None of this is tied to one editor or one vendor. Whether you live in Claude Code, GitHub Copilot, Kiro, or anything else on the supported list, the pattern holds: Your agent connects to the New Relic MCP Server, and the operational context follows you into whichever coding surface you're in. The scenarios above are built in specific tools only because that's where those teams happen to work. The deployment health check that runs in Claude Code works just as well from Copilot, and the observability gap analysis isn't Copilot-specific either. Every one of them is available in every supported environment.

That's deliberate. We're not asking developers to move somewhere new to get production context; we're bringing production context to where they live. Since MCP is an open standard, new AI development environments inherit this the moment they are spun up. Pick the tools your team likes. New Relic Ground Truth meets you there.

Getting started

Getting started with New Relic Ground Truth’s MCP Server capability is simple. Full prerequisites, authentication options, and client-specific configuration are covered in the setup guide. For a full reference of available tools and capabilities (including 3 skills and 42 tools), see the MCP Server tool reference.

Looking to track cost, efficiency, and tool usage for your AI agents? Check out Preflight, our open-source observability solution for Claude Code, Copilot, Cursor, and more.

Zero is a powerful number in container security. Run a container image through a tool such as Trivy or Docker Scout and imagine the result comes back with no known vulnerabilities. It is tempting to treat that number as if the image is clean and safe to deploy, but what did the scan actually prove?

It proved only that the scanner found no known vulnerabilities in the components it could identify. That is useful information, but it’s not evidence that the artifact itself is trustworthy. The distinction matters because an image can have zero known CVEs while its origin, build process, dependency selection and software composition remain largely unknown.

Recent supply-chain incidents show why. In the 2026 Nx Console compromise, an attacker used a stolen GitHub credential to publish a malicious version of the extension through legitimate publishing channels. The publishing system accepted the release because the attacker was operating with a compromised legitimate contributor credential.

Container images face the same fundamental problem, because a vulnerability scan doesn’t tell us everything we need to know before trusting an artifact in production.

Zero CVEs Only Tells You What Is Known Today

Vulnerability scanners generally identify packages and other software components in an image and correlate their versions against vulnerability databases and security advisories. Docker Scout, for example, analyzes image contents using an SBOM and evaluates the components against vulnerability data. Trivy can similarly scan image and SBOM contents for known vulnerabilities.

So when the result says no known vulnerabilities, the useful interpretation is: “No known vulnerabilities were identified in the components the scanner could identify.” But that’s different from saying the software is safe.

New vulnerabilities are disclosed continuously, so a package that produces a clean result today may have a vulnerability disclosed tomorrow. In fact, the image itself doesn’t have to change for its vulnerability assessment to change. A scanner can reassess an image as new vulnerability data becomes available, using the metadata associated with the artifact.

Scanner results also depend on accurately identifying the software present in the image. Unexpected binaries, incomplete metadata or software installed outside standard package management mechanisms can make that harder.

There is another, more important limitation: malicious software doesn’t need a CVE.

A compromised dependency can contain malicious code without having a vulnerability identifier at all. The scanner may correctly report zero known CVEs because it’s looking for known vulnerabilities, not malicious intent.

That is what makes supply-chain compromises particularly difficult. In the Nx Console incident, a malicious @tanstack/zod-adapter package was installed on a contributor’s machine during a routine pnpm install. Its payload stole a GitHub CLI OAuth token, which was later used to publish a malicious Nx Console release. The eventual release was accepted through the legitimate publishing process and distributed through marketplaces.

The scanner isn’t wrong in this situation, because we’re asking it to answer a question it was never designed to answer.

Trust Starts With Knowing Where the Image Came From

Suppose an image passes the scan.

The next question should be: Who built it, and how?

Public registries make image consumption easy, but a familiar image name doesn’t establish who produced an artifact, what source code was used, what dependencies were incorporated or whether the publisher intended to release it.

There are several different pieces of evidence here, and they are easy to conflate.

An image digest identifies the exact artifact being consumed. A cryptographic signature can establish that a particular identity signed a specific artifact. Whether that identity is one an organization trusts is a separate policy decision.

Provenance provides evidence about the source, builder and process used to produce the artifact.

These are complementary controls.

A signature can establish that an image was signed by an expected identity. It doesn’t, by itself, prove that the image was built from the source code or dependencies you expected. For container images, we need evidence about both the artifact and the process that produced it.

What Is Inside the Image Matters Beyond Vulnerabilities

Now look at the image itself.

A typical container image can contain considerably more software than the application actually requires: shells, package managers, compilers, network utilities, debugging tools and other operating system packages inherited from a base image.

Removing a shell, compiler or package manager doesn’t necessarily remove a vulnerability. If an attacker does gain execution inside a container, having fewer tools available can constrain what they can do next.

The objective should be to produce an artifact where the software that remains is intentional, identifiable and tied back to a controlled build process.

Trust Has to Continue After Deployment

Even a well-built image doesn’t remain permanently secure.

A new vulnerability can be disclosed tomorrow. A dependency can reach end of life. A security advisory can change the assessment of software that passed every check when the image was released.

This is why artifact trust has to continue beyond the initial build.

An important distinction is that the artifact and its security assessment have different lifecycles. An image digest identifies an immutable artifact, but the vulnerability information used to assess that artifact changes over time. A new advisory can change the risk assessment without changing a single byte of the image.

Give Developers a Better Default

A developer should not need to become a software-supply-chain investigator every time they need a container image. Security teams cannot manually investigate every image pulled into an enterprise environment either.

Give developers access to images that already carry evidence about their provenance, composition and build process. Make those requirements part of the platform rather than another checklist developers have to navigate.

Vulnerability scanning remains an important part of that process. A low or zero known-CVE count can help teams assess known exposure and prioritize remediation. But it should be one piece of evidence, not the entire trust model.

A clean vulnerability report is valuable, but it only answers one question. Trust also depends on evidence about where the image came from, what it contains and how it was built.

One Agent Development Kit, five languages

When we talk about agents, that doesn't necessarily mean we're talking about Python. In today's modern ecosystem, an agent might run as a high-concurrency Go microservice on Cloud Run, or live inside a browser tab calling Chrome's on-device model. Yet many agent frameworks assume a Python-only stack. That assumption forces teams to either spin up a Python service just for their agent workloads, or to build agent orchestration logic from scratch in their own language.

Google's Agent Development Kit (ADK) removes that trade-off with coverage for Python, Go, TypeScript, Java, and Kotlin. Each version of the ADK is built from the ground up for its own runtime and type system, while sharing a unified architecture with interoperability through open protocols.

A shared foundation

Picking a language doesn't lock in your architecture: all five support the same open protocols. ADK includes a Model Context Protocol client in every language, so a custom tool exposed as an MCP server can be called from any of them immediately.

For multi-agent systems, ADK supports the Agent2Agent (A2A) protocol, allowing an agent to communicate with a remote agent over HTTP. Through A2A, agents written in different languages or frameworks are interoperable.

Starting agent-first

To get started, run the one-time setup command to check your Google Cloud authentication, install the CLI, and then set up first-party ADK skills in coding agents like Antigravity, Claude Code, or Codex:

uvx google-agents-cli setup

Then, run the scaffold command to generate a working project with A2A endpoints and deployment config for Cloud Run, Gemini Enterprise Agent Runtime, or Google Kubernetes Engine (GKE):

# Scaffold an ADK project (use --agent adk for Python or --agent adk_go for Go) 
agents-cli scaffold create my-go-agent \ 
  --agent adk_go \ 
  --deployment-target cloud_run \ 
  --region us-central1

Both the ADK for Python and ADK for Kotlin repositories also ship agent builder skills in their source trees, and the adk.dev/llms.txt resource gives any coding agent on-demand API documentation across all five languages.

Coding-agent skills scaffold your project while MCP and A2A connect all five language runtimes.

Hands-on view of each ADK

The clearest way to feel the difference between the five languages is to define the same weather tool in each one and see how its design fits the runtime.

Python

In ADK for Python, you don't need a decorator or a schema builder class. You pass a plain Python function into the tools parameter, and ADK inspects its signature, type hints, and docstring at startup to automatically construct the tool declaration and Pydantic parameter schema:

from google.adk.agents.llm_agent import Agent

def get_weather(city: str) -> dict:
    """Retrieves the current weather report for a specified city.
    Args:
        city (str): The name of the city for which to retrieve the weather report.
    """
    return {"status": "success", "report": f"Sunny and 72F in {city}."}

root_agent = Agent(
    name="weather_agent",
    model="gemini-3.8-flash",
    instruction="Answer questions about the weather in a city.",
    tools=[get_weather],
)

Beyond fast prototyping, Python is ADK's reference implementation, shipping the broadest set of Google Cloud integrations (BigQuery, Spanner, Bigtable) and model adapters (Gemini, Anthropic, OpenAI, LiteLLM). Paired with the adk eval or agents-cli eval run commands, its evaluation tooling scores agent trajectories against golden datasets in continuous integration (CI) to test the whole system end to end, even when your production service runs in another language.

Go

While Python leads in experimentation and evaluation, ADK for Go hits the sweet spot for production backend engineering.

A Go tool function takes the agent.Context configuration and a strongly typed input struct. The functiontool.New function uses Go generics to infer both the input and output schemas from your struct tags:

type GetWeatherArgs struct {
    City string `json:"city" jsonschema:"The city to look up weather for."`
}
type GetWeatherResult struct {
    Report string `json:"report"`
}

func getWeather(_ agent.Context, args GetWeatherArgs) (GetWeatherResult, error) {
    return GetWeatherResult{Report: "Sunny and 72F in " + args.City}, nil
}

weatherTool, err := functiontool.New(functiontool.Config{
    Name:        "get_weather",
    Description: "Retrieves the current weather report for a city.",
}, getWeather)

Because the functiontool.New function is generic over your handler's signature, the Go compiler verifies that your function matches the expected contract before your binary ever runs.

Go was also the first language after Python to ship the ADK 2.0 Graph Workflow runtime. When you need deterministic control flow alongside LLM reasoning, the workflow.New function wires agents and plain Go functions into a directed graph with conditional routing and parallel steps:

ag, err := workflowagent.New("order_pipeline", []workflow.Edge{
    {From: workflow.Start, To: triageNode},
    {From: triageNode, To: fulfillNode, Route: workflow.StringRoute("fulfill")},
    {From: fulfillNode, To: notifyNode},
})

Where Go really stands out is in production on Cloud Run and GKE. A compiled Go agent packages onto a roughly 2 MiB Google distroless static base image, with no language VM or heavy dependency tree. A static Go binary binds to its port as soon as the container starts, skipping the module imports of dynamic languages and the JAR unpacking and JIT warmup of JVM services.

After it's running, Go's concurrency model fits agent workloads naturally. Each goroutine starts with a 2 KB stack and minimal scheduling overhead, so a single instance can multiplex thousands of concurrent streaming requests and tool calls. ADK for Go also includes built-in A2A server handlers and OpenTelemetry tracing out of the box.

TypeScript

TypeScript erases type annotations when compiling to JavaScript, so runtime reflection isn't an option. Instead, ADK for TypeScript pairs with Zod. You pass a z.object(...) schema to the FunctionTool function, and TypeScript statically infers the parameter types of your execute callback from that single declaration:

import { FunctionTool, LlmAgent } from "@google/adk";
import { z } from "zod";

const getWeatherTool = new FunctionTool({
  name: "get_weather",
  description: "Retrieves the current weather report for a city.",
  parameters: z.object({
    city: z.string().describe("The name of the city."),
  }),
  execute: async ({ city }) => {
    return { status: "success", report: `Sunny and 72F in ${city}.` };
  },
});

export const rootAgent = new LlmAgent({
  name: "weather_agent",
  model: "gemini-3.8-flash",
  instruction: "Answer questions about the weather in a city.",
  tools: [getWeatherTool],
});

Your runtime input validation, the JSON Schema sent to the model, and your IDE autocomplete inside the execute callback all flow from one source of truth.

TypeScript also runs on the ADK 2.0 Graph Workflow engine and ships a standalone browser bundle. Beyond Node.js services, you can run an ADK agent directly in a browser tab using WebMCP for page-level tools and Chrome's on-device Prompt API.

Java

ADK for Java snaps right in for teams who have standardized on Maven, Gradle, and Spring Boot. You add @Schema annotations to method parameters and assemble agents by using a fluent LlmAgent.builder() API:

public class WeatherTools {
    public static Map<String, String> getWeather(
        @Schema(name = "city", description = "The name of the city") String city) {
        return Map.of("status", "success", "report", "Sunny and 72F in " + city);
    }
}

LlmAgent agent = LlmAgent.builder()
    .name("weather_agent")
    .model("gemini-3.8-flash")
    .instruction("Answer questions about the weather in a city.")
    .tools(FunctionTool.create(WeatherTools.class, "getWeather"))
    .build();

At runtime, the runner streams agent events reactively through RxJava. For local testing and existing JVM apps, ADK for Java ships a Maven Dev UI plugin alongside first-party bridges for Spring AI and LangChain4j.

Kotlin

ADK for Kotlin takes a strong stance on build-time safety. Its Kotlin Symbol Processing (KSP) compiler plugin reads @Tool and @Param annotations at build time and emits the tool schema directly into bytecode:

class WeatherTools {
    @Tool(
        name = "get_weather",
        description = "Retrieves the current weather report for a city.",
    )
    suspend fun getWeather(
        @Param("The name of the city.") city: String
    ): Map<String, String> = mapOf("status" to "success", "report" to "Sunny and 72F in $city")
}

val agent = LlmAgent(
    model = Gemini("gemini-3.8-flash"),
    name = "weather_agent",
    instruction = Instruction("Answer questions about the weather in a city."),
    tools = WeatherTools().generatedTools(),
)

If a tool signature uses an unsupported type or declares a default value on a non-nullable parameter, your Gradle build fails immediately instead of breaking in production. Because the .generatedTools() function uses zero runtime reflection, it survives Android R8 shrinking.

That zero-reflection design makes it possible to run the entire agent loop locally on Android. It offers JVM server support, including a Spring AI bridge and experimental ADK 2.0 graph workflows. On-device, ADK for Kotlin integrates with LiteRT-LM for tool calling and ML Kit (Gemini Nano) for text generation. It also works with Room and AppSearch for local state, and Android AppFunctions for OS-level actions, all without requiring a network connection.

Choice without compromise

Having five native languages in ADK means you don't have to pick a single runtime for your entire organization. Because every version of ADK speaks MCP and A2A, each component of your system can use the language that fits it best.

What languages have you used to build agents with Google's ADK? Let me know what stack you're running on X, LinkedIn, or Bluesky.

Design Systems Directory

Design systems, component libraries, and UI toolkits from around the world.

Elastic UI framework

Sainsbury's Design System

Ariakit

SubZero

Web Awesome

Red Hat design system

HeroUI

Morningstar Product System

Visa Product Design System

Elisa Design System

NewsKit

Momentum Design

Chakra UI

Blueprint

eBay Playbook

Flowbite

Cauldron

Nessie

PatternFly

Cedar

Headless UI

Gestalt

Geist Design System

Pajamas

Base Web

Porsche Design System

shadcn/ui

Clarity Design System

Wanda

Coral

Backpack

Polaris

Primer

Nord Design System

Material Design

Carbon Design System

Wise Design

Ant Design

Decathlon Design System

Helios

Mozilla Protocol

ONS Design System

giffgaff design system

Crayons

Workday Canvas Design System

GOLD Design System

Evergreen

Dell Design System

Fluent UI

Quasar Framework

Jøkul Designsystem

Atlassian Design System

Forma 36

Auro

Source

Spectrum

W3C design system

Pharos

Bolt Design System

Ontario Design System

Nucleus Design System

West Midlands Network Design System

Helsinki Design System

Radix Primitives

NHS Digital service manual

Instructure-UI

Shoelace

generic-components

Purple3

Aurora

Paste

uStyle

98.css

eBay MIND Patterns

BBC Global Experience Language

Seeds

Thumbprint

Duet Design System

Bulma

Bootstrap

Grommet

Seek style guide

United States Web Design System

Stacks

Brighton & Hove City Council Website pattern library

Lightning Design System

A11Y Style Guide

Inclusive Components

GOV.UK Design System

FutureLearn design system

MUI (Material UI)

Mantine

Radix UI

daisyUI

Storybook Design System

Tremor

Park UI

Reshaped

Nuxt UI

KoliBri

Aalto University

AWS Cloudscape

Apple

Audi

Australian Gov

Aviva

Basis

BBC

Biings

Brainly

Carrefour

Cash App

Datadog

Denmark

Esri

General Electric

Ideagen

INPS

Kajabi

Lemon Squeezy

Line

Michelin

Vibe (monday.com)

MongoDB

Morningstar

Motorway

NS

NSW

NYS

Orange

Palantir

Pando (Pluralsight)

Playbook (Power Home Remodeling)

Razorpay

Samsung

SAP

Seek

ServiceNow

Sigma

Skoda

SumUp

Sunrise

Tailwind

Teamleader

UAE

Vercel

Visa

VTS

Washington Post

Wikimedia

Zendesk

Mews Design System

Telekom Scale

Element Plus

Arco Design

Semi Design

TDesign

PrimeReact

Naive UI

DSFR (Système de Design de l'État)

NL Design System

SGDS (Singapore Government Design System)

Designsystemet (Norway)

Emplifi Soul

Microsoft Power Platform

Serendie

Astryx

CMS Design System

Maersk Design System

Arbetsförmedlingen Designsystem

Paysafe

Amplitude

Siemens Industrial Experience (iX)

Delphinus (Aarhus University)

Rivet (Indiana University)

Canada.ca Design System

VA.gov Design System

Europa Component Library

Scottish Government Design System

New Zealand Government Design System

Westpac GEL

Klarna

DHL Wording

Fit height to background pattern size with calc-size()

The problem

I have a section with a background pattern that repeats on both axes. I wanted the vertical repeat to be aligned with the pattern’s height, so the pattern won’t have cut-offs.

Take the following design.

.section {
  --tile-width: 52px;
  --tile-height: 44px;
  background-image: url("pattern.png");
  background-size: var(--tile-width) var(--tile-height);
}

Do you notice the cut-off at the bottom? This doesn’t look good. The first thing I thought about was to use the round value for the background-repeat property.

.section {
  background-repeat: round;
}

That worked.

But my inner self said no. This is not the best solution, given the modern CSS tools we have. What I don’t like about this solution is that using the round keyword will force the pattern to shrink or grow.

Let’s explore a better solution.

Using calc-size() and round()

The following solution works only in Chrome at the time of writing this article.

We can use calc-size() to get the auto height value, then use it with the round() function to round it to the nearest value we want. In this case, the value is the pattern’s height.

.section {
  height: calc-size(auto, round(up, size, var(--tile-height)));
}

At first glance, it looks like setting a fixed height. In fact, it works like min-height, as it calculates the auto height, then rounds it up to the nearest multiple of the tile height.

It works great for different pattern sizes. All we need to provide is the tile height.

I wrote about the round() function if you want to dig more into it.

Rounding up or down?

Rounding the height up to the nearest multiple of the tile height will increase the height in most cases.

I prefer down, as it doesn’t increase the height when it’s not needed, but it’s not a big deal.

What about padding?

We need to add vertical padding, so I thought about using round() for it, too. See below:

.section {
  height: calc-size(auto, round(up, size, var(--tile-height)));
  padding-block: round(up, 1rem, var(--tile-height));
}

The top and bottom padding is equal to the pattern’s tile height.

Where rounding down feels a bit odd

Notice how the content is too close to the edge. This is where it starts to feel a bit odd, so in this case, I’d switch to rounding up for more expected behavior.

The complete solution

Here is the complete CSS for the solution in the previews demos.

.section {
  /* This must match the size of the pattern image. */
  --tile-width: 52px;
  --tile-height: 44px;
  background-image: url("pattern.svg");
  background-size: var(--tile-width) var(--tile-height);
  padding-block: round(up, 1rem, var(--tile-height));
  height: calc-size(auto, round(up, size, var(--tile-height)));
}

Full article content is not available for inline reading.

Read the original article →

An update on independent evaluators

Last month, we committed to stronger independent, third-party assessment of our models and safety practices. We also signed the White House Accord on Super Intelligence, in which companies pledged to partner with independent external auditors or evaluators. Today, we’re announcing that we’ve started to give employee-like access to a team from METR, a nonprofit that evaluates frontier AI models.

METR will publish the results of its risk assessments. Over the past year, we’ve worked with METR on several reviews of our Risk Reports. We’re also working with SecureBio and Accenture as evaluators, and are in early discussions with several other organizations, including Hugging Face.

Here, we explain our approach to working with evaluators.

Why we work with independent evaluators

As models have become more powerful, they’ve required stronger safeguards and more challenging risk assessments. We want to be publicly accountable for the decisions we’re making. But we can’t publicly share all the information relevant to our safety work. Some of it could be useful to malicious actors, or is commercially sensitive, like compute or training information. Instead, we’re giving third-party evaluators deep access to our information, and are having them share evidence and reasoned conclusions about the risks involved in building our models. We aim to close the gap between what evaluators understand about a frontier AI company’s systems and what the company itself understands.

To evaluate frontier models, evaluators need the following:

• Expertise in assessing the capabilities of frontier models, including understanding the way evaluations can go wrong, such as alignment faking.
• Independence: they shouldn’t hold equity in the company they’re evaluating, or be financially or operationally dependent on it.
• Enough access to meaningfully evaluate risks. Beyond access to the latest models, evaluators also need visibility into how those models were trained, into how our safeguards function, and more.
• Freedom to publish observations and assessments of a model or a process.

Although the analogy isn’t exact, bank supervision operates on similar logic. Banks know more about their own risk positions than outsiders do, so regulators assign resident examiners to work inside the largest banks. If a bank’s risk-taking could threaten the public, third parties try to see that in advance.

For the AI industry, this is a step toward robust safety oversight. We’re moving quickly with multiple evaluators and learning actively about what kinds of partnership work. We’re committed to sharing updates as we go.

What we mean by “employee-like access”

We’ve onboarded six METR employees with company hardware, office space, and access to internal tools and systems that are roughly on par with what our own risk assessment teams have access to. Outside a small number of clearly defined categories (e.g., data about customers, attorney-client privileged information, or information about our model architectures), Anthropic employees are encouraged to share information with them just as they would with a fellow employee working on alignment risk.

METR will publish its analysis of our risk levels, incidents, or practices. Anthropic can’t veto publication, but we can require them to redact information if it falls into predefined categories like the ones above. (Anthropic ultimately decides what information does and does not fall into these categories, although METR reserves the right to make certain statements about the level of access and information it received.)

Our engagement with METR is limited to a small team operating under enforceable agreements. Other METR employees are siloed from that team’s work, including researchers who work with us on other projects. Embedded evaluation gives a few qualified people working under Anthropic’s security standards a close view of how we operate. It’s not a way for other organizations to obtain our information, or that of our customers and partners.

Our other evaluator partnerships

We’re working on several other partnerships. Because these other partnerships are still in early stages, we’re providing these evaluators with more limited information, and they have more limited publication rights. Our goal with these early engagements is to make sure that our partners protect confidential information and have the requisite resources and expertise to evaluate our models. As partnerships progress and we build trust, we hope to fully embed more independent evaluators.

We’re developing three tiers of access for evaluators. Over time, we hope to have several evaluators at the third tier, with each team publishing their own analysis of our risk posture and key decisions.

In the first tier, evaluators conduct narrowly scoped risk investigations. We decide what access to provide a first-tier evaluator, and evaluators can publish their findings with our approval.

In the second tier, evaluators can publish without our approval, and they have access to more information. We can still require removal of specific categories of information, though evaluators can publish their views on whether our redactions were justified.

The third tier is the arrangement we’ve described with METR, with employee-like access and publication rights similar to those of the second tier.

SecureBio is in the process of reviewing the chemical and biological weapons chapter of our August Risk Report, after completing and publishing a pilot review of our previous risk report. After this review is complete, we will consider moving to projects with increased access.

With Accenture and Faculty, we're working toward an ambitious partnership that includes evaluating and red-teaming models, conducting alignment assessments, testing model safeguards, and building capabilities across our organizations.

We’re embedding evaluators voluntarily, and we believe every frontier AI company should do the same.

Footnotes

1. Because these evaluators (like bank examiners) may also work with other companies, we exercise caution around the possibility of inadvertent leaks. Information about architectures presents a high risk of inadvertent leaks, and its relevance to risk assessment is typically low. When it is highly relevant, we will generally seek to share it, though this will require explicit internal approvals.
2. Updated on October 16 to clarify METR's current project terms with Anthropic.
3. Updated on October 15 to clarify SecureBio's current project terms with Anthropic.

When I visited Ramp’s headquarters last month in New York, I talked with the four-person development team behind ‘Glass,’ which is Ramp’s “internal Lovable.” This name refers to Lovable.dev, the leading AI vibe-coding tool.

Ramp’s ‘Glass’ is basically a vibe-coding platform for non-engineers to create complex applications which run internally. They can share the applications they make with others to use and modify.

Users can connect various internal systems as “connectors” that can then be used as data sources for their apps.

The team showed me a few popular apps, including a custom CRM built by someone in marketing, which had 50+ fields and filters on-screen. It seemed overwhelming to me, but the creator reportedly finds it extremely powerful to use.

Since then, I’ve learned that many other mid-sized-and-above companies are building similar “internal Lovables” that don’t usually see the light of day in the outside world.

As of today, Ramp has launched Glass as an external product, describing it as "An AI workspace where knowledge compounds." It's interesting to see Ramp productionizing a formerly internal-only tool!

Stripe’s “internal Lovable”, Harbor

Stripe shared details last month about its own vibe-coding tool (my words) that they refer to more elegantly as an “AI-assisted prototyping tool” called ‘Harbor’. It was built by a single engineer in around six weeks.

Why build your own internal “prototyping tool?”

Lovable, Replit, Retool, and other vendors specializing in prototyping and building internal tools already exist, so what’s the point in bothering to build your own? Here are the main reasons I’ve gathered:

• A tiny investment. Stripe had a single engineer build Harbor v1, and Ramp had two engineers put it together in a matter of weeks. For companies with hundreds of engineers or more, this is a very reasonable time/effort investment.
• Expose internal data to be used. Because the tool produces internal-facing prototypes/tools, it’s a lot safer to expose internal data sources to it. Doing the same with third-party tools like Lovable or Replit would be risky.
• Use the existing design library. Both Ramp and Stripe paid special attention to ensuring that the apps generated do follow internal design guidelines. Third-party apps are not great at this, out of the box.
• Onboard most non-engineers, when done right. Ramp’s Glass project is extremely widespread, to the point that onboarding to Glass and creating a simple tool is now part of new hire onboarding. Pretty much every non-engineer uses Glass to build their own tools.
• Don’t wait on vendors to build what’s needed. Why spend time and effort on tailoring a vendor solution which is straightforward enough to build? It would probably be a comparable effort to select the right vendor to go with, in terms of compliance reviews and price negotiations.
• It’s educational and helps engineers to upskill. Engineers building these tools get better at building and maintaining AI infra at a time when recruiting engineers who are experts at AI infra is hard enough. Internal Lovables are a way to train more devs in-house by investing a few weeks in putting a solution together.

Among tech companies that have built their own coding agent harness that runs in their cloud, I expect many to build internal vibe-coding tools for non-engineers like Stripe and Ramp have done. After all, it’s not that complicated, gets much higher adoption than vendor products, and helps build out more AI infra expertise in-house.

Key Points

• Signs of distressed deepened in CoreWeave on Thursday.
• Yields on its 2031 bonds topped 15%.
• Spreads in credit default swaps, which is insurance against defaults, widened to levels generally viewed as distressed.

CoreWeave's credit stress continued to spiral on Thursday amid a broader sell-off in AI-related names.

CoreWeave — a so-called neocloud company — has taken on more than $51 billion of debt and obligations to finance its AI infrastructure build-out. It's now facing a rebellion from stock and bond investors as concerns mount about AI overspending and debt levels.

Yields on CoreWeave debt have continued to climb, with the yield to maturity on its 9.75% bond due in 2031 reaching 15.78%. Bond yields move inversely to prices. Meanwhile, the spread on its five-year credit default swaps, which are insurance against default, at one point soared above 1,140 basis points. It last stood at nearly 1,038 bps — a 16% week-over-week jump and 164% higher than a year ago.

Credit concerns are spreading into AI equities too. CoreWeave's stock dropped during trading Thursday before recovering and closing 5.94% higher. The moved followed Wednesday's 15.5% nosedive.

Other neocloud stocks were under similar pressure. Nebius also sank before climbing higher to close 1.29% and Iren ultimately ended the day down less than 1%.

Anxiety about AI overspending is also spreading. For instance, shares of Micron Technology and Advanced Micro Devices sold off following a big sell-off in South Korean chip stocks. Micron closed up 1.87%, while AMD fell 1.5%.

While AI debt worries have been rising steadily, investors have suddenly grown skittish about the underlying finances of companies borrowing heavily to buy chips, build data centers, and rent out computing power.

Coreweave's credit default swaps are one place to see that stress.

Credit default swaps are contracts where buyers pay for protection in case the borrower can't repay its debt. The spread widening in CoreWeave's CDS means investors view the debt as riskier.

The market is "basically saying 'no mas' to AI debt without operating cash flow support," the Evercore ISI trading desk said.

A spread level of 1,000 basis points or above is generally a sign of credit distress, said Gavan Nolan, executive director of fixed income pricing at S&P Global Market Intelligence.

No one is questioning whether demand for CoreWeave's services is real, since the company has contracts with all the big hyperscalers, Nolan said. The question is whether the cash flows will ultimately arrive.

"It's obviously heavily negative free cash flow at the moment, and it needs that cash flow to arrive really to justify all the capex that it's spending," Nolan said.

'Self-fulfilling prophecy'

CEO Mike Intrator dismissed the market moves on Wednesday, according to a memo obtained by CNBC.

"When you are building a new type of technology company there will be periods of time when the market misunderstands and acts irrationally," he wrote. "We are confident in our backlog which more than supports all of our debt obligations."

The backlog is over $100 billion, with around two-thirds of it with investment-grade counterparts including Microsoft, Meta, IBM and Caterpillar, Deutsche Bank said in a note Thursday. The firm reiterated its buy-rating on the stock Wednesday, calling the dislocation a buying opportunity.

"The problem is this kind of concern can prove to be a self-fulfilling prophecy, and risks triggering broader issues in other AI-related names," said Jim Reid, global head of macro and thematic research.

Domino effects may already be happening. Oracle's five-year CDS spread also spiked this week. It was last at 282 basis points, a roughly 10% increase in one week.

"A downgrade from its BBB rating would risk pressuring IG-only investors into forced sales," Reid said.

Just below the broad smile he’s been wearing all morning, John Ternus’ hands are shaking.

On this September day in Silicon Valley, Apple’s new CEO is surrounded by celebrities, tech enthusiasts, and reporters who have flocked to Apple Park from around the world to catch a glimpse of the company’s new and long-awaited foldable phone, the Duo. After Ternus takes a few photos with the device, an Apple PR rep pulls the influencer known as IShowSpeed out of the crowd to meet him. Winning over the 21-year-old isn’t exactly the toughest assignment Ternus has faced in his 25-year Apple career: Speed barks when he sees the phone and then gleefully springs into a backflip.

But IShowSpeed also happens to be livestreaming the encounter to his 61 million subscribers on YouTube. And as Ternus thumbs through the $1,999-and-up phone, attempting to drag a photo into a text-message box, eagle-eyed viewers notice his hands quivering. Ternus later explains that he didn’t sleep much the night before. “I was on such a moment of adrenaline and caffeine and excitement for the day,” he tells TIME.

Ternus, who took over as CEO on Sept. 1, was chosen to run Apple because he has ably applied his engineering and design expertise to shape products that redefined entire categories of technology, from the Mac to iPad to AirPods and the iPhone. But becoming the face and front man for the second biggest company in the world by market cap is another proposition entirely. Ternus’ hard technical skills may not translate easily to a position that involves toggling from Hollywood to stockholders, to politicians shaping policy, to rival CEOs in the constellation of tech companies whose power challenges nation-states.

The recently unveiled iPhone Duo.

His predecessor, Tim Cook, gained stature on the world stage as a power broker deftly navigating global politics, markets, and supply chains. Cook’s predecessor, Steve Jobs, was a rock-star visionary who sold the world on the deeply personal and even spiritual nature of tech creations. Ternus is different from both: a private, detail-oriented engineer, little known outside the company until he became a senior hardware executive in 2021.

Rather than try to replicate his forerunners’ styles of success, Ternus has so far played to his own strengths—which are also those of Apple, a company that first and foremost builds beautiful products that spark productivity, creativity, and joy. Ternus’ interaction with IShowSpeed is clearly designed to capture this (and, of course, to goose preorder numbers among the influencer’s young and massive audience). To some critics, however, the scene might read as a legacy company aura-farming to sell expensive gizmos instead of building the actual technological future, which is artificial intelligence.

Virtually every one of Apple’s fellow Goliaths—Microsoft, Google, Meta, Amazon, and now Nvidia, OpenAI, and Anthropic—have flung themselves into the AI arms race, pouring hundreds of billions of dollars into training ever more intelligent thinking machines. While Apple trains its own models, it is nowhere near the technology’s frontier, leaving the company vulnerable to being left behind if super-smart AI transforms industry and society in the coming years. At the end of September, Apple was conspicuously absent when President Donald Trump hosted a luncheon for AI power players like SpaceX’s Elon Musk, Meta’s Mark Zuckerberg, and Google’s Sundar Pichai, who lobbied him on their visions for America’s tech future.

But Ternus and other Apple executives say they aren’t concerned. In the midst of a populist AI backlash, they contend that they are building a modest but personalized and privacy-preserving form of AI that will actually help ordinary people as opposed to merely dazzling Silicon Valley’s “lunatic fringe,” as Greg Joswiak, senior vice president of worldwide marketing, puts it. And besides, Ternus argues, even if other companies succeed in their mission of building super-intelligence, users will still want to talk to it on Apple devices.

“People want physical things,” he says a week later at the company’s flagship store on Fifth Avenue in New York City, wearing an amiable but guarded smile. He spent the morning greeting superfans who had lined up all night to get their hands on the iPhone 18. “They want to talk, to see images of loved ones, to take pictures. So I don’t see a scenario where all of a sudden, our devices disappear and we don’t have a need for an iPhone.”

Ternus is barely older than the company he’s running. Born in 1975 in Orange County—Apple was founded in ’76, just 400 miles north—he grew up as Steve Jobs was redefining the personal computer as a societal force. When Ternus was 8 years old, his father bought an Apple II. “I started playing games on it and fell in love with Apple,” he says. A decade later, he took his family’s newer Mac, an SE model, to college at the University of Pennsylvania, where he studied mechanical engineering.

In 2001, Ternus left a virtual reality company to join Apple at 26 as an engineer. He has now spent half his life at the company, rising through the ranks on his propensity to consider both how products are made and how the user will experience them. “John was really at the center of driving the spirit of collaboration across the company to make sure that end-to-end these products give you that delightful experience—and the technology kind of disappears as you’re using it,” says Richard Dinh, Apple’s vice president of product design.

Over the past 15 years, some critics have accused Apple of failing to innovate the way it did in previous eras, when the company popularized the home computer, the MP3 player, and the smartphone. But it is undeniable that Apple’s core products—its computers and phones—have remained elite thanks to behind-the-scenes innovations in hardware and software. And Ternus, Apple executives say, has been a big part of these improvements.

When Cook, 65, announced in April he would step down following a 15-year tenure, it was no surprise that the company tapped Ternus as his successor. Other executives describe him as pragmatic, meticulous, and unifying. “Ternus is a logical pick,” says Neil Cybart, an analyst who has been following Apple for more than a decade on his site Above Avalon. “He has been a big believer in this design-first culture that defines Apple, as well as the idea that it’s not just technology here; there needs to be something more.”

In the late 2010s, Apple executives shifted their messaging to focus more on fast-growing services like iCloud, the App Store, and Apple Music. But growth in service revenue has moderated this year, and the promotion of Ternus, a hardware and product guru, signals a return to the company’s roots.

So it’s fitting that his ascension coincided with the announcement of the Duo, Apple’s most radical design change to the iPhone since its 2007 release. Apple has been exploring the concept for years. In 2011, when many people still carried around flip phones, Ternus and five other engineers filed a patent application for foldable electronic devices, exploring what kind of batteries and circuit boards might be needed for such an idea. “It felt clear that we were going to eventually get to a place where OLED displays were going to be flexible,” Ternus says.

Many challenges stood in between the patent and the final product. The first attempts at a foldable smartphone in the 2010s from other companies tended to be fragile and not all that useful, with a big, ugly crease in the center interrupting the experience. “The other examples in the world felt kind of cheap and crappy,” Ternus says.

Across hardware and software teams, Apple engineers and designers got to work. With advancements in Apple silicon chips, they created a new display engine that facilitates seamless transitions between the inner and outer screens. They redesigned iOS to move the phone’s main controls from the bottom to the right side for easier thumb access.

Along the way, Ternus encouraged the team to invest more time and effort in the project, even as other foldables beat them to market. Dinh says that at one point, he asked Ternus whether his team should proceed with a top surface material similar to those of its competitors, which Dinh felt had flaws but was potentially good enough.

“And he was like, ‘Look, Rich, you’ll know when it’s the right thing.’ And that gave us the confidence to really push the boundaries on innovation and manufacturing, and fly worldwide to connect our suppliers together and come up with the nanotexture [a surface technology],” Dinh says. The result—a surface made from a custom polymer and built atop a hinge with more than 100 tiny interior precision components—has impressed its first users, who have said they can barely detect a crease at all.

Ultimately, the Duo promises a passport-size phone that opens into a powerful tablet. Users can stow it in their pockets, then open it up and watch a movie on its matte, light-deflecting surface. They can compare websites side by side, play touchscreen games with intricate detail, and use a dual-camera setup to simultaneously show themselves and what they’re seeing on FaceTime.

Of course, the Duo’s success is far from guaranteed. It could go the way of Apple flops like the Apple Car, which was never released, or the Vision Pro, which had staggeringly low sales in the last year. Certainly many prospective buyers will blanch at the device’s $2,000-plus price tag.

But the phone’s first public testers have given glowing feedback. And demand for the Duo’s closest rival, Samsung’s Galaxy Z Fold8, has been strong, suggesting there is a growing appetite for the foldables category, especially in Asia.

Apple also hopes that the Duo will earn a cultural cachet among storytellers and other artists, many of whom, like loyal Apple customers, still regard the company more as a maker of useful tools rather than as the enemy Big Tech has become. Four-time Oscar winner Sean Baker (Anora), who spoke at a “no-AI” film festival earlier this year, attended the Duo announcement in Cupertino and described his relationship with Apple as collaborative. “It’s not just ‘buy or use our product’—it’s much more of a working relationship where they stay in touch and ask us for feedback,” he says.

Baker noted that Apple’s products often underpin many parts of the filmmaking process. For instance, when moviemakers need to add dialogue in postproduction, they often simply record on an iPhone, because it’s convenient and nearly production-grade.

In person, Ternus presents as a guy trying to blend in: he dresses in nondescript clothes and speaks in platitudes, often about the brilliance of Apple’s internal culture. He has devoted much of his early tenure as CEO to obsessing over design, spending multiple days a week in the studio with the industrial-design and human-interface teams.

But beyond Apple Park’s gleaming, curved walls, Ternus has tough choices to make. Case in point: the ongoing global shortage of memory chips, which has raised costs and delayed production timelines. In July, the Financial Times reported that Apple had begun testing memory chips made by a Chinese company as a way to increase supply. That drew a swift rebuke from a bipartisan group of Senators, who wrote a letter to Apple arguing that to do so would imperil national security.

Ternus pushes back on that idea. “The constraints in memory are absolutely a crisis,” he tells TIME in the second of two interviews during September. “So our fundamental belief is that all options should be on the table.”

Cook was skilled at managing relationships with governments around the world, vigorously defending Apple’s interests even if it made him a target of ridicule. His courtship of Trump, which included a widely maligned presentation of a disc with a 24-karat gold base at the White House in 2025, helped Apple secure relief from some of the Administration’s most punishing proposed levies.

Ternus is tight-lipped about how his approach to government affairs might differ from Cook’s. He says that he’s met with members of the Trump Administration to “make sure we’re kind of working through things together.” But as Ternus transitions into the role, Cook has continued to represent Apple on the world stage, meeting with Australian Prime Minister Anthony Albanese, European Commission President Ursula von der Leyen, and then attending President Trump’s state dinner for Xi Jinping.

“He has great relationships and loves Apple,” Ternus says of Cook, who is staying on as Apple’s executive chairman. “So wherever he can help, he is happy to help.”

Big Tech has been hammered by lawsuits, public campaigns, and regulation over the past few years from critics who feel that screens have poisoned the cognitive development of an entire generation. From Australia and the U.K. to California and New York, governments have restricted children’s social media and AI use and phones in schools, while pressuring Apple and Google to build stronger protections into their devices.

Apple has traditionally framed many of these harms as arising from particular apps and content. But the company rolled out a redesigned Screen Time experience in September to give parents greater control over their children’s app access. Ternus pledges that child safety will be central to Apple’s approach going forward. He says that in the past, he used Apple’s Screen Time to monitor his kids’ device usage, and that it was “good but not great.” The company’s new updates, he says, stem from a desire “to build tools that make that really easy for parents—and I think we’ve done just that.”

Questions around geopolitical tensions, supply chains, and child safety only become more urgent as Apple wades into its next frontier: artificial intelligence.

Since the rise of ChatGPT in November 2022, the generative AI boom has unfolded with Apple largely sitting on the sidelines. OpenAI brought AI chatbots to the masses; Nvidia built the chips powering them. Microsoft, Google, Meta, Amazon, Anthropic, and SpaceX now all either compete at the technology’s frontier or are printing revenue from a key part of its supply chain.

Apple, in contrast, did not unveil Apple Intelligence until June 2024—18 months after ChatGPT’s release—and then spent more than two years struggling to deliver the personalized version of Siri it had promised. In 2025, realizing the existing Siri couldn’t grow into the AI era, the company scrapped it in favor of another version that had been two-tracked. The resulting new Siri, which launched in the U.S. in September, is powered by an Apple AI model that leverages Google’s Gemini technology. This arrangement puts Apple in an unusual position: reliant on a rival for a key building block of its products. This sort of outsourcing famously angered Jobs, who went to great lengths to own the “whole widget,” from hardware to software to services.

But Ternus rejects the idea that its AI strategy represents a retreat from that philosophy. After all, Apple relies on suppliers for many of its components, including screens, memory, and modems. “There’s not really a goal that we have to make every single thing in a product,” he says. “We make the things where we can really bring differentiation.”

Apple has said that AI is core to the company’s focus, and that it has significantly increased its investments in AI. The company also argues that the usefulness of AI comes less from its underlying raw power than what surrounds it—the chip it runs on, the operating system it inhabits, and the knowledge it has gathered about its user. The new Siri cannot code like OpenAI’s Codex or write as elegantly as Anthropic’s Claude. But with a user’s permission, it can draw connections across messages, emails, calendars, documents, and other information stored on Apple devices. It is designed to help prepare someone for a meeting, find a flight confirmation buried in an inbox, or help interpret health data collected by an Apple Watch.

“Everything in your life is on your phone,” Ternus says. “With Siri AI, you can leverage all that information to do things, learn things, and figure out things in a way that nobody else can do.”

Cybart, the analyst, argues that Apple has seen this kind of competition before. “People tend to think that if someone else does great, that must mean Apple has a problem, like in music or television streaming,” he says. “But how do you access Spotify or Netflix? When other companies do great things, that doesn’t change the fundamental story with Apple in terms of having the ecosystem that people want to be in.”

And Apple executives argue that some of Siri’s limitations are actually strengths. For instance, Siri’s lack of thinking power compared with other ultra-smart AI agents means that there’s far less of a chance of going rogue, or behaving in eerie ways. “We’ve purposely designed it not to be a companion, and we don’t want to replace human connection,” says Lilian Rincon, who leads Apple’s AI product efforts.

Privacy is another advantage Apple executives stress. The company has a decades-long reputation for protecting user data more aggressively than its competitors. Siri processes requests on-device where possible, and more computationally demanding asks are sent to Private Cloud Compute. Apple’s system of servers is designed so that queries are not stored or made available to the company. Apple has also made portions of that system inspectable by independent security researchers, some of whom have praised Apple’s approach while noting limits to its transparency.

Some investors are already treating Apple as a hedge against the other AI companies dominating the Nasdaq; if those companies wobble, Apple could be the big stock-market winner. Conversely, if AI progress continues apace, smaller but capable models could soon become able to run cheaply on devices built to handle them. In this future, the scarce asset would be not the chatbot, but the device, operating system, and personal context through which the chatbot becomes useful. And Apple has built a significant moat in the form of loyal customers across the world.

This thesis was already borne out earlier this year, when a wave of AI enthusiasts bought Mac Minis in order to set up personal AI agents through the open-source project OpenClaw.

While Apple had nothing to do with the project, users quickly figured out that the energy-efficient Apple silicon, compact form, and unified memory of the $600 computers allowed them to run AI workloads locally, continuously, and for minimal electricity costs. This spike in demand caused some Mac Mini models to be unavailable for months.

More AI-oriented hardware products are on the way. On Oct. 13, Apple announced a new smart-home hub called the HomeView, powered by Apple intelligence. Bloomberg also reported the company is readying smart glasses for 2027; the company declined to comment.

But Apple will soon face more competition in the hardware space. Last year, OpenAI acquired the startup of Jony Ive, Apple’s longtime chief design officer, who is now building AI-first devices for the ChatGPT maker alongside several other prominent former Apple veterans. In July, Apple sued OpenAI in federal court, alleging the theft of hardware trade secrets. (OpenAI calls the lawsuit “baseless.”)

Despite the lawsuit, Apple executives say they’re not worried about the newfound rivals. After all, Apple devices have never been more popular. With 2.5 billion active devices, the company rakes in nearly half a trillion dollars in annual revenue. And when new users convert to Apple, they tend to stick around. “We don’t feel threatened,” says Joswiak, the longtime executive. “The fact that we can do the integration that we can means we can create just a better experience than anybody else.”

Joswiak adds that Apple isn’t making products for the die-hard AI believers chasing superintelligence. “We love them, and they can use it too, but that’s not our target,” he says. “It’s how to make it better for everyday people.”

On Sept. 18, lines formed outside Apple stores from Hangzhou to London to Mumbai, with scores of those “everyday people” spending hundreds of their hard-earned dollars on the newest iPhone. Ternus, as Tim Cook and Steve Jobs did before him, showed up to the Fifth Avenue store to take pictures with superfans, who treated him like a movie star. Most of them probably didn’t know much about Ternus himself. But they didn’t need to, because they saw him reflected through devices that had become omnipresent in their lives.

AI may yet be a part of their lives too. But no one knows the future—and at present, handheld devices are still king. “You can’t do everything,” Ternus says. “So we kind of try to constrain ourselves to the stuff that matters most—and then put all of our effort into those things.”

Amazon S3 is now available on the Vercel Marketplace and v0 by Vercel

Developers building web and AI applications on Vercel can now add Amazon S3 storage resources directly from the Vercel Marketplace and v0, combining the durability, scalability, and performance of S3 with the convenience of Vercel’s provisioning workflow. Starting today, Amazon Simple Storage Service (Amazon S3), including Amazon S3 Tables and Amazon S3 Vectors, is natively available on the Vercel Marketplace and v0 by Vercel, covering three distinct workloads: everyday storage, analytics, and AI.

Why this matters

Millions of developers build, deploy, and scale web and AI applications on Vercel. Storage is essential to performant applications, but provisioning it previously required navigating to the AWS Management Console to configure policies and credentials. Now, S3 storage integrates directly into the Vercel developer experience, with resource provisioning and connection configuration handled automatically. Together with existing integrations for Amazon Aurora, Amazon DynamoDB, and Amazon OpenSearch Serverless, you get a complete application stack inside Vercel (from your database layer through to object, vector, and open table storage), all on one AWS account with one bill.

Storage for every application

• S3 general purpose buckets: Store virtually unlimited objects, including files, images, videos, and documents, for data lakes, cloud-native applications, and mobile apps. S3 delivers the most durable storage in the cloud, designed to provide 99.999999999% (11 nines) data durability and 99.99% availability by default.
• S3 Tables: Fully managed Apache Iceberg storage optimized for analytics with automated table maintenance and compaction. S3 Tables are directly queryable from any Iceberg REST catalog API-compatible engine, including DuckDB.
• S3 Vectors: Purpose-built vector storage and query for AI applications such as retrieval-augmented generation (RAG) and semantic search, reducing the cost of uploading, storing, and querying vectors by up to 90%.

Getting started

From the Vercel dashboard, go to the Storage tab and choose Create. Select AWS, pick your S3 bucket type, choose a Region, and create the bucket. Provisioning takes a few seconds.

Connect the bucket to a project, then pull the environment variables with the Vercel CLI:

vercel link 
vercel env pull .env.local

The integration injects AWS_REGION, AWS_ROLE_ARN, AWS_RESOURCE_ARN, and a bucket name such as S3_BUCKET_NAME, S3_TABLE_BUCKET_NAME, or S3_VECTOR_BUCKET_NAME. At runtime, the awsCredentialsProvider from @vercel/functions/oidc exchanges a short-lived OIDC token with AWS Security Token Service (AWS STS) for temporary credentials scoped to your resource. No static keys are stored anywhere.

Object storage with S3

This Next.js route uploads a file to a general purpose bucket:

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { awsCredentialsProvider } from "@vercel/functions/oidc";

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  credentials: awsCredentialsProvider({
    roleArn: process.env.AWS_ROLE_ARN,
  }),
});

export async function POST(request) {
  const formData = await request.formData();
  const file = formData.get("file");

  await s3.send(new PutObjectCommand({
    Bucket: process.env.S3_BUCKET_NAME,
    Key: file.name,
    Body: Buffer.from(await file.arrayBuffer()),
    ContentType: file.type,
  }));

  return Response.json({ success: true });
}

Use ListObjectsV2Command and GetObjectCommand with the same client to list and read objects.

Analytics with S3 Tables

S3 Tables offer S3 storage purpose-built for analytics and are designed to automatically optimize query performance as data volumes grow. They provide fully managed Apache Iceberg tables with automated table maintenance and compaction, and they work with any Apache Iceberg REST Catalog-compliant engine. You can use DuckDB inside a Vercel Function for lightweight queries. This example attaches the table bucket by its ARN and runs an aggregate query:

import { DuckDBInstance } from "@duckdb/node-api";
import { awsCredentialsProvider } from "@vercel/functions/oidc";

const creds = await awsCredentialsProvider({
  roleArn: process.env.AWS_ROLE_ARN,
})();

const db = await DuckDBInstance.create();
const conn = await db.connect();

await conn.run("INSTALL aws; LOAD aws; INSTALL httpfs; LOAD httpfs; INSTALL iceberg; LOAD iceberg;");

await conn.run(`
  CREATE SECRET s3cred (
    TYPE s3,
    KEY_ID '${creds.accessKeyId}',
    SECRET '${creds.secretAccessKey}',
    SESSION_TOKEN '${creds.sessionToken}',
    REGION '${process.env.AWS_REGION}'
  );
`);

await conn.run(`
  ATTACH '${process.env.AWS_RESOURCE_ARN}' AS s3tables (
    TYPE iceberg,
    ENDPOINT_TYPE s3_tables
  );
`);

const rows = await (await conn.run(`
  SELECT town_city, AVG(median_price) AS avg_price
  FROM s3tables."${process.env.S3_TABLES_NAMESPACE}".town_monthly
  GROUP BY town_city
  ORDER BY avg_price DESC
  LIMIT 10
`)).getRowObjects();

If you run DuckDB inside a Next.js route, add @duckdb/node-api to serverExternalPackages in next.config.mjs so the native bindings are not bundled.

Vector search with S3 Vectors

S3 Vectors deliver purpose-built, cost-optimized vector storage for AI agents, AI inference, and semantic search of your content stored in Amazon S3. S3 Vectors provides native support for storing and querying vectors without a separate vector database, but it does not generate them. Create the embeddings with your model of choice, then query the index for the closest matches:

import { S3VectorsClient, QueryVectorsCommand } from "@aws-sdk/client-s3vectors";
import { awsCredentialsProvider } from "@vercel/functions/oidc";

const vectors = new S3VectorsClient({
  region: process.env.AWS_REGION,
  credentials: awsCredentialsProvider({
    roleArn: process.env.AWS_ROLE_ARN,
  }),
});

export async function POST(request) {
  const { embedding } = await request.json();

  const result = await vectors.send(new QueryVectorsCommand({
    vectorBucketName: process.env.S3_VECTOR_BUCKET_NAME,
    indexName: "documents",
    queryVector: { float32: embedding },
    topK: 5,
    returnMetadata: true,
  }));

  return Response.json({ matches: result.vectors });
}

Building with v0 by Vercel

From v0, describe what you want to build and it provisions the right resource. Ask for “a photo gallery where users upload images using AWS” to get a general purpose bucket, “a support bot that answers questions from my PDFs” to get S3 Vectors, or “an analytics dashboard for my clickstream data” to get S3 Tables. v0 then configures the connection and generates working code.

Combining S3 with AWS databases

Every AWS service on Vercel shares the same account and authentication model, so you can combine them with no extra setup. For example, store file metadata in Aurora with the files in S3, or store document text in DynamoDB with embeddings in S3 Vectors for RAG retrieval.

Using Vercel with your AWS account

You can create a new AWS account through the integration, which supports S3 features for building and testing. You can also link an existing AWS account to use your full set of S3 and AWS capabilities and keep your current billing. If you link an existing AWS account, Vercel works only with new buckets that you create through the platform. Because every S3 resource lives in your own AWS account, you keep AWS-native tooling and a single place to manage storage.

Pricing and availability

There is no charge for the integration, and you pay standard Amazon S3 pricing for the resources you use. New AWS accounts created through Vercel receive $100 in credits valid for six months. Amazon S3, including S3 Tables and S3 Vectors, is available from v0 and the Vercel Marketplace in all commercial AWS Regions supported by Vercel. To get started, visit v0 or the AWS page on the Vercel Marketplace.

Join the conversation

If you want to call APIs, search documentation, find regional availability, and check troubleshooting about this new feature, try using the AWS MCP Server and plugins with your preferred AI tool. Share your feedback on AWS re:Post for Amazon S3.

Hyperdrive support for MySQL is now generally available (GA). Hyperdrive, Cloudflare’s database connection pooler for Workers, initially only supported Postgres. Since we announced the beta for MySQL protocol support last year, we have been working on making our MySQL handling as reliable as our Postgres support.

We’ve learned some things from building and running a MySQL connection pooler on Cloudflare’s network. It turns out that the difference between beta and GA is just 20 bug fixes in a trenchcoat.

During the beta, our goal was to validate that real-world customer usage hit our production availability target of 99.99%. We knew there were gaps in our reliability since we track how often client connections from a Worker end in an error related to Hyperdrive. In addition, we had feedback from customers in the beta that led to the discovery of a rare and subtle bug in our production environment. To convey the complexity of this particular bug, we first need to explain how Hyperdrive works.

What is Hyperdrive?

Hyperdrive is a distributed database connection pooler that connects Cloudflare Workers to SQL databases. Hyperdrive holds warm SQL connections open on our global network and shares these connections across many Worker requests. In addition to connection pooling, Hyperdrive can also cache common queries in our global CDN cache, reducing latency and load on the origin database.

Establishing a connection to a SQL database is expensive, taking a minimum of five network round trips. On a serverless compute platform like Workers, establishing a connection for each request would be impractical, both because of the amount of latency added to each request, but also because a large number of open connections can incur high per-connection resource costs. While MySQL fares better here than Postgres since MySQL doesn’t use an entire OS process per connection, a typical deployment can still run into resource issues with a large number of connections.

A journey to stability

Some of the issues we ran into were small and easy to fix once reported. One of the first problems reported in the beta was that emojis written to the database were not being correctly returned. This was a bug related to character set encoding. Hyperdrive negotiates the encoding for the connection in the startup handshake, and we negotiate utf8 as the encoding. As it turns out, utf8 is a deprecated alias for utf8mb3 which is not all of UTF-8: it only supports 1–3 bytes per Unicode code point, while full UTF-8 has a max of 4 bytes per code point. To get all of UTF-8, we needed to change our handshake to use utf8mb4, which fixed the emoji issue.

Another encoding-related bug occurred when we added connection cleanup via COM_RESET_CONNECTION. This is a lightweight MySQL command to clear any session state in a given connection, which is exactly what Hyperdrive needs to ensure that no session-level state leaks between checkouts. But adding this connection cleanup step caused encoding issues once again! It turns out that COM_RESET_CONNECTION does not reset the connection state to the character set encoding we negotiated at connection startup; it instead resets the connection to the server default. This means that after negotiating utf8mb4 on a MySQL server instance that defaults to latin1, once we run COM_RESET_CONNECTION, our connection now has an encoding of latin1. The fix is to always run a command setting the encoding to utf8mb4 after a connection reset.

A hard-to-find bug

During the beta, our highest-volume MySQL customer reported a small but consistent rate of empty query responses for which a non-empty response was always expected. We started the investigation by combing through all of our telemetry and understanding the user’s logic. Then we also increased the trace rate for this Hyperdrive instance, hoping to see the execution flow of this bug. We searched for network issues that could explain the issue — but nothing seemed to account for this behavior. This error occurred about 0.001% of the time, which made it very hard to find relevant end-to-end execution traces.

But we did have one clue: the bug only occurred on Hyperdrive instances with caching enabled. This made us suspect an issue with Hyperdrive’s MySQL caching.

Hyperdrive caching

To understand Hyperdrive's caching, we first need to talk about Hyperdrive's topology and how it interacts with the CDN cache. Every Hyperdrive instance has a split architecture with two nodes: a Client and an Endpoint. The Client node runs on the same machine as the user’s Worker, handles local caching (if enabled), and forwards queries to the Endpoint if needed. The Endpoint node holds the connection pool and is placed in the data center closest to the origin database. Each node uses the Cloudflare CDN cache local to the data center it is running in. Importantly, this means that if the two nodes are running in the same data center, they will share a cache.

Sharing the cache between Client and Endpoint is generally good for performance. We use a hash of the contents of the query’s wire messages with some normalization and store the result bytes in the cache. If a Hyperdrive Client running in the same data center as its Endpoint receives a request, and that Endpoint has recently cached the results of the same query, the Client can serve the response directly from the shared data center cache. To work correctly, this scheme requires that the same bytes are cached for the same response, whether stored by the Client or the Endpoint.

MySQL client/server protocol

Since we couldn’t get a direct trace of the issue, we decided to spend time looking at the MySQL client / server protocol. Perhaps we had an assumption about Postgres’ protocol built into our MySQL handling that was causing these issues.

The protocol describes how a MySQL client communicates with a MySQL server. This can be somewhat confusing in the case of database connection pooling since we have two components acting like clients and two acting like servers. The first client is the Worker using a MySQL client library, and it connects to the first component acting like a server, the Hyperdrive Client. The Hyperdrive Client connects to the Hyperdrive Endpoint which acts as a MySQL client connecting to the actual MySQL origin server.

This protocol starts with the Connection Phase where the MySQL client and the MySQL server agree upon a common authentication method and capabilities. Once a connection is established, the client and server begin the Command Phase, where the client sends commands (e.g. SQL statements) to be executed with the server responding and waiting for the next command.

Capabilities are exchanged during the Connection Phase to support protocol evolution while remaining backwards compatible. Each capability is a feature, like supporting compression during the Command Phase. The clients and servers must only use features which the other supports, otherwise a protocol desync can occur, leading to unexpected behavior.

This capability negotiation can be tricky for Hyperdrive, since Hyperdrive acts as a client to the origin database, and acts as a server to the Worker. To ensure that Hyperdrive can parse the messages from the origin MySQL server, Hyperdrive has a fixed set of capability flags which are used to communicate to the origin database. Since it is possible for Hyperdrive’s set of capabilities to be different from the Worker client’s set of capabilities, Hyperdrive needs to translate all origin responses encoded by its capabilities to the capabilities that the Worker expects.

While trying to debug this issue, we found that a popular MySQL client library for Node.js (mysql2) does not currently support the CLIENT_DEPRECATE_EOF capability, which changes how result sets are terminated. Without this capability, the server sends an EOF packet between the column definitions and rows, then another EOF packet at the end of the result set. When CLIENT_DEPRECATE_EOF is negotiated, the metadata EOF packet is omitted and the final EOF packet is replaced with an OK packet. MySQL server 5.7.5 and later supports this capability, but the newer OK-packet behavior is only used when the client negotiates it.

While Hyperdrive does support the CLIENT_DEPRECATE_EOF capability, it’s possible that the Worker’s MySQL driver does not. To handle this, Hyperdrive adjusts its behavior to support the capabilities that the Worker client driver expects. This means rewriting result sets to include EOFs and to translate all OK packets to EOFs when required.

After days of looking through telemetry, documentation, and Hyperdrive code, everything clicked. We finally came across the last missing piece: a faulty assumption in our Hyperdrive Client caching code. We baked in an assumption when implementing the Client caching for Postgres: that we can cache exactly the bytes we wrote back to the Worker. While this was a reasonable assumption to make with Postgres’ wire protocol, it no longer holds due to the mismatch in MySQL capabilities. If a Client node is caching exactly what it writes back to the Worker, these bytes are in a different format than what an Endpoint node would be expecting. That’s because the Endpoint node is always working with Hyperdrive’s capabilities, while the Client node writes back bytes that are translated to match the Worker’s capabilities.

Now if we put all these clues together, we can understand scenarios where Hyperdrive can return a MySQL response encoded with the wrong capabilities, leading to the unexpected empty results and hangs. The bug occurs when both the Client and Endpoint are in the same data center, sharing a cache, and when one reads a value cached by the other.

Endpoint caches, Client reads

1. The Endpoint is in the middle of servicing a request for query Q. It gets back a response R from the origin server and caches response result bytes R_hyperdrive (Response encoded with Hyperdrive capabilities) into the shared cache.
2. Client receives another request for query Q, checks the shared cache and finds R_hyperdrive.
3. Client responds to the Worker with R_hyperdrive without translating to the Worker’s capabilities.

Client caches, Endpoint reads

1. Client is finishing serving a request for query Q, and it writes the translated response bytes R_worker to cache.
2. Endpoint is in the middle of servicing a request for Q, and checks the cache to find R_worker.
3. Endpoint sends R_worker to a different Client, and this Client tries to translate R_worker to the Worker’s capabilities, and ends up sending an early EOF or hangs.

Once we found this type of failure mode, the fix was simple. We now make sure that we always write back a canonical version of the query result bytes in the cache encoded using Hyperdrive’s capabilities, no matter which node stored it. After releasing this change, the empty row problem went away — and our success rate for Client connections reached 99.99%.

Reliability is a feature

You can use Hyperdrive support for MySQL today! Create a Hyperdrive to an existing MySQL-compatible database via the dashboard or wrangler, or connect to a new PlanetScale database. If you run into any issues or have questions, reach out on our Discord.

npx wrangler hyperdrive create HYPERDRIVE_NAME --connection-string="mysql://user:password@db-host.example.com:3306/dbname"

Announcing Snowflake Dynamic and Rotated Credentials with Pulumi ESC

Secure your Snowflake access with Pulumi ESC's new dynamic OIDC login for temporary credentials and automated RSA keypair rotation for user secrets.

Will AI vibecoded apps soon displace paid legacy software like Photoshop and Microsoft Word? Or will the deluge of apps spun up by AI models show that those legacy apps are superior products?

Earlier this month, Reddit user u/ai_art_is_art, who Tom's Hardware revealed to be a software engineer named Brandon Thomas, shared a project he's been working on: A rebuild of the entire Adobe Creative Suite, under the banner ArtCraft. There is one key difference between ArtCraft and Adobe software, however: You have to pay to access Adobe.

From Adobe Premiere to Photoshop to After Effects, Thomas recreated them all with help from AI in Rust, a programming language. Named PhotoCraft, FilmCraft, DesignCraft, EffectCraft, PdfCraft, VectorCraft, and LightCraft, each duplicated one of the signature apps found in Adobe's software suite for creatives.

And Thomas has released them all as free, open-source tools.

Now, the developer is taking on the Microsoft Office suite.

Thomas has just released WordCraft, DeckCraft, and GridCraft (along with additional creative duplicates like SoundCraft, DesignCraft, and CADCraft), which are his take on Microsoft Word, PowerPoint, and Excel.

These apps have caused a bit of a controversy on Reddit and elsewhere on the internet over AI-generated code. While Thomas has clarified that he is a software developer who knows how to code, concerns over cybersecurity and the ethics of AI-generated code have resulted in posts about his apps being removed from some subreddits.

In addition, there is concern that Thomas is now overseeing a suite of apps by himself, although the community has been invited to contribute to the open-source project as well.

Thomas is clearly pro-AI, which has also been a point of contention for some critics. He has released these apps under his ArtCraft brand, a for-profit suite of tools for AI-generated art. Also, if you take another look at Thomas' Reddit username, it is "AI art is art."

Supporters of Thomas and his apps, however, view this as an act of protest against companies like Adobe and what they see as unscrupulous subscription tactics, in which Adobe makes it difficult for customers to cancel plans.

It's unclear if Thomas's apps have staying power at this point, but that hasn't stopped social media users from speculating that the era of paid legacy software is over. Indeed, we're seeing a similar trend in the gaming world, where AI-assisted jailbreaks and decompiled games are now readily available.

AI can produce twelve credible variations before the coffee cools, so the design crit should become a debrief, like an F1 team studying telemetry over finishing position. Teams ask about reasoning, friction with the model and verification, and also critique the setup: briefs, constraints, and rubrics. AI critics tend to agree with the designer, and great AI-first designers sense limits, give precise feedback, and know when to override the model.

Every page, UI, and campaign your team ships is another decision about what your brand is. We’re making more of those decisions than ever, with more people, and more agents involved.

Agentic Brand Systems are how we can turn each touchpoint into a stronger extension of a brand’s living source of truth.

Most brand owners reading this will think it can wait. It can't. The math is already starting to compound against you.

Picture a company a month after a successful raise. Two founders and a couple of engineers have become a team with new hires in product, marketing, and brand.

In their first week, the product designer opens the design system. It has not been updated in months, although the live product has changed twice since. The new marketer opens the Canva folder for an upcoming campaign. They find several logos, copy from different eras of the company, and a read-only voice document owned by someone who no longer works there. Across the room, the head of brand opens the agency's Figma guidelines. The founder has been making new pages with AI every week, so the file is already behind the site.

Everyone is trying to do good work. They are just working from different versions of the company.

The usual fix is another meeting, a new PDF, or one person who becomes the answer to every brand question. Then that person goes on vacation, a new tool arrives, or the company launches something the guidelines never anticipated.

Now an agent is in the room too. It does not know which file is current. It will make a reasonable choice and keep going.

I have sat on both sides of this table, at an agency and inside a company. I know exactly how each of those people feels, and I want to say:

These scenarios can end. Today.

Full article content is not available for inline reading.

Read the original article →

GPT-Live-1.1 features new voices in French, German, Hindi, Mexican Spanish, Korean, and Japanese. The voices have better accents, and there is less 'checking'. The model is available in ChatGPT, Codex, and dots. It is rolling out now to all users and coming soon to the API.

Tesla unveiled its second-generation Roadster sports car on Thursday by having the vehicle launch itself up a curved ramp into a controlled landing. It later leapt over five other Tesla vehicles while twisting in the air. The car is scheduled to enter production in 2028. Tesla has yet to release a price tag. The Roadster can go from a stop to 60 miles an hour in about one second, and it has a top speed of over 230 miles an hour.

Stablecoins are the public cloud moment for global money movement.

Every job expects employees to be able to use AI, but using AI costs money, which adds pressure when job seeking.

Google's redesigned Play Store for cars with Google built-in adapts to different display sizes and orientations, rolling out automatically over the coming weeks. Entertainment collections for parked use sit apart from driving-related options, and a left-side rail becomes a top bar in portrait. YouTube Premium subscribers get background audio when the car shifts into drive, and Firefox and STARZ join the lineup.

Opera Air has been redesigned around distraction-free browsing, using a cleaner interface that keeps additional tools hidden until they are needed. New wallpapers use open spaces, perspective, and natural textures that adapt to light and dark themes, while built-in Soundscapes combine ambient sounds and original music to encourage focus or relaxation. The browser also includes customizable break reminders, breathing exercises, neck stretches, an ad blocker, and a free VPN while allowing messaging and social apps to remain accessible through the sidebar.

Static personas capture real-world complexity poorly in intelligence-driven products, where outcomes are probabilistic and vary across people and contexts. AI lets teams build products that adapt to individual users at scale, which means designing for a spectrum of outcomes.

Recent Design is a curated platform that focuses on the latest trends in web, interface, and interaction design.

Studio Herrström's identity for Sony Music imprint Milkshake combines references from car culture, flavored drinks, and pop music into a single energetic visual language. The original splash mark has been redrawn as a rounded emblem inspired by fluid, metal, and reflections, accompanied by neon pinks, high-vis yellows, asphalt blacks, soft lilacs, and the neo-grotesque Gates typeface. Information layouts borrow from car window permits and milk carton nutrition tables, creating structured containers for track titles, release dates, copyright information, and other details across digital and promotional materials.

Designers at any level can build a more open design culture by creating psychological safety one conversation at a time.