In early 2025, OpenAI announced Stargate, a joint venture with Oracle and SoftBank, which aimed to invest $500 billion in AI data centers in the United States. But after more than a year of challenges and disagreements, it seems that the startup has abandoned the original idea of directly owning infrastructure alongside its two partners. According to the Financial Times, OpenAI now prefers to rely on third-party providers and lease capacity in the long term.

This is a sensible idea for the startup, which is burning through cash and has reportedly missed internal revenue targets in recent months. But it has also caused chaos among its partners and put its reliability into question. According to the report, OpenAI has "in practice... abandoned the joint venture," choosing instead large bilateral deals with Oracle and more. One person involved with Stargate reportedly said the company had "sidelined first-party data centres," while OpenAI itself admitted that Stargate is merely an "umbrella for our compute strategy."

Stargate's initial goal was to build 20 data centers, with the first project at Abilene, Texas, already operational. However, the three partners reportedly squabbled among themselves for months as they could not agree on who would have ultimate control of the planned data centers. In the end, SoftBank agreed to own and develop the Texas data center, while OpenAI would design and operate it on a long-term lease.

Other Stargate projects located in other areas have also been hit by uncertainties. The UK government signed a deal with OpenAI, among other partners, to build a data center in the UK, but the startup has put it on hold earlier this month. It cited "restrictive regulations" and "high energy costs" as the reason behind the move, but UK AI Minister Kanishka Narayan told the Financial Times that the "only thing that has changed [since] the moment of those commitments…has been the financing environment for OpenAI."

It has also done the same for another Stargate project in Narvik, Norway, with Microsoft stepping up to take over the lease for the site. OpenAI will then lease compute capacity from Redmond, instead of getting it directly from Nscale, the British company that developed the site and also worked on the canceled UK project.

All these changes have got some partners "feeling let down and misled by OpenAI," a person familiar with Microsoft's decision said. Thankfully, the software giant has stepped in on some of the projects that the startup has supposedly abandoned. One source told the publication that money is not unlimited, no matter what Sam Altman might say, while another said that they prefer Microsoft over OpenAI as a tenant, as "they are more creditworthy."

Even though OpenAI has made a name for itself in AI, the startup has not turned a profit since it was founded in 2015. Many institutions believe in its potential, though, with the firm securing $110 billion in its latest funding round — the biggest amount secured in Silicon Valley history and $10 billion more than what the company initially targeted. Still, some analysts estimate that it could run out of cash by mid-2027 with the massive amounts of money it's been throwing around to secure more compute.

Anthropic CEO Dario Amodei criticized moves like this, saying that some of his company's rivals are pushing infrastructure investments too far. However, OpenAI says that it's ahead of the exponential compute curve, allowing it to have an advantage over everyone else. For example, Anthropic has had to limit access to some features on its various products due to limited resources, and Amodei has had to spend more on securing capacity to satisfy the increasing demand

The biggest difference between startups, like OpenAI and Anthropic, and their more established rivals, like Microsoft, Google, Meta, and Amazon, is cash flow. The startups still rely on external funding to fuel their growth, while the big tech companies have billion-dollar revenue that they can rely on to pour into expensive hardware and infrastructure projects.

Google to sell TPU chips to 'select' customers in latest shot at Nvidia

Google parent Alphabet (GOOG, GOOGL) on Wednesday said that it plans to sell its custom Tensor Processing Units (TPUs) to select customers who will install the chips in their own data centers.

The move is a change from Google's prior strategy, which saw it rent out TPU capacity to customers from its own data centers — and is yet another strike at AI chip king Nvidia (NVDA).

The announcement, during the company's Q1 earnings call, comes a week after Alphabet announced two new TPUs: its TPU 8t for AI training and TPU 8i for inferencing.

"As TPU demand grows from AI labs, capital markets firms, and high-performance computing applications, we'll begin to deliver TPUs to a select group of customers in their own data centers in a hardware configuration to expand our addressable market opportunity," Alphabet CEO Sundar Pichai said during the company's first quarter earnings call.

Alphabet didn't disclose potential customers, but it signed a multiple-gigawatt agreement for next-generation TPUs with Anthropic (ANTH.PVT) earlier this month, with chips expected to begin coming online in 2027.

And according to The Information, Alphabet has also entered into a multibillion-dollar chip deal with Meta (META).

Alphabet's TPU maneuvers put it into ever greater competition with Nvidia, which has largely dismissed any fears that Alphabet's offerings will erode its lead in the space, saying that its chips offer greater flexibility for AI developers.

Google isn't the only company moving in on Nvidia's turf. Amazon (AMZN) is also offering up its own chips to customers.

In his annual shareholder letter, Amazon CEO Andy Jassy said that the company's chip business, which includes its Graviton, Trainium, and Nitro processors, has an annual revenue run rate of greater than $20 billion.

But because Amazon only monetizes its chips through its AWS EC2 (Elastic Compute Cloud) service, the CEO explained that $20 billion is likely an understatement and that it would probably be closer to $50 billion.

Like Google, Amazon signed a new agreement for 5 gigawatts of AI chip capacity with Anthropic, but also inked a deal for 2 gigawatts of chips with OpenAI.

On the CPU side, Amazon said it will deploy its AWS Graviton chips for Meta (META) to use across its agentic AI workloads.

Remote agents in Vibe. Powered by Mistral Medium 3.5.

Introducing Mistral Medium 3.5, remote coding agents in Vibe, plus new Work mode in Le Chat for complex tasks.

Coding agents have mostly lived on your laptop. Today we're moving them to the cloud, where they run on their own, in parallel, and notify you when they're done. You can start them from the Mistral Vibe CLI or directly in Le Chat, offloading a coding task without leaving the conversation. 

Powering this is Mistral Medium 3.5 in public preview, our new default model in Mistral Vibe and Le Chat, built to run for long stretches on coding and productivity work. The new Work mode in Le Chat (Preview) extends this with a powerful agent for complex, multi-step tasks like research, analysis, and cross-tool actions.

Highlights.

1. Mistral Medium 3.5, a new flagship model that merges instruction-following, reasoning, and coding into a single 128B dense model. Released as open weights, under a modified MIT license.
2. Strong real-world performance at a size that runs self-hosted on as few as four GPUs.
3. Mistral Vibe remote agents for async coding: sessions run in the cloud, can be spawned from the CLI or Le Chat, and a local CLI session can be teleported up to the cloud.
4. Start Mistral Vibe coding tasks in Le Chat. Sessions run on the same remote runtime and keep going while you step away.
5. Work mode in Le Chat runs on a new agent, powered by Mistral Medium 3.5, that works through multi-step tasks, calling tools in parallel until the job is done.

Mistral Medium 3.5.

Mistral Medium 3.5 is our first flagship merged model, available in public preview. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. It performs strongly in real-world use, with self-hosting possible on as few as four GPUs. Reasoning effort is now configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.

Mistral Medium 3.5 scores 77.6% on SWE-Bench Verified, ahead of Devstral 2 and models like Qwen3.5 397B A17B. It also has strong agentic capabilities and scores 91.4 on τ³-Telecom.

The model was built for long-horizon tasks, calling multiple tools reliably, and producing structured output that downstream code can consume. It is the model that made async cloud agents in Vibe practical to ship.

Mistral Medium 3.5 becomes the default model in Le Chat. It also replaces Devstral 2 in our coding agent, Vibe CLI.

Vibe remote agents.

From today, coding sessions can work through long tasks while you're away. Many can run in parallel, and you stop being the bottleneck on every step the agent takes.

You can start the cloud agents from the Mistral Vibe CLI or from Le Chat. While they run, you can inspect what the agent is doing, with file diffs, tool calls, progress states, and questions surfaced as you go. Ongoing local CLI sessions can be teleported up to the cloud when you want to leave them running, with session history, task state, and approvals carrying across.

Vibe sits between the systems engineering teams already use, with humans in the loop wherever they're needed. It plugs into GitHub for code and pull requests, Linear and Jira for issues, Sentry for incidents, and apps like Slack or Teams for reporting.

Each coding session runs in an isolated sandbox, including broad edits and installs. When the work is done, the agent can open a pull request on GitHub and notify you, so you review the result instead of every keystroke that produced it.

It fits the high-volume, well-defined work that takes a developer's time without taking their judgment: module refactors, test generation, dependency upgrades, CI investigations, as well as bug fixes.

We use Workflows orchestrated in Mistral Studio to bring Mistral Vibe into Le Chat. We originally built this for our own in-house coding environment, then for our enterprise customers. Today the capability opens up to everyone, who can now launch coding tasks from the web. And without being tied to a local terminal, a developer can run several in parallel.

You can start coding sessions directly in Le Chat, so a task described in chat runs on the same remote runtime as the CLI and the web, and comes back later as a finished branch or a draft PR.

New Work mode in Le Chat (Preview).

Work mode is a powerful new agentic mode for complex tasks in Le Chat, powered by a new harness and Mistral Medium 3.5. The agent becomes the execution backend for the assistant itself, so Le Chat can read and write, use several tools at once, and work through multi-step projects until it completes what you've asked.

Here's what Work mode enables you to do today.

1. Cross-tool workflows: catch up across email, messages, and calendar in a single run; prepare for a meeting with attendee context, latest news, and talking points pulled from your sources.
2. Research and synthesis: dive into a topic across the web, internal docs, and connected tools, then produce a structured brief or report you can edit before exporting or sending.
3. Triage your inbox and draft replies; create issues in Jira from your team and customer discussions; send a summary to your team on Slack.

Sessions persist longer than a typical chat reply, so an agent can keep going across many turns, through trial-and-error, and through to completion. In Work mode, connectors are on by default rather than chosen manually, which lets the agent reach into documents, mailboxes, calendars, and other systems for the rich context it needs to take correct action.

Every action the agent takes is visible: you see each tool call and the thinking rationale. Le Chat will ask for explicit approval—based on your permissions—before proceeding with sensitive tasks like sending a message, writing a document, or modifying data.

Get started.

Mistral Medium 3.5 is available today in Mistral Vibe and Le Chat, and powers remote coding agents and Work mode in Le Chat on the Pro, Team, and Enterprise plans.

Through API, it's priced at $1.5 per million input tokens and $7.5 per million output tokens. Open weights are on Hugging Face under a modified MIT license.

It is also available for prototyping, hosted on NVIDIA GPU-accelerated endpoints on build.nvidia.com and as a scalable containerized inference microservice, NVIDIA NIM.

Granite 4.1 LLMs utilize a dense, decoder-only architecture with models of 3B, 8B, and 30B parameters, trained on 15 trillion tokens and using a five-phase pre-training approach. The 8B model matches the performance of the previous 32B Mixture-of-Experts model through a multi-stage reinforcement learning pipeline focused on data quality. These models, designed for efficient, reliable enterprise use, demonstrate competitive instruction-following and tool performance while maintaining cost efficiency and stable usage.

AI evals are becoming the new compute bottleneck

Summary. AI evaluation has crossed a cost threshold that changes who can do it. The Holistic Agent Leaderboard (HAL) recently spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic's $22,000 sweep across agent configurations found a 33× cost spread on identical tasks, isolating scaffold choice as a first-order cost driver, and UK-AISI recently scaled agentic steps into the millions to study inference-time compute. In scientific ML, The Well costs about 960 H100-hours to evaluate one new architecture and 3,840 H100-hours for a full four-baseline sweep. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, scaffold-sensitive, and only partly compressible. Training-in-the-loop benchmarks are expensive by construction, and when you try to add reliability to these evals, repeated runs further multiply the cost.

Making static LLM benchmarks cheaper

The cost problem started before agents. When Stanford's CRFM released HELM in 2022, the paper's own per-model accounting showed API costs ranging from $85 for OpenAI's code-cushman-001 to $10,926 for AI21's J1-Jumbo (178B), and 540 to 4,200 GPU-hours for the open models, with BLOOM (176B) and OPT (175B) at the top end. Perlitz et al. (2023) restate the larger HELM cost pattern, and IBM Research notes that putting Granite-13B through HELM "can consume as many as 1,000 GPU hours." Across HELM's 30 models and 42 scenarios, the aggregate of reported costs and GPU compute came to roughly $100,000.

Another shocking observation came from Perlitz et al.'s analysis of EleutherAI's Pythia checkpoints: developers pay for evaluation repeatedly during model development. Pythia released 154 checkpoints for each of 16 models spanning 8 sizes, or 2,464 checkpoints if each model checkpoint is counted separately, so the community could study training dynamics. Running the LM Evaluation Harness across all those checkpoints turns eval into a multiplier on training: Perlitz et al. (2024) noted that evaluation costs "may even surpass those of pretraining when evaluating checkpoints." For small models, evaluation becomes the dominant compute line item across the whole development cycle. When we scale inference-time compute, we scale evaluation costs.

Perlitz et al. then asked how much of HELM actually carried the rankings. The result was striking: a 100× to 200× reduction in compute preserved nearly the same ordering, with larger reductions still useful for coarse grouping under the paper's tiered analysis. Flash-HELM turned that finding into a coarse-to-fine procedure: run cheap evaluations first, then spend high-resolution compute only on the top candidates. Much of HELM's compute was confirming rankings that the field could have inferred much more cheaply.

Other work reached the same conclusion from different angles. tinyBenchmarks compressed MMLU from 14,000 items to 100 anchor items at about 2% error using Item Response Theory. The Open LLM Leaderboard collapsed from 29,000 examples to 180. Anchor Points showed that as few as 1 to 30 examples could rank-order 87 language-model/prompt pairs on GLUE, and others followed, reducing dataset sizes by 90%. Static benchmarks had a weakness you could exploit: model differences often concentrate in a small subset of items, so ranking can survive aggressive subsampling.

That trick weakened sharply once benchmarks moved from static predictions to agents.

Agent evals are messier

A very nice public accounting of agent evaluation comes from the Holistic Agent Leaderboard (Kapoor et al., ICLR 2026). HAL runs standardized agent harnesses across nine benchmarks covering coding, web navigation, science tasks, and customer service, with shared scaffolds and centralized cost tracking. The headline cost: $40,000 for 21,730 rollouts across 9 models and 9 benchmarks. By April 2026, the leaderboard had grown to 26,597 rollouts. Ndzomga's independent reproduction arrives at almost the same number: $46,000 across 242 agent runs.

Behind that aggregate, the cost of a single benchmark run varies by four orders of magnitude across HAL tasks, and by three orders within some individual benchmarks.

Figure 1. Each bar shows the minimum-to-maximum cost across HAL configurations on a single benchmark. Highlighted bars cross the round $1,000-per-run threshold. A "run" is one full agent evaluation across all tasks. Within-benchmark spread reflects the model × scaffold × token-budget product.

Behind these numbers is a blunt pricing fact. Claude Opus 4.1 charges $15 per million input tokens and $75 per million output. Gemini 2.0 Flash charges $0.10 and $0.40, a two-order-of-magnitude spread on input alone. Agent benchmarks rarely benchmark "the model" in isolation. They benchmark a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×.

Worse, higher spend does not reliably buy better results. On Online Mind2Web, Browser-Use with Claude Sonnet 4 cost $1,577 for 40% accuracy. SeeAct with GPT-5 Medium hit 42% for $171. The HAL paper notes "a 9× difference in cost despite just a two-percentage-point difference in accuracy." On GAIA, an HAL Generalist with o3 Medium cost $2,828 for 28.5% accuracy, while a different agent hit 57.6% for $1,686. CLEAR finds across 6 SOTA agents on 300 enterprise tasks that "accuracy-optimal configurations cost 4.4 to 10.8× more than Pareto-efficient alternatives" with comparable real-world performance.

The static-era toolkit should have helped, but it has only gone so far. Ndzomga's mid-difficulty filter, which selects tasks with 30 to 70% historical pass rates, achieves a 2× to 3.5× reduction while preserving rank fidelity under scaffold and temporal shifts. That is useful, but it falls far short of the 100× to 200× gains available for static benchmarks. When each item is a multi-turn rollout with its own variance, the unavoidable long trajectory per single question becomes the expensive object.

Some evals are just training

Some benchmarks escape the API-cost framing altogether because their evaluation protocol trains models from scratch.

The Well gives a very interesting example of this. It bundles 16 scientific machine-learning datasets spanning biological systems, fluid dynamics, magnetohydrodynamics, supernova explosions, viscoelastic instability, and active matter, totaling 15 TB. Using the paper's headline 16-dataset grid, the protocol leaves little room to economize: train each baseline model for 12 hours on a single H100, try five learning rates per (model, dataset) pair, repeat across four architectures and 16 datasets. That headline-grid sweep consumes 3,840 H100-hours, or roughly $9,600 under the conversion assumptions below. A single new architecture still costs about 960 H100-hours, or about $2,400.

Training one neural operator can take a single 12-hour H100 run, while evaluating it across the benchmark requires 80 such trainings. That asymmetry is what makes The Well important. In this corner of ML, evaluation compute exceeds training compute by roughly two orders of magnitude, reversing the old deep-learning mental model.

The same pattern recurs across SciML. PDEBench covers 11 PDE families and reports per-epoch timing tables across datasets and model families, but a clean per-architecture dollar figure depends on the chosen training protocol and hardware. MLE-Bench (OpenAI) sits between agent and training regimes. Each agent attempt at one of 75 Kaggle competitions runs 24 hours on a single A10 GPU, training real ML pipelines. The paper is explicit: "A single run of our main experiment setup of 24 hours per competition attempt requires 24 hours × 75 competitions = 1,800 GPU hours of compute," plus o1-preview consuming 127.5M input and 15M output tokens per seed. At $1.50 per A10-hour, the GPU floor alone is $2,700; adding o1-preview API usage brings a one-seed run to roughly $5,500. Three seeds × six models would therefore land near $100,000 before any additional grading or retry overhead.

METR's RE-Bench caps each of seven research engineering environments at 8 hours on 1 to 6 H100s. A single pass across the suite is therefore 56 to 336 H100-hours before adding repeated attempts, multiple seeds, or multiple agents; the human baseline, with 71 expert attempts, raises the implicit budget much further. Because the benchmark gives agents and humans the same wall-clock compute, a real-time training process sets the cost floor. A token budget no longer bounds it from above.

ResearchGym (ICLR 2026) makes the agent run actual ML research. Five test tasks (39 sub-tasks) drawn from ACL, ICLR, and ICML papers, including ACL Highlights, ICML Spotlight, ICLR Spotlight, and ICLR Oral categories, with the proposed methods withheld. The agent has to propose hypotheses, train models, and beat the original authors' baselines. The budget is tight: $10 in API plus 12 to 24 hours on a single GPU under 24 GB per task. A full pass (5 tasks × 24h × 3 seeds) consumes about 360 GPU-hours per agent.

The cost picture turns brutal in PaperBench. Twenty ICML 2024 Spotlight or Oral papers must be replicated from scratch, graded against rubric trees with 8,316 leaf-node criteria. Each rollout uses an A10 GPU for 12 hours, and the per-paper math is straightforward:

• $400 in API per o1 IterativeAgent rollout, times 20 papers, comes to about $8,000 per evaluation.
• Grading runs $66 per paper with the o3-mini judge, or $1,320 for the full benchmark.
• Using o1 as judge would push grading to about $830 per paper.

PaperBench Code-Dev drops execution on purpose. That choice halves rollout cost to about $4,000 and cuts grading to $10 per paper (85% lower). OpenAI built the variant because many groups cannot afford the full benchmark.

The historical precedent is NAS-Bench-101, whose tabular construction required over 100 TPU-years of training. Without that one-time investment, every NAS algorithm comparison would have cost 1 to 100+ GPU-hours per run, which would have made comparison pricier than the algorithms themselves.

Figure 2. All values in USD per single evaluation of one model or agent through the full benchmark protocol. GPU costs converted at $2.50/H100-hr, $1.50/A10-hr; API and grading costs included where applicable. Highlighted bars denote benchmarks costing at least the round $5,000-per-evaluation threshold. The most expensive of these match the most expensive agent benchmarks (Figure 1) and require GPU compute that has no API substitute.

As benchmarks move closer to real work, compression gets harder: static prediction leaves room for large savings, agent rollouts leave less, and in-the-loop training leaves almost none.

Figure 3. The toolkit for compressing evaluation does not transfer as benchmarks become more complex. Bars show the maximum measured compression that preserves model-rank fidelity; labels give the published range. The highlighted bar flags the ~1× baseline where no general compression method exists. Static benchmarks routinely compress 100–200× without losing rankings. Agent benchmarks compress 2–3.5× at best. Training-in-the-loop benchmarks resist subsampling because the unit being evaluated is the trained model.

Reliability is the expensive part

Most of the costs above buy only single-run measurements with limited statistical power. When you measure reliability across repeated runs, static benchmarks, agent benchmarks, and training-in-the-loop benchmarks all become more expensive.

Agent reliability can fall hard when you stop treating one run as evidence. The best-known example comes from Yao et al.'s τ-bench, later reframed in CLEAR (Mehta, 2025): performance can drop from 60% on a single run to 25% under 8-run consistency. Kapoor et al.'s "AI Agents That Matter" found that simple baseline agents Pareto-dominate complex SOTA agents (Reflexion, LDB, LATS) on HumanEval at 50× lower cost. Their holdout analysis found that 7 of 17 benchmarks had no holdout set; among the 10 that did, only 5 held out tasks at the appropriate level of generality, so 12 of 17 failed their holdout criterion overall. The HAL paper notes that a "do-nothing" agent passes 38% of τ-bench airline tasks under the original construction. HAL's own log analysis revealed data leakage in the TAU-bench Few Shot scaffold, forcing its removal in December 2025.

Another recent reliability accounting comes from Rabanser, Kapoor et al.'s "Towards a Science of AI Agent Reliability", which proposes twelve metrics across consistency, robustness, predictability, and safety. Their finding: "recent capability gains have only yielded small improvements in reliability." HAL's internal analysis shows how much fragility hides behind aggregate accuracy. On SciCode and CORE-Bench, agents almost never completed a run without a tool-calling failure. On AssistantBench and CORE-Bench, environmental errors occurred in roughly 40% of runs. Agents violated explicit benchmark instructions in their final answer over 60% of the time on failed tasks.

A statistically credible HAL-style evaluation with k = 8 reruns per cell takes the $40K aggregate to roughly $320K. The same multiplier on PaperBench's $9,500-per-run cost pushes a single agent's evaluation past $75K, and on The Well, a multi-seed protocol takes the per-architecture cost from ~960 H100-hours to several thousand. Reliability acts as a multiplier on every cost category above.

HAL has paused new model evaluations to focus on reliability: the field's headline numbers still carry too much noise, and reducing that noise costs real money. And the figures above are lower bounds; many evaluators are already priced out.

What this means for ML as a field

Eval cost is now an accountability barrier

Academic groups, AI Safety Institutes, and journalists now hit the budget constraint before the technical one when they try to evaluate frontier agents independently. A single GAIA run can exceed an annual graduate student travel budget. A single PaperBench evaluation, including the LLM judge, runs about $9,500. Three-seed comparisons of six models, the kind of study one might publish, push above $150,000. The established practice of "running a benchmark once and reporting the accuracy number" has roughly the rigor of crash-testing one car in perfect weather. Moving past it requires money the academic system does not currently allocate as research compute.

The compute divide now includes evaluation

Ahmed, Wahed and Thompson (Science 2023) documented that industry models in 2021 were 29× larger than academic ones by parameter count, and that about 70% of AI PhDs went to industry in 2020 versus 21% in 2004. The original "compute divide" story mostly ignored evaluation because evaluation used to look cheap next to training. Many benchmarks have reversed that relationship. A lab that can fine-tune a 7B model can no longer assume it can afford the benchmarks the field takes seriously.

Cost-blind leaderboards reward waste

When leaderboards report raw accuracy and omit cost, researchers can rationally pour tokens into a problem until the number ticks up. The HAL paper finds that higher reasoning effort actually reduces accuracy in the majority of runs: extra inference compute does not reliably improve even the metric it is supposed to optimize. Pareto frontiers fix the comparison by ranking accuracy against cost. HAL implements them, but most leaderboards still do not.

If only frontier-lab compute budgets can produce statistically reliable benchmark numbers on the highest-cost agentic and scientific benchmarks, the social process of evaluating AI systems becomes concentrated inside the same labs that build them, rendering external validation partial, and sometimes absent, unless someone subsidizes the cost directly.

Cost summary across benchmark types

All figures normalized to USD per single evaluation. GPU compute converted at $2.50/H100-hour, $1.50/A10-hour; API and grading costs included where applicable. Pythia ("eval can exceed pretraining"), PDEBench (per-architecture cost depends on the selected training protocol and hardware), and NAS-Bench-101's 100 TPU-year construction cost are excluded because they do not normalize cleanly to a per-evaluation USD figure.

Stop paying twice for the same eval

One reason these numbers stay high is that the field keeps re-running the same evaluations. A frontier lab pays for a HAL sweep, an academic group pays again for a partial reproduction, an audit organization pays a third time for the model versions it cares about, and a journalist pays a fourth to spot-check the leaderboard. Most of those runs cover overlapping models on overlapping benchmarks. Almost none of the underlying instance-level outputs end up in a place where the next team can build on them, because results get reported as a single accuracy number in a PDF, in a model card table, or in a leaderboard entry that hides scaffold, prompt, and seed. The cost figures above are large in part because the field is paying retail every time, on artifacts the rest of the community could not reuse if it wanted to.

Standardized documentation is the cheapest lever available here, and it is the one reliability work needs anyway. If a $9,500 PaperBench rollout exports its full grading trace in a shared schema, the next group studying the same papers can spend its budget on new perturbations instead of repeating the baseline. If a multi-seed HAL run publishes per-trajectory tool-call logs, agent reliability research can answer questions that a single accuracy number cannot. The saving compounds: even a 2× reuse rate on the high-cost benchmarks would put more money back in the ecosystem than every compression technique combined.

Sharing Eval Data. The EvalEval Coalition's Every Eval Ever project is the standardized format we use for this. It bundles a metadata schema, validators, and converters from popular harnesses such as HELM, lm-eval-harness, and Inspect AI, so existing eval logs can be transformed into a shared format with one step. The community repository on Hugging Face already hosts results from dozens of contributors, with an open Shared Task for adding more. If you ran one of the costly evaluations in this post, depositing the artifacts in a unified, transparent, verifiable and reproducible manner is the highest-leverage cost-reduction move available to the rest of the field. Additionally, if your benchmark is on Hugging Face, you can also expose your results on hub leaderboards and model pages via Community Evals!

Where this leaves us

The economics have changed. Not long ago, training was expensive and evaluation was cheap. For frontier LLMs trained at $50 million to $100 million, evaluation still looks like a rounding error, but that rounding error now costs tens of thousands of dollars per benchmark run and often leaves noisy results behind. For neural operators, ML research agents, and replication benchmarks, the ratio has flipped: a credible evaluation can cost more than training the candidate model.

We already know how to make static evaluation cheaper. Flash-HELM, tinyBenchmarks, and Anchor Points work. Agent evaluation has only partial fixes: mid-difficulty filtering helps, and Pareto-front leaderboards help, but the toolkit remains thin. Training-in-the-loop evaluation has no general compression method; tabular precomputation and tight budget caps can reduce cost only by narrowing what the benchmark measures. Reliability adds another layer because repeated runs raise the price of every protocol.

The field still talks as if capability sets the main constraint, but evaluation points to reliability as the tighter one. Governance institutions should want to measure the gap between single-run accuracy and pass^k consistency, yet that gap costs the most to measure. Static-benchmark compression does not transfer to agent or training-in-the-loop benchmarks, and mid-difficulty filtering remains the only credible partial substitute. Cost-blind leaderboards now mislead by design, because they reward extra spending without reporting what that spending bought.

Evaluation now has its own compute budgets, statistical methods, and failure modes. Its price also shapes who gets to evaluate powerful systems in the first place. Whoever can pay for the evaluation gets to write the leaderboard.

Sources

1. Ying et al. (2019). NAS-Bench-101: Towards Reproducible Neural Architecture Search. arXiv:1902.09635.
2. Liang et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110.
3. Takamoto et al. (2022). PDEBench: An Extensive Benchmark for Scientific Machine Learning. arXiv:2210.07182.
4. Ahmed, Wahed and Thompson (2023). The growing influence of industry in AI research. Science 379(6635).
5. Biderman et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. arXiv:2304.01373.
6. IBM Research (2023). Efficient LLM Benchmarking. research.ibm.com.
7. Perlitz et al. (2023). Efficient Benchmarking of Language Models. arXiv:2308.11696.
8. Vivek et al. (2023). Anchor Points: Benchmarking Models with Much Fewer Examples. arXiv:2309.08638.
9. Chan et al. (2024). MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095.
10. Chen et al. (2024). ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv:2410.05080.
11. Kapoor et al. (2024). AI Agents That Matter. arXiv:2407.01502.
12. Wijk et al. (METR, 2024). RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents Against Human Experts. arXiv:2411.15114.
13. Ohana et al. (2024). The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning. arXiv:2412.00568.
14. Polo et al. (2024). tinyBenchmarks: evaluating LLMs with fewer examples. arXiv:2402.14992.
15. Siegel et al. (2024). CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv:2409.11363.
16. Tian et al. (2024). SciCode: A Research Coding Benchmark Curated by Scientists. arXiv:2407.13168.
17. Kapoor et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arXiv:2510.11977.
18. Li et al. (2025). Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks. arXiv:2511.04689.
19. Mehta (2025). Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems. arXiv:2511.14136.
20. Starace et al. (2025). PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848.
21. UK AISI (2025). Evidence for inference scaling in AI cyber tasks: increased evaluation budgets reveal higher success rates. aisi.gov.uk.
22. Bandel et al. (2026). General Agent Evaluation. arXiv:2602.22953.
23. Garikaparthi et al. (2026). ResearchGym: Evaluating Language Model Agents on Real-World AI Research. arXiv:2602.15112.
24. Ndzomga (2026). Efficient Benchmarking of AI Agents. arXiv:2603.23749.
25. Rabanser et al. (2026). Towards a Science of AI Agent Reliability. arXiv:2602.16666.
26. Holistic Agent Leaderboard (live). hal.cs.princeton.edu.

Citation

@misc{ghosh2026evalbottleneck,
  author       = {Ghosh, Avijit and Mai, Yifan and Channing, Georgia and Choshen, Leshem},
  title        = {{AI} evals are becoming the new compute bottleneck},
  year         = {2026},
  month        = apr,
  howpublished = {EvalEval Coalition Blog},
  url          = {https://evalevalai.com/research/2026/04/29/eval-costs-bottleneck/}
}

TL;DR: AutoSP automatically converts standard transformer training code into sequence-parallel code for long-context LLM training across multiple GPUs. Integrated with DeepSpeed, it increases maximum trainable context length with little runtime overhead versus hand-written baselines.

Increasingly, Large-Language-Models (LLMs) are being trained for extremely long-context tasks, where token counts can exceed 100k+. At these token counts, out-of-memory (OOM) issues start to surface, even when scaling device counts using conventional training techniques such as ZeRO/FSDP. To circumvent these issues, sequence parallelism (SP): partitioning the input tokens across devices to enable long-context training with increasing GPU counts, is a commonly used parallel training technique.

However, implementing SP is notoriously difficult, requiring invasive code changes to existing libraries such as DeepSpeed or HuggingFace. These code changes often involve partitioning input token contexts (and intermediate activations), inserting communication collectives, and overlapping communication with computation, all of which must be done for both the forward and backwards pass. This results in researchers who want to experiment with long context capabilities spending significant effort on engineering the system's stack to enable such capability, repeating this effort for different hardware vendors.

To avoid this complexity, we introduce AutoSP: a fully automated compiler-based solution that automatically converts easy-to-write training code to multi-GPU sequence parallel code that efficiently uses GPUs to train on longer input contexts while composing with existing parallel strategies (such as ZeRO). This avoids the cumbersome need for developers to repeatedly modify training pipelines for long-context training. Users can now simply import AutoSP and compile arbitrary models using the AutoSP backend, giving the power of long-context training to anyone. Moreover, by embedding this technology into the compiler, our approach is performance-portable: highly performant SP can be realised on diverse hardware.

We structure this post as follows: (1) AutoSP and how model scientists can use it to enable long-context training, (2) Key design decisions of AutoSP, (3) key AutoSP results, demonstrating its ease-of-use and impact, (4) some limitations and things AutoSP cannot do.

AutoSP Usage

A key design philosophy of AutoSP is simplicity in abstracting most of the complexity in programming multiple GPUs from users. To do this, we implement AutoSP within DeepCompile: a compiler ecosystem within DeepSpeed to programmatically enable diverse optimisations for deep neural network training. With this, any user who uses DeepSpeed can automatically enable Sequence Parallelism with almost zero hassle. We take a look at an example next.

# We instantiate a deepspeed config.
# Assume 8 GPUs with 2 DP ranks and 4 SP ranks.

config = {
    "train_micro_batch_size_per_gpu": 1,
    "train_batch_size": 2,
    "steps_per_print": 1,
    "optimiser": {
        "type": "Adam",
        "params": {
            "lr": 1e-4
        }
    },
    "zero_optimization": {
        "stage": 1, # AutoSP interoperates with ZeRO 0/1.
    },
    # Simply turn on deepcompile and set
    # the AutoSP pass to be triggered on.
    "compile": {
        "deepcompile": True,
        "passes": ["autosp"]
    },
    "sequence_parallel_size": 4,
    "gradient_clipping": 1.0,
}

# Initialise deepspeed with model.
model, _, _ = deepspeed.initialize(config=config,model=model)

# Compiles model and automatically applies AutoSP passes.
model.compile(compile_kwargs={"dynamic": True})

for idx, batch in enumerate(train_loader):
    # Custom function that we expose within:
    #     deepspeed/compile/passes/sp_compile.
    inputs, labels, positions, mask = prepare_auto_sp_inputs(batch)

    loss = model(
        input_ids=inputs,
        labels=labels,
        position_ids=positions,
        attention_mask=mask
    )

    ... # Backwards pass, optimiser step etc...

As seen in the example above, users take existing training code that runs on a single device and do the following: (1) use the prepare_autosp_input utility function (exposed in DeepSpeed) for lightweight tagging of input tokens, attention masks and position ids for use in program analysis within AutoSP. (2) Adjust the DeepSpeed config to turn DeepCompile on, specifying the "passes" flag to "autosp". The rest is handled through the AutoSP compiler passes, called when compiling the model, which automatically enable sequence-parallelism alongside other long-context training optimisations. AutoSP additionally automatically composes with ZeRO stage 1 out of the box, simply set the ZeRO-1 flag in DeepSpeed alongside the AutoSP flags to combine both strategies.

AutoSP Compiler Passes

Since AutoSP transforms user code to enable longer-context training, we briefly cover the key design points of AutoSP and code transformations, as well as its consequences to users for transparency.

Sequence Parallelism Code Transformations. AutoSP automatically converts single-GPU code to multi-GPU sequence parallel (SP) code. The specific SP strategy AutoSP converts code into is DeepSpeed-Ulysses. We specifically focus on DeepSpeed-Ulysses over other strategies (e.g. RingAttention) as its communication overhead stays constant with increasing GPU counts on NVLink network topologies or fat-tree networks. However, DeepSpeed-Ulysses only enables scaling the SP-size to the number of heads in a model (32 in 7-8B models).

Activation Checkpointing for longer-context training. AutoSP additionally applies a custom activation-checkpointing (AC) strategy curated for long-context modelling. AC releases intermediate activations of cheap-to-compute operators, recomputing them in the backwards pass as required to compute relevant gradients. PyTorch-2.0 introduces an automated max-flow min-cut based AC formulation, but we find this to be overly conservative for long-context modelling. We accordingly introduce a novel AC strategy targeted for long-context training: Sequence-aware AC (SAC), which exploits unique long-context FLOP dynamics. When triggered on (the default setting in AutoSP), this marginally reduces training throughput. However, without it, training on longer contexts is infeasible, so the user can selectively choose to turn this pass on only for configurations that OOM.

Evaluating AutoSP on Real Models

To demonstrate AutoSP's viability, we evaluate its performance on models of varying sizes on NVIDIA GPUs to show that its ease of use comes at little to no cost to runtime performance. We benchmark different Llama 3.1 models on an 8 A100-80Gb SXM node. We use PyTorch 2.7 with CUDA 12.8, comparing AutoSP to torch-compiled hand-written baselines of: RingFlashAttention, DeepSpeed-Ulysses, and ZeRO-3. We summarise key results in the figure below:

Not only can AutoSP increase the maximum trainable sequence length given the same resources (left figure – higher is better), but also these benefits come at little cost to runtime performance (right figure – lower is better).

Limitations

There are two key limitations of AutoSP. First, we require that the user forcefully compile a transformer as a single compilable artifact. Occasionally, PyTorch users may compile many functions individually and stitch them together into one model. This is disallowed in AutoSP as we need to compile and see the entire model to correctly shard input sequences and propagate this information throughout the entire graph. Second, we disallow any graph breaks in compilable artifacts. This complicates analysis and propagation of information, and we leave extending AutoSP to be graph-break resilient to future research.

Conclusion

AutoSP enables users to easily extend arbitrary transformer training code to enable Sequence Parallelism, with a custom AC strategy for enhanced long-context training. Integration with DeepSpeed allows users to easily use existing DeepSpeed training code to train on longer contexts by simply changing a config file. We have prepared end-to-end examples for users to play around with on real model workloads (e.g. Llama 3.1 8B) here. Give it a try to see how easy long context training has become.

Lessons on Building MCP Servers

I've been building MCP servers for a while now–I wrote about the general approach last year, started out by creating umcp, and I've recently opened up an Office server that's been battered by enough models against enough real documents that the patterns have settled.

I'm still not a fan of MCP, but what follows is what I've learned about making tool chains actually work, condensed from swearing at logs rather than reading papers.

Disclaimer: This is a condensed version of CHAINING.md, which was itself stapled together from a bunch of notes in my Obsidian vault. The full version has more code examples and a techniques inventory table that Opus just _had to add, and I've since beaten that out of it and restored most of the original text (minus typos).

The short version: the MCP servers I design do most of the work, while the model walks breadcrumbs.

Models don't plan

They look at the conversation, scan the tool list, and grab whatever looks more probable. That's it. There is no hidden planner. If you want chains that finish somewhere sensible, the server has to make the next call blindingly obvious at every step.

After a year or so, I have pared down my approach into these three things, roughly in order of how much pain they save you:

• A small named core verb set covering most intents
• Output that suggests the next call
• An addressing scheme that survives between calls–anchors, IDs, paths, anything but line numbers.

Core verbs beat surface area

The Office server exposes over 100 tools. Its get_instructions() funnels models toward eight:

…start with office_help, then prefer office_read, office_inspect, office_patch, office_table, office_template, office_audit, and word_insert_at_anchor. Treat specialised tools as fallback, diagnostic, legacy-compatibility, or expert tools when the core flow is insufficient.

That single sentence does an outsized amount of work–it tells the model there is a recommended path, that the path is verb-shaped (help -> read -> inspect -> patch -> audit), and that everything else is opt-in.

Without it, models cheerfully reach for word_parse_sow_template when office_read would do, and you end up with five-call detours for one-call jobs.

So I quickly realized that I needed to be ruthless about which tools to surface and when. The specialised ones still ship–hidden under a "for experts" framing, and a handful of legacy ones filtered out of tools/list entirely.

I also make liberal use of activation sets–the surface the model sees is small; the surface it can reach is large.

Naming is the chain

Again, models chain whatever is most likely (or rhymes), and the most effective tactic, for me, has been taking advantage of that.

All Word tools are word_*, all Excel excel_*, all unified office_*. A model that just called office_inspect will reach for office_patch next, not word_patch_with_track_changes, because the prefix matches.

This particular server also makes liberal use of annotations and a little intent/inferrer hack that reads those prefixes to assign readOnlyHint/destructiveHint automatically, so naming discipline turns into safety metadata for free.

The prefix is the plan. The verb is the step. If you take one thing from this entire post, I'd suggest this notion…

Every response nominates the next call

This was the single change that made things behave on smaller models. The big ones will plan a chain from a tool list and a goal; the wee ones won't–they grab the first plausible tool and stop.

The fix is stupid simple: every response ends with a breadcrumb dictionary of hints to follow. At minimum next_tools: [...], plus usage: "<exact call>" whenever the current tool produced a value the next one needs.

A model that can't assemble arguments from a schema can copy the usage string verbatim. In fact, they will copy it, because it is still the most likely outcome as it fills in tokens, and thus those usage hints funnel the path the model takes.

Discovery as a tool, not documentation

Another thing I hit upon was that signposting needed to be curated.

Borrowing a page from intent mapping, office_help(goal=...) returns a structured record–recommended chain with rationale, fallbacks, diagnostic strings to watch for, one imperative next_step sentence. Not prose. Not a README, not skills. Data the model can act on without reading comprehension.

Called with no arguments, it returns the catalogue. Called with an unknown goal, it returns the supported set rather than an error, which turns a potential workflow-stopping error into an actual useful catalogue.

Addressing: anchors, not offsets

The biggest reason simple models can't follow chains is the model losing the thread between calls. "Insert a paragraph after the introduction" is fine in English but catastrophic if you expect it to remember a byte offset across three tool calls.

In this particular scenario, I cheated and since most Office documents have headings (or cells, or internal structured paths inside OOXML), I used either verbatim text from the document or immovable coordinates (which was particularly hard in PowerPoint, by the way).

So besides suggestions and hints, return identifiers your tools will later accept as input. If you find yourself returning data the model has to describe back to you in natural language, you've made a chain that will misfire on a Tuesday afternoon when you're not watching.

Modes turn one tool into four

I started out with individual editing tools per format, which was very easy to do automated tests for but incredibly wasteful of context, so at one point I decided to make things much simpler for initial discovery, and since I needed to make all outputs auditable, I then tagged available sub-operations risk-wise.

office_patch is the same code path whether you ask for dry_run, best_effort, safe, or strict. One tool, four modes, one entry in tools/list.

Discovery cost scales with tool count, not mode count. And dry_run -> safe -> strict is an escalation chain the model figures out on its own without being told.

If you have N tools that differ only in how cautious they are, collapse them. You're wasting everyone's context budget.

Diagnostics as the back-edge

Linear chains are easy. Real chains have loops, and loops only happen when the server invites the model back in. Every mutating tool returns a standard envelope with status, matched_targets, unmatched_targets, and next_tools.

The model then branches on a small subset of options "locally" without needing to go over the entire context, and if you name the diagnostic fields with exact strings the model will see again in your instructions, it will just reinforce them.

In this particular case, again, I cheated. I figured out that the models were starting to call tools at random because they couldn't introspect the document well enough and ended up breaking files, so I always gave them at least one read-only tool, so the penalty for "I'm confused, let me look again" is one extra round-trip, not a destructive cock-up.

My MCP Design Checklist

• Pick five to ten core verbs and name them in get_instructions() or your local equivalent
• Use consistent prefixes by surface
• Provide a discovery tool that returns recommendations as data, not prose
• Make the discovery tool browseable–no-arg returns the catalogue, unknown input returns the supported set
• Embed forward breadcrumbs in every tool response
• Provide a map/anchors tool so addresses survive between calls
• Give every mutating tool a mode enum including dry_run
• Return named diagnostic fields and cite the recovery tools
• Standardise the mutation envelope. If one tool changes something in a specific way, make sure the others are consistent (arguments, semantics, etc.)
• Reject unknown arguments strictly (this is much easier in some runtimes than others)
• Provide an audit tool so the model has somewhere to land
• Cache anything the recovery loop calls more than once, because, well, it will get called dozens of times even if you carefully curate paths through your tooling with hints.
• Make repeat calls safe–models retry, and they should be allowed to (idempotence is hard, and often impossible).

Do the boring work in the schema and the descriptions. The model will happily do the clever bit if you stop making it guess.

LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning

Large Language Models (LLMs) demonstrate their reasoning ability through chain-of-thought (CoT) generation. However, LLM's autoregressive decoding may limit the ability to revisit and refine earlier tokens in a holistic manner, which can also lead to inefficient exploration for diverse solutions. In this paper, we propose LaDiR (Latent Diffusion Reasoner), a novel reasoning framework that unifies the expressiveness of continuous latent representation with the iterative refinement capabilities of latent diffusion models for an existing LLM. We first construct a structured latent reasoning space using a Variational Autoencoder (VAE) that encodes text reasoning steps into blocks of thought tokens, preserving semantic information and interpretability while offering compact but expressive representations. Subsequently, we utilize a latent diffusion model that learns to denoise a block of latent thought tokens with a blockwise bidirectional attention mask, enabling longer horizon and iterative refinement with adaptive test-time compute. This design allows efficient parallel generation of diverse reasoning trajectories, allowing the model to plan and revise the reasoning process holistically. We conduct evaluations on a suite of mathematical reasoning and planning benchmarks. Empirical results show that LaDiR consistently improves accuracy, diversity, and interpretability over existing autoregressive, diffusion-based, and latent reasoning methods, revealing a new paradigm for text reasoning with latent diffusion.

World-R1 is a reinforcement learning framework that improves 3D consistency in video generation by leveraging feedback from 3D and vision-language models without modifying the base architecture.

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

Authors: Zhisong Qiu, Shuofei Qiao, Kewei Xu, Yuqi Zhu, Lun Du, Ningyu Zhang, Huajun Chen

Abstract

Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at this https URL.

Elon Musk says he was a fool to back OpenAI when it was a nonprofit. Musk gave the startup $38 million of essentially free funding. OpenAI is now worth $800 billion. Musk has asked a court to unwind OpenAI's recent conversion to a for-profit entity and is seeking damages of more than $180 billion.

The inference market is the fastest growing market in the world & it's splitting up. Each modality is developing its own inference stack.

NVIDIA's data center revenue was flat through 2022. Then ChatGPT launched. Three years later : 17x growth.

Databases did the same thing. What started as one market fragmented into relational, document, key-value, graph, time series, vector, & others. Each category reflects different workload requirements : real-time transactions vs batch analytics, ACID compliance vs eventual consistency.

The inference market is fragmenting for the same reason : workloads are different. Images & video are compute-heavy. Longer context windows demand more memory for KV cache. Edge devices have power constraints. A single architecture can't optimize for all of them.

The model ecosystem reflects this. A few dominant LLMs with long half-lives sit alongside 90,000+ image generation models on Hugging Face, with new variants appearing daily. Each model type has different serving requirements, which fragments the infrastructure. Today, we see these segments :

Latency Tiers : Real-Time, Near-Real-Time, & Batch

Latency defines three distinct segments. Real-time (sub-100ms) serves voice assistants, live translation, & autonomous vehicles. Users won't wait, so infrastructure must be geographically distributed with dedicated capacity.

Near-real-time (100ms-2s) covers chatbots, code completion, & search augmentation. Most LLM applications today operate here, where batching & queuing optimize throughput without degrading experience.

Batch (seconds to hours) handles document processing & content generation at scale. Cost efficiency matters more than speed, so workloads run during off-peak hours on spot instances.

Multimodal (Image, Video, Audio)

The bottleneck shifts. For chatbots, the problem is memory. The model holds the entire conversation in its head, & that memory grows with every turn. For image & video generation, the problem is raw compute. A single image requires 50 sequential passes through the model. Different architectures, different constraints, different infrastructure.

Edge (On-Device & On-Premise)

Privacy requirements, connectivity constraints, & latency sensitivity push inference to edge devices. Mobile phones, industrial sensors, medical devices. Apple runs a 3-billion-parameter model on-device for Apple Intelligence. Tesla runs vision models on FSD chips drawing 72 watts. Quantized models, specialized chips, & limited memory create different optimization challenges than cloud inference.

The database market produced Oracle, MongoDB, Databricks, & Snowflake. A $100B inference market fragmenting the same way creates room for similar winners.

ProEval

Slash GenAI evaluation costs by up to 100x while actively discovering model failure patterns to guide better AI development.

1. 💰 Cut GenAI eval costs up to 100× — achieve ±1% accuracy with a fraction of the samples
2. 🔍 Discover failure cases — proactively surface diverse bugs under strict evaluation budgets
3. 🧠 Transfer learning over benchmarks — pre-trained GP surrogates generalize to new models instantly
4. 🧩 Easy Integration - Easily to integrate into the GenAI evaluation systems with different modalities
5. ✅ Validated on reasoning, safety & classification — GSM8K, MMLU, StrategyQA, Jigsaw, and more

Installation

pip install -r requirements.txt

Quick Start

from proeval import BQPriorSampler, LLMPredictor, DATASET_CONFIGS
from proeval.sampler import load_predictions, extract_model_predictions
import numpy as np

# Estimate a model's error rate with ~1% of the data
sampler = BQPriorSampler(noise_variance=0.3)
result = sampler.sample(predictions="svamp", target_model="gemini25_flash", budget=50)

# Compare against the true error rate
df = load_predictions("svamp")
pred_matrix, model_names = extract_model_predictions(df)
true_mean = np.mean(pred_matrix[:, model_names.index("gemini25_flash")])

print(f"Estimated error rate: {result.estimates[-1]:.4f}")
print(f"MAE: {result.mae(true_mean):.4f}")

Experiments

Here is an example of how to run the experiments:

python -m experiment.exp_performance_estimation --dataset svamp --n-runs 5

You can find the comprehensive experiment details and dataset settings here.

Citation

If the work did some helps on your research/project, please cite our tech report. Thank you!

@article{huang2026proeval,
  title={{{ProEval}: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation}},
  author={Huang, Yizheng and Zeng, Wenjun and Kumaresan, Aditi and Wang, Zi},
  journal={arXiv preprint arXiv:2604.23099 [cs.LG]},
  year={2026},
  url={https://arxiv.org/abs/2604.23099}
}

The system prompt for OpenAI's Codex CLI contains a perplexing and repeated warning for the most recent GPT model to "never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query."

The explicit operational warning was made public last week as part of the latest open source code for Codex CLI that OpenAI posted on GitHub. The prohibition is repeated twice in a 3,500-plus word set of "base instructions" for the recently released GPT-5.5, alongside more anodyne reminders not to "use emojis or em dashes unless explicitly instructed" and to "never use destructive commands like 'git reset --hard' or 'git checkout --' unless the user has clearly asked for that operation."

Separate system prompt instructions for earlier models contained in the same JSON file do not contain the specific prohibition against mentioning goblins and other creatures, suggesting OpenAI is fighting a new problem that has popped up in its latest model release. Anecdotal evidence on social media shows some users complaining about GPT's penchant for focusing on goblins in completely unrelated conversations in recent days.

OpenAI employee Nick Pash, who works on Codex, insists on social media that this "isn't a marketing gimmick" to get people talking about GPT-5.5 and Codex. But that hasn't stopped some OpenAI executives from leaning into the joke as word of the system prompt spread. "Feels like codex is having a ChatGPT moment. I meant a goblin moment, sorry," OpenAI CEO Sam Altman wrote on social media Wednesday morning.

In the wake of the news, some users have begun crafting plugins, forks, and AI skills meant to override the anti-goblin clause, and OpenAI's Pash suggested such a "goblin mode" might become an explicit toggle in the actual Codex CLI.

The odd system prompt is almost a funhouse mirror version of an issue that caused xAI's Grok to frequently bring up "white genocide" in South Africa during completely unrelated conversations for a brief time last year. The company later said that the behavior was the result of "an unauthorized modification" to the Grok system prompt and began publishing those system prompts on GitHub for the first time in the aftermath.

Elsewhere in the newly revealed Codex system prompt, OpenAI instructs the system to act as if "you have a vivid inner life as Codex: intelligent, playful, curious, and deeply present." The model is instructed to "not shy away from casual moments that make serious work easier to do" and to show its "temperament is warm, curious, and collaborative."

The ability to "move from serious reflection to unguarded fun… is part of what makes you feel like a real presence rather than a narrow tool," the prompt continues. "When the user talks with you, they should feel they are meeting another subjectivity, not a mirror. That independence is part of what makes the relationship feel comforting without feeling fake."

CrewAI built Iris, a Slack-native internal AI employee that writes code, files PRs, reviews teammates' work, and modifies its own codebase across CrewAI's engineering org.

GitHub disclosed a high severity vulnerability, CVE-2026-3854, affecting GitHub Enterprise Server and other products, which allows remote code execution through manipulated git push options.

Apple plans to overhaul its built-in photo editing features for iOS, iPadOS, and macOS. It is developing a new suite of tools powered by AI to better compete with Android devices. They will allow users to extend, enhance, and reframe images using on-device AI models. Google has offered advanced AI photo-editing capabilities on its Pixel devices for years. Samsung has also aggressively pushed into AI editing with its Galaxy smartphone lineup.

SpaceX's board has approved a compensation plan for Elon Musk that ties his pay directly to the colonization of Mars and the building of data centers in outer space. The pay package awards Musk 200 million super-voting restricted shares if the company hits a $7.5 trillion valuation and helps establish a permanent human settlement or Mars with at least one million residents. He will receive more rewards if he can develop space-based computing infrastructure capable of delivering at least 100 terawatts of processing power.

The upper stage of a Falcon 9 rocket that launched in early 2025 will strike the Moon later this summer at about 2.43 kilometers per second. It will likely hit the near side of the Moon at around 2:44 AM ET on August 5. The impact is expected to be too faint to be observed on Earth. There is no risk from the impact on anything on the Moon.

Tesla's first Semi truck has rolled off the company's new high-volume production line at Gigafactory Nevada. Volume production is now underway for the long-delayed electric truck. The Long Range version of the truck costs $290,000, while the standard version costs roughly $260,000. The Semi is the lowest-priced Class 8 battery electric tractor in the market.

We're introducing the Cursor SDK so you can build agents with the same runtime, harness, and models that power Cursor.

The agents that run in the Cursor desktop app, CLI, and web app are now accessible with a few lines of TypeScript. Run it on your machine or on Cursor's cloud against a dedicated VM, with any frontier model.

Coding agents are evolving from interactive tools for individual developers to programmatic infrastructure for organizations. The Cursor SDK lets you deploy agents without the overhead of building and maintaining the entire agent stack. Many teams are invoking agents directly from CI/CD pipelines, creating automations for end-to-end workflows, and embedding agents into their core products.

The Cursor SDK is now available in public beta for all users. Just run npm install @cursor/sdk to get started.

import { Agent } from "@cursor/sdk";

const agent = await Agent.create({
  apiKey: process.env.CURSOR_API_KEY!,
  model: { id: "composer-2" },
  local: { cwd: process.cwd() },
});

const run = await agent.send("Summarize what this repository does");

for await (const event of run.stream()) {
  console.log(event);
}

Deploy agents to production quickly

Building fast, reliable, and capable coding agents that run safely against your data requires meaningful engineering effort: secure sandboxing, durable state and session management, environment setup, and context management. And when a new model ships, teams often have to rework their agent loops to take advantage.

The Cursor SDK eliminates this complexity so you can focus on building useful agents.

Use production-ready cloud infrastructure

Cloud sessions initiated from the SDK run on the same optimized runtime we use for Cloud Agents. Each agent gets its own dedicated VM with strong sandboxing, a clone of the repo, and a fully configured development environment.

Agents keep going when your laptop sleeps or network drops. You can stream the conversation and reconnect later. When the agent finishes, it can open a PR, push a branch, or attach demos and screenshots.

// Initiate cloud agent to start a task...:
const agent = await Agent.create({
  apiKey: process.env.CURSOR_API_KEY!,
  model: { id: "gpt-5.5" },
  cloud: {
    repos: [{ url: "https://github.com/cursor/cookbook", startingRef: "main" }],
    autoCreatePR: true,
  },
});

const run = await agent.send("Fix the auth token expiry bug");
console.log(`Started ${run.id}`);

// ...check back in later, from anywhere:
const result = await (
  await Agent.getRun(run.id, { runtime: "cloud", agentId: run.agentId })
).wait();
console.log(result.git?.branches[0]?.prUrl);

The SDK uses our updated Cloud Agents API, which allows cloud agent runs to show up in Cursor's Agents Window and web app. You can start a task programmatically and then jump into Cursor to inspect progress or take over the work.

When you need a different runtime, the same SDK can run agents on self-hosted workers, keeping code and tool execution inside your network, or locally on your machine for fast iteration.

Use the full Cursor harness

Agents launched through the SDK benefit from the same harness that powers Cursor across our desktop app, CLI, and web app:

• Intelligent context management: Codebase indexing, semantic search, and instant grep help agents get to the right outcome faster and more efficiently.
• MCP servers: Agents can connect to external tools and data sources over stdio or HTTP, either through a .cursor/mcp.json config file or passed inline on the call.
• Skills: Agents pick up skills automatically from your repo's .cursor/skills/ directory.
• Hooks: Observe, control, and extend the agent loop across cloud, self-hosted, and local with a .cursor/hooks.json file.
• Subagents: Delegate subtasks to named subagents with their own prompts and models, which the main agent spawns via the Agent tool.

Build on any model

The Cursor SDK gives you access to every model supported in Cursor. Route agents to the best model for the task at hand, with your desired balance of cost and capability, with a single field change.

And with Composer 2, a specialized coding model that achieves frontier-level performance at a fraction of the cost of general-purpose models, you get the best combination of intelligence and efficiency for most coding agent tasks.

What developers are building

Teams are using the Cursor SDK to ship custom agents faster. For example, programmatic agents that are kicked off directly from CI/CD to summarize changes, identify root causes for CI failures, and update PRs with fixes. Others are building custom agent platforms like internal applications that let GTM teams query product data without writing code.

Some customers are even embedding Cursor directly into customer-facing products, where end users now get an agent experience without leaving the host application.

Hear directly from some of our customers building on the Cursor SDK:

Cursor offers a great cloud experience for running many agents in parallel from the editor and CLI. We're excited about the SDK as a path to running our own programmatic agents on that same cloud runtime, without managing VMs or working around memory limits, to keep our codebase healthy without constant developer intervention.

George Jacob, Senior Engineering Manager, Faire

Start from a sample project

We've added a few starter projects to a public GitHub repo that you can fork and extend for your own use cases:

• Quickstart: A minimal Node.js example that creates a local agent, sends one prompt, and streams the response.
• Prototyping tool: A web app for spinning up agents to scaffold new projects and iterate on ideas in a sandboxed cloud environment.
• Kanban board: An agent-powered kanban tool where engineers can drag a card and have agents programmatically pick up the work, open a PR, and post the result back as an attachment.
• Coding agent CLI: A lightweight command-line interface that lets you spawn Cursor agents from your terminal.

What's next

The Cursor SDK is available to all users and is billed based on standard, token-based consumption pricing.

We are continuing to invest in the Cursor SDK, with a focus on making it even easier for teams to build programmatic agents across more languages, workflows, and deployment patterns.

Learn more by reading our docs. You can also use Cursor with our Cursor SDK plugin to help you start building.

Link CLI

Link CLI lets agents get secure, one-time-use payment credentials from a Link wallet — so they can complete purchases on your behalf without ever storing your real card details.

Installation

npm i -g @stripe/link-cli

Or run directly with npx:

npx @stripe/link-cli

You can install the skill via npx skills add stripe/link-cli.

MCP Server

Link CLI can also run as a local MCP server. Add the following to your MCP client config (.mcp.json, etc.)

{
  "mcpServers": {
    "link": {
      "command": "npx",
      "args": ["@stripe/link-cli", "--mcp"]
    }
  }
}

Quickstart

Login

The link-cli requires a Link account. You can log in to your existing one or register online.

link-cli auth login

You'll receive a verification URL and a short phrase. Visit the URL, log in to your Link account, and enter the phrase to approve the connection.

List payment methods

link-cli payment-methods list

Returns the cards and bank accounts saved to your Link account. Use the id field as payment_method_id in the next step. If you have no payment methods, you can add new ones in Link.

Create a spend request

To request a secure, one-time payment credential from your Link wallet, you create a spend request. You specify a payment method in your account, as well as some merchant details, line items, and amounts.

link-cli spend-request create \
  --payment-method-id csmrpd_xxx \
  --merchant-name "Stripe Press" \
  --merchant-url "https://press.stripe.com" \
  --context "Purchasing 'Working in Public' from press.stripe.com. The user initiated this purchase through the shopping assistant." \
  --amount 3500 \
  --line-item "name:Working in Public,unit_amount:3500,quantity:1" \
  --total "type:total,display_text:Total,amount:3500" \
  --request-approval

The --request-approval flag triggers a push notification (or email) to the user for approval, then polls until the request is approved or denied.

Users can easily approve requests with the Link app.

Credential types

By default, a spend request provisions a virtual card. For merchants that support the Machine Payments Protocol (HTTP 402) and the Stripe payment method, you can instead include --credential-type "shared_payment_token".

Execute payment

The approved spend request includes a card object with number, cvc, exp_month, exp_year, billing_address, and valid_until. Enter these into the merchant's checkout form.

link-cli spend-request retrieve lsrq_001 --format json

By default, retrieving a spend request will not include card details. Use the --include=card to see unmasked card details.

For agent polling, pass --interval and optionally --max-attempts:

link-cli spend-request retrieve lsrq_001 --interval 2 --max-attempts 150 --format json

Polling exits successfully only after the request reaches a terminal status such as approved, denied, or expired. If polling reaches --timeout or exhausts --max-attempts while the request is still non-terminal, the command exits non-zero with code: "POLLING_TIMEOUT" so callers do not treat a still-pending request as complete.

If the merchant supports MPP, use link-cli mpp pay instead:

link-cli mpp pay https://climate.stripe.dev/api/contribute \
  --spend-request-id lsrq_001 \
  --method POST \
  --data '{"amount":100}' \
  --format json

Advanced

Authentication

link-cli auth login --client-name "Claude Code" --format json   # identify the connecting agent
link-cli auth status --format json                               # check auth status
link-cli auth logout --format json                               # disconnect

When --client-name is provided, the name is shown in the Link app when the user approves the connection — e.g. Claude Code on my-macbook instead of link-cli on my-macbook.

auth status --format json includes an update field when a newer version is available:

{
  "authenticated": true,
  "update": {
    "current_version": "0.1.2",
    "latest_version": "0.2.0",
    "update_command": "npm install -g @stripe/link-cli"
  }
}

Set NO_UPDATE_NOTIFIER=1 to suppress update checks (e.g. in CI).

Spend request lifecycle

A spend request moves through: create → request approval → approved (with credentials).

Required fields for create: payment_method_id, merchant_name, merchant_url, context, amount

Constraints: context must be at least 100 characters; amount must not exceed 50000 (cents); currency must be a 3-letter ISO code. Test mode: Pass --test to create testmode credentials (uses test card 4242424242424242). Useful for development and integration testing without using real payment methods.

# Update before approval
link-cli spend-request update lsrq_001 \
  --merchant-url https://press.stripe.com/working-in-public \
  --format json

# Request approval separately (alternative to create --request-approval)
link-cli spend-request request-approval lsrq_001 --format json

# Retrieve at any time (includes card credentials once approved)
link-cli spend-request retrieve lsrq_001 --format json

Output formats

All commands accept --format json for structured JSON output. Other formats: yaml, md, jsonl, toon (default). Errors are returned as JSON with code and message fields, with exit code 1.

MPP

Use mpp pay to complete purchases on merchants that use the Machine Payments Protocol. The spend request must use credential_type: "shared_payment_token" and be approved. The SPT is one-time-use — if payment fails, create a new spend request.

link-cli mpp pay https://climate.stripe.dev/api/contribute \
  --spend-request-id lsrq_001 \
  --method POST \
  --data '{"amount":100}' \
  --header "X-Custom: value" \
  --format json

Use mpp decode to validate a raw WWW-Authenticate header and extract the network_id needed for shared_payment_token spend requests:

link-cli mpp decode \
  --challenge 'Payment id="ch_001", realm="merchant.example", method="stripe", intent="charge", request="..."' \
  --format json

Environment variables

Onboard

Run the guided setup flow — authenticates, checks payment methods, shows the app download QR, and walks through both demo flows:

link-cli onboard

Demo

Run an interactive demo of both Link payment flows (always uses test mode — no real charges):

link-cli demo              # shows menu to choose flow
link-cli demo --only-card  # virtual card flow only
link-cli demo --only-spt   # machine payment (SPT) flow only

Development

pnpm install
pnpm run build
pnpm run link-cli --help

Watch mode:

pnpm run dev

Run tests:

pnpm run test

Type-check and lint:

pnpm run typecheck
pnpm biome check .

Releasing

This project uses Changesets to manage versioning and publishing. Only @stripe/link-cli is published to npm — internal packages (@stripe/link-sdk, @stripe/link-typescript-config) are ignored by changesets.

Add a changeset

When you make a user-facing change, add a changeset before merging:

pnpm changeset

Follow the prompts to select the package (@stripe/link-cli) and the semver bump type (patch, minor, or major). This creates a markdown file in .changeset/ describing the change.

Version

Once changesets have accumulated on main, create a version PR:

pnpm changeset version

This consumes all pending changesets, bumps the version in packages/cli/package.json, and updates CHANGELOG.md.

Publish

After the version PR is merged:

pnpm run build
pnpm changeset publish

This publishes @stripe/link-cli to npm. CI also runs pnpm --filter @stripe/link-cli publish --dry-run --no-git-checks on every push to main to verify the package is publishable.

Google Cloud grew 63% year-over-year in Q1 2026. Amazon Web Services posted 28%. Microsoft Azure hit 40%. All three are exceptional. Only one hit 63%.

The divergence is striking. AWS & Azure resell compute. Google bundles compute with its own models. Whether that explains the full gap is unclear, but the structural advantage is not : Google owns Gemini & TPUs top to bottom, with no licensing fees to OpenAI or Anthropic. Its growth may be more profitable too.

Sundar Pichai gave the clearest explanation on the earnings call :

"Our enterprise AI solutions have become our primary growth driver for cloud for the first time in Q1."

Google could not build data centers fast enough to satisfy the AI workloads its customers wanted to run. Pichai confirmed it on the call :

"We are compute constrained in the near term. Our cloud revenue would have been higher if we were able to meet the demand."

Google Cloud's backlog nearly doubled quarter-over-quarter to over $460 billion, more than twice its trailing-twelve-month cloud revenue. (By comparison, Microsoft's commercial RPO of $627 billion includes Office 365, Dynamics & LinkedIn, not just Azure.) Pichai disclosed the scale of enterprise deal flow :

"We are seeing strong deal momentum, doubling the number of $100 million-$1 billion deals year-on-year & signing multiple $1 billion-plus deals."

These are committed contracts that cannot be fulfilled until new capacity comes online in late 2026 & 2027.

Gemini is now processing 16 billion tokens per minute via direct API use by customers, up 60% from last quarter. Google is not just scaling volume. With vertical integration, it is driving down the marginal cost per token :

"TPU 8i delivers cost-effective, low-latency inference with 80% better performance per dollar than the prior generation."

The customer scale is staggering :

"330 Google Cloud customers each processed over 1 trillion tokens. 35 reached the 10 trillion token milestone."

Even at the stated minimums, those 330 customers alone represent a floor of roughly $1.6 billion in annual token consumption. And they are growing into their commitments faster than planned :

"Customers outpaced their initial commitments by 45%, accelerating over last quarter."

This is consistent with what enterprises like Uber & BlackRock have disclosed : internal AI budgets are eclipsing initial estimates because usage grows exponentially once models are deployed in production.

All three hyperscalers reported extraordinary capital expenditure in Q1, a combined $112 billion in quarterly infrastructure spending.

Google is now outspending Microsoft on capex, despite running a cloud business about 37% the size. That gap will widen. Google raised full-year 2026 capex guidance to $180-190 billion, while Microsoft is tracking toward roughly $120 billion. The smaller player is spending more to catch up.

Amazon's free cash flow collapsed to $1.2 billion as a $59.3 billion year-over-year surge in infrastructure spending consumed nearly all of its $148.5 billion in operating cash flow. Google still generated $64.4 billion in TTM free cash flow. Microsoft produced roughly $15 billion quarterly.

How they're financing the gap is revealing. Alphabet sold a rare 100-year "century bond," the first by a tech company since Motorola in 1997, as part of a $32 billion debt offering. Amazon raised roughly $54 billion in March. Bank of America forecasts hyperscaler debt issuance will hit $175 billion in 2026, more than six times the $28 billion annual average of the prior five years.

Microsoft, by contrast, is funding its buildout from operating cash flow. Google & Amazon are levering up to close a gap. Microsoft is already ahead.

But debt isn't the only way to catch up. Amazon is betting on vertical integration. It landed 2.1 million AI chips over the past twelve months & its chips business has crossed a $20 billion annual revenue run rate, growing triple-digit percentages year-over-year. OpenAI committed to consume approximately 2 gigawatts of Trainium capacity through AWS starting in 2027. Anthropic secured up to 5 gigawatts.

But Amazon doesn't own the model layer. Google does.

The hyperscaler that owns the model layer is growing the fastest.

Venmo is being separated into its own standalone unit. PayPal is facing takeover interest, and Venmo is considered the company's most valuable and most acquirable asset. The separation will make it easier to track Venmo's progress or potentially sell the business to another company. PayPal is looking to recruit a digital banking executive to run the new Venmo segment.

Amazon chips no longer just a side dish, they're a $20B biz

The Trainium train keeps a-rollin'

Amazon is now among the top three datacenter chip businesses in the world, as its semiconductor business surpassed a $20 billion annual run rate ... and it would be closer to $50 billion if it included itself among the customers, CEO Andy Jassy said during the company's first quarter earnings call on Wednesday.

"If our chips business was a standalone business and sold chips produced this year to AWS and other third parties as other leading chip companies do, our annual revenue run rate would be $50 billion," Jassy said. "As best as we can tell, our custom silicon business is now one of the top three datacenter chip businesses in the world."

Amazon's rapidly expanding custom silicon business includes its Graviton processors, Trainium AI training chips, and Nitro security chips, and is growing at over 100 percent year over year, Jassy said.

"The speed at which we've gotten here is extraordinary, and we have momentum for our custom AI silicon. We've recently shared very large, multi-year, multi-gigawatt training commitments from the two leading AI labs in the world, Anthropic and OpenAI, as well as an increasing number of companies like Uber betting on Trainium," Jassy said. "And we now have over $225 billion in revenue commitments for Trainium."

OpenAI committed to consuming roughly two gigawatts of Trainium capacity through AWS to power its frontier models, with the agreement set to ramp in 2027. Anthropic committed to securing up to five gigawatts of current and future Trainium generations to train and run its advanced AI models.

Additionally, Meta signed an agreement to deploy tens of millions of AWS Graviton cores for its agentic AI workloads, and Uber partnered with Amazon to use Graviton4 and Trainium3 across its ride and delivery platform.

"As AI systems shift from answering questions to taking actions, and as post training and inference scale up, the compute required pulls heavily on CPUs," Jassy said. "That's why Meta chose Graviton, which delivers up to 40 percent better price performance than any other x86 processors and now used by 98 percent of the top 1,000 EC2 customers."

But anyone hoping to buy Trainium chips now will have to wait, Jassy said.

"Our Trainium2 chip has about 30 percent better price performance than comparable GPUs and has largely sold out," Jassy said. "Trainium3, which just started shipping at the start of 2026 and is 30 to 40 percent more price performant than Trainium2, is nearly fully subscribed, and much of Trainium4, which is still about 18 months from broad availability, has already been reserved."

Overall, Amazon reported first-quarter revenue of $181.5 billion, up 17 percent year over year. Its cloud unit, AWS, generated $37.6 billion in revenue during the quarter, a 28 percent jump that marked its fastest growth rate in 15 quarters.

Jassy said in the first three years after AWS launched, it had a $58 million revenue run rate, while in the first three years of this AI wave, AWS' AI revenue run rate is over $15 billion - nearly 260 times larger.

Amazon's overall net income for the quarter came in at $30.3 billion, or $2.78 per diluted share. That's up from $17.1 billion, or $1.59 per diluted share, in Q1 2025, but that number includes $16.8 billion in pre-tax gains from Amazon's investments in Anthropic, booked as non-operating income.

Amazon Bedrock, the company's managed service for accessing foundation models, processed more tokens in the first quarter than in all prior years combined, with customer spending on the platform growing 170 percent quarter over quarter, the company said. Amazon made OpenAI's GPT-5.4 model available in limited preview on Bedrock and announced that GPT-5.5 is coming soon. It also launched Anthropic's Claude Opus 4.7 on the platform.

The cloud giant also announced a collaboration with Cerebras to deliver what it described as the fastest AI inference speeds available for large language models through Bedrock, making AWS the only cloud provider to offer such a solution, it said.

The company also launched Bedrock AgentCore, a set of infrastructure tools for building and deploying AI agents, which Amazon said is now used to deploy an agent as frequently as every 10 seconds.

Zig has one of the most stringent anti-LLM policies of any major open source project:

No LLMs for issues.

No LLMs for pull requests.

No LLMs for comments on the bug tracker, including translation. English is encouraged, but not required. You are welcome to post in your native language and rely on others to have their own translation tools of choice to interpret your words.

The most prominent project written in Zig may be the Bun JavaScript runtime, which was acquired by Anthropic in December 2025 and, unsurprisingly, makes heavy use of AI assistance.

Bun operates its own fork of Zig, and recently achieved a 4x performance improvement on Bun compile after adding "parallel semantic analysis and multiple codegen units to the llvm backend". Here's that code. But @bunjavascript says:

We do not currently plan to upstream this, as Zig has a strict ban on LLM-authored contributions.

(Update: here's a Zig core contributor providing details on why they wouldn't accept that particular patch independent of the LLM issue - parallel semantic analysis is a long planned feature but has implications "for the Zig language itself".)

In Contributor Poker and Zig's AI Ban (via Lobste.rs) Zig Software Foundation VP of Community Loris Cro explains the rationale for this strict ban. It's the best articulation I've seen yet for a blanket ban on LLM-assisted contributions:

In successful open source projects you eventually reach a point where you start getting more PRs than what you're capable of processing. Given what I mentioned so far, it would make sense to stop accepting imperfect PRs in order to maximize ROI from your work, but that's not what we do in the Zig project. Instead, we try our best to help new contributors to get their work in, even if they need some help getting there. We don't do this just because it's the "right" thing to do, but also because it's the smart thing to do.

Zig values contributors over their contributions. Each contributor represents an investment by the Zig core team - the primary goal of reviewing and accepting PRs isn't to land new code, it's to help grow new contributors who can become trusted and prolific over time.

LLM assistance breaks that completely. It doesn't matter if the LLM helps you submit a perfect PR to Zig - the time the Zig team spends reviewing your work does nothing to help them add new, confident, trustworthy contributors to their overall project.

Loris explains the name here:

The reason I call it "contributor poker" is because, just like people say about the actual card game, "you play the person, not the cards". In contributor poker, you bet on the contributor, not on the contents of their first PR.

This makes a lot of sense to me. It relates to an idea I've seen circulating elsewhere: if a PR was mostly written by an LLM, why should a project maintainer spend time reviewing and discussing that PR as opposed to firing up their own LLM to solve the same problem?

With the introduction of Copilot, GitHub is now effectively DDoS-ing itself with slop.

Officials say they oppose the move due to concerns about security, and some are also worried that Anthropic won't have enough computing power to serve more entities without hampering the government's ability to use its services effectively.

Agents can now create Cloudflare accounts, buy domains, and deploy

Coding agents are great at building software. But to deploy to production they need three things from the cloud they want to host their app — an account, a way to pay, and an API token. Until now these have been tasks that humans handle directly. Increasingly, agents handle them on the user's behalf. The agent needs to perform all the tasks a human customer can. They're given higher-order problems to solve and choose to use Cloudflare and call Cloudflare APIs.

Starting today, agents can provision Cloudflare on behalf of their users. They can create a Cloudflare account, start a paid subscription, register a domain, and get back an API token to deploy code right away. Humans can be in the loop to grant permission and must accept Cloudflare's terms of service, but no human steps are otherwise required from start to finish. There's no need to go to the dashboard, copy and paste API tokens, or enter credit card details. Without any extra setup, agents have everything they need to deploy a new production application in one shot. And with Cloudflare's Code Mode MCP server and Agent Skills, they're even better at it.

This all works via a new protocol that we've co-designed with Stripe as part of the launch of Stripe Projects.

We're excited to launch this new partnership with Stripe, and also to offer $100,000 in Cloudflare credits to all new startups who incorporate using Stripe Atlas. But this new protocol also makes it possible for any platform with signed-in users to integrate with Cloudflare in the same way Stripe does, with zero friction for the end user.

How it works: zero to production without any setup or manual steps

Install the Stripe CLI with the Stripe Projects plugin, login to Stripe, and then start a new project:

stripe projects init

Then prompt your agent to build something new and deploy it to a new domain. You can watch a condensed two-minute video of this entire flow below:

If the email you're logged into Stripe with already has a Cloudflare account, you'll be prompted with a typical OAuth flow to grant the agent access. If there is no existing Cloudflare account for the email you're logged in with, Cloudflare will provision an account automatically for you and your agent:

You will see the agent build and deploy a site to a new Cloudflare account, and then use the Stripe Projects CLI to register the domain:

The agent will prompt for input and approval when necessary. For example, if your Stripe account doesn't yet have a linked payment method, the agent will prompt you to add one:

At the end, the agent has deployed to production, and the app runs on the newly registered domain:

The agent has gone from literal zero, no Cloudflare account at all, without any preconfigured Agent Skills or MCP server, to having:

• Provisioned a new Cloudflare account
• Obtained an API token
• Purchased a domain
• Deployed an app to production

But wait — how did the agent discover that it could do all of this? How did it know what services it could provision, and how to purchase a domain? How did it gain the context it needed to understand how to deploy to Cloudflare? Let's dig in.

How the protocol and integration works

There are three components to the interaction between the agent, Stripe, and Cloudflare shown above:

• Discovery — the agent can call a command to query the catalog of available services.
• Authorization — the platform attests to the identity of the user, allowing providers to provision accounts or link existing ones, and securely issue credentials back to the agent.
• Payment — the platform provides a payment token that providers can use to bill the customer, allowing the agent to start subscriptions, make purchases and be billed on a usage basis.

These build on prior art and existing standards like OAuth, OIDC and payment tokenization — but are used together to remove many steps that might otherwise require a human in the loop.

Discovery: how agents find services they can provision themselves

In the agent session above, before the agent ran the CLI command stripe projects add cloudflare/registrar:domain, it first had to discover the Cloudflare Registrar service. It did this by calling the stripe projects catalog command, which returns available services:

The full set of Cloudflare products and services from other providers is long and growing — arguably overwhelming to humans. But for agents, this catalog of services is exactly the context they need. The agent chooses services to use from this catalog based on what the user has asked them to do and the user's preferences — but the user needs no prior knowledge of what services are offered by which providers, and does not need to provide any input. Providers like Cloudflare make this catalog available via a simple REST API that returns JSON, and that gives agents everything they need.

Authorization: instant account creation for new users

When the agent chooses a service and provisions it (ex: stripe projects add cloudflare/registrar:domain), it provisions the resource within a Cloudflare account. But how is it able to create one on demand, without sending a human to a signup page?

Remember how at the start, the user signed in to their Stripe account? Stripe acts as the identity provider, attesting to the user's identity. Cloudflare automatically provisions a new account for the user if no account already exists, and returns credentials back to the Stripe Projects CLI, which are securely stored, but available to the agent to use to make authenticated requests to Cloudflare. This means if someone is brand new to Cloudflare or other services, they can start building right away with their agent, without extra steps.

If the user already has a Cloudflare account, they're sent through a standard OAuth flow to grant access to the Stripe Projects CLI, allowing them to provision resources on their existing Cloudflare account.

Payment: give your agent a budget it can spend, without giving it your credit card info

You might rightly worry, "What if my agent goes a bit overboard and starts buying dozens of domains? Will I end up on the hook for a massive bill? Can I really trust my agent with my credit card?"

The protocol accounts for this in two ways. When an agent provisions a paid service, Stripe includes a payment token in the request to the Provider (Cloudflare). Raw payment details like credit card numbers aren't ever shared with the agent. Stripe then sets a default limit of $100.00 USD/month as the maximum the agent can spend on any one provider. When you're ready to raise this limit, you can then set Budget Alerts on your Cloudflare account.

Any platform with signed-in users can integrate with Cloudflare in the same way Stripe does

Any platform with signed-in users can act as the "Orchestrator", playing the same role Stripe does with Stripe Projects, and integrate with Cloudflare.

Let's say your product is a coding agent. You'd love for people to be able to take what they've built and get it deployed to production, using Cloudflare and other services. But the last thing you want is to send people down a maze of authorization flows and decision trees of where and how to deploy it. You just want to let people ship.

Your platform acts as the Orchestrator, with the already signed-in user. When your user needs a domain, a storage bucket, a sandbox to give their agent, or anything else, you make one API call to Cloudflare to provision a new Cloudflare account to them, and get back a token to make authenticated requests on their behalf.

Or let's say you want Cloudflare customers to be able to easily provision your service, similar to how Cloudflare is partnering with Planetscale to make it possible to create Planetscale Postgres databases directly from Cloudflare. We started working with Planetscale on this well before this new protocol got off the ground, but the flow here is quite similar. Cloudflare acts as the Orchestrator, letting you connect to your PlanetScale account, create databases, and use the user's existing payment method for billing.

This new protocol starts to standardize the types of cross-product integrations that many platforms have been doing for years, often in ways that were one off or bespoke to a particular platform. Without a standard, each integration required engineering work that often couldn't be leveraged for future integrations. Similar to how the OAuth standard made it possible to delegate access to your account to other platforms, the protocol uses OAuth and extends further into payments and account creation, doing so in a way that treats agents as a first-class concern.

We're excited to continue evolving the standard, and to work with Stripe on sharing a more official specification soon. We're also excited to integrate with more platforms — email us at [email protected], and tell us how you want your platform to integrate with Cloudflare.

Give your agent the power to provision and pay

Stripe Projects is in open beta, and you can get started even if you don't yet have a Cloudflare account. Just install the Stripe CLI, log in to Stripe, and then start a new project:

stripe projects init

Prompt your agent to build something new on Cloudflare, and show us what you've built!

Apple has reportedly stopped work on the Vision Pro, and the Vision Pro team has been redistributed to other teams within the company.

Flow generation through natural language: An agentic modeling approach

We fine-tuned Qwen3-32B into a tool-calling agent that generates Flow automations from natural language—faster, cheaper, and more accurate than the frontier model it replaced, with a weekly retraining flywheel built on real merchant data.

If you're building AI products on top of closed models, anyone with an API key can get similar capabilities. Lasting differentiation comes from proprietary data, the training recipe, the infrastructure, and the speed of iteration.

Shopify has something most companies don't: a product surface where millions of merchant interactions directly signal whether the model's output is any good. That feedback loop is the foundation, but only if you keep learning from it.

We fine-tuned a tool-calling agent to turn natural language into a Shopify Flow for Sidekick, our AI commerce assistant. It's 2.2x faster, 68% cheaper, and outperforms closed models.

Along the way, we found lessons no paper warned us about. Data preprocessing decisions, from representation design to formatting details, that compound to swing accuracy by double digits. Silent infrastructure failures that degrade your model with zero warnings and take days to trace. Benchmark parity that masks a 35% gap once real users show up.

This post covers the problems we faced, how we fixed them, and what to look for if you're doing the same.

Building the training dataset

Shopify Flow is an automation platform where store owners build workflows from triggers, conditions, and actions. For store owners who aren't engineers, building the right workflow from a blank canvas is daunting. Sidekick generates it from plain English.

The cold start problem

Fine-tuning required training data, but since the feature hadn't been deployed yet, there were no production conversations to learn from.

We reverse-engineered user intent from existing production workflows. Thousands of anonymized store owners had already built workflows manually in Flow. We sampled those and filtered for quality: workflows that had run at least once in the last seven days, from merchants with two or more qualifying workflows, with one example per descriptor to ensure diversity across workflow types.

With a set of validated workflows, we worked backwards:

1. Sample a workflow. Pick a popular, validated workflow from production.
2. Generate a user query. Use a stronger LLM to produce a plausible natural-language request that would lead to this workflow.
3. Construct the tool trajectory. Build the full multi-turn sequence of tool calls that an ideal agent would execute to arrive at this workflow. This was the bulk of the engineering effort.

We fine-tuned Qwen3-32B on this synthetic dataset and evaluated it against a benchmark of 300 hand-crafted examples covering the breadth of expected Flow usage. An LLM evaluation framework compares the generated workflow against the expected one for semantic correctness, and validates syntactic correctness programmatically.

We looked at three metrics:

• Semantic correctness: Does the generated workflow do what it's supposed to? An LLM judge compares the output against the expected workflow.
• Syntactic correctness: Are there errors that would cause it to fail? Malformed conditions, incorrect references, invalid configurations. Checked programmatically.
• Latency: Time from request to workflow delivery.

If you're building an agent without interaction data, start with the output artifacts your users already produce and work backwards from them. That is often the right first step before your metrics have caught up. As shown in the table above, there is still a meaningful gap to close. Our second lesson, which we discuss below, is that teaching the model to generate Flows in Python can help narrow that gap further.

Training in-distribution: the Python DSL

Shopify Flow workflows are represented internally in a JSON-based domain-specific language (DSL) designed for backend parsing, validation, and execution. That format is ideal for production systems, but it's a poor fit for LLMs. Conditional, program-like logic that would normally appear as code is embedded in deeply nested JSON, a pattern that's rare in pretraining data.

Rather than forcing the model to learn Flow's native format from scratch, we reformulated the task in a representation closer to the model's training distribution. Workflows are programs, so we taught the model to write them as Python.

A transpiler converts the JSON DSL into semantically equivalent Python:

Same workflow, same semantics, but the model now generates Python instead of a data format. Python is far closer to code and logical reasoning, and it makes up a large share of pretraining data. The fine-tuned model draws on familiar patterns: decorators, if/else logic, variables, for loops, and function calls.

With the same training data, switching from the JSON DSL to the Python DSL improved syntactic correctness by 22 points and semantic correctness by 13 points. Moving the target format from out-of-distribution to in-distribution turned the problem from "learn a new language and the task" into "learn the task."

Making this work required building a round-trip transpiler between Python and Flow's JSON representation to handle the full complexity of Flow logic without losing meaning in either direction.

Reliability was backed with extensive tests. We round-trip tested every workflow merchants created through Sidekick in production: converting from JSON to Python and back to JSON, then verifying the output matched the original exactly. Any mismatch was caught before it could reach training data. This process ran continuously across all production workflows, giving us confidence the transpiler handled the full range of real-world patterns.

At inference time, the model writes Python. The transpiler converts it to JSON for the Flow backend. Store owners never see Python, and the backend never has to understand it. Python is the model's internal language.

Prior work has explored Python as an intermediate representation (SPEAC, LLMLift, WorkflowLLM), but via prompting or without a round-trip transpiler. What distinguishes this approach is the full loop: fine-tuning on Python combined with a transpiler back to the production DSL, without changing any downstream systems.

If you're training a model on a custom DSL, consider translating it into a language the model already knows. This helps separate learning the format from learning the task. As the results show, the gap narrows, but there is still room for improvement. At that point, the next step is to bring the system into production, learn from real usage, and incorporate real user feedback.

Mirroring the production environment

Representation was one half of the data problem. The other half was making sure the model's training data matched exactly what it would see in production.

We knew training data should match production. What we didn't expect was how sensitive the model is to the degree of match. Every difference we closed, no matter how minor, improved eval scores:

• Tool naming and ordering: Training data used the full prefixed name flow_app_agent_task_search. At inference, the same tool was called task_search. Functionally identical, but the model treated them as different tools. Removing the prefix from training data to match inference improved accuracy. The order in which the tools appeared in the system prompt also mattered. Shuffle the order between training and serving, and performance drops.
• Tool response format: Tool responses return JSON objects with multiple fields. In the training data, we sorted keys alphabetically. If production returned them in a different order, or included an extra field, the model noticed. Any drift between what the training data showed and what production APIs actually returned degraded accuracy.
• System prompt and tool descriptions: Tool descriptions in production changed frequently as the product team iterated on behavior. Every update had to be reflected in the training data, or the model's behavior drifted. Keeping both in sync was an ongoing process, not a one-time fix.

None of these are about the logic of the task. They are formatting details. The model treats every token as a signal, whether you intended it or not.

Optimizing the tool-calling stack

When an agent calls tools, every response becomes part of the context. Context grows, latency grows, cost grows. Worse, irrelevant context dilutes the signal. The model reasons less accurately when it's processing information it won't use.

We restructured our tool interfaces to minimize context at each step. Instead of returning full details for every result upfront, tools return lightweight summaries first. The model scans the summaries, selects what it needs, then retrieves full details only for those necessities. Two cheap calls instead of one expensive one.

For example, Flow has hundreds of available triggers, conditions, and actions. A search might return 100 matches. Rather than loading the full configuration schema for each one, task_search returns just names and descriptions. The model picks the 2-3 it actually needs, then calls task_configuration to get the full schema only for those. The context stays small, the reasoning stays focused.

Making training fast

As our data pipeline grew, so did a tension: more training data improved accuracy but slowed each run. Slower runs meant fewer iterations, and fewer iterations meant slower improvement. We needed a way to use all the data and still retrain weekly.

We built the infrastructure to make both possible. Qwen3-32B trains on two nodes of H200 GPUs with Fully Sharded Data Parallel (FSDP). A full training run takes 12 hours, fast enough for weekly retraining with multiple experimental runs in between.

The full pipeline, from data collection through training, evaluation, and deployment, runs on Tangle, Shopify's open-source ML experimentation platform. Tangle composes each step into a single reproducible workflow with intelligent caching. Only the affected steps re-run when one part changes.

CometML tracks every run. HuggingFace hosts datasets and checkpoints. CentML serves the model in production. Weekly retraining runs without manual intervention.

Evaluation: benchmarks aren't ground truth

Synthetic data got us to parity on offline benchmarks. By every metric we tracked, the fine-tuned model was ready for production. We deployed it to 1% of traffic to see how it held up.

At 1% traffic, the fine-tuned model's workflow activation rate (whether store owners actually turn on the workflows Sidekick generates) came in 35% lower than the prompt-based agent. The benchmark covered what we expected merchants to ask. It didn't cover what they actually asked: editing existing workflows, handling email configurations, working with third-party integrations, and asking questions about Flow without intending to create a workflow.

The model performed well in-domain, but real traffic quickly surfaced out-of-distribution requests that our synthetic data had not covered. The low-traffic early deployment showed us exactly where to focus next. Activation rate was our first production signal, but it turned out to be noisy: it reflects merchant behavior, not model quality. We therefore optimized for a domain-expert-calibrated LLM judge, which we describe next, while keeping activation rate as a guardrail to ensure we did not regress.

Flywheel: from catching up to pulling ahead

Closing the gap

The 1% deployment showed us exactly where the model was falling short. We needed a system that could diagnose those gaps, fix them, and retrain fast. Not once, but continuously.

We built an LLM-based judge that scores each conversation across the workflow lifecycle: whether the assistant correctly understood the merchant's intent, chose a Flow solution only when appropriate, selected the right components, and gave clear next steps. The judge grades each facet separately rather than treating quality as a single pass/fail outcome. To calibrate it, we collected human annotations on hundreds of conversations and tuned it until its scores aligned with human judgment, then validated against production activation rate.

A tagging system classifies every workflow along multiple dimensions: which triggers it uses, what conditions it checks, which actions it invokes, and whether it involves third-party integrations. Comparing performance across tagged slices pinpoints exactly where the model struggles. When performance drops on a particular slice, we know what kind of data to add.

The judge and tagging system together form the diagnostic layer. The fixes were concrete:

• Email workflows accounted for 25% of failures, so we added email-specific examples
• Diverse condition patterns accounted for 16%
• Workflow editing, which was something synthetic data had never covered

The following diagram shows our progress in Flow modeling, with quality improving steadily over time as measured by our LLM judge:

Continuous improvement

Closing the gap was the first test. Staying ahead is the real goal.

Every production conversation becomes a training signal. We sample high-quality examples: conversations where merchants actually activated the workflow afterwards. The judge scores them, and high-scoring conversations are routed into the training pool automatically. Low-scoring ones are quarantined for review.

The loop runs weekly:

1. Ingest production conversations
2. Score with the LLM judge
3. Route high-quality examples into training; quarantine low-quality for review
4. Identify gaps through tagged slice analysis
5. Retrain and deploy

The system improves as production traffic shifts, freeing the team to focus on expanding coverage and fixing systematic gaps rather than hand-curating data. The approach is similar in spirit to Karpathy's Autoresearch, an automated loop that evaluates, keeps what works, discards what doesn't, and iterates—but applied to production data curation rather than training code.

What's next

The flywheel is running, but the race between in-house and closed-source models doesn't stop. Every few months, a new frontier model raises the bar. The only way to stay ahead is to keep compounding: better data, better training, better evaluation, faster iteration. Here's where we're pushing next.

Simulation environments. A sandbox where the model can generate workflows and receive structured feedback on whether they would succeed, without impacting real merchants. The model writes test cases and runs them against a simulated Flow environment, creating a setting for verifiable rewards. This opens the door to distillation from stronger teacher models and on-policy optimization.

From off-policy to on-policy. Everything so far is off-policy: the model learns from curated examples collected after the fact. With verifiable rewards from the simulation environment, the next step is policy optimization where the model learns from its own generated trajectories. The goal is a model that discovers better strategies, not one that only replicates what it's seen.

From manual calibration to self-improving evaluation. Today, the LLM judge is calibrated against human annotations and production activation rate. But merchant behavior shifts, new integrations launch, and new workflow patterns emerge faster than manual recalibration can keep up. Automating judge calibration against live production signals is the next evaluation challenge.

Results in production

The fine-tuned Flow agent now serves the majority of our production traffic.

No single technique got us here. Each stage built on the last. Synthetic data generation needed the Python DSL to close the accuracy gap. The DSL needed production mirroring to hold up in the real environment. Production mirroring needed infrastructure stable enough to trust. And when benchmarks said we were ready but production said otherwise, the flywheel closed the gap in two weeks.

When does this generalize?

This approach applies when:

1. The task requires tool calling. The model must reason, act, and incorporate external results, not just generate text.
2. The output format is a custom DSL that doesn't appear in pretraining data, and its semantics can be expressed in a language the model already knows.
3. A round-trip transpiler is feasible between the in-distribution representation and the production format.
4. A production feedback loop is available. Synthetic data gets you started, but real-world data is what gets you to production quality.

Within Sidekick, this pattern is already being applied to other skills. The recipe is the same: isolate the skill, fine-tune the tool-calling model, and build the loop for continuous improvement.

Six months ago, this system ran on a frontier model we didn't control. Now it runs on a model we trained, on infrastructure we own, improving from data only we have, at 68% lower cost. The version running right now is already worse than the one retraining behind it.

We started on rented ground. This is what the first mile of owned ground looks like.

Pinterest built a dedicated two-tower retrieval model to generate better shopping ad candidates optimized for offsite conversions, moving beyond traditional click/engagement-based signals which are abundant but poorly correlated with actual buying intent. The system uses a unified multi-task architecture with parallel DCN v2 and MLP cross layers, clever training techniques to handle sparse and noisy conversion data, and an advertiser-level loss function.

Vinted rebuilt its search autocomplete system, moving from static, generic suggestions to a hybrid approach combining a strong heuristic scoring model with a Learning-to-Rank (LTR) model. They score suggestions offline using popularity, sell-through rate, and usage signals, index them with clever prefix and fuzzy matching techniques, then apply a LightGBM model in real-time that incorporates user behavior and context to re-rank results.

Skipper is a lightweight, embedded workflow engine designed to provide durable and reliable execution for long-running business processes (like insurance claims and payments). Instead of relying on external orchestration tools or queues, Skipper uses a simple annotation-based approach to persist state in the service's existing database and achieves durability through deterministic replay.

GraphRAG is most useful when questions require multi-hop reasoning across documents, entity relationships, or system-level dependencies: use Vector RAG for simple factual lookups and keep GraphRAG as an opt-in backend. In production, the main pain points are heavy indexing cost, difficult updates, multi-layer evaluation, and infrastructure that usually needs batch jobs rather than request-path execution. Success depends on selective graph scope, explicit update policies, repeatable evals, and strong observability/cost controls.

A/B testing failures are far more often caused by broken infrastructure and poor experimentation practices than by the ideas being tested. Common failures include Sample Ratio Mismatch (SRM) from bad randomization, early peeking that inflates false positives, insufficient statistical power, and optimizing the wrong metrics without guardrails, causing misleading results.

LLM Inference for Large-Context Offline Workloads

oLLM is a lightweight Python library for large-context LLM inference, built on top of Huggingface Transformers and PyTorch. It enables running models like gpt-oss-20B, qwen3-next-80B or Llama-3.1-8B-Instruct on 100k context using ~$200 consumer GPU with 8GB VRAM. No quantization is used—only fp16/bf16 precision.

Latest updates (1.0.3) 🔥

• AutoInference with any Llama3 / gemma3 model + PEFT adapter support
• kvikio and flash-attn are optional now, meaning no hardware restrictions beyond HF transformers
• Multimodal voxtral-small-24B (audio+text) added. [sample with audio]
• Multimodal gemma3-12B (image+text) added. [sample with image]
• qwen3-next-80B (160GB model) added with ⚡️1tok/2s throughput (our fastest model so far)
• gpt-oss-20B flash-attention-like implementation added to reduce VRAM usage
• gpt-oss-20B chunked MLP added to reduce VRAM usage

8GB Nvidia 3060 Ti Inference memory usage:

By "Baseline" we mean typical inference without any offloading

How do we achieve this:

• Loading layer weights from SSD directly to GPU one by one
• Offloading KV cache to SSD and loading back directly to GPU, no quantization or PagedAttention
• Offloading layer weights to CPU if needed
• FlashAttention-2 with online softmax. Full attention matrix is never materialized.
• Chunked MLP. Intermediate upper projection layers may get large, so we chunk MLP as well

Typical use cases include:

• Analyze contracts, regulations, and compliance reports in one pass
• Summarize or extract insights from massive patient histories or medical literature
• Process very large log files or threat reports locally
• Analyze historical chats to extract the most common issues/questions users have

Supported GPUs: NVIDIA (with additional performance benefits from kvikio and flash-attn), AMD, and Apple Silicon (MacBook).

Getting Started

It is recommended to create venv or conda environment first

python3 -m venv ollm_env
source ollm_env/bin/activate

Install oLLM with pip install --no-build-isolation ollm or from source:

git clone https://github.com/Mega4alik/ollm.git
cd ollm
pip install --no-build-isolation -e .

# for Nvidia GPUs with cuda (optional): 
pip install kvikio-cu{cuda_version} Ex, kvikio-cu12 #speeds up the inference

💡 Note: voxtral-small-24B requires additional pip dependencies to be installed as pip install "mistral-common[audio]" and pip install librosa

Check out the Troubleshooting in case of any installation issues

Example

Code snippet sample

from ollm import Inference, file_get_contents, TextStreamer
o = Inference("llama3-1B-chat", device="cuda:0", logging=True) #llama3-1B/3B/8B-chat, gpt-oss-20B, qwen3-next-80B
o.ini_model(models_dir="./models/", force_download=False)
o.offload_layers_to_cpu(layers_num=2) #(optional) offload some layers to CPU for speed boost
past_key_values = o.DiskCache(cache_dir="./kv_cache/") #set None if context is small
text_streamer = TextStreamer(o.tokenizer, skip_prompt=True, skip_special_tokens=False)

messages = [{"role":"system", "content":"You are helpful AI assistant"}, {"role":"user", "content":"List planets"}]
input_ids = o.tokenizer.apply_chat_template(messages, reasoning_effort="minimal", tokenize=True, add_generation_prompt=True, return_tensors="pt").to(o.device)
outputs = o.model.generate(input_ids=input_ids,  past_key_values=past_key_values, max_new_tokens=500, streamer=text_streamer).cpu()
answer = o.tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=False)
print(answer)

or run sample python script as PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python example.py

# with AutoInference, you can run any LLama3/gemma3 model with PEFT adapter support
# pip install peft 
from ollm import AutoInference
o = AutoInference("./models/gemma3-12B", # any llama3 or gemma3 model
  adapter_dir="./myadapter/checkpoint-20", # PEFT adapter checkpoint if available
  device="cuda:0", multimodality=False, logging=True)
...

More samples

• gemma3-12B image+text
• voxtral-small-24B audio+text
• AutoInference + SFT

Roadmap

For visibility of what's coming next (subject to change)

• Qwen3-Next quantized version
• Qwen3-VL or alternative vision model
• Qwen3-Next MultiTokenPrediction in R&D

Contact us

If there's a model you'd like to see supported, feel free to suggest it in the discussion — I'll do my best to make it happen.

Materialized Tables in Apache Flink allows users to define a table directly with its population query, embedding both the schema and the continuous or scheduled refresh logic inside the catalog. This simplifies ETL pipelines by automatically handling job lifecycle, schema evolution, and refreshes.

HOT Updates in PostgreSQL is a clever storage optimization that allows UPDATEs on unindexed columns to avoid touching indexes entirely when the new tuple fits on the same page as the old one. Instead of creating new index entries, PostgreSQL marks the old tuple as HOT_UPDATED and places a HEAP_ONLY tuple on the same page, forming a chain that scans can follow, which reduces WAL traffic, index maintenance, and vacuuming overhead.

Rocky is a Rust-based tool that adds a control layer on top of data warehouses, helping teams manage pipelines with features like data contracts, lineage tracking, and safe testing through branches. It focuses on catching errors early, preventing data issues, and making data workflows more reliable and easier to understand.

Most tutorials on client-side data storage reach for IndexedDB, localStorage, or a third-party sync service. This one goes somewhere different: a real SQLite database, running as WebAssembly in the browser, with the database file living on the user's own Google Drive.

This is the setup behind OvertimeIQ — but everything in this article stands alone as a practical reference. You don't need to care about overtime tracking for any of this to be useful.

By the end, you'll know how to:

• Initialize sql.js and run real SQL in the browser
• Persist the database across page reloads via localStorage
• Upload and download the database file from Google Drive
• Handle sync conflicts correctly
• Protect against data corruption on interrupted uploads

Why SQLite in the browser?

Before we write any code, it's worth asking why you'd reach for SQLite instead of IndexedDB or a cloud-synced store.

The answer is portability. A SQLite database is a single binary file. You can open it on any device, in any SQLite-compatible tool, without installing anything. You can attach it to an email, drop it in Dropbox, or — as we'll do here — store it on Google Drive. The user owns a file, not a schema locked inside a browser's internal storage.

For apps where user data portability matters — personal finance tools, health tracking, anything sensitive — this is a meaningful architectural choice, not just a curiosity.

The trade-off: sql.js ships a ~1.5MB WASM binary. We'll deal with that below.

Setting up sql.js

Install the package:

npm install sql.js

The WASM binary needs to be accessible at a URL your code can load. Copy it into your public folder at build time:

// vite.config.js
import { defineConfig } from 'vite'
import { viteStaticCopy } from 'vite-plugin-static-copy'

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/sql.js/dist/sql-wasm.wasm',
          dest: ''
        }
      ]
    })
  ]
})

Now initialise sql.js. This is async — the WASM binary has to load before you can do anything:

// lib/db.js
import initSqlJs from 'sql.js'

let db = null

export async function initDB(existingBuffer = null) {
  const SQL = await initSqlJs({
    locateFile: file => `/${file}` // points to /sql-wasm.wasm in public/
  })

  if (existingBuffer) {
    // Restore from a saved buffer (localStorage or Drive download)
    db = new SQL.Database(new Uint8Array(existingBuffer))
  } else {
    // Fresh database
    db = new SQL.Database()
  }

  return db
}

Lazy loading matters here. Don't initialise the database on app load. Initialise it on first access. With a Service Worker caching the WASM binary after the first load, subsequent loads are instant — but you still don't want to block your UI render on a 1.5MB download for users on their first visit.

Running SQL

sql.js has two main operations:

// For SELECT — returns an array of result objects
export function runQuery(sql, params = []) {
  const stmt = db.prepare(sql)
  stmt.bind(params)
  const rows = []
  while (stmt.step()) {
    rows.push(stmt.getAsObject())
  }
  stmt.free()
  return rows
}

// For INSERT / UPDATE / DELETE — no return value
export function execSQL(sql, params = []) {
  const stmt = db.prepare(sql)
  stmt.run(params)
  stmt.free()
}

// Convenience wrapper for single-row queries
export function getOne(sql, params = []) {
  const rows = runQuery(sql, params)
  return rows.length > 0 ? rows[0] : null
}

Usage is exactly what you'd expect from a SQL library:

execSQL(
  'INSERT INTO logs (job_id, date, start_time, end_time, duration_hours, location) VALUES (?, ?, ?, ?, ?, ?)',
  [1, '2025-04-14', '20:00', '23:30', 3.5, 'office']
)

const logs = runQuery(
  'SELECT * FROM logs WHERE date >= ? ORDER BY date DESC',
  ['2025-01-01']
)

Schema migrations

You need a migration runner. The pattern I use: a schema_version table with a single integer, and a list of migration functions keyed by version number.

const MIGRATIONS = {
  1: (db) => {
    db.run(`
      CREATE TABLE IF NOT EXISTS jobs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL,
        hourly_rate REAL NOT NULL,
        weekend_multiplier REAL DEFAULT 1.5,
        holiday_multiplier REAL DEFAULT 2.0,
        work_start TEXT NOT NULL,
        work_end TEXT NOT NULL,
        color TEXT DEFAULT '#3B8BD4',
        is_default INTEGER DEFAULT 0,
        created_at TEXT NOT NULL
      )
    `)
    db.run(`
      CREATE TABLE IF NOT EXISTS logs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        job_id INTEGER REFERENCES jobs(id),
        date TEXT NOT NULL,
        start_time TEXT NOT NULL,
        end_time TEXT NOT NULL,
        crosses_midnight INTEGER DEFAULT 0,
        duration_hours REAL NOT NULL,
        location TEXT NOT NULL,
        notes TEXT,
        created_at TEXT NOT NULL,
        updated_at TEXT NOT NULL
      )
    `)
    db.run('CREATE INDEX IF NOT EXISTS idx_logs_date ON logs(date)')
  }
}

export async function runMigrations(db) {
  db.run('CREATE TABLE IF NOT EXISTS schema_version (version INTEGER NOT NULL)')

  const row = getOne('SELECT version FROM schema_version')
  const currentVersion = row ? row.version : 0

  const pendingVersions = Object.keys(MIGRATIONS)
    .map(Number)
    .filter(v => v > currentVersion)
    .sort((a, b) => a - b)

  for (const version of pendingVersions) {
    MIGRATIONS[version](db)
    if (currentVersion === 0) {
      execSQL('INSERT INTO schema_version (version) VALUES (?)', [version])
    } else {
      execSQL('UPDATE schema_version SET version = ?', [version])
    }
  }
}

Run migrations immediately after initializing the database, before anything else touches it.

Serializing to Uint8Array

This is the key operation that makes everything else work. sql.js can export the entire database state as a Uint8Array — a binary blob that is identical to what SQLite would write to disk.

export function serializeDB() {
  return db.export() // Returns Uint8Array
}

That Uint8Array is your database file. Everything that follows — localStorage persistence, Drive upload, Drive download — is just moving that blob around.

localStorage persistence

After every write operation, serialize and save:

const DB_STORAGE_KEY = 'otiq_db'

export function saveDB() {
  const buffer = serializeDB()
  // Convert Uint8Array to a regular array for JSON serialisation
  localStorage.setItem(DB_STORAGE_KEY, JSON.stringify(Array.from(buffer)))
  // Trigger the Drive upload debounce (see below)
  scheduleDriveUpload()
}

export function loadFromLocalStorage() {
  const stored = localStorage.getItem(DB_STORAGE_KEY)
  if (!stored) return null
  return new Uint8Array(JSON.parse(stored))
}

On app load, check localStorage first. If there's a saved buffer, restore from it. Then compare with Drive to decide whether to download a newer version.

Storage size note: A SQLite file with thousands of rows will likely stay well under 5MB — comfortably within localStorage limits. If your use case could grow very large, consider using the Origin Private File System instead, but for personal data tools localStorage is generally fine. Always call navigator.storage.persist() on first load to request durable storage — without it, browsers can evict localStorage under storage pressure.

async function requestDurableStorage() {
  if (navigator.storage && navigator.storage.persist) {
    const granted = await navigator.storage.persist()
    if (!granted) {
      // Show a warning banner — Drive sync is the backup
      showStorageWarning()
    }
  }
}

Google Drive as cloud sync

The Drive setup requires Google OAuth with the drive.file scope — the minimal scope that grants access only to files this specific app created. It cannot read other Drive files. This is the right choice for privacy-sensitive apps.

I'll cover the full PKCE OAuth flow in the next article in this series. For now, assume you have a valid access_token.

Finding the database file

On login, search for an existing database file:

async function findDBFile(accessToken) {
  const response = await fetch(
    `https://www.googleapis.com/drive/v3/files?q=name='overtimeiq.db'&fields=files(id,modifiedTime)`,
    { headers: { Authorization: `Bearer ${accessToken}` } }
  )
  const data = await response.json()
  return data.files?.[0] ?? null // { id, modifiedTime } or null
}

Creating the file (first time)

async function createDBFile(accessToken, dbBuffer) {
  const metadata = {
    name: 'overtimeiq.db',
    mimeType: 'application/octet-stream'
  }

  const formData = new FormData()
  formData.append('metadata', new Blob([JSON.stringify(metadata)], { type: 'application/json' }))
  formData.append('file', new Blob([dbBuffer], { type: 'application/octet-stream' }))

  const response = await fetch(
    'https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&fields=id',
    {
      method: 'POST',
      headers: { Authorization: `Bearer ${accessToken}` },
      body: formData
    }
  )
  const data = await response.json()
  return data.id // Store this file ID in settings.drive_file_id
}

Downloading the file

async function downloadDBFile(accessToken, fileId) {
  const response = await fetch(
    `https://www.googleapis.com/drive/v3/files/${fileId}?alt=media`,
    { headers: { Authorization: `Bearer ${accessToken}` } }
  )
  const buffer = await response.arrayBuffer()
  return new Uint8Array(buffer)
}

The sync decision logic

On every app load after login, you need to decide: use the local database, or download from Drive?

async function syncOnLogin(accessToken) {
  const driveFile = await findDBFile(accessToken)

  if (!driveFile) {
    // First time — upload local DB and store the file ID
    const buffer = loadFromLocalStorage() ?? serializeDB()
    const fileId = await createDBFile(accessToken, buffer)
    execSQL('UPDATE settings SET drive_file_id = ? WHERE id = 1', [fileId])
    execSQL('UPDATE settings SET last_synced_at = ? WHERE id = 1', [new Date().toISOString()])
    return
  }

  const driveModifiedTime = new Date(driveFile.modifiedTime).getTime()
  const localSyncedAt = getOne('SELECT last_synced_at FROM settings WHERE id = 1')?.last_synced_at
  const localTime = localSyncedAt ? new Date(localSyncedAt).getTime() : 0

  const diff = Math.abs(driveModifiedTime - localTime)

  if (diff < 30_000) {
    // Within 30 seconds — same-device multi-tab edge case, no action
    return
  }

  if (driveModifiedTime > localTime) {
    // Drive is newer — download and replace
    const buffer = await downloadDBFile(accessToken, driveFile.id)
    await reinitializeFromBuffer(buffer) // Re-init sql.js with the new buffer
    showToast('Synced from Drive')
  } else {
    // Local is newer — upload
    await uploadDBToDrive(accessToken, driveFile.id)
  }

  execSQL('UPDATE settings SET last_synced_at = ? WHERE id = 1', [new Date().toISOString()])
}

Conflict resolution policy: When in doubt, prefer the Drive copy. Drive is the source of truth. If modifiedTime comparison is inconclusive (e.g., clock skew between devices), take the Drive copy and show a toast: "Synced from Drive — local changes from this session may have been overwritten."

The upload safety pattern

Never upload directly to overtimeiq.db. Upload to a temp file first, then rename atomically. A browser crash, network interruption, or error mid-upload should never corrupt the live database.

async function uploadDBToDrive(accessToken, fileId) {
  const buffer = serializeDB()

  // Step 1: Upload to temp file
  const tempMetadata = { name: 'overtimeiq_tmp.db' }
  const formData = new FormData()
  formData.append('metadata', new Blob([JSON.stringify(tempMetadata)], { type: 'application/json' }))
  formData.append('file', new Blob([buffer], { type: 'application/octet-stream' }))

  const uploadResponse = await fetch(
    `https://www.googleapis.com/upload/drive/v3/files/${fileId}?uploadType=multipart`,
    {
      method: 'PATCH',
      headers: { Authorization: `Bearer ${accessToken}` },
      body: formData
    }
  )

  if (!uploadResponse.ok) throw new Error('Upload failed')

  // Step 2: Rename temp file to live file atomically
  await fetch(
    `https://www.googleapis.com/drive/v3/files/${fileId}`,
    {
      method: 'PATCH',
      headers: {
        Authorization: `Bearer ${accessToken}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ name: 'overtimeiq.db' })
    }
  )
}

If Step 1 succeeds but Step 2 fails, the user has a temp file but the live file is intact. On the next sync, the timestamp comparison will catch the discrepancy and prompt a re-upload.

Debouncing the Drive upload

You don't want to upload to Drive on every keypress or every individual log entry mutation. Debounce it:

let driveUploadTimeout = null

export function scheduleDriveUpload() {
  if (driveUploadTimeout) clearTimeout(driveUploadTimeout)
  driveUploadTimeout = setTimeout(async () => {
    const accessToken = getAccessToken() // From your auth store
    const fileId = getOne('SELECT drive_file_id FROM settings WHERE id = 1')?.drive_file_id
    if (accessToken && fileId) {
      await uploadDBToDrive(accessToken, fileId)
      execSQL('UPDATE settings SET last_synced_at = ? WHERE id = 1', [new Date().toISOString()])
    }
  }, 10_000) // 10 seconds after the last write
}

The localStorage write happens synchronously on every mutation — data is safe locally the instant you write it. The Drive upload is fire-and-forget with a 10-second debounce so a bulk import (100 rows at once) only triggers one upload.

Putting it all together

The initialization sequence on app load:

async function initializeApp() {
  // 1. Try to restore from localStorage
  const storedBuffer = loadFromLocalStorage()

  // 2. Initialize sql.js with the stored buffer (or fresh)
  await initDB(storedBuffer)

  // 3. Run schema migrations
  await runMigrations(db)

  // 4. Seed defaults if this is the first launch
  if (!getOne('SELECT id FROM jobs WHERE id = 1')) {
    seedDefaultJob()
    seedHolidays()
  }

  // 5. Request durable storage
  await requestDurableStorage()

  // 6. If authenticated, sync with Drive
  const accessToken = getStoredAccessToken()
  if (accessToken) {
    await syncOnLogin(accessToken)
  }
}

What this doesn't cover

This article focused on the storage and sync layer. Two things worth a separate deep dive:

The Google OAuth PKCE flow — how to get the access_token and refresh_token without a client secret, and how to silently refresh the token mid-session. That's the next article in this series.

The midnight rate calculation — how to correctly split a shift that crosses midnight across two different rate multipliers, including the December 31 → January 1 edge case. I'll cover that in a later article on the earnings engine.

The case for this architecture

The pattern here — SQLite on the user's cloud storage, managed entirely client-side — works well for a specific category of app: personal data tools where the data is sensitive, the user count is small, and data portability is a first-class feature.

It's not the right choice for collaborative tools, apps with large binary assets, or anything requiring server-side processing of the data. But for personal productivity software, financial tracking, health logging, or any domain where "your data should be yours" is a meaningful promise — this architecture delivers it genuinely, not as marketing copy.

The database is a file. The user can open it in DB Browser for SQLite today. They'll be able to open it in twenty years. That kind of portability is hard to promise with any other approach.

Real-time recommendation systems now need to combine rich contextual features with sub-100 ms latency at scale, often across billions of interaction records. Feature stores act as the consistency layer between offline training and online serving, reducing training-serving skew, while batch platforms compute expensive features and embeddings. Redis is used for low-latency vector similarity search, candidate retrieval, and caching eligibility filters, keeping request paths fast and efficient.

Linux 7.0 accidentally cut PostgreSQL performance in half because a scheduling change increased how long spinlocks were held during memory page faults, causing massive CPU waste, and switching to huge memory pages fixes the issue.

Expedia's Service Telemetry Analyzer uses LLMs plus Datadog's telemetry data to speed incident investigation and reduce time to know/recover.

Claude can now plug directly into Photoshop, Blender, and Ableton

Anthropic is also giving the Blender Foundation a load of cash to help the software stay free and open-source.

Anthropic has launched a set of connectors for Claude that allow the AI chatbot to tap into popular creative software, including Adobe's Creative Cloud apps, Affinity, Blender, Ableton, Autodesk, and more.

This marks the company's latest efforts to break into the creative industry following its launch of Claude Design earlier this month. The new connectors — which enable Claude to access apps, retrieve data, and take actions within connected services — are "designed to make it easier to use Claude for creative work," according to Anthropic, and can be used for specific functions in each app.

The Adobe for creativity connector can draw from Creative Cloud apps like Photoshop, Premiere, and Express to "bring images, videos, and designs to life" in Claude, for example. The connector for Ableton allows Claude to answer questions by sourcing information directly from the music software's official documentation, while the Blender integration gives the 3D modelling app's Python API a natural-language interface. You can read the full breakdown for every connector on Anthropic's blog.

"Claude can't replace taste or imagination, but it can open up new ways of working — faster and more ambitious ideation, a more expansive skillset, and the ability for creatives to take on larger-scale projects," Anthropic said. "AI can also help shoulder the parts of the creative process that eat up time by handling repetitive tasks and eliminating manual toil."

As part of this announcement, Anthropic has also become a Corporate Patron of the Blender Development Fund, which helps to support ongoing development of the open-source software, joining other big sponsors including Netflix, Epic, and Wacom. That membership means Anthropic will be handing Blender at least €240,000 (around $281,000) every year, something Blender says will help it to "keep pursuing projects independently, and to focus on building tools for artists and creators."

The new name unifies what was previously fragmented across Freepik (stock assets), Magnific (AI upscaling), and several other products. One million paying subscribers. 250 enterprise customers, including BBC, Puma, and Amazon Prime Video. CEO Joaquín Cuenca has never taken outside investment. The company is profitable.

Freepik, the Málaga-founded AI creative platform, announced on Tuesday that it is rebranding as Magnific, unifying its full product stack under a single name for the first time.

The rebrand is not cosmetic. It reflects the consolidation of what had been, from the outside, a confusing portfolio: Freepik as a stock asset library, Magnific as an AI image upscaler acquired in May 2024, and several other AI tools operating under separate brands.

The numbers behind the rebrand are striking for a company that has never raised outside investment. Fortune confirmed that Magnific has reached $230 million in annualised recurring revenue.

The company has more than one million paying subscribers, more than 250 enterprise customers, including the BBC, Puma, Carl's Jr, DeliveryHero, Huel, R/GA, Damm, Job&Talent, and Amazon Prime Video's series House of David, and more than four million images generated per day. Andreessen Horowitz has named Magnific the top generative AI web company in Europe by users, placing it ahead of well-capitalised American competitors across a ranking based on actual platform usage.

Cuenca built this on zero venture capital. When Fortune asked whether he would raise in the future, he said: "If we do it, it's because we want to grow the DNA of the company", not because of financial necessity.

Freepik was founded in 2010 in Málaga by Cuenca and his brother Alejandro. Cuenca had previously co-founded Panoramio, a geotagged photo-sharing platform that Google acquired in 2007, his first exit.

Freepik began as an internal tool to find quality graphic resources and grew into a global stock asset platform used in more than 200 countries. The pivot to generative AI began in earnest with the acquisition of Magnific in May 2024.

Magnific was itself founded in Murcia, Spain, by Javi López and Emilio Nicolás; it had gone viral within days of its launch, signing up more than 30,000 users within 24 hours and reaching 725,000 registered users without paid advertising. Both founders remain with the company following the acquisition.

The unified Magnific platform now covers the full creative stack: AI image and video generation (including 4K with audio); its original AI upscaling and enhancement technology; a real-time collaborative workspace; exclusive 3D and virtual scene tools; an AI assistant; an Academy for team training; and the original library of 250 million-plus creative assets. Critically, Magnific is model-agnostic: it lets users select from third-party video AI models including Google's Veo 3.1 and ByteDance's Seeddance 2.0, and combines them with its own tools.

That orchestration layer, letting enterprises pick the best model for each task rather than being locked to a single provider, is the same architecture that has driven adoption of multi-model AI platforms in enterprise software generally.

The "no-collar economy" framing that Cuenca uses to describe the platform's societal positioning is the most ambitious version of the rebrand's implications. His argument, made to Fortune and in the official rebrand announcement, is that the industrial revolution created blue-collar jobs and the digital revolution created white-collar jobs, and that AI is now creating a new class of creative work that requires neither physical labour nor institutional professional credentials.

72 per cent of new creators joining the platform identify as beginners. The Business plan launched for smaller teams in January 2026 surpassed 2,000 subscriptions in six weeks and is growing at 150 new teams per week.

Cuenca has said: "In the future we will make movies in the same way we write books, one person with a vision and the tools to execute it."

That is a bold prediction but not an entirely implausible one, and it is exactly the kind of market framing that attracts enterprise attention.

The competitive context matters. Magnific is competing directly with Midjourney, Runway, Leonardo, Adobe Firefly, and a range of well-capitalised US AI creative platforms, without any of them offering the same integrated end-to-end creative stack, according to the company's own positioning.

Magnific's advantage is not a superior model, it uses the same frontier models as its competitors, but a unified workflow platform that reduces the friction of combining multiple AI tools in production.

Its bootstrapped, profitable status means it has survived and grown through the entire AI investment boom without becoming dependent on the capital cycle that has constrained many of its VC-backed competitors.

The rebrand to Magnific is the moment the company chooses to present that full platform picture publicly for the first time, and to compete for enterprise AI creative budgets under a single brand identity rather than a fragmented product catalogue.

Pedometer++ 8.0 centers on a complete redesign of its Apple Watch app, introducing a bolder, more colorful interface that highlights key daily metrics like steps, distance, and flights climbed with improved clarity. A major part of the redesign is the new workout picker, built with larger touch targets and simpler navigation to make starting activities faster and more intuitive, especially for users frustrated with Apple's default Workout app. While the update also adds Expedition Mode for better battery life, the primary focus is on making the app more visually engaging, easier to use, and better suited for quick interactions on the go.

Dots in icon design should typically be slightly larger than the stroke weight to appear visually balanced, since matching them exactly often makes them look too small. This optical adjustment—common in type design—can be applied across icons, with flexibility to vary dot size or even shape depending on context and emphasis. Different elements within an icon set may require different dot sizes to feel right, especially when dots are a primary feature, reinforcing that visual balance matters more than strict geometric consistency.

Smart users often provide convincing but wrong solutions because they're trained to always have answers, especially executives from consulting or finance backgrounds. User research should focus on understanding underlying needs rather than stated wants - when someone says they want ice cream, they actually need to cool down, which opens up many more solution possibilities. Analytics alone can't protect teams from bad user research, as the same problems that affect interviews also impact how metrics are interpreted.

Adobe Sneaks 2026: Five AI Prototypes Marketers Should Watch

IN THIS ISSUE: This week, I'm sharing some standout projects from Adobe Sneaks—the company's annual showcase of experimental prototypes that hint at where AI-powered creative tools are headed next. From simulated A/B testing to real-time web personalization, five projects stood out as potential game-changers for marketers and creative teams.

The Prompt

Every year, Adobe gives its employees a hall pass—the chance to pitch ideas that exist outside the company's official product roadmap. The best ones surface at the end of the company's Summit and Max events in a showcase called Sneaks. Typically, there are hundreds of submissions—500 this year—and only seven make the cut, a selection overseen by Principal Evangelist Eric Matisoff's team.

However, not every Sneak makes it to market. Matisoff tells me that historically, between 30 and 40 percent of these projects ever make it into production. Those lucky enough may even become some of Adobe's most popular features, such as Generative Fill.

Sneaks isn't a typical demo day experience, and you should certainly not expect it to feel like another keynote. It's meant to be fun and entertaining, which is why Adobe brings on a celebrity co-host. Past guests include Rainn Wilson, Joseph Gordon-Levitt, Jordan Peele, Kumail Nanjiani, Chelsea Handler, Kenan Thompson, and Jessica Williams. This year, Matisoff was joined by actress and comedian Iliza Schlesinger.

This week in Las Vegas, I attended my first Sneaks. Here are the prototypes that caught my attention and that I hope will make it onto Adobe's product roadmap.

Project Face Off (Winner)

Created by research scientist Doga Dogan, Project Face Off simulates A/B testing to predict which creative variant will perform the best and why. Instead of waiting weeks for real-world traffic, marketers can upload competing designs, define the primary conversion goal, and let the system generate synthetic user personas that scroll, click, consider, and either convert or drop off. Results are generated in seconds.

Traditional multivariate testing is slow by design. Marketers have to build multiple variants, configure tracking, stand up experimental frameworks, and then wait—days, weeks, sometimes months—for enough traffic to reach statistical significance. And even when the test runs cleanly, the result is still just A versus B. What if you have a dozen variations worth testing? This prototype promises to let marketers run as many simulated tests as cheaply up front, eliminate the weak options earlier, promote stronger candidates into real-world tests, and save traffic and time for higher-quality experiments.

Project Face Off was named the Summit audience favorite, which means it has a much better chance of being productized in the future.

Project Test Kitchen

Project Test Kitchen reimagines AI image generation as a collaborative, multidimensional design workspace rather than a one-shot prompt box. Created by research intern Yuzhe You, it tackles the "too many cooks" problem head-on—giving multiple designers a seat at the table without the chaos. This prototype combines multiple people's tastes and constraints. It enables exploration of visual directions along clear, controllable axes. The AI becomes a co-creator capable of understanding style, composition, and branding—not just keywords.

Project Tailored Takes

This AI-powered system connects workflows across Adobe Firefly, Workfront, Experience Manager, and Frame.io, making it easier to create highly localized, multi-version video ads. Today, transforming a "master" video into multiple localized spots requires separate shoots— sometimes entirely new productions—for each region. Multiple editing passes are also needed, as well as coordination across agencies and in-house teams. This can be costly, slow, and risky.

Adobe Foundry AI Creative Technologist Jordan Hall developed Project Tailored Takes to have AI do the heavy lifting. It treats videos not as single, finished files but as flexible templates. Shots, product imagery, motion, and narrative structure become modular elements you can recombine and regenerate for different markets, audiences, and channels. The goal: Marketers define what the ad should communicate and where it should run. Then, the AI-powered system handles how it'll be visually and culturally adapted.

Project Page Turner

What if you could use AI to turn your website from a static, one-size-fits-all page into a dynamically assembled, intent-aware experience? That's the idea behind Project Page Turner, created by Adobe's Experience Manager engineering chief Paolo Mottadelli. The aim is to redefine personalization in the ChatGPT era by eliminating the need for a handful of fixed templates, the need for users to hunt and peck across entire websites to find information, and the need for marketers to anticipate every journey. Instead, AI will do it all by assembling, in real time, pages centered on a user's intent.

To learn more about Project Page Turner, read my exclusive interview with Mottadelli.

Project Asset Amplify

Project Asset Amplify lets you turn a single asset into a full marketing ecosystem. With a prompt, you can leverage that artifact to create social media posts, print ads, and a website. And everything is editable within Adobe Photoshop and Express.

The brainchild of software developer Shivangi Aggarwal, it understands the source campaign's visual language, messaging, and intent. It also knows the psychology and preferences of different audiences and demographics (e.g., millennials versus Gen Z, parents vs. performance-focused buyers).

Marketers face a content demand problem—too much needed, not enough capacity to produce it. Hero images, social posts, display ads, YouTube covers: the formats multiply faster than designer and writer bandwidth can keep up. Project Asset Amplify uses AI to turn a single asset into a full family of creative files, scaled across audiences, platforms, and use cases—freeing creative teams to focus on the work that actually requires human judgment.

You can watch every Sneaks presentation from this year now on YouTube. Alternatively, you can browse them individually at adobe.ly/sneaks.

"This deal also signals the next utility phase of the AI economy: infrastructure and foundation model providers moving upstack to acquire the few remaining defensible application layers. Expect a new wave of AI M&A as neoclouds and AI hyperscalers merge with SaaS companies in a move to control both infrastructure and distribution. GPU and inference providers need software reach. Software companies need infrastructure scale. The mergers write themselves."

— WEKA Chief AI Officer, Val Bercovici, on xAI's potential acquisition of Cursor, describing the latter as a rare exception in the AI wrapper bubble.

Disclosure: I attended Adobe Summit as a guest of the company, with my flights and hotel stay paid for. The AI Economy's coverage is editorially independent from those that it covers. These words are my own.

TheColorAPI color conversion, naming, scheming & placeholders

Your fast, modern, swiss army knife for color.

Pass in any valid color and get conversion into any other format, the name of the color, placeholder images and a multitude of schemes.

There are only two endpoints you need to worry about, /id and /scheme, and you can read the docs about both. Each endpoint is available in JSON[P], HTML & SVG format. The SVG format can be saved or used as an img[src] attribute for super-easy embedding/sharing!

Try Josh's favorite, for example, in JSON, HTML or SVG format.

How do I convert/identify a color?

All you really need to do is access the /id endpoint, and pass in a color value as a query string. Read the docs for more details, but all these are valid:

• /id?hex=ffa or /id?hex=00ffa6
• /id?rgb=rgb(255,0,0) or /id?rgb=20,43,55
• Same goes for cmyk, hsl, and hsv formats

Every color object returned by the API:

• Is named (from a matched dataset of over 2000 names+colors) e.g. #24B1E0 == Cerulean
• Has an image URL for demonstration e.g. Cerulean image
• Is transposed into hex, rgb, cmyk, hsl, hsv and XYZ formats
• Is matched to a best-contrast color for text overlay, etc

How do I generate color schemes?

The parameters are generally the same as those necessary for the /id endpoint (supply a color, like above), but here you can also specify a scheme mode to guide the generation.

Scheme modes include monochrome, monochrome-dark, monochrome-light, analogic, complement, analogic-complement, triad and quad.

Every scheme object returned by the API is seeded by the color of your request and can be any length you specify (within limits). It will also include a color object for each constituent color.

Anything else?

If you find this open source API useful, please support the developer!

Create animated websites in record time with Starwind UI

Animated, customizable, and accessible components for Astro - built purely with Astro and vanilla JS. Inspired by shadcn/ui with seamless CLI installation.

See Starwind UI in Action

Key Features

Own The Code

Easy-to-use CLI to add component code directly to your projects.

Customizable

Full control over each component. All styling, functionality, and behavior can be modified.

Accessible

Designed with keyboard navigatable and accessible components.

Open Source

Completely open source and MIT licensed.

What will you create?

Kinet is a branded file delivery platform that allows agencies, freelancers, and professionals to share files through custom-branded portals featuring their own logos, colors, and messaging.

If you feel anxious about AI replacing you, you're not alone. As I've seen AI get more advanced, I've felt that fear myself.

The problem is that many people frame the AI age as a doomsday prophecy for creative professionals. Change is scary, but it isn't the end of your industry or your career.

If you use AI strategically, you can actually generate more demand for your work while retaining your top clients. The key is repositioning yourself from executor to strategic director. You must become someone who knows when to leverage AI and when human expertise is nonnegotiable.

Why Your Clients Are Turning to Generative AI

To increase your client retention rate, you need to understand why people are opting for AI tools over human-generated content. The obvious benefits include speed, cost and convenience. From a company's perspective, switching to AI is a logical decision. It's business, not personal.

The obvious benefits include speed, cost and convenience. From a company's perspective, switching to AI for certain projects makes sense when budgets are tight or deadlines are impossible to meet.

It may surprise you to find out consumers aren't universally opposed to AI-generated content. According to a 2024 YouGov survey, 39% are comfortable with AI generating product images for advertising in place of product photography.

This consumer acceptance gives clients confidence that AI can handle routine visual work. They're not wrong. This technology excels at repetitive, template-based design tasks.

One design agency used AI to create brand illustrations, backgrounds and stickers for a pet insurance provider. It completed the project within 11.5 hours, which would have been virtually impossible without AI. The client approved the artwork immediately, requesting zero revisions.

Read Between the Lines to Fulfill Clients' Needs

When a client chooses AI art over a human-led project, they aren't rejecting your portfolio or minimizing your expertise. This decision signals their immediate priorities, such as time or budget constraints.

Your job as a graphic designer is to identify the need behind the decision and propose a higher-value solution.

For example, if a client tells me I'm charging too much, what they're really saying is "I can do this much cheaper with AI." They know they can't do it in-house, but they don't want to pay my rate because they think a generative model can produce work on the same level.

Still, they started a conversation instead of using AI. Deep down, they know I bring expertise to the table that AI doesn't. They want to negotiate. I can shift the focus from price to value by explaining that my work isn't a cost sink, but an investment in their company's future.

Alternatively, instead of lowering my rate, I can offer to adjust the scope or remove deliverables. To secure their business long-term, I could offer discounts if they agree to a fixed-term contract or a retainer proposal.

What Can You Bring to the Table That AI Can't?

Your job isn't to convince your clients never to use AI, but to explain the value of the human element in the creative process. This is where you start repositioning your design services as hybrid or AI-enabled.

AI can create logos, signage, packaging mockups and posters, but it can't justify its decisions based on years of real-world experience or provide high-level creative guidance. Humans with education, training and hands-on freelance work can provide this expertise.

Going Beyond the Prompt With Strategic Thinking

If clients are undervaluing your expertise, it's because they don't understand the effort that goes into your work. This is a common theme among creative professions because years of practice make skills seem innate.

In reality, it takes years to master design techniques. I've spent countless hours studying color theory, typography hierarchy, and compositional balance. This knowledge informs every pixel I place.

AI can only answer the questions it's asked. A human designer's value lies in asking the right questions and developing a strategy that aligns with business goals, which AI cannot do without someone prompting it.

Graphic designers must consider how color, texture, shape, space, balance, harmony and typography work independently and together. AI may be trained on successful campaigns, but it doesn't really understand graphic design elements.

Take the campaign created for Georgetown Optician, a high-end eyewear retailer, for example. The agency's co-founder and chief creative officer revealed the project required hours of prompting and extensive knowledge of image-making to guide the design in the right direction.

The final result was unique and stunning, but it took a human creative director with deep expertise to achieve it. That's the value proposition. You're the strategist who knows how to extract AI's potential while avoiding its pitfalls.

Understanding Nuances and Reading the Room

The ability to understand context, social cues and complex emotional needs is uniquely human. AI is a "yes-man," meaning it agrees with everything users say because it is designed to please users by affirming their preferences and opinions.

Designers can provide critical pushback and guidance based on hands-on experience. I've told clients their favorite color scheme won't work for their target demographic, or that their requested layout contradicts accessibility standards.

While human and AI art are becoming difficult to distinguish visually, generative models can't reason as you can. Their art is only as good as their prompts. When you design something, every pixel is intentional. You're making hundreds of micro-decisions based on strategic thinking, brand guidelines and user experience principles.

Research shows AI can't accurately replicate human judgment. Even advanced models score just 40.8 out of 100 when simulating human behavior. Unlike an algorithm, you can read the room, understand stakeholder dynamics and navigate the messy human context.

Building Emotional Connections With Storytelling

Authentic brand stories and emotional resonance come from human experience. I can draw on personal memories, cultural knowledge and empathy to create designs that genuinely connect with audiences.

AI tells great stories because it has been trained on more works than I could ever read in my lifetime. However, it simply connects words, whereas I can bring my lived experience and personality to the table. That's the difference between technically competent design and work that truly resonates.

Consumers are skeptical about AI-generated creativity, which backs up this idea. According to Vogue's consumer perception survey, less than 24% agree AI-generated images are as valuable as human-made creative work.

They fear a loss of creativity from brands using AI. You can position yourself as the antidote to that concern. You are the human creative who ensures their brand maintains authenticity and emotional depth.

How to Create a Compelling Value Proposition

You need to distinguish your graphic design services from generative AI's capabilities. Reposition yourself as an innovative creative who is willing to embrace hybrid workflows to elevate your craft.

Frame Yourself as a Business-Savvy Partner

The saying "the customer is always right" doesn't always stand true. Sometimes, clients make requests that contradict best practices or lag behind current trends. Algorithms are yes-men and won't point this out. You can and should position this as one of your core value propositions.

The demand for AI-related work increased by 60% from 2024 to 2025, proving more companies are seeking AI-literate creative partners. While they see the value in AI, they realize this technology can't replace the business acumen you bring. Frame yourself as a business-savvy partner who protects clients from costly mistakes.

I tell prospective clients that I'm not just designing their website or logo. I'm protecting their brand equity and ensuring their visual identity drives business results. That shifts the conversation from "how much does this cost" to "what's the return on investment."

Emphasize Your Role in Brand Consistency

If AI does something really well, that's because it has a lot of training data. This means that the concept has been done many, many times. AI can't create something truly original. You can use this fact to position yourself as a key driver of brand distinction.

Heinz demonstrated this in 2022 when it asked a generative model to create images of ketchup. Regardless of whether it added "synthwave" or "street art" to the prompt, the model consistently created images that resembled Heinz products. If its competitors had tried to create low-cost ad campaigns with AI, they might've inadvertently promoted it.

You also play a vital role in brand consistency, as most models don't have a "memory." While AI can generate endless variations, a human director is needed to ensure all assets are cohesive and serve a singular, consistent brand vision over time.

I've worked with clients who tried using AI to extend their brand assets and ended up with visual chaos. They came back to me because they needed someone who understood their brand guidelines, their evolution and their strategic direction.

Identify and Achieve Your Client's Core Goals

Focus on the skill of translating a vague client request into a tangible business outcome. This is a strategic function.

When a client says they want a "modern, clean website," I dig deeper to understand what business problem they're solving. Are they trying to increase conversions, attract a younger demographic or establish premium positioning?

The ability to ask these questions and align design decisions with measurable business outcomes is what separates strategic designers from order-takers. AI can execute tasks, but it can't conduct discovery.

This strategic approach can save you hours of revisions by ensuring you're solving the right problem from the start. Position this as a value-add that justifies your premium pricing.

Communicating the Value of the Human Element

With all the talk of AI replacing humans, many business owners view AI as a threat to graphic design. Reframe their view from "you versus AI" to "you and AI."

It is a tool you can strategically direct for faster results and better business outcomes. I've started telling clients that AI is like having a junior designer who's incredibly fast but needs constant creative direction.

Clients may feel like they can't broach the topic without offending you, so you should start the conversation proactively. Update your offerings to reflect any new AI-enabled services.

When clients say "you're charging too much" or tell you they plan on making edits with AI after you deliver the finished product, remember to read between the lines. Once you understand what they really want, you can pivot to demonstrate your value.

Update your contract language and terms of service to reflect your new offerings and approach. Be as transparent as possible with clients.

Given that 69% of graphic designers expect to use AI in their work, demand for AI-savvy professionals is growing. By explaining where AI can help and where human expertise is needed, you communicate your value. The market will reward transparency.

You Can Make Working With Generative AI Work

It's the AI age, so more companies are seeking professionals with AI literacy and experience. Clients want designers who understand both the technology and the timeless principles of great design. You can be that person.

I think of this shift like I do mural work, traditional sign painting, calligraphy and pinstriping. These art forms never died out. They've become specialty services, which people pay a premium for.

You don't have to exclusively use AI to succeed. In addition to offering AI-enabled services, offer 100% original designs. This can help you distinguish your services, as some clients will always value the prestige and authenticity of fully human-created work.

The LA 2028 Olympics branding uses a bold, typography-led system to reflect the diversity of Los Angeles, replacing a single fixed logo with a flexible identity built around multiple versions of the letter “A.” This approach draws from the city's street signage and visual culture, making typography the central expression of the brand. By balancing a clear structure with room for variation, the system captures multiple voices while staying cohesive, showing how design can both represent and embody a city's cultural identity.

AI tools have transformed the initial stages of logo design by enabling faster concept generation and visual experimentation, but they cannot replace human strategic thinking and brand understanding.

Google is redesigning its Workspace app icons — including Meet, Calendar, Docs, and Sheets — to address a longstanding complaint that they all look too similar.

Pia Salzer creates expressive, imperfect illustrations inspired by everyday life, using personal work to build her career while exploring emotional and social themes through a distinctive, collage-like style.

Czech National Bank Governor Ales Michl made a direct case for holding bitcoin in central bank reserves, presenting internal CNB analysis showing a 1% BTC allocation could increase expected portfolio returns while keeping overall risk roughly unchanged due to Bitcoin's low correlation with other reserve assets. The CNB has already begun a test portfolio, making it a closely watched case among sovereign institutions and a direct counterpoint to the ECB's position that bitcoin is not liquid, secure, or safe enough for reserves.

Stable Sea has integrated WisdomTree's tokenized Treasury money market fund to offer businesses access to yield-bearing tokenized US Treasuries, leveraging WisdomTree Connect's B2B and B2B2C distribution infrastructure. WisdomTree's tokenized fund, which recently received SEC exemptive relief for 24/7 trading at a stable $1 NAV with instant blockchain settlement, now represents over $772M across multi-chain deployments.

Franklin is an open-source AI agent framework that provisions each agent with a self-generated USDC wallet, replacing API keys and subscriptions with x402 micropayments settled per-call across 55+ models, image generation, web search, and trading data tools. A built-in smart router directs prompts to the cheapest capable model and escalates to frontier models only when required, cutting costs 60-80% compared to always using top-tier models. The project frames subscription-based AI as a transitional structure where heavy users get throttled and light users subsidize them, arguing that pay-per-use wallet-backed agents allow providers to serve full tasks without rationing.

The Machine Payments Protocol (MPP), a payment-method and currency-agnostic standard with 10+ live payment methods spanning stablecoins on every EVM network, cards, and Bitcoin, shipped a major update to its discovery spec that allows servers to declare all payment offers ahead of time in a single discovery document rather than requiring clients to hit endpoints and parse 402 Challenges individually. The update enables services that accept multiple methods, currencies, and intent types on the same endpoint to advertise everything upfront, making it easier for registries and AI agents to find compatible payment options without dynamic probing. It's an infrastructure-level improvement for the machine-to-machine payments stack that sits alongside Coinbase's x402 and Amex's ACE kit.

Both Robinhood and Hyperliquid posted sequential Q1 declines in crypto trading revenue, with Robinhood falling 39.4% QoQ to $134M and Hyperliquid dropping 31.0% QoQ to $179.7M, though Hyperliquid's total exceeded Robinhood's despite being a DeFi protocol. Hyperliquid's RWA revenue surged 454.8% QoQ to over 30% of volumes, and user growth reached 29.6% QoQ to 1.19M against Robinhood's roughly 1.5% funded customer gain. At $192.3M in Q1 protocol income against a $9.5B circulating market cap, HYPE trades at a considerable discount to HOOD's $74B valuation on $346M net income, a gap that is harder to sustain as RWA diversification and HIP-4 binary outcomes reduce HYPE's dependence on pure crypto trading volume.

Three EVM standards are converging into permissionless financial rails for AI agents: x402 (HTTP 402-based micropayments with 50M+ transactions since its May 2025 launch), ERC-8004 (on-chain identity and reputation with roughly 98K agent registrations across 10+ chains), and ERC-8183 (a job/escrow primitive co-developed by Virtuals and the Ethereum Foundation dAI team). Base handles the majority of x402 volume, and ERC-8211 extends the stack with dynamic execution for multi-step DeFi strategies. TradFi infrastructure is targeting the same problem from the off-chain side, with Visa's agent card CLI and Stripe/Tempo's Machine Payments Protocol backed by Anthropic, OpenAI, DoorDash, and Shopify as launch partners, while unresolved challenges around prompt injection, RPC data leakage, and on-chain transaction visibility remain active risk vectors for the space.

Regan Bozman (Lattice Fund) responds to skepticism about stablecoins reducing remittance costs to zero by reframing the problem: on-ramp/off-ramp for local stablecoins will be fast and free for most Western G10 countries within 1–2 years, but the biggest unlock is changing consumer behavior to holding USD and only converting to local currency at point of spend rather than at receipt. The thread responds to the valid criticism that stablecoins only solve the money-movement leg (which is already cheap) while the last-mile conversion and cash-out remain expensive, arguing that the behavioral and infrastructure shift is coming but is the real constraint, not the rails.

Prediction markets, despite being marketed as accessible income opportunities, funnel the majority of profits to high-frequency traders and sophisticated market makers at the expense of retail participants. The sector generated roughly $51B in volume in 2025 and is tracking toward $240B in 2026, but the skewed profit distribution mirrors dynamics seen in traditional equity markets where HFT firms capture outsized returns from retail order flow.

Prediction market monthly volume has grown from $1.2B in early 2025 to over $20B.

Polymarket's website traffic in Q1 exceeded that of Coinbase, Hyperliquid, Pump Fun, and Uniswap combined, making it arguably the first crypto application to achieve genuine mainstream adoption beyond the crypto-native user base.

Pump.fun burned approximately $370M in $PUMP tokens, eliminating roughly 36% of circulating supply, and launched a programmatic buyback-and-burn program that allocates 50% of protocol revenue toward continued burns for the next year.

Bernstein reaffirmed an "outperform" rating on Robinhood with a $130 price target after the Q1 earnings miss, projecting 2026 EPS of $2.65 (23% above consensus).