Back to Devoured - November 04, 2026
DEVOURED
How much do tokens really cost?

How much do tokens really cost?

AI llmenterprise Substack
Current AI token pricing is likely sustainable, as fierce competition among infrastructure providers balances costs despite significant upfront training investments.
What: Hadley Wickham analyzed token pricing for the Kimi K3 model across various providers, concluding that current commercial API rates, typically around $1.13 per million tokens, align reasonably with the hardware costs of self-hosting.
Why it matters: While individual developer subscription plans may be heavily subsidized by VCs, the per-token pricing for enterprises reflects a competitive, maturing market rather than a temporary marketing tactic.
Deep dive
  • Kimi K3 serves as a cost-effective alternative to larger frontier models like Opus 5.5.
  • Competitive pricing from providers like BaseTen, Fireworks, and Together suggests a market-driven equilibrium.
  • The capital expenditure (CAPEX) for self-hosting high-end inference hardware, such as 8x GB300 servers, remains significant at approximately $550,000.
  • Operational expenses, including colocation, power, and bandwidth, bring the total cost of ownership to roughly $23,000 per month for a high-capacity cluster.
  • High utilization (75%) allows for a per-token cost of ~$0.25/million for self-hosting, which is lower but within the same order of magnitude as commercial APIs.
  • Training costs remain the primary unrecovered expense, currently borne by investors rather than reflected in inference margins.
Decoder
  • vLLM: A high-throughput, memory-efficient library for serving and running large language models.
  • Colocation: A data center facility where companies rent space for their own servers, benefiting from high-end power and cooling infrastructure.
Original article

How much do tokens really cost?

I hear a lot of suspicion that AI tokens are unsustainably priced and that frontier labs are heavily subsidising costs with their VC money. They want to get us hooked now, so they can pull the rug out from under us whenever they want to make a bunch of money. Is that true? My sense is that it’s mostly not. But can I back that up with some data? That’s my goal with this post.

There are many open models available, but here I’m going to focus on Kimi K3. I’ve been using it heavily for the past couple of months, and interestingly, I’ve continued to use it even after Opus 5.5 was released. Opus 5.5 definitely feels a bit smarter than Kimi K3 to me, but I still find myself drawn to Kimi K3 for its cost effectiveness. I want to talk more about this in a future post, because it connects to one of my favourite Posit Assistant features: the cost display. This is a powerful tool for building up your sense of where models spend money and how much a given task should cost.

The Posit Assistant chat box, showing the running cost of the current conversation (34K tokens, $0.94) alongside the model selector set to Kimi K3.

So Kimi K3 is a reasonable proxy for a solid frontier model: maybe it’s not quite as good, but it’s pretty darn close. How much does it cost?

OpenRouter pricing

Because it’s an open model, you can buy Kimi K3 from a range of providers. Here’s the current pricing from OpenRouter. I’ve included a blended column which uses typical ratios for agentic coding: 20:1 input to output and 95% of input from cache. We’ll need that later when we compare costs to self-hosting.

#>  Provider       Input /M Output /M Cache /M Blended /M
#>  Wafer          0.28     9.60      0.13     0.59      
#>  Sail Research  0.22     10.72     0.11     0.62      
#>  Relace         0.21     9.33      0.21     0.64      
#>  Makora         0.36     12.00     0.12     0.70      
#>  Chutes         1.85     9.50      0.18     0.71      
#>  Decart         1.95     9.75      0.20     0.73      
#>  ParetoAlpha    1.00     11.25     0.23     0.79      
#>  Fireworks      2.10     10.50     0.21     0.79      
#>  Phala          2.55     12.75     0.26     0.96      
#>  DigitalOcean   2.55     12.95     0.26     0.97      
#>  InferenceNet   0.23     16.50     0.20     0.97      
#>  Morph          1.42     12.00     0.40     1.00      
#>  DeepInfra      2.85     14.25     0.28     1.07      
#>  Amazon Bedrock 3.00     15.00     0.30     1.13      
#>  Parasail       3.00     15.00     0.30     1.13      
#>  Modal          3.00     15.00     0.30     1.13      
#>  Together       3.00     15.00     0.30     1.13      
#>  BaseTen        3.00     15.00     0.30     1.13      
#>  Moonshot AI    3.00     15.00     0.30     1.13      
#>  Alibaba        3.45     17.25     0.34     1.30      
#>  AkashML        1.20     14.00     1.20     1.81

You’ll notice that there are a lot of options! And I bet most purchasing decisions are driven by cost, which means there’s a lot of pressure on these providers to compete on price.

What stops some sort of subsidy pushing prices unsustainably low? Well, I bet some of the cheapest providers are spending their VC cash to try and get customers. But there are also six providers all selling around the same price at the high end. One of those providers is BaseTen who we work with to provide models for Posit AI Pass. I can personally vouch for the quality of this service; it’s not the cheapest but our experience with them has been great.

That’s buying Kimi K3 “off the rack”. What if we tried to host it ourselves?

Self-hosting

I’m no expert on local models, so I used Claude to help figure out the costs. I’ve carefully checked individual facts as I wrote this article, but it’s possible Claude steered me in completely the wrong direction and there’s a deeper flaw that I missed. But let’s have a go.

We need to start by buying a computer that can host Kimi K3. How much will that cost? Moonshot AI (the authors of Kimi K3) provide a vLLM recipe that recommends “at least 8x GB300”. One possible option for that is a Supermicro 8U GPU SuperServer, which costs a whopping $550,000! I don’t have half a million dollars lying around, so we’ll need to borrow the money for it. If we assume an 8% interest rate (which is maybe optimistic) for 3 years, that’s going to cost us around $17,000 a month.

You could instead rent the 8 GPUs for around $4/hour each, which adds up to around the same amount when you include the cost of hosting (discussed next). The advantage of borrowing the money and buying it is that you could sell it at the end of the three years, netting you maybe $150,000. The advantage of renting is that you can stop at any time.

Now we need to put that server somewhere. That’s called a server colocation facility, and while pricing information isn’t generally available on the internet, Claude claims it will cost around $3,750 / month, with another $600 for bandwidth, and ~$700 for power. You might consider keeping the server at your house, but it produces a tremendous amount of heat, and you’ll need to buy a dedicated A/C unit for it.

There are a few other smaller bits and pieces that will bring the total monthly cost to around $23,000.

How many tokens can we serve in a month? SemiAnalysis suggests ~6000 tokens/s per GPU, so 48,000 tokens/second, which is 124 billion tokens/month. If you assume we get 75% utilisation (reasonable for an API provider), that’s $0.25 / million tokens. If you assume typical ratios for agentic coding (20:1 input to output, 95% of input from cache), the equivalent API cost (for BaseTen and friends) would be $1.13 per million tokens. Assuming we have high capacity, it seems like the cost of running the server is significantly cheaper than what providers are charging.

Obviously this is a crude analysis, but it’s a reasonable top-level bound on the cost of serving tokens. A real business would benefit from economies of scale, and the ability to negotiate with vendors, but would also have to pay staff.

Subsidies

So I think I’m convinced that token prices are reasonably accurate measurements of the underlying cost. There’s a lot of competition pushing the cost down, but there still seems to be room for a healthy profit margin based on the underlying costs.

That’s not to say subsidies aren’t still happening! I just don’t think they’re on per-token API costs. Instead, they’re happening on subscription plans. A recent SemiAnalysis post suggests that a $200 / month plan gets you somewhere between $2,000 and $10,000 worth of tokens. These plans are only available to individuals and small businesses, not enterprises, which have to pay per-token prices.

It’s also worth noting that my analysis ignores the cost of model training which might have cost several hundred million dollars. Currently that cost is paid by the investors in Moonshot AI (including the Chinese government). You could argue that this is also a subsidy, which is fair. My hope is that more governments decide that training open models is a societal good (compared to having frontier models owned exclusively by private industry) and decide to fund it directly.

What do you think? Did you find this analysis convincing? Do you have other evidence that might suggest how much tokens really cost?

Explore all 53 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi