Inference cost at scale with napkin math

If you serve AI models as a part of your product's stack (hard not to in 2026), you've done the math on how much juice you can get out of a single A100/H100/H200/B200/whatever. This directly affects the pricing for subscription-based products.

I want to show that even as models, hardware, and inference engines evolve; the dollar price-per-user remains straightforward to work out on paper. This exercise should also reveal how various optimizations in inference engines help SaaS products remain profitable.

If you were actually working this out on paper, you'd need only the following information:

1. GPU hardware specs: Memory bandwidth and peak throughput (explained below).
2. Context length: assumed 200k tokens.
3. Active parameter count of the model: Assumed 32B to keep things simple on a single GPU.
4. Some idea about your product: Whether it's driven by user prompts or programmed loops, duty cycle of your user profile (explained at the end), etc.

The specifics of the model architecture matter surprisingly little, unless it's something entirely different like diffusion.

Resources on a single GPU

For any GPU on the market, you can find on its spec sheet two key metrics:

1. Peak throughput: Number of floating-point operations executed per second. Usually in TeraFLOPs (1 TFLOP/s = 10^9 ops/sec).
2. Memory bandwidth: Amount of data that can be moved from global memory (VRAM) to registers (SRAM). Usually in TB/sec.

We'll assume FP-8 quantization to compute throughput, though it's easy to adjust the math for FP-16 as well.

Cost of a Matrix Multiplication

If you bothered to click on this article, you know that AI models do many matrix multiplications on massive matrices. That we start by finding the cost of a matmul should be no surprise then.

Assume two matrices: A_{N \times d} and B_{d \times M}. Let their product be the matrix O_{N \times M}. From high school algebra, we know that each element of O can be computed as:

O^{i,k} = \sum_{j=1}^{d} A^{i,j} * {B}^{j,k}

In this, we find our first insight into the "cost" of a matrix multiplication. For each O^{i,k}, we need to start with an initial value of 0 and:

1. Load A^{i,j} from memory.
2. Load (B^{j,k}) from memory.
3. Multiply them together.
4. Add the result of #3 to the cumulative sum so far.

And this is done a total of d times per item. So, the cost of a (N,d)*(d,M) matrix product is 2NMd memory accesses and 2NMd floating-point operations.

With an optimization called tiling, the memory access goes down to about d(N+M). The details aren't necessary to proceed.

An Overview of Language Models

At their core, LLMs are simple – they receive a sequence of N words and generate the N+1th. Each word is represented as a vector with d entries. Using repeated applications of a function called "attention" (explained later), they predict the next word.

A single forward pass roughly looks like this:

y = input() # y = matrix of size N x d
for each layer in the network:
  y = attention(y)

# Convert the final layer's output to word-probs.
# W_vocab = matrix of size d x vocab_len,
# and vocab_len is the number of all words
# in the model's vocabulary.
token_probs = softmax(y * W_vocab)
next_tok    = token_probs(argmax(token_probs))
# next_tok is a (1 x d) vector

This is also why LLMs are called auto-regressive. They can keep doing multiple forward passes over their own output until a <stop> token is generated.

Attention in Greater Detail

Let's place the attention function under a magnifying glass.

As you saw, the input is a matrix X \in \mathbb{R}^{N \times d}, and X_i is a single d dimensional vector. For every "layer" in the network, the model stores matrices W_Q,W_K, W_V \in \mathbb{R}^{d \times d}, and computes "attention" as follows:

Q = X.W_Q, K = X.W_K and V = X.W_v

Attention(Q,K,V) = softmax(Q.K^T/\sqrt{d}).V

Or, in python:

def attention(X, W_q, W_k, W_v):
    Q,K,V = X @ W_q, X @ W_k, X @ W_v
    Q_KT = Q @ K.transpose(2,1)
    return softmax(Q_KT / sqrt(d_model)) @ V

Where @ is the dot-product of two matrices.

In reality, multiple LLM conversations are processed in parallel. So inference is batched—where we process B chats concurrently. This means our input sequence X \in \mathbb{R}_{B \times N \times d}.

For a batch size of B, we get:

1. Floating-point operations: 2BNd^{2}.
2. Memory accesses: Bd(N+d).

Assume N to be roughly 200k, and d to be 8192. Meaning that to generate one token for a single user, we need 26 trillion floating-point ops and 1.7 billion memory accesses. This is with the tiled matmuls.

Reducing Compute with KV-Cache

The intermediate output on every chat, namely K and V, is cached at every layer, and stored in a region of VRAM called the KV Cache.

For our napkin math, the existence of KV-cache allows one simplification: for every forward pass, we get to process only the most recently generated word, rather than the entire history. i.e., instead of processing a X \in R_{N \times d}, we get X \in R_{1 \times d} (the most recent token).

For a batch size of B, we get:

1. ~26.2 million memory accesses
2. ~52.4 million ops

How much does a token cost?

Let's take the NVIDIA B200 as our leading example. It has a memory bandwidth of 8 TB/s and a compute intensity of 4500 TFLOP/s.

A Blackwell class GPU can crunch bytes 562 times faster than it can load them. We're doing 2*B operations per memory access. So, how many users should we serve to fully exhaust a B200's compute and bandwidth budget?

2B = 562 \implies B = 331

With a single NVIDIA B200 GPU, we should be serving 331 users concurrently to get the most out of our investment.

How many users can you serve realistically?

We'll assume a 32B dense model. This is 32GB in VRAM. Let's assume a context window of N=200k tokens. For each layer, we need to store 2Nd bytes for a pair of K and V matrices. With a model of d=8192 and L=64, that is 210 GB of VRAM—more than the GPU has.

Using Grouped-Query-Attention (GQA) cuts down the KV cache size by about 8x. With GQA, our KV-cache is now at ~26GB per chat sequence. We're already using 32GB for weights, so how many concurrent chat contexts can we store in the remaining 160GB? That's 160/26 = 6.

Optimizing for hundreds of users on a GPU

Most contexts will never reach the 200k token limit. We can split the KV-cache into chunks, and then allocate those chunks to different users as their token use increases. Conversation threads that are abandoned/cold can be flushed out of the cache. Depending on the median user activity, you can serve anywhere between 40-60 users per Blackwell chip.

Remember that the nature of your product matters too. In most ChatGPT-style apps, the user spends more time reading than prompting. For a median chat session, a user will likely have 80% idle time. So realistically, one chip can serve ~300-800 users comfortably depending on the style of app.

Tokens Per Second

Earlier, we saw that we can comfortably support 6 users at 100% duty cycle. Every 24ms, we generate B=6 tokens. For 1s, we generate roughly 250 tokens for 6 users, or about 40 tokens per user per second. 40 tokens per second is beyond most people's reading speed.

Dollar cost per user

This largely depends on whether you own or rent your hardware. At $40,000 per B200, serving 500 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill.

If you rent the GPU, the cost is more straightforward. At $3 per hour for a B200, your cost per user per hour is 3/num_users. For num_users=500, you get a cost of about $0.006 per user per hour, or $4.32 per month. As long as you charge them more than $4.32, your operating costs are covered.