AI Inference Cost: Why the Per-Token Price Is the Least Important Number

AI inference cost is the total spend required to run a model in production, and it is the product of three numbers: the price per token, the number of tokens each task consumes, and the number of tasks you run. Providers publish the first number. Your architecture sets the second. Most teams negotiate the sticker price and never measure the count, and the count is where the money goes.

The ordering matters because token cost is a cost of goods sold, the case we made in our pillar on AI token economics. COGS you don't measure is margin you can't defend. And the count side of the multiplication is dominated by tokens nobody asked for: resent conversation history, duplicated instructions, tool definitions shipped to requests that never call them, and output that runs long because nothing told it to stop.

#The Sticker Price Is a Decoy

Cost per million tokens is the number everyone compares, and it hides more than it shows. Output tokens typically price three to five times higher than input tokens, because generating text takes more compute than reading it (Source: Microsoft Mechanics). You are billed on every token processed, not on what the user typed. The prompt the user sees is often a minority of what the model reads.

So the real unit is cost per task: price per token, times tokens per task. A team that migrates to a model 30% cheaper per million tokens while unmanaged context doubles tokens per task has made the bill worse and called it optimization. The price lever is public and small. The count lever is private, architectural, and much larger.

#Token Bloat Is Waste, in the Classical Sense

Our founder is a Lean Six Sigma Master Black Belt who has spent thousands of hours operating AI systems. Through that lens, an inflated inference bill is not a pricing problem. It is waste in the strict manufacturing sense: resources consumed without creating value anyone would pay for. The classical waste categories map onto token spend with almost no translation required.

The best public evidence is Microsoft's July 2026 Mechanics demo on tokenomics, where April Gittens walks through real workloads with the token counts on screen (Source: Microsoft Mechanics). Every number below comes from that demo. Read them as one team's measurements, not universal constants. The mechanisms are universal.

Inventory: context accumulating across turns. Language models are stateless, so conversation continuity means resending the full history with every request. In the demo, a trivial three-turn exchange grew from 34 prompt tokens to 52 to 71, even as the user's messages got shorter. Tool outputs, retrieved documents, and hidden metadata all count as input too. This is work-in-progress piling up between stations, carried at full cost. Summarizing the prior turns instead of resending them cut a comparable conversation to 118 tokens by turn three.

Transport: shipping full context when a summary would do. The demo found roughly 300 tokens of system-prompt instructions duplicating what the base model already does, resent on every turn, which Microsoft put at thousands of dollars per billing cycle at application scale (Source: Microsoft Mechanics). Tool definitions are heavier freight: give an agent 30 tools and every description ships as input on every request, whether the task needs one tool or none. The demo agent without dynamic tool routing consumed almost 4,700 tokens per request. With a router, 467. A 90% reduction, no quality trade-off. Caching is the other transport fix: a cache hit bills at roughly 10% of normal input rates, though the prefix has to match exactly.

Overproduction: output nobody reads. Unbounded generation is the most expensive waste per unit, because output carries the higher price multiplier. The same prompt in the demo consumed 675 tokens unrestricted and 93 tokens with a 200-token completion cap and matching instructions. The pairing matters. A cap without instructions stops responses mid-sentence, which trades a cost problem for a quality problem.

Over-processing: a frontier model doing a small-model task. In the demo's side-by-side, the mini model ran 72% cheaper per token than its larger sibling, with a measured quality gap of 0.67 against 0.81. Routing by task complexity, frontier models for hard prompts and small models for routine ones, is the fix. Think of it like machining a bracket on a five-axis mill when a drill press would do.

Defects: retries and reruns. "Cheaper models sometimes require a few retries to get what you want," as the demo puts it, which is why the lowest per-token price is not always the lowest total cost (Source: Microsoft Mechanics). A failed generation is a defect, and defects in token systems cost you twice: once for the bad output, again for the rework. Agent loops compound this, because a retry storm has no natural ceiling unless you build one.

#Count First, Then Price

Notice what the big demo numbers have in common. The 90% tool-routing cut, the 86% output cut, the context summarization cut: all reductions in token count, at zero quality cost. The one price lever in the demo, dropping to the mini model, was the only lever that came with a measured quality trade-off.

That is the general shape of inference cost optimization. Count levers are bigger and entirely under your control. Price levers are bounded by whatever the provider charges everyone else. So the sequence is: measure tokens per task by workflow, split input from output from cached, find which waste category dominates, and only then talk about model pricing. Most teams run this backwards, because the price is printed on the invoice and the count is not.

The instrumentation is not exotic. Every model response already returns input, output, and cached token counts per request. Tag each request with the workflow it belongs to, aggregate weekly, and you have a token bill of materials: which workflow consumes what, and which token type dominates it. That one report turns an undifferentiated invoice into a ranked fix list, and it usually surprises whoever reads it first.

#Common Mistakes in Managing LLM Serving Cost

We should be plain about standing. We have not run AI cost engagements, and these patterns come from operating our own AI systems daily, not from client work. The mechanics above still make the failure modes predictable.

Negotiating the price while ignoring the count. Enterprise discounts on per-token rates feel like progress and cap out fast. A 20% discount on a workload whose tokens per task doubled last quarter is a worse bill wearing a better rate.

Capping output without matching instructions. The demo showed why: restriction without instruction stops responses mid-sentence (Source: Microsoft Mechanics). The cap is a budget. The instructions are how the model learns to live inside it.

Caching without measuring the hit rate. Cache pricing only pays off on hits, and a hit requires an exact prefix match. Teams flag everything as cacheable, match on almost nothing, and pay the higher write price for the privilege.

Giving every agent every tool. Tool definitions are per-request freight. A 30-tool agent pays for all 30 descriptions on a request that calls none of them. Route tools the way you route models: by what the task actually needs.

Treating retries as free. Every rerun consumes the full token load of the task again and shows up nowhere, because dashboards report requests, not first-pass yield. If you track one quality metric on a generation workload, track how often output gets regenerated.

#Frequently Asked Questions

#What is AI inference cost?

AI inference cost is the spend required to run a trained model in production: tokens processed per task, multiplied by per-token prices, multiplied by task volume. It excludes training cost. For an AI product it behaves as a cost of goods sold, scaling with every unit of usage rather than sitting in a fixed infrastructure line.

#What does cost per million tokens actually tell you?

It tells you the price of the unit, not the size of the order. Providers quote input and output rates per million tokens, with output typically three to five times higher (Source: Microsoft Mechanics). Comparing models on this number alone ignores tokens per task, which the Microsoft demo showed varying tenfold on identical requests depending on architecture.

#Why does inference cost grow faster than usage?

Because tokens per task grow on their own. Context accumulates across turns, agents add steps and tool calls, and retrieval appends documents to every request. A user base that doubles while tokens per task also doubles produces a 4x bill. Flat usage with growing context still produces a rising bill.

#Is a cheaper model always the cheaper option?

No. Cheaper models sometimes need retries to reach acceptable quality, and each retry consumes the full token load of the task again. Microsoft's demo lands on the same point: the lowest per-token price is not always the lowest total cost (Source: Microsoft Mechanics). Compare models on cost per successful task, not cost per token.

#What should you measure before optimizing inference cost?

Tokens per task, by workflow, split into input, output, and cached. Then first-pass yield: how often a generation is accepted without rework. Those two measurements tell you which waste dominates, and the fix for accumulating context is different from the fix for unbounded output or retry loops.

#The Count Is the Business

The per-token price will keep falling, and it will keep getting the headlines. The count will keep rising quietly underneath, because near every product decision, longer context, more tools, more agent steps, pushes it up. Teams that measure tokens per task will watch their unit economics improve as prices drop. Teams that don't will wonder why cheaper tokens somehow produced a bigger bill.

If you're building an AI product and want to know where your inference spend actually goes, book a discovery call for a token cost assessment. We'll walk through your tokens per task, your usage distribution, and which of the five wastes is eating your margin, and tell you whether we're the right fit. Sometimes we're not. We'll tell you that too.