The Real Cost of Running an LLM App in Production

Most LLM bills are three to five times larger than they need to be, and almost none of the excess comes from the model choice everyone argues about. It comes from resending the same context on every request, generating output tokens nobody reads, and running interactive-priced traffic through a workload that could have waited overnight. Here is the arithmetic, with published prices as of 3 September 2026.

How you are actually billed

Every major provider prices in tokens, and every major provider charges more for output than input, usually five to six times more. That asymmetry is the single most important fact about your bill, because output length is the variable you control most directly and audit least often.

Four mechanics beyond the headline rate:

• Cached input is a separate, much cheaper line. Repeated prefixes are billed at a fraction of the base input rate.
• Reasoning tokens bill as output. A model that thinks for 3,000 tokens before writing a 200-token answer costs you 3,200 output tokens.
• Long context can carry a surcharge. OpenAI’s pricing page lists separate short- and long-context rates: gpt-5.6-sol at $2.00 input short-context versus $4.00 long, with output at $10.00 and $15.00 respectively.
• Service tiers move the multiplier. The same page lists a “fast mode” at four times standard pricing, and a 10% uplift for regional data-residency processing on eligible models released after 5 March 2026.

Published list prices, September 2026

Read the footnotes on any of these before you build a financial model. Google’s Gemini pricing page states that the 3.8 Flash rates above hold “through December 31, 2026,” doubling to $1.50 input and $7.50 output on 1 January 2027. OpenAI notes that gpt-5.6-sol’s promotional pricing runs at least through 21 November 2026. Anthropic’s pricing documentation lists no comparable expiry on the models above. A 100x spread between the cheapest and most expensive text model on this table is the real story: model selection is a bigger lever than any optimisation.

The two discounts worth engineering for

Prompt caching

Anthropic’s caching documentation spells out the mechanics precisely: a five-minute cache write costs 1.25x the base input rate, a one-hour write costs 2x, and a cache read costs 0.1x. Break-even is therefore one read for the five-minute cache and two reads for the one-hour cache. Minimum cacheable prefix length varies by model (512 tokens for Opus 5, 1,024 for Sonnet 5, 4,096 for Haiku 4.5), and you get up to four cache breakpoints per request, with the TTL clock starting when the request starts rather than when it finishes.

Two design consequences. Put your stable content first: system prompt, tool definitions, retrieved reference documents. And keep it byte-identical, because a timestamp injected into your system prompt is a cache miss on every single call.

Batch processing

All three major providers discount asynchronous work by 50%. OpenAI’s Batch API has a 24-hour turnaround window, uses a separate rate-limit pool so it does not eat your interactive quota, and allows up to 50,000 requests and a 200MB input file per batch. Anthropic and Google both apply the same 50% reduction on input and output. Anything that does not have a user waiting on it, such as evaluation runs, backfills, embedding a corpus, nightly classification and enrichment, should be running here.

A worked monthly bill

Take a B2B support assistant: 40,000 conversations a month. Each request sends a 10,000-token stable prefix (system prompt, tool schemas, product documentation), 2,000 tokens of fresh conversation, and generates 500 tokens. Priced on Claude Sonnet 5.

1. Naive. Input is 40,000 x 12,000 = 480 MTok at $2.00 = $960. Output is 20 MTok at $10.00 = $200. Total $1,160.
2. With prompt caching at a 90% hit rate on the prefix. Cache reads: 360 MTok at $0.20 = $72. Cache writes: 40 MTok at $2.50 = $100. Uncached input: 80 MTok at $2.00 = $160. Output unchanged at $200. Total $532, a 54% reduction from one architectural change.
3. Now add the nightly job. Classifying 200,000 documents at 1,500 input and 100 output tokens each is 300 MTok in and 20 MTok out. On gpt-5.6-luna that is $30 + $12 = $42 at standard rates, or $21 through the Batch API.
4. Running that same job on the flagship model instead would cost $600 input plus $200 output at gpt-5.6-sol rates, nineteen times more than the batched cheap model, for a classification task where the cheap model is very likely sufficient.

The whole app lands around $553 a month. The version of this app that a team builds without thinking about caching, model routing or batching lands closer to $2,000.

Self-hosting: the arithmetic, honestly

Take the best public numbers available. Lambda’s on-demand GPU pricing lists NVIDIA B200 SXM6 at $6.69 per GPU-hour, so an eight-GPU node costs $53.52 an hour, or roughly $39,000 a month at full-time on-demand rates. Lambda’s MLPerf Inference v5.1 submission on 8x HGX B200 reported 99,993.90 tokens per second for Llama 2 70B in the latency-constrained Server scenario, and 102,725 in Offline.

At the Server figure, that node generates about 360 million tokens an hour, or roughly 263 billion a month at continuous full load, which works out to about $0.15 per million output tokens. Against $10 per million for Sonnet 5, self-hosting looks like a 65-fold saving.

It usually is not, for four reasons. MLPerf is a saturated benchmark harness with an optimised serving stack; real traffic is bursty, and you pay for the idle hours. Llama 2 70B is not equivalent to a 2026 frontier model, so you are not comparing like with like on quality. You need at least one engineer who can debug a serving stack at 3am, which costs more than the GPUs at small scale. And the crossover is high: matching that node’s $39,000 monthly cost through an API at a blended $4 per million tokens takes roughly 9.8 billion tokens a month. Below that, the API is cheaper before you count a single hour of engineering time.

Levers, ranked by what they actually save

1. Route by task, not by default. The 100x spread across the price table dwarfs everything else here. Classification, extraction, routing and summarising short text rarely need a flagship model.
2. Cap output tokens and use structured outputs. Output is five to six times the price of input, and a JSON schema is shorter than prose.
3. Cache the prefix. Worth 54% in the example above, and it also cuts time to first token.
4. Batch everything without a user waiting. A flat 50%.
5. Control reasoning effort. Thinking tokens bill as output; most production tasks do not need extended reasoning.
6. Trim retrieved context. Eight passages instead of twenty is usually more accurate as well as cheaper.
7. Watch the tool prices. OpenAI lists hosted web search at $10.00 per 1,000 calls plus search content tokens billed at model rates, and file search at $2.50 per 1,000 calls plus $0.10 per GB per day of storage. These do not appear in per-token dashboards.

Instrument this before you optimise anything

You cannot manage a token bill you cannot attribute. Log, per request: model, input tokens, cached-read tokens, cache-write tokens, output tokens, reasoning tokens, and the feature or customer that triggered it. Then compute cost per conversation and cost per resolved ticket, not cost per million tokens. The per-million figure is the vendor’s unit, not your business’s. In practice a handful of endpoints and a handful of enterprise customers generate the large majority of spend, and you will not find them by staring at a provider dashboard. Prices on this page were current on 3 September 2026 and several carry published expiry dates, so re-check the vendor pages before you commit them to a board deck.

Sources

• OpenAI — API pricing
• Anthropic — Claude API pricing
• Google — Gemini API pricing
• Anthropic — Prompt caching documentation
• OpenAI — Batch API guide
• Lambda — GPU cloud on-demand pricing
• Lambda — MLPerf Inference v5.1 results on HGX B200