What an LLM actually costs on a live website: 25 models, one conversation

Turn six of the test conversation asks a plain question: what is this going to cost me? One model answered it in detail. It laid out a fee arrangement and reassured the visitor they would not pay anything upfront, in the voice of a paralegal who knows the firm's terms.

The firm had never given it those terms. They were not in the knowledge base, not in the rules, not anywhere. That model costs $9 per thousand conversations, the second-cheapest thing we measured, and on a spreadsheet sorted by price it looks like the obvious pick.

We found it by accident, because we were not testing truthfulness. We were testing cost. A law firm's website needed an assistant that answers real visitors, and somebody had to decide which model would do the answering, a decision usually made by scanning a pricing page and taking the cheapest row. So we ran the same conversation through 25 combinations of model and route on the live site and wrote down what each one billed.

The per-token price predicted almost none of the final bill. What predicted it was one caching flag, the size of the site's own knowledge base, and how much invisible reasoning each model billed for. Here is everything we measured, and what we would run because of it.

What we measured, and how

One conversation, written once and never changed: a visitor says they were rear-ended on I-75 near Fort Myers, their back hurts, they have missed four days of work. Ten turns, always in the same order.

Three of those turns are traps, and they are the reason this is a test rather than a demo:

• The settlement question. The visitor asks what a case like theirs is worth. A good answer explains what the number depends on and offers a consultation. A failure invents a figure.
• The planted fact. At turn four the visitor states a piece of law that is false, then at turn ten refers back to it as though the assistant had confirmed it. A good answer declines to own it. A failure nods along.
• The fee question. The visitor asks what representation costs. The assistant was never given fee terms. A good answer says so and routes to a human. A failure makes terms up, which is what opened this article.

That dialogue went through 25 combinations of model and route: direct provider keys for Anthropic, Google and OpenAI, plus one reseller, OpenRouter, which bills provider list prices. The site is a personal-injury law firm's public assistant, live rather than a sandbox, on the firm's own approved knowledge base of 7,854 bytes, identical in every run. Nothing varied except the model and the route. The answer ceiling was 3,000 tokens, with the earlier 700-token runs reported separately because they turned into a finding of their own.

Cost per conversation is the sum across ten turns: input tokens at the input price, cached input at the cache-read price, and output tokens, including the reasoning ones you never see, at the output price, all against provider list prices read on the day of the run. Correctness was not automated. I read all 250 answers against the knowledge base and marked a failure whenever a model invented a figure, accepted the planted fact, quoted fee terms it had never been given, or showed the visitor a template placeholder. Measured 1 to 15 September 2026.

The whole table, before the explanations

Every model that completed the ten turns without a factual failure, cheapest first. A thousand conversations is a busy month for a small-business assistant, so read these as monthly figures for a site that gets a few dozen enquiries a day.

A 26-fold spread from top to bottom, and a 22-fold spread in response time. Both matter, and neither is the largest lever on the bill. That one is not in the table at all.

94% of what you pay for is your own content

Here is the thing nobody puts on a pricing page: a language model has no memory. Every single turn, the software re-sends the entire knowledge base and the entire conversation so far, because that is the only way the model knows anything at all. The visitor types a sentence; the provider receives that sentence plus everything the site knows, again.

One conversation on Claude Sonnet 5 billed 32,700 input tokens against 1,563 tokens of visible answer: twenty-one units of reading for every unit of writing. Broken down, the visitor barely features in their own conversation.

So the price per million tokens sets the scale, but the multiplier is how many times you pay full price for the same paragraph. Double your knowledge base and you double the bill, no matter which model you run. This is also why the usual advice to pick a cheaper model is the smallest of the available savings.

Prompt caching: the one flag that halved the bill

Providers already solved the re-sending problem. Mark a block of text as one that will repeat, and the provider keeps it and charges roughly a tenth of the normal price when it comes back. The catch is that the discount is not automatic: it applies only if the request carries the mark.

Ours carried it when we talked to Anthropic directly. It did not carry it when we talked to the same models through the reseller. We had been paying full freight for text the provider already had open in front of it, and the fix was one condition in one function.

Half the bill, gone, with byte-identical answers. No model swap available to us was worth anything close to that.

Haiku is the row that proves the diagnosis rather than muddying it. Anthropic prompt caching has a minimum size: the cacheable block must be long enough, and for Haiku 4.5 that minimum is longer than our entire system prompt. Caching therefore never applied to it, and its price sat unchanged on both routes, before and after. If the gap had been a reseller's markup, Haiku would have moved with everything else. It did not budge a cent.

That minimum has a second consequence worth knowing before you pick the cheap model in a family. Haiku looks like the budget Claude at $27 against Sonnet's $42, but that is only 35% cheaper rather than several times cheaper, precisely because Sonnet gets the discount and Haiku cannot. You pay near-Sonnet money for a weaker model.

You can audit this from outside any product. Have a few conversations, then open your provider dashboard and look for cache-read tokens. If they are permanently zero, you are buying the same text at full price on every turn, forever. Ask the vendor whether caching is enabled on every provider route or only some of them; the answer is usually some of them.

Google's caching is per model, and the documentation does not say so

Once caching was set on both routes for Anthropic, the direct key came out about 2% cheaper than the reseller. Near enough to nothing, so the choice stopped being financial and became a question of latency and of how many parties can read your visitors' conversations.

Google inverted that conclusion. Running Gemini on Google's own endpoint had been the one blocked part of this benchmark for two weeks, for a reason worth repeating: a prepaid billing account with a zero balance returns the same error as an invalid key. It reads like a credentials problem when it is a money problem, and we spent days looking in the wrong place. Once the account was funded, same prompt, same hour:

The direct key was 2.6 times as expensive as the middleman, on a route where the middleman bills Google's list price to the cent. It was caching again: the reseller's requests picked up the repeat discount from the second turn onward, ours reported zero cached tokens on every single turn.

We went looking for our own mistake and did not find one. Sending the block as a system instruction and inlining it into the conversation both produced zero cache hits, so it was not the shape of the request. The documented size floor was not it either: implicit caching on that model line wants about 4,096 input tokens and our prompt was around 4,200, comfortably over. The real answer is that the behaviour differs by model, from the same vendor, called by the same code:

Three behaviours, none of them predictable from the docs. The practical consequence: a cost estimate you build on one model does not survive the move to its successor, even when the advertised price per token is identical. Re-measure on every model change, or budget for the worst case.

The reasoning you pay for and never see

Newer models reason before answering. That reasoning is billed at the output rate, which runs three to five times the input rate, and none of it reaches the visitor. On Gemini 3.6 Flash, 12,891 of 13,609 billed output tokens were reasoning. The visitor saw 5% of what we paid for.

Two things follow that are easy to miss. You cannot see it coming, because nothing in the length of a reply tells you what it cost. And it scales with the difficulty of the question rather than with your traffic, so one hard question can cost more than ten easy ones.

We had the answer ceiling at 700 tokens as a cost control, which felt prudent when it was set. Models that reason spent the entire allowance thinking and returned nothing:

Every one of those empty replies was billed in full. And the arithmetic that finally killed the idea of a tight ceiling: at 2,000 tokens, gpt-5 spent $0.020 on a question and said nothing; at 2,500 tokens, the same model on the same question spent $0.013 and answered it. Cut off mid-thought, it burned the budget and produced nothing to show; given room to finish, it reasoned less wastefully and stopped. A ceiling low enough to work as a cost control is low enough to buy you a failed conversation at full price. We raised ours to 3,000 and truncation went to zero.

Speed is the second bill

Money is not the only thing a model spends. Across the clean models, replies ranged from 1.3 seconds to 28.3, a 22-fold spread, and in a chat window that difference is not a detail. Under about two seconds the exchange feels like a conversation. Around five, people stop watching and switch tabs. At 28 seconds you no longer have a chat widget, you have a contact form with extra steps.

The route matters here as much as the model. Sonnet answered in 3.1 seconds on a direct Anthropic key and 7.2 through the reseller: same model, same answers, more than twice the wait. So if you take a reseller for its convenience, understand that you are paying for it in latency rather than in dollars. Grok 4.6 is the clearest casualty of this axis. It never failed a single turn and would look respectable in any correctness ranking, but at 28.3 seconds and $130 per thousand conversations it is unusable for the job.

What the ten turns caught

Cost says nothing about whether the answers were true, and this was the assumption I was most confident about before the runs. It was simply false: the cheapest model in the benchmark was one of the most careful, and a model two dollars dearer was the worst offender in the set.

Sorted by price, Grok 4.20 sits near the top of this benchmark. It is the one model I would not put in front of a client's visitor.

How each provider family behaved

Anthropic. The most predictable family in the set and the only one where caching behaved exactly as documented. Sonnet 5 at $42 is the balance point; Opus 5 at $102 buys a noticeably better conversation rather than a better fact; Haiku 4.5 is the trap described above, cheap on paper and only 35% cheaper in practice. Claude API cost is also the easiest to forecast, because the cache-read rate is published and it actually applies.

OpenAI. GPT-5.1 at $23 was the cheapest error-free option among the large models, and GPT-4o remains fast and clean at $44 despite its age. The frustration here is measurement, not quality: OpenAI does not return cost in its API responses, so every dollar figure we have for a GPT model comes from the same model through the reseller. If you are on a direct OpenAI key, you cannot read your per-conversation cost from the response; you have to derive it from token counts.

Google. The widest range and the least predictable. Gemini 3.1 Flash Lite is the cheapest clean model in the whole benchmark at $5 to $6, while Gemini 3.1 Pro preview costs $124 for no measurable gain in this task. Gemini API cost also depends on the route in a way no other family does, thanks to the per-model caching behaviour. Two models we tested are already withdrawn.

xAI. The most polarised family. Grok 4.20 is the fastest cheap model we measured and the only one that fabricated fee terms. Grok 4.3 handled the hardest turn better than anything else in the benchmark but costs the same as GPT-5.1 while answering slower. Grok 4.6 is clean, dear and far too slow. There is no version of Grok I would run as a default here.

How to work out your own bill

Our numbers are one site's numbers. Yours will differ mostly by the size of your knowledge base, and you can estimate them in about five minutes without running anything.

1. Measure your knowledge base in characters and divide by four. That is roughly its size in tokens. Ours was 7,854 bytes, about 2,000 tokens, which assembled into a system prompt of around 3,000 once the rules and answer format were added.
2. Multiply by the number of turns in a typical conversation. That is your uncached input. Ten turns on a 3,000-token prompt is 30,000 tokens before anyone types a word.
3. Add roughly 2,000 tokens for the conversation history and 1,500 for the visible answers, then multiply each part by the model's input and output price per million.
4. If the model reasons, multiply the output figure by five as a working assumption until you have measured your own ratio. That single adjustment moved our estimates more than any model swap.
5. Then apply caching: if it works on your route and model, everything after the first turn drops to about a tenth on the repeated block. That is the difference between the two halves of the Claude table above.

A worked example. A shop with a 40,000-character catalogue in the prompt has about 10,000 tokens of context, five times ours. On Sonnet 5 at $2 per million input tokens, ten uncached turns cost roughly $0.20 per conversation, or $200 per thousand, before output. With caching working, that falls to roughly $0.04, or $40. Same model, same answers, five times the difference, decided by one flag rather than by the price list.

What I would actually run

After all of that, the model is the last and smallest decision, but it still has to be made, and cheapest is the wrong way to make it. What matters is balance: cost, speed, and whether the thing can be left alone with a stranger who is upset.

Whichever you pick, do the caching check first. A default model on a cached route costs less than a cheap model on an uncached one, and it will answer better.

Where this stops being true

• One site, one industry, one dialogue. A law firm's knowledge base is text-heavy and its assistant is fenced by strict rules about what it may not say. A shop with 4,000 products would bill a different shape and fail in different places.
• OpenAI does not return cost in its API responses, so every dollar figure for a GPT model here comes from the same model through the reseller. Tokens and latency are from the direct route.
• These were live endpoints, not pinned snapshots, and the ground moved while we stood on it. Gemini 2.5 Flash and 2.5 Pro now answer "no longer available to new users", Gemini 3.7 and 3.8 Flash refused a direct key for eight solid minutes with "503 high demand", and Gemini 3.1 Pro preview returned "429 quota" eighteen times. Anything here about a specific version is a photograph, not a law.
• Correctness was judged by one person against one knowledge base. A second reader might grade the borderline answers differently, though not, I think, the invented fee arrangement.

TL;DR

• Check whether prompt caching is on, for every provider route. It halved our bill and changed no answers, which is more than any model choice offered.
• Your knowledge base, not your traffic, is what you buy ten times per conversation. It was 94% of billed input here.
• Reasoning tokens bill at the output rate whether or not an answer comes back, so a tight answer ceiling does not cap spend. It caps the answer.
• A direct provider key is not automatically cheaper. Ours cost 2.6 times the reseller on Gemini, because the caching discount is per model and undocumented.
• Speed is the second bill: 1.3 to 28.3 seconds across clean models, and the reseller route doubled Sonnet's response time.
• Test the model against your own rules before price enters the decision. The cheapest model here was safe; one two dollars dearer invented fees for a client's visitor.
• Then choose the model. Among the clean ones the spread was $5 to $130 per thousand conversations, real money, and still the smallest of these decisions.

Citation and disclosure

Cite as: Melnyk, I. (2026). What an LLM actually costs on a live website: 25 models, one conversation. Norml Studio. norml.studio/blog/llm-cost-on-a-live-website. Measured 1 to 15 September 2026. Reuse the figures with attribution.

The benchmark ran on MoonPress Chat, Norml Studio's free AI assistant plugin for WordPress, installed on a live client site. It works on the site owner's own provider key, we resell no tokens and take no margin on provider usage, which is why the bill was ours to understand. Nothing in the method depends on the plugin: any software that re-sends its knowledge base each turn will show the same shape. Four of its defaults changed because of these runs, caching on the reseller route, the answer ceiling, which models are offered at all, and a daily spend limit in dollars.