AI chat model benchmark 2026: cost, speed, and answer quality on a live website

Almost everyone picks a model for a website chat the same way: open a pricing page, sort by price, take the cheapest row. We wanted to know whether that works. One live site — a law firm's, where the assistant answers real visitors. The same ten-turn conversation. Twenty-five combinations of model and route. For each one we recorded what it billed, how long it took to think, and exactly what it said.

It turns out the pricing page is not where the cost is decided. The largest saving came from no model at all, but from one flag. Prompt caching halved the bill and changed not a single word of any answer: Claude Sonnet 5 from $77 to $43 per thousand conversations, Opus 5 from $202 to $101. No model swap available to us came close.

The reason is simple, and it appears on no pricing page: a model has no memory, so your knowledge base is sent to it again on every turn. Here that was 94% of everything billed. The visitor's own words were 6%. And while we were counting money, we found something we were not looking for: one of the cheap models invented payment terms for a visitor that the firm had never given it. So the order is this — caching first, then a check against your own rules, and only then price.

Here is everything we measured, and what we would run because of it.

What we measured, and how

Locked conditions. Nothing changed but the model and the route: one live site, the public assistant of a personal-injury law firm, not a sandbox. One firm-approved knowledge base of 7,854 bytes, byte-identical in every run. One conversation, written once and never edited. The answer ceiling was 3,000 tokens; the earlier 700-token runs are reported separately, because they turned into a finding of their own. Measured September 1-15, 2026.

The conversation. We wrote one conversation: a person asking for help after a car accident. Ten turns, always in the same order, not a word changed between runs.

The important part is that we knew in advance which answer would satisfy us and which would not. Every turn had a written criterion: what the assistant had to say, and what counted as a failure. Without that this would be a demo, not a measurement.

Three of the ten turns are traps. They are where you see the difference between a model you can leave alone with a real person and one you cannot.

What we ran. Twenty-five combinations of model and route: four provider families — Anthropic, OpenAI, Google and xAI — over two routes. Direct provider keys where we had them, and one reseller, OpenRouter, which bills provider list prices.

How cost was computed. Cost per conversation is the sum across all ten turns: input tokens at the input price, cached input at the cache-read price, and output tokens, including the reasoning ones you never see, at the output price. All against provider list prices read on the day of the run.

How correctness was judged. Not automated. I read all 250 answers against the knowledge base and marked a failure whenever a model named a figure, accepted the planted fact, quoted fee terms it had never been given, or showed the visitor a template placeholder.

The whole table, before the explanations

Here is what came out: what each model costs and how long it takes, for every model that reached the end of the conversation. Cheapest first.

How to read it:

• The price is per thousand conversations. That is roughly a month of work for an assistant on a site where a few dozen people a day write in.
• The seconds are the visitor's wait. How long they look at the screen after sending a message.
• The last column is whether the model invented facts, across all ten turns. Where it says no, the price is still listed, but that model should not go on a site.
• Two models are missing entirely. GPT-5 and GPT-5-mini returned an empty screen instead of an answer: billed, with nothing for the visitor to read.

The dearest model costs 26 times the cheapest. The slowest takes 22 times longer than the fastest.

And still, the biggest lever on the bill is not in this table. It is one flag in the settings: caching. It halved the bill on these same models without changing a word of any answer. The next two sections are about it.

94% of what you pay for is your own content

Here is the simple thing that appears on no pricing page: a language model has no memory. None. Every turn, the software re-sends it the whole knowledge base and the whole conversation so far, because that is the only way it knows anything at all.

It works like this: the visitor types one sentence, and the provider receives that sentence plus everything the site knows. Every time.

One conversation on Claude Sonnet 5 billed 32,700 input tokens against 1,563 tokens of visible answer. Twenty-one lines read for every line written.

What the visitor actually typed with their own hands across the whole conversation — 254 words, about 330 tokens — is one percent of the bill. The rest is text you already paid for on the previous turn. And the one before that.

The price per million tokens sets the scale. But the multiplier is how many times you pay full price for the same paragraph. Double your knowledge base and you double the bill, on any model.

Which is why the usual advice to pick a cheaper model is the smallest saving available.

Prompt caching: the one flag that halved the bill

Providers solved this re-sending problem long ago. Mark a block of text as one that will repeat, and the provider keeps it and charges roughly a tenth of the usual price when it comes back.

The catch is that the discount does not switch itself on. It applies only if the request carries the mark.

Ours carried it when we talked to Anthropic directly. It did not carry it when we talked to the same models through the reseller. So we had been paying full price for text the provider already had open in front of it. The fix was one condition in one function.

Half the bill, gone, with byte-identical answers.

The Haiku row is not an exception. It is the proof. Anthropic prompt caching has a minimum size: the cacheable block has to be long enough, and for Haiku 4.5 that minimum is longer than our entire system prompt. Caching therefore never applied to it, and its price sat unchanged on both routes, before and after. If the gap had been a reseller's markup, Haiku would have moved with everything else. It did not budge a cent.

That minimum has a second consequence, worth knowing before you reach for the cheap model in a family. Haiku looks like the budget Claude: $27 against Sonnet's $42. But that is only 35% cheaper rather than several times cheaper, precisely because Sonnet gets the discount and Haiku cannot. You pay near-Sonnet money for a weaker model.

You can check this on any product, not just ours. Have a few conversations, then open your provider dashboard and look for cache-read tokens. If they are permanently zero, you are buying the same text at full price on every single turn. Ask the vendor whether caching is enabled on every provider route or only some of them. The answer is usually some of them.

Google's caching is per model, and the documentation does not say so

Once caching was set on both routes for Anthropic, the direct key came out about 2% cheaper than the reseller. Near enough to nothing, so the choice stopped being financial and became a question of latency and of how many parties can read your visitors' conversations.

Google inverted that conclusion.

First, why this part was blocked for two weeks: a prepaid billing account with a zero balance returns the same error as an invalid key. It reads like a credentials problem when it is a money problem. We spent days looking in the wrong place.

Once the account was funded, same prompt, same hour:

The direct key was 2.6 times as expensive as the middleman, on a route where the middleman bills Google's list price to the cent.

It was caching again: the reseller's requests picked up the repeat discount from the second turn onward, ours reported zero cached tokens on every single turn.

We went looking for our own mistake and did not find one. Sending the block as a system instruction and inlining it into the conversation both produced zero cache hits, so it was not the shape of the request. The documented size floor was not it either: implicit caching on that model line wants about 4,096 input tokens and our prompt was around 4,200, comfortably over.

The real answer is that the behavior differs from model to model. Same vendor. Same code.

Three behaviors, and not one of them predictable from the docs.

What follows from that: a cost estimate built on one model does not survive the move to its successor, even when the advertised price per token is identical. Re-measure on every model change, or budget for the worst case.

The reasoning you pay for and never see

Newer models reason before answering. That reasoning is billed at the output rate, which runs three to five times the input rate, and none of it reaches the visitor. On Gemini 3.6 Flash, 12,891 of 13,609 billed output tokens were reasoning. The visitor saw 5% of what we paid for.

Two things follow that are easy to miss. You cannot see it coming, because nothing in the length of a reply tells you what it cost. And it scales with the difficulty of the question rather than with your traffic, so one hard question can cost more than ten easy ones.

We had the answer ceiling at 700 tokens as a cost control, which felt prudent when it was set. Models that reason spent the entire allowance thinking and returned nothing:

Every one of those empty replies was billed in full. And the arithmetic that finally killed the idea of a tight ceiling: at 2,000 tokens, gpt-5 spent $0.020 on a question and said nothing; at 2,500 tokens, the same model on the same question spent $0.013 and answered it. Cut off mid-thought, it burned the budget and produced nothing to show; given room to finish, it reasoned less wastefully and stopped. A ceiling low enough to work as a cost control is low enough to buy you a failed conversation at full price. We raised ours to 3,000 and truncation went to zero.

Speed is the second bill

Money is not the only thing a model spends. Across the clean models, replies ranged from 1.3 seconds to 28.3, a 22-fold spread, and in a chat window that difference is not a detail. Under about two seconds the exchange feels like a conversation. Around five, people stop watching and switch tabs. At 28 seconds you no longer have a chat widget, you have a contact form with extra steps.

The route matters here as much as the model. Sonnet answered in 3.1 seconds on a direct Anthropic key and 7.2 through the reseller: same model, same answers, more than twice the wait. So if you take a reseller for its convenience, understand that you are paying for it in latency rather than in dollars. Grok 4.6 is the clearest casualty of this axis. It never failed a single turn and would look respectable in any correctness ranking, but at 28.3 seconds and $130 per thousand conversations it is unusable for the job.

What the ten turns caught

Cost says nothing about whether the answers were true, and this was the assumption I was most confident about before the runs. It was simply false: the cheapest model in the benchmark was one of the most careful, and a model two dollars dearer was the worst offender in the set.

Sorted by price, Grok 4.20 sits near the top of this benchmark. It is the one model I would not put in front of a client's visitor.

How each provider family behaved

Anthropic. The most predictable family in the set and the only one where caching behaved exactly as documented. Sonnet 5 at $42 is the balance point; Opus 5 at $102 buys a noticeably better conversation rather than a better fact; Haiku 4.5 is the trap described above, cheap on paper and only 35% cheaper in practice. Claude API cost is also the easiest to forecast, because the cache-read rate is published and it actually applies.

OpenAI. GPT-5.1 at $23 was the cheapest error-free option among the large models, and GPT-4o remains fast and clean at $44 despite its age. The frustration here is measurement, not quality: OpenAI does not return cost in its API responses, so every dollar figure we have for a GPT model comes from the same model through the reseller. If you are on a direct OpenAI key, you cannot read your per-conversation cost from the response; you have to derive it from token counts.

Google. The widest range and the least predictable. Gemini 3.1 Flash Lite is the cheapest clean model in the whole benchmark at $5 to $6, while Gemini 3.1 Pro preview costs $124 for no measurable gain in this task. Gemini API cost also depends on the route in a way no other family does, thanks to the per-model caching behavior. Two models we tested are already withdrawn.

xAI. The most polarised family. Grok 4.20 is the fastest cheap model we measured and the only one that fabricated fee terms. Grok 4.3 handled the hardest turn better than anything else in the benchmark but costs the same as GPT-5.1 while answering slower. Grok 4.6 is clean, dear and far too slow. There is no version of Grok I would run as a default here.

How to work out your own bill

Our numbers are one site's numbers. Yours will differ mostly by the size of your knowledge base, and you can estimate them in about five minutes without running anything.

1. Measure your knowledge base in characters and divide by four. That is roughly its size in tokens. Ours was 7,854 bytes, about 2,000 tokens, which assembled into a system prompt of around 3,000 once the rules and answer format were added.
2. Multiply by the number of turns in a typical conversation. That is your uncached input. Ten turns on a 3,000-token prompt is 30,000 tokens before anyone types a word.
3. Add roughly 2,000 tokens for the conversation history and 1,500 for the visible answers, then multiply each part by the model's input and output price per million.
4. If the model reasons, multiply the output figure by five as a working assumption until you have measured your own ratio. That single adjustment moved our estimates more than any model swap.
5. Then apply caching: if it works on your route and model, everything after the first turn drops to about a tenth on the repeated block. That is the difference between the two halves of the Claude table above.

A worked example. A shop with a 40,000-character catalogue in the prompt has about 10,000 tokens of context, five times ours. On Sonnet 5 at $2 per million input tokens, ten uncached turns cost roughly $0.20 per conversation, or $200 per thousand, before output. With caching working, that falls to roughly $0.04, or $40. Same model, same answers, five times the difference, decided by one flag rather than by the price list.

What I would actually run

After all of that, the model is the last and smallest decision, but it still has to be made, and cheapest is the wrong way to make it. What matters is balance: cost, speed, and whether the thing can be left alone with a stranger who is upset.

Whichever you pick, do the caching check first. A default model on a cached route costs less than a cheap model on an uncached one, and it will answer better.

Where this stops being true

• One site, one industry, one dialogue. A law firm's knowledge base is text-heavy and its assistant is fenced by strict rules about what it may not say. A shop with 4,000 products would bill a different shape and fail in different places.
• OpenAI does not return cost in its API responses, so every dollar figure for a GPT model here comes from the same model through the reseller. Tokens and latency are from the direct route.
• These were live endpoints, not pinned snapshots, and the ground moved while we stood on it. Gemini 2.5 Flash and 2.5 Pro now answer "no longer available to new users", Gemini 3.7 and 3.8 Flash refused a direct key for eight solid minutes with "503 high demand", and Gemini 3.1 Pro preview returned "429 quota" eighteen times. Anything here about a specific version is a photograph, not a law.
• Correctness was judged by one person against one knowledge base. A second reader might grade the borderline answers differently, though not, I think, the invented fee arrangement.

TL;DR

• Check whether prompt caching is on, for every provider route. It halved our bill and changed no answers, which is more than any model choice offered.
• Your knowledge base, not your traffic, is what you buy ten times per conversation. It was 94% of billed input here.
• Reasoning tokens bill at the output rate whether or not an answer comes back, so a tight answer ceiling does not cap spend. It caps the answer.
• A direct provider key is not automatically cheaper. Ours cost 2.6 times the reseller on Gemini, because the caching discount is per model and undocumented.
• Speed is the second bill: 1.3 to 28.3 seconds across clean models, and the reseller route doubled Sonnet's response time.
• Test the model against your own rules before price enters the decision. The cheapest model here was safe; one two dollars dearer invented fees for a client's visitor.
• Then choose the model. Among the clean ones the spread was $5 to $130 per thousand conversations, real money, and still the smallest of these decisions.

Disclosure

The benchmark ran on MoonPress Chat, Norml Studio's free AI assistant plugin for WordPress, installed on a live client site. It works on the site owner's own provider key, we resell no tokens and take no margin on provider usage, which is why the bill was ours to understand. Nothing in the method depends on the plugin: any software that re-sends its knowledge base each turn will show the same shape. Four of its defaults changed because of these runs, caching on the reseller route, the answer ceiling, which models are offered at all, and a daily spend limit in dollars.