The real cost of conversation history

Here is a bill that surprises people. A chat assistant with modest traffic, priced on a cheap model, quietly costing several times what the forecast said. The traffic estimate was right. The token estimate was wrong, and it was wrong in a specific, predictable way.

In a stateless chat API, the model has no memory. To continue a conversation, you re-send the entire history on every turn. So the tokens you pay for on turn n include everything from turns 1 through n.

Turn 1:  send [msg1]                        → 1 unit
Turn 2:  send [msg1, msg2]                   → 2 units
Turn 3:  send [msg1, msg2, msg3]             → 3 units
...
Turn 10: send [msg1 ... msg10]               → 10 units

Total input tokens ≈ n(n+1)/2, quadratic in turns

A ten-turn conversation doesn't cost ten times a single turn. It costs roughly fifty-five times a single message, because the early messages are paid for again and again.

You cannot make the model stateful, but you can control what "history" means.

1. Truncate. Keep the last k turns and drop the rest. Simple, and correct for most assistant use cases where distant context stops mattering.
2. Summarise. Periodically replace old turns with a compact summary. Trades a little quality for a large, sustained token saving on long conversations.
3. Cache the prefix. If your provider supports prompt caching, a stable conversation prefix bills below the normal input rate on repeat turns. This is the highest-leverage option when it's available, and what it costs across the catalogue is worth checking before you restructure anything else.
4. Cap the window. Enforce a hard ceiling on history length at the application layer, so a runaway conversation can't produce a runaway bill.

The single most common source of a surprising AI bill is not traffic you didn't expect. It is history you re-sent without noticing.

The fix that matters most is the one you do before shipping: estimate both directions of your workload separately, using a realistic conversation length rather than a single-turn sample. A summarisation workload reads a lot and writes little; a multi-turn assistant re-reads its own history on every turn. Those are different bills, and only one of them is obvious.