Normally you send a single block of text to an AI and get a response back. In real LLM APIs (e.g. OpenAI, Anthropic), however, the communication is structured: the request consists of multiple messages with different roles.

Example of a typical chat request

Systemprompt: "Du bist ein freundlicher Assistent. Antworte auf Deutsch."
User:         "Was ist die Hauptstadt von Frankreich?"
KI-Antwort:   "Paris ist die Hauptstadt."
User:         "Und von Deutschland?"

Why is this relevant for cost estimation?

Every message counts as input tokens. In ongoing conversations (multi-turn chats), the entire prior history is sent along with every new request — so the cost grows with each round. On top of that, the model adds a small overhead per message (typically 3–4 tokens) for the role markers.

When should you use chat mode?

• You're planning a chatbot with a system prompt and want a realistic per-request cost estimate
• You want to know what a multi-turn conversation costs
• You want to see the impact of the system prompt on the total cost

When is single-text mode enough?

If you just want a rough estimate of how expensive processing some text would be — for example an article to summarize or translate — the normal mode is perfectly fine.

Every message in an AI conversation gets a role. The role tells the model who's speaking and how the message should be interpreted.

Typical structure

1. 1× system prompt (rules)
2. 1× user (question)
3. For multi-turn: alternating user / AI response / user / AI response, ends with user

Prompt caching is a cost optimization offered by major providers like OpenAI and Anthropic — and for anyone running chatbots or agents in production, it's practically indispensable.

The problem caching solves

Every request in a chat sends the entire prior history. The system prompt never changes — it gets repeated identically on every request. If your system prompt is 1000 tokens and you make 100 requests per day, you pay for 100,000 tokens daily just for the unchanging system prompt.

What caching does

After the first request, the provider "remembers" the system prompt in a cache for 5–10 minutes. If another request with exactly the same beginning arrives within that window, the cached tokens aren't reprocessed but pulled from the cache — and billed at a steep discount.

The discounts at a glance

• Anthropic (Claude): approx. 10% of the normal price for cached tokens — i.e. 90% discount
• OpenAI (GPT): approx. 50% of the normal price — i.e. 50% discount
• Other providers: varies, often no caching available

How this tool calculates

In cumulative mode you see two numbers side by side: the normal input cost, and the cost with caching. The calculation rule is:

• Turn 1: System prompt is billed at full price (cache is filled)
• Turn 2+: System prompt is billed at the cached price (loaded from cache)
• The rest of the chat history (user messages and AI responses) always stays at the normal price

What this tool simplifies

In practice, advanced users can cache larger parts of the chat (entire first turns, knowledge base blocks, etc.), and Anthropic charges a small premium when writing to the cache for the first time. To keep the display manageable, we only cache the system prompt here and ignore the minimal write overhead. For a realistic budget estimate that's perfectly sufficient.

Availability

This caching calculator only appears for models that support prompt caching and have a cached price set. For other models, the caching block isn't shown — there's simply no saving to calculate.

When you send a chat request to an LLM, the cost overview at the top always shows a few more tokens than the pure text sum would suggest. Those are the "overhead tokens" — a small but constant addition found in almost every modern chat API.

Why is there overhead?

An LLM doesn't just receive a flat text, but a structured list of messages with role markers. The model needs to know where a message starts, which role it has (system prompt, user, AI response) and where it ends. To handle this, invisible "control characters" are attached to every message — each of them is itself a token.

How much overhead exactly?

For most common APIs (OpenAI, Anthropic, Mistral, …) you can use this rule of thumb:

• 3 Tokens pro Nachricht — for role markers and separators
• + 3 Tokens pro API-Anfrage — one-time overhead for the full request

Concrete example: A request with system prompt + 3 user messages + 2 AI responses = 6 messages. Overhead = 6 × 3 + 3 = 21 tokens.

Is this the same for all models?

No — the exact value varies slightly between providers and models, and the documentation is often imprecise. The formula used here (3+3) is the OpenAI standard and works as a good approximation for practically all chat APIs. For very short texts the overhead plays a relatively larger role; in long conversations it gets lost in the noise.

Why should I care?

In a single case the overhead is negligible. But for a real application with thousands of chat requests per day, it adds up: 20 tokens × 100,000 requests = 2 million tokens of overhead per day, just for the role markers. Good to know when you're doing budget calculations for your chatbot.

Especially visible in cumulative mode

In the turn-by-turn table above you see the breakdown per turn, e.g. "250 (230 + 20 OH)". The "OH" number is the overhead for that turn. The more messages the API sends in a turn, the higher this item gets.

In a real chat application, every new user message sends the entire prior conversation history along — otherwise the AI wouldn't know what's being talked about. That has a massive impact on cost, and you can measure exactly that effect here.

🔹 "Single API request" mode

Calculates the cost for exactly one API request, as the chat below currently looks. Useful when you want to test a specific state: "What does it cost to send exactly this sequence of messages to the API once?"

🔹 "Whole conversation (cumulative)" mode

Computes what the conversation costs in total if it unfolds turn by turn as shown here. For each new user turn the full history is counted again. This shows you the reality of an ongoing chat session.

A concrete example

Suppose you have a system prompt (60 tokens), the user asks 3 questions of 20 tokens each, and the AI responds with 80 tokens each time:

Turn 1 (1. User-Frage):       60 + 20                           =  80 Tokens
Turn 2 (2. User-Frage):       60 + 20 + 80 + 20                 = 180 Tokens
Turn 3 (3. User-Frage):       60 + 20 + 80 + 20 + 80 + 20      = 280 Tokens
----------------------------------------------------------------
Summe der Input-Tokens:                                           540 Tokens

"Eine API-Anfrage"-Modus würde   280 Tokens zeigen (nur Turn 3)
"Kumulativ"-Modus zeigt          540 Tokens (alle Turns summiert)

⚠️ Costs grow quadratically

A 10-round conversation doesn't cost 10× more than a 1-round one, but often 30–50× more. This is one of the main cost drivers in chatbot applications. Anyone building chatbots in production should keep an eye on it.

💡 What to do about the costs?

• Use prompt caching (available at OpenAI & Anthropic) — drastically reduces costs for recurring system prompts
• Summarize older messages before the context gets too long
• Add a max-turn limit — e.g. start a new conversation after 10 messages
• Use smaller models for simple turns, large models only for complex questions