How Much Does an LLM API Call Really Cost? Tokens, Caching, and Batching Explained

Anyone building an application using the OpenAI, Anthropic, or Google API will sooner or later ask themselves: How much does this actually cost? The official price lists are transparent, but they don’t reveal how costs accumulate in a real conversation. This guide explains the mechanics behind it—from tokens to input/output to caching, batching, and overhead—and concludes with a sample calculation that shows why chatbot costs often end up being higher than expected.

1. What are tokens, and how are they calculated?

A token is the smallest unit into which a language model breaks down text. It is neither a word nor a letter, but a subword unit—typically 3 to 5 characters long, often a syllable or a frequently occurring fragment. English texts usually end up with about four characters per token, while German texts tend to have three due to the many compound words and umlauts.

Example: The sentence "Künstliche Intelligenz verändert die Arbeitswelt." is split into 9 to 12 tokens, depending on the model. The “ä” in “verändert” alone can be counted as a separate unit depending on the tokenizer, because it occurs less frequently than in the English training data corpus.

Important: Each provider uses its own tokenizer. OpenAI uses the tiktokenlibrary, while Anthropic and Google use proprietary methods. The same text might result in 140 tokens with OpenAI and 180 with Claude. If you want to calculate across models, you should determine the token count for each model separately.

Model

Qwen 3 72B Qwen 2.5 72B Command R+ Command R Command R7B DeepSeek V3 DeepSeek R1 Gemma 3 27B Gemma 2 27B Llama 3.3 70B Llama 3.2 90B Llama 3.2 11B Llama 3.1 405B Llama 3.1 70B Llama 3.1 8B Phi-4 Phi-3.5 Mistral Large Mistral Medium Mistral Small Mixtral 8x22B GPT-5.5 pro GPT-5.5 GPT-5.4 pro GPT-5.4 GPT-5.4 Mini GPT-5.4 Nano GPT-4.1 GPT-4o GPT-4o Mini o1 o3 o4-mini Grok 2

Calculate tokens and costs

0 / 10,000

Load sample text Clear

💬 Switch to chat ?

Chat conversation ← Back to single text

+ Add message 🗑 Reset all

Calculation mode: Single API request Whole conversation (cumulative) ?

🤖Discover all LLM models now🧾Token Knowledge

Expected response length: 500 Output tokens (estimated)

02.0004.0006.0008.000

Tokens 0 —

Characters / Words 0 0 Words

Estimated cost —  

Cost breakdown (per request)

Input —

Output —

Total —

🔍 Compare with other models

Model comparison ×

Select models for comparison

×

Pick up to 5 models. Each model will re-tokenize your current input text.

What is chat mode?

×

Normally you send a single block of text to an AI and get a response back. In real LLM APIs (e.g. OpenAI, Anthropic), however, the communication is structured: the request consists of multiple messages with different roles.

Example of a typical chat request

Systemprompt: "Du bist ein freundlicher Assistent. Antworte auf Deutsch."
User:         "Was ist die Hauptstadt von Frankreich?"
KI-Antwort:   "Paris ist die Hauptstadt."
User:         "Und von Deutschland?"

Why is this relevant for cost estimation?

Every message counts as input tokens. In ongoing conversations (multi-turn chats), the entire prior history is sent along with every new request — so the cost grows with each round. On top of that, the model adds a small overhead per message (typically 3–4 tokens) for the role markers.

When should you use chat mode?

• You're planning a chatbot with a system prompt and want a realistic per-request cost estimate
• You want to know what a multi-turn conversation costs
• You want to see the impact of the system prompt on the total cost

When is single-text mode enough?

If you just want a rough estimate of how expensive processing some text would be — for example an article to summarize or translate — the normal mode is perfectly fine.

The roles in chat mode

×

Every message in an AI conversation gets a role. The role tells the model who's speaking and how the message should be interpreted.

🛠️ System prompt

The "rules of the game" for the model. Here you define how the AI should behave — personality, response style, constraints, format. Set only once at the start of the conversation and sent in the API as "system".

Example: "You are a concise assistant. Reply in no more than three sentences."

👤 User

The user's message — the actual question or instruction. In most applications this is what the end user types into a chat box. Sent in the API as "user".

Example: "Explain quantum physics in simple terms."

🤖 AI Response

Previous AI responses. You only need to include these if you're simulating an ongoing conversation. The model needs this context to keep track of multi-turn dialogues. In the API this role is called "assistant".

Example: "Paris is the capital of France."

Typical structure

1. 1× system prompt (rules)
2. 1× user (question)
3. For multi-turn: alternating user / AI response / user / AI response, ends with user

What is prompt caching?

×

Prompt caching is a cost optimization offered by major providers like OpenAI and Anthropic — and for anyone running chatbots or agents in production, it's practically indispensable.

The problem caching solves

Every request in a chat sends the entire prior history. The system prompt never changes — it gets repeated identically on every request. If your system prompt is 1000 tokens and you make 100 requests per day, you pay for 100,000 tokens daily just for the unchanging system prompt.

What caching does

After the first request, the provider "remembers" the system prompt in a cache for 5–10 minutes. If another request with exactly the same beginning arrives within that window, the cached tokens aren't reprocessed but pulled from the cache — and billed at a steep discount.

The discounts at a glance

• Anthropic (Claude): approx. 10% of the normal price for cached tokens — i.e. 90% discount
• OpenAI (GPT): approx. 50% of the normal price — i.e. 50% discount
• Other providers: varies, often no caching available

How this tool calculates

In cumulative mode you see two numbers side by side: the normal input cost, and the cost with caching. The calculation rule is:

• Turn 1: System prompt is billed at full price (cache is filled)
• Turn 2+: System prompt is billed at the cached price (loaded from cache)
• The rest of the chat history (user messages and AI responses) always stays at the normal price

What this tool simplifies

In practice, advanced users can cache larger parts of the chat (entire first turns, knowledge base blocks, etc.), and Anthropic charges a small premium when writing to the cache for the first time. To keep the display manageable, we only cache the system prompt here and ignore the minimal write overhead. For a realistic budget estimate that's perfectly sufficient.

Availability

This caching calculator only appears for models that support prompt caching and have a cached price set. For other models, the caching block isn't shown — there's simply no saving to calculate.

What are "overhead tokens" in chat requests?

×

When you send a chat request to an LLM, the cost overview at the top always shows a few more tokens than the pure text sum would suggest. Those are the "overhead tokens" — a small but constant addition found in almost every modern chat API.

Why is there overhead?

An LLM doesn't just receive a flat text, but a structured list of messages with role markers. The model needs to know where a message starts, which role it has (system prompt, user, AI response) and where it ends. To handle this, invisible "control characters" are attached to every message — each of them is itself a token.

How much overhead exactly?

For most common APIs (OpenAI, Anthropic, Mistral, …) you can use this rule of thumb:

• 3 Tokens pro Nachricht — for role markers and separators
• + 3 Tokens pro API-Anfrage — one-time overhead for the full request

Concrete example: A request with system prompt + 3 user messages + 2 AI responses = 6 messages. Overhead = 6 × 3 + 3 = 21 tokens.

Is this the same for all models?

No — the exact value varies slightly between providers and models, and the documentation is often imprecise. The formula used here (3+3) is the OpenAI standard and works as a good approximation for practically all chat APIs. For very short texts the overhead plays a relatively larger role; in long conversations it gets lost in the noise.

Why should I care?

In a single case the overhead is negligible. But for a real application with thousands of chat requests per day, it adds up: 20 tokens × 100,000 requests = 2 million tokens of overhead per day, just for the role markers. Good to know when you're doing budget calculations for your chatbot.

Especially visible in cumulative mode

In the turn-by-turn table above you see the breakdown per turn, e.g. "250 (230 + 20 OH)". The "OH" number is the overhead for that turn. The more messages the API sends in a turn, the higher this item gets.

Calculation mode: single request vs. cumulative

×

In a real chat application, every new user message sends the entire prior conversation history along — otherwise the AI wouldn't know what's being talked about. That has a massive impact on cost, and you can measure exactly that effect here.

🔹 "Single API request" mode

Calculates the cost for exactly one API request, as the chat below currently looks. Useful when you want to test a specific state: "What does it cost to send exactly this sequence of messages to the API once?"

🔹 "Whole conversation (cumulative)" mode

Computes what the conversation costs in total if it unfolds turn by turn as shown here. For each new user turn the full history is counted again. This shows you the reality of an ongoing chat session.

A concrete example

Suppose you have a system prompt (60 tokens), the user asks 3 questions of 20 tokens each, and the AI responds with 80 tokens each time:

Turn 1 (1. User-Frage):       60 + 20                           =  80 Tokens
Turn 2 (2. User-Frage):       60 + 20 + 80 + 20                 = 180 Tokens
Turn 3 (3. User-Frage):       60 + 20 + 80 + 20 + 80 + 20      = 280 Tokens
----------------------------------------------------------------
Summe der Input-Tokens:                                           540 Tokens

"Eine API-Anfrage"-Modus würde   280 Tokens zeigen (nur Turn 3)
"Kumulativ"-Modus zeigt          540 Tokens (alle Turns summiert)

⚠️ Costs grow quadratically

A 10-round conversation doesn't cost 10× more than a 1-round one, but often 30–50× more. This is one of the main cost drivers in chatbot applications. Anyone building chatbots in production should keep an eye on it.

💡 What to do about the costs?

• Use prompt caching (available at OpenAI & Anthropic) — drastically reduces costs for recurring system prompts
• Summarize older messages before the context gets too long
• Add a max-turn limit — e.g. start a new conversation after 10 messages
• Use smaller models for simple turns, large models only for complex questions

2. Input tokens vs. output tokens

Every API request generates two types of data: input tokens (everything you send to the AI—system prompt, user question, chat history) and output tokens (the AI’s response). Both are billed separately, and output is typically 4–5 times more expensive than input.

The reason: Output tokens require more computational power from the provider. When processing the input, the model analyzes the entire text in a single pass. For the output, it must generate tokens sequentially, one by one—each new token requires a complete computational pass through the model. This does not scale linearly.

Practical implication: If your application generates long AI responses (such as summaries or creative text), output costs will be the dominant factor. For classification or routing tasks (short responses, lots of context), input costs will be the dominant factor.

3. Conversations: Why every request includes the entire conversation history

This is the biggest cost obstacle. An LLM has no memory between API calls. For a chat AI to know what was just discussed, the entire history up to that point must be sent along with every new message—including the system prompt, all previous user messages, and all previous AI responses.

Specifically: If a chatbot runs with an 800-token system prompt and 5 message exchanges, the system prompt is billed five times—once per request. Each new user question contributes not only its own text but also the entire stack preceding it.

Mathematically speaking, this is exponential growth: A conversation with 10 rounds doesn’t cost 10 times as much as one with a single round, but often 30–50 times as much. That’s why chatbots operated productively can quickly become expensive without optimization.

4. Cached Input and Batch Pricing — the two most important levers

Prompt Caching: Up to 90% off repeat inputs

Both OpenAI and Anthropic offer prompt caching. The idea is that the provider stores the beginning of a request (typically the system prompt) in a cache for 5 to 10 minutes. If another request with an identical beginning arrives within that time, the cached tokens are not reprocessed but retrieved from the cache—and billed at about 10% of the normal price (i.e., a 90% discount on the cached portion).

Important: The discount applies only to the cached portion, not to the entire request. For an 800-token system prompt within a 2,000-token request, the system prompt is discounted, while the rest of the request remains at the standard price. The actual cost savings are typically between 20% and 40%, not 90%.

Batch API: 50% discount for asynchronous workloads

If you don’t need real-time responses—such as for nightly data analysis, bulk classification, or machine translation—the Batch API offers a flat 50% discount on input and output. In return, you agree that results will be delivered within 24 hours, not in seconds.

Caching and batching can be combined. With properly configured pipelines, combined savings of 60–70% are realistic.

5. Context Window, Max Output, and Overhead Tokens

Three terms that appear in price lists and are often confused:

• Context Window — the maximum number of tokens a model can process in a single request (input and output combined). Current models such as Claude Sonnet 4.6 or GPT-5.4 support 200K to 1M tokens. If your conversation exceeds this limit, the API will fail—you’ll need to summarize or shorten the conversation.
• Max Output — the maximum length of a single response. Typically ranges from 8K to 128K tokens, which is significantly shorter than the context window, even if there were theoretically still space available.
• Overhead tokens — invisible control tokens that are added to every message. The model needs to know where a message begins, what its role is (system, user, assistant), and where it ends. Rule of thumb: 3 tokens per message plus 3 tokens for the request. In a chat with 6 messages, that’s 6×3+3 = 21 additional tokens that you don’t see anywhere, but you pay for.

6. Sample calculation: Customer support chatbot with 5 turns

Enough theory. Let’s look at a realistic use case: a support chatbot based on Claude Sonnet 4.6 ($3 / $15 per 1M tokens, cached input $0.30).

Assumptions:

• System prompt: 800 tokens (persona, instructions, knowledge snippet)
• Per turn: 50 tokens for user questions + 200 tokens for AI responses
• 5 rounds of conversation, cumulative scoring

Three cost scenarios for this very conversation:

Two observations from the table:

1. Caching alone yields about a 24% reduction—not 90%, because only the system prompt portion benefits. The effect becomes more significant with longer system prompts or more turns.
2. Caching and batch processing combined yield savings of about 62%. Those who can use both optimizations (i.e., don’t necessarily need real-time processing) can save around $2,200 per month—enough to more than justify the investment in a robust caching solution.

Conclusion

Token costs aren’t a mystery; they follow clear rules. Anyone who understands the three mechanisms—tokenization, conversation accumulation, and special pricing—can plan budgets realistically and implement targeted optimizations. Key takeaways:

• Output costs 4–5 times more than input — short answers save the most.
• In conversations, costs increase quadratically, not linearly.
• Prompt caching typically reduces costs by 20–40%, while batch processing reduces them by 50%.
• Overhead tokens are invisible but countable—and become significant in large numbers.

Glossary

Batch API

Asynchronous processing mode with a 50% discount on input and output. Responses are provided within 24 hours instead of in real time.

Cached Input

Input tokens that have already been processed in a previous call and cached by the provider. Billed at approximately 10% of the normal price.

Context Window

Maximum number of tokens a model can process in a single request (input + output combined).

Input Token

Token sent to the AI — everything in the prompt: system prompt, user message, chat history.

Max Output

Maximum number of tokens the model can generate in a single response. Must be less than the Context Window.

Output Token

Token generated by the AI in its response. Typically 4–5 times more expensive than the input.

Overhead Tokens

Invisible control tokens that every message in a chat request receives—typically 3 per message plus 3 for the entire request.

Prompt Caching

A mechanism that allows providers to cache recurring prompt components (typically system prompts) and bill them at the cached price for the next request.

System prompt

The instruction at the beginning of an AI request that defines the model’s behavior (persona, response style, constraints). Is sent again with every request in a conversation.

Token

The smallest processing unit of a language model. Subword-based, typically 3–5 characters long. Each provider has its own tokenizer.

Tokenizer

An algorithm that breaks text down into tokens. OpenAI uses tiktoken; other providers use proprietary methods.

Prices as of May 2026. Check the providers’ official websites for current rates at any time.