LLM cost in production: what drives the bill and how to control it

How the bill is built: tokens, calls and model

LLM providers charge per token, the unit of text a model reads and writes, with different prices for input and output and for each model. The cost of a task is the sum of the calls it makes; the monthly cost is that figure times the volume.

• Input tokens: Everything sent to the model on each call, such as system instructions, tool definitions, retrieved documents and the conversation history.
• Output tokens: The generated text; they usually cost more than input tokens, and the reasoning of models that think before answering is billed as output too.
• Calls per task: A chatbot makes one call per message; an agent makes several, and each step sends the accumulated context again.
• Model: Larger models cost more per token than smaller ones from the same provider, and price lists change often.

Beyond tokens, the estimate also covers embeddings, the vector store, infrastructure and observability.

What weighs most on the bill

In practice, the bill goes off forecast less because of the token price and more because of how the system is designed. These are the usual culprits:

• Context that only grows: The full history, too many documents and long instructions sent again on every call.
• Agent loops: Tasks that take many steps, retries after an error and tools called more than once for the same information.
• A model larger than the task needs: Using the most capable model to classify, extract or summarize, tasks a smaller model handles.
• Long output: Lengthy answers, verbose formats and long reasoning on simple tasks.
• Wasted cache: Prompts that change right at the start on every call, which prevents the prefix from being reused.

How to cut cost without losing quality

Every cost cut goes through the same evals as quality: a cut that lowers the success rate ends up costing more, in rework and human review.

• Prompt caching: The provider reuses an identical prefix (instructions, tools, fixed documents) and charges less for tokens read from the cache, so stable content goes at the start of the prompt and variable content at the end.
• Semantic caching: A question similar to one already answered gets the stored answer, with no model call; it needs a similarity threshold and expiry rules.
• Routing by task: A smaller model classifies, extracts and summarizes, and the larger model takes the hard steps.
• A lean context: Compact the history, retrieve fewer and more relevant passages (reranking) and cap the output length.
• Batch processing: Work that does not need an immediate answer goes to the providers' batch APIs, which charge less in exchange for delivering later.

How to forecast the cost before you build

A good estimate comes from measurements, not assumptions: first measure the tokens of a real task, then apply the provider's current price list.

1. Define the task and the volume: How many conversations, documents or runs per day, and with what peaks.
2. Measure on a prototype: Run real samples and record the input, output and cache tokens the API reports in every response.
3. Count the calls per task: The average and the worst case, with retries and extra agent steps.
4. Apply today's prices: Multiply by the current prices of each candidate model; since they change, the spreadsheet keeps the date they were checked.
5. Compare scenarios: A larger or smaller model, with and without caching, always by cost per completed task, not per call.
6. Keep a margin: Usage growth, context that grows over time and prompt changes.

How to track cost in production

In production, the bill has to be read per task, per flow and per customer, not only as a monthly total. That means recording the tokens of each call as it happens.

• Traces: Every run becomes a trace with the model calls, the tools, the tokens and the cost of each step; OpenTelemetry's generative AI conventions standardize those fields.
• Cost per completed task: It combines cost with eval results and shows when a cheaper model ends up costing more because it fails more often.
• Cache hit rate: It shows whether prompt caching and semantic caching are working.
• Limits and alerts: A budget per flow or per customer, a step limit per task and an alert when cost drifts from the average.

In Master Timm's own products, Zorbit audits the cost and the tokens of every agent run, and the Contas Casal assistant tracks cost per model and uses a semantic cache.