LLM Cost Optimization: Prompt Caching, Batching and Model Routing

LLM cost optimization is the work of getting the same quality from an AI application for fewer tokens, cheaper tokens, or both. The biggest levers in 2026 are prompt caching, batch processing, routing requests between large and small models, and trimming what goes into each prompt. Each one has published discount figures behind it, and applied together they can cut bills substantially without users noticing a difference.

Costs matter more now because usage patterns changed. Agents make many model calls per task, retrieval stuffs long documents into prompts, and multi-agent systems multiply everything. Anthropic reported that its agents used about 4 times the tokens of a chat interaction and its multi-agent system about 15 times. A feature that was cheap as a chatbot can become expensive as an agent.

This guide explains where LLM costs come from, each major lever with the vendor figures behind it, the trade-offs, and a step-by-step plan.

Where the money goes

Every model call is billed on tokens in and tokens out, and output tokens usually cost several times more than input tokens. In a typical business application, input dominates by volume.

Measure before you optimise

Start with cost per successful task, not cost per call. An agent that is cheap per call but needs three attempts is expensive. Track, per feature:

• Calls per task, tokens in and out per call, model used.
• Cache hit rate.
• Success rate from your evaluation suite.
• Latency.

Tracing tools record most of this automatically. Our guide to LLM evaluation and observability covers the setup. Without it, you will optimise the wrong thing.

Lever 1: Prompt caching

Most prompts start with the same long prefix: instructions, examples, tool definitions, sometimes a reference document. Prompt caching lets the provider store that processed prefix and reuse it.

Anthropic's prompt caching documentation lists cache reads at 0.1 times the base input price for standard models, with even lower multipliers on some newer models, while writing to the cache costs 1.25 times base for a 5-minute lifetime or 2 times for one hour. The cache lifetime refreshes each time it is used. OpenAI applies caching automatically to prompts above a minimum length, with a discount on cached input.

To get high hit rates:

• Put stable content first and variable content last: instructions, then examples, then tools, then documents, then the user's message.
• Do not put timestamps, request IDs or user names at the top of the prompt; they break the shared prefix.
• Keep tool definitions in a stable order.
• For long reference documents used across many questions, cache the document once and ask many questions against it.

Lever 2: Batch processing

If a result is not needed immediately, send it as a batch. Anthropic's Message Batches API and OpenAI's Batch API both charge 50% of standard prices for asynchronous work. Anthropic's batch processing documentation says most batches finish in under an hour, and its caching discounts stack with the batch discount.

Good batch candidates:

• Overnight document classification, tagging and extraction.
• Generating summaries for yesterday's records or tickets.
• Running evaluation suites.
• Back-filling embeddings or contextual descriptions for a RAG index.
• Bulk translation and content preparation.

Audit your workloads for jobs that run on a schedule or feed reports. Teams are often surprised how much of their volume does not need a real-time answer.

Lever 3: Routing between models

Not every request needs the most capable model. A router sends simple requests to a smaller, cheaper model and hard ones to a larger model.

The research case is well established. LMSYS's RouteLLM work trained routers between a strong and a weak model; its best router reached 95% of the strong model's MT Bench performance while sending only 14% of calls to the strong model. The accompanying paper describes cost reductions of more than 2 times without substantially compromising quality.

Common routing designs:

Step-level routing is often the biggest win in agents, because most steps (parse this, format that, classify the tool result) are narrow. Our post on small language models vs frontier APIs covers which steps suit smaller models.

Lever 4: Send less context

• Retrieve better, not more. Reranking lets you send 5 strong chunks instead of 20 weak ones. See our post on GraphRAG, agentic RAG and long context.
• Summarise history. Replace old conversation turns with a running summary.
• Trim tool definitions. Agents with dozens of tools pay for every definition on every step. Load tools per task, or use protocols that reveal tools progressively.
• Strip boilerplate. Remove HTML, repeated headers and signatures before documents reach the prompt.

Lever 5: Control output

• Set sensible maximum output lengths per feature.
• Ask for structured output (JSON with defined fields) instead of prose where software consumes the result.
• Use reasoning or "thinking" budgets deliberately: high for hard planning steps, low or off for simple ones.
• Avoid asking the model to repeat input back.

Lever 6: Application-level caching

If many users ask the same question ("what are OPD timings on Sunday?"), cache the final answer and skip the model entirely. Exact-match caching is safe. Semantic caching (reusing answers for similar questions) saves more but can return a wrong answer for a question that is similar but different, so use it only for low-risk, non-personal content and expire entries when the underlying data changes.

Lever 7: Self-host where volume justifies it

For steady, high-volume narrow tasks, an open-weight model on your own GPU or a managed open-model host can be cheaper per token than any API. The trade-off is operations: capacity planning, upgrades, monitoring and security. It also helps with data residency, which matters for hospitals and banks in India.

Putting it together: a worked example

Consider a support assistant for a clinic chain that answers patient queries on WhatsApp and drafts replies for staff. Before optimisation, every message goes to a large model with a long system prompt, the full conversation and 15 retrieved chunks.

A typical optimisation pass:

1. Restructure the prompt so instructions, examples and policy text form a stable, cached prefix.
2. Add an exact-match cache for the 50 most common questions (timings, locations, fees).
3. Route greetings, confirmations and simple FAQs to a small model; keep the large model for complaints and complex queries.
4. Add reranking and cut retrieved chunks from 15 to 5.
5. Summarise conversations after 10 turns.
6. Move the nightly summary of all conversations for managers to the batch API.

Each step is checked against the evaluation suite before release. The result is usually a much lower cost per conversation with the same or better quality, because shorter, cleaner prompts also tend to produce better answers. For the WhatsApp side of this setup, see our guide to WhatsApp automation for hospitals and clinics.

Governance that keeps costs down

• Attribute every call to a feature and a customer or department.
• Set daily budgets and alerts per feature.
• Cap steps, tokens and time per agent task.
• Review the top 10 most expensive features monthly.
• Re-evaluate model choices when providers release new models or change prices; the cheapest adequate model changes often.

Mistakes that raise costs quietly

• Retries without limits. A failing tool call retried indefinitely inside an agent loop can burn more tokens in an hour than a feature normally uses in a week.
• Debug logging of full prompts to another model. Some teams summarise every conversation with a second model "for analytics" and double their spend without noticing.
• Using the largest model by default. New features often launch on the most capable model and are never revisited once they work.
• Ignoring output tokens. Verbose answers and long reasoning cost more per token than input. Ask for concise output where users do not need detail.
• Prompt changes that break caching. Adding a date or user name at the top of a prompt can wipe out cache savings overnight.

How to start

1. Add tracing with token and cost attribution per feature.
2. Find the three most expensive features by monthly spend.
3. For each, check cache hit rate, average input size and model choice.
4. Apply caching and context trimming first; they rarely affect quality.
5. Move non-urgent work to batch.
6. Introduce routing last, with careful evaluation, since it is the lever most likely to affect quality.

Algon's LLM development and AWS and cloud teams run this kind of cost review on existing AI products, starting from the tracing data and ending with changes verified against your evaluation suite.

Conclusion

LLM costs are mostly a design problem, not a pricing problem. Stable prompt prefixes make caching work, patience makes batching work, and knowing which steps need a large model makes routing work.

Measure cost per successful task, apply the safe levers first, and keep your evaluation suite running so savings never come at the expense of quality.