Why LLM API Costs Surprise Mid-Market Teams
LLM API costs at development scale are negligible — a developer running 1,000 test requests per day spends a few dollars. The same application serving 50,000 daily active users with even moderate AI feature usage can easily reach $20,000-$50,000 per month in API costs.
The surprise is predictable: teams build features at development scale where cost is invisible, ship to production, watch adoption grow, and encounter costs that were not in the budget. The moment of discovery is usually a billing alert or a CFO question at the end of the month.
This post covers a four-lever framework for LLM API cost optimization in production — applied after features are live, not before they are built.
The Four-Lever LLM Cost Framework
LLM API cost has four controllable components: input token volume, output token volume, model selection, and request volume. Each lever provides independent cost reduction without requiring the others.
Lever 1: Input Token Optimization
Input tokens (the content you send to the LLM, including system prompt, user message, and retrieved context) typically represent 60-80% of total token cost for RAG-based applications. They are the highest-leverage optimization target.
System prompt audit. Review your system prompts for redundancy and verbosity. Most system prompts accumulate instructions over time as edge cases are addressed. A prompt that has grown to 800 tokens through iterative additions often contains 200-300 tokens of redundant or superseded instructions. A structured audit of system prompts typically finds 20-35% reduction without any change in behavior.
Context compression for RAG. In RAG systems, the retrieved context is often the largest component of the input. Two optimizations reduce context token cost without degrading retrieval quality: - Chunk size tuning: Smaller chunks reduce the context tokens per retrieval while increasing the number of retrieved chunks needed for equivalent coverage. Find the chunk size that balances retrieval precision against total context tokens. - Context relevance filtering: After retrieval, run a lightweight relevance filter that drops retrieved chunks below a similarity threshold. A retrieved chunk with 0.6 cosine similarity to the query adds tokens without meaningful information.
Conversation history management. Multi-turn conversational features accumulate history in the context window. Full conversation history sent on every turn grows unboundedly. Implement a conversation compression strategy: summarize older turns (beyond the last 3-5 exchanges) using a cheap, fast model, and include the summary rather than the full history.
Lever 2: Output Token Optimization
Output tokens are typically 15-30% of total token cost but are harder to optimize because they represent the value the model is generating.
Max token calibration. Set max_tokens at the 95th percentile of your observed output length for each prompt, not at an arbitrary large value. A summarization prompt that produces outputs of 200-400 tokens in practice should have max_tokens set to 450-500, not 2,000. Unused token capacity does not cost money, but it is a signal that the output length control is not being managed deliberately.
Structured output enforcement. For classification, extraction, and structured generation tasks, use JSON mode or structured output features (available on Claude and OpenAI APIs) to constrain the output to the required fields only. A classification task that only needs a label and a confidence score should return a 30-token JSON object, not a 200-token explanation followed by the label.
Output length instructions. Explicit instructions about output length are effective: "Respond in 2-3 sentences" or "Return a JSON object with exactly these fields" consistently produce shorter, denser outputs than unconstrained generation. These instructions are low-cost to add and produce 20-40% output token reduction for tasks where the instruction is appropriate.
Lever 3: Model Selection and Routing
Different LLM models have dramatically different cost-per-token profiles. As of 2026, the cost difference between frontier models and efficient mid-tier models is roughly 10-30x per token.
Task-appropriate model selection. Not all tasks require frontier model capability. Classify your AI features by reasoning complexity: - Simple classification, extraction, formatting: mid-tier models (Claude Haiku, GPT-4o-mini) are appropriate and cost 10-20x less per token than frontier models - Standard content generation, summarization, Q&A: mid-tier models with strong context handling perform well - Complex reasoning, multi-step analysis, nuanced judgment: frontier models (Claude Sonnet/Opus, GPT-4o) are justified
Run a structured evaluation comparing output quality between a mid-tier and frontier model on your specific task using your evaluation test set. In our experience, 60-70% of production LLM tasks at mid-market companies can move to a mid-tier model with acceptable quality loss or no measurable quality loss.
Prompt routing. For features that receive diverse input complexity, implement prompt routing: a lightweight classifier (using an LLM call on a fast cheap model, or a traditional ML classifier) that routes simple requests to a cheap model and complex requests to an expensive model. A customer support feature where 70% of queries are simple factual lookups and 30% require nuanced multi-step reasoning can route the 70% to a 10x cheaper model, yielding a blended cost reduction of 60-65%.
Lever 4: Semantic Caching
Many LLM applications receive semantically similar queries repeatedly. A FAQ chatbot, a search assist feature, or an internal knowledge base receives clusters of related questions. If two questions are semantically equivalent, the response to the first can serve the second without a new LLM call.
Semantic caching stores recent prompt-response pairs alongside their embeddings. Incoming queries are compared against cached entries by cosine similarity. If a cached entry with similarity above a defined threshold exists (typically 0.95+ for tight caching, 0.90 for broader caching), the cached response is returned.
When semantic caching is appropriate: - Features with repetitive query patterns (FAQs, knowledge bases, product documentation) - Features where the LLM response does not need to reflect real-time information - High-volume features where even a 20-30% cache hit rate produces significant cost reduction
When semantic caching is not appropriate: - Features requiring personalized or context-dependent responses - Features requiring real-time information - Conversational features where query context changes the appropriate response
GPTCache (open-source) and several managed semantic caching services integrate with the major LLM provider APIs. For mid-market teams, an in-process semantic cache using a vector database you already operate is often the simplest implementation.
Establishing Cost Governance Before Problems Arise
The levers above apply after problems are discovered. The governance practices below prevent budget surprises:
Per-feature cost tracking. Configure cost attribution at the feature level, not just at the API account level. Use tags in LLM API calls (or a thin wrapper that adds cost tracking) to attribute cost to each feature. Knowing that feature A costs $3,000/month and feature B costs $800/month is more actionable than knowing total API costs are $3,800/month.
Cost-per-user metric. Track LLM API cost per monthly active user as a product metric. Set a target cost-per-user budget per feature during design. If a feature is designed to generate $0.50 LLM API cost per user per month, alert when it exceeds $0.65.
Model pricing change monitoring. LLM provider pricing changes. Anthropic, OpenAI, and Google have adjusted model pricing multiple times since 2023. Subscribe to provider changelogs and update your cost models when pricing changes occur. A pricing change that reduces your costs by 30% may justify reconsidering a self-hosting decision you made when prices were higher.
Frequently Asked Questions
What is the typical LLM API cost per monthly active user for a B2B SaaS product? For a B2B SaaS product with 1-2 AI features of moderate usage (one AI-assisted action per user session), we see $0.05-$0.25 per monthly active user per month using mid-tier models for appropriate tasks. Products where AI is the core interaction model run $0.50-$2.00 per user per month.
Does using smaller models meaningfully degrade user-perceived quality? For most production use cases we evaluate, users cannot distinguish between frontier and mid-tier model quality for standard content generation and Q&A tasks. The quality gap is observable in complex reasoning tasks and in head-to-head blind comparisons by expert reviewers. Run your own evaluation on your specific task rather than relying on model benchmarks.
Is it worth building semantic caching infrastructure for $2,000/month in LLM costs? At $2,000/month, a 25% cache hit rate saves $500/month — $6,000 per year. If semantic caching implementation requires one engineer-week and minimal ongoing maintenance, the payback period is 1-2 months. The economics are favorable at this scale if your feature has the repetitive query patterns that semantic caching benefits from.
How do we handle LLM provider cost increases? Maintain a model abstraction layer in your application code so that changing the underlying model provider does not require changes throughout the codebase. A thin wrapper that accepts model-agnostic parameters and translates them to provider-specific API calls reduces switching costs significantly when provider pricing or capability changes make switching attractive.
If you want a structured review of your current LLM API cost baseline and a prioritized optimization plan, we offer a free AI cost review for mid-market engineering teams.
