Cutting LLM API Costs by 70%: Caching, Model Routing, and Prompt Compression

Your LLM bill hit $12,000 last month. This guide shows how to cut it to $3,600 without degrading quality. You’re not alone—GPT-4o alone processes 1 trillion tokens monthly (OpenAI DevDay 2025), and that scale means even minor inefficiencies bleed real money. The brute-force approach—throwing every query at your most expensive model—is a financial aneurysm. We’ll stop the bleeding with three surgical techniques: intelligent model routing, semantic caching, and aggressive prompt compression. This isn't theoretical; it's the production playbook for keeping your CFO from asking why your "chat feature" costs more than your entire data center.

Profiling Your LLM Spend: Where Do Your Tokens Actually Go?

Before you optimize a single line of code, you need to know what you're paying for. Your $12,000 bill is a black box from OpenAI. The first step is to instrument your application to break down costs by user, feature, and query type. You'll likely find a Pareto distribution: 20% of your query types (complex analysis, code generation) consume 80% of your budget, while simple paraphrasing, formatting, and classification tasks burn cash on GPT-4o when a cheaper model would suffice.

You need observability. Don't just use the vanilla OpenAI SDK; wrap it with a tool like Helicone or LiteLLM to get granular logging. Here’s a quick setup with LiteLLM that logs cost per call:

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-key"


response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize the latest quarterly report."}],
)

# You can also use it for routing and cost tracking
def track_cost(model, messages):
    resp = completion(model=model, messages=messages)
    # Cost is automatically calculated and can be exported
    print(f"Model: {model}, Cost: ${resp._response_ms['cost']}")
    return resp

The insight you're looking for? The distribution between "simple" and "complex" tasks. If 60% of your queries are "rewrite this sentence" or "categorize this ticket," you're hemorrhaging money. The average LLM API cost per 1M tokens is GPT-4o $5, Claude 3.5 Sonnet $3, Gemini 1.5 Pro $3.50 (as of Jan 2026). Routing that 60% to a cheaper model like gpt-4o-mini (at a fraction of the cost) is your first major win.

Model Routing: Send the Right Query to the Right (Cheap) Model

Model routing is the practice of classifying an incoming query and directing it to the most cost-effective model that can handle it with sufficient quality. You don't need a 7-year-old to solve 2+2, and you don't need GPT-4o to fix a comma.

The strategy is simple:

1. Classify the query intent. Is it simple (grammar, formatting, basic Q&A), moderate (multi-step reasoning, brief analysis), or complex (creative generation, advanced code, nuanced reasoning)?
2. Route accordingly.
   • Simple: gpt-4o-mini, claude-3-haiku
   • Moderate: claude-3.5-sonnet, gpt-4o
   • Complex: gpt-4o, claude-3-opus

Implement a lightweight classifier. You can even use a cheap model to classify queries for you—a meta-optimization.

from openai import OpenAI
client = OpenAI()

def route_query(user_query: str) -> str:
    """Classify query and return recommended model name."""
    classification_prompt = f"""
    Categorize this user query for the purpose of choosing a cost-effective LLM.
    Categories: 'simple' (spelling, formatting, simple fact), 'moderate' (multi-step instruction, basic analysis), 'complex' (creative, advanced reasoning, code generation).
    Query: {user_query}
    Return ONLY the category word.
    """
    
    # Use a very cheap model for the classification itself
    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",  # Cheap classifier
            messages=[{"role": "user", "content": classification_prompt}],
            max_tokens=10,
        )
        category = response.choices[0].message.content.strip().lower()
    except Exception as e:
        # Fallback to a moderate model if classification fails
        category = "moderate"
    
    # Route based on category
    route_map = {
        "simple": "gpt-4o-mini",       # ~10x cheaper than GPT-4o
        "moderate": "claude-3-5-sonnet-20241022", # Excellent price/performance
        "complex": "gpt-4o"            # Bring out the big guns only when needed
    }
    return route_map.get(category, "gpt-4o")  # Default to GPT-4o

# Example usage
user_ask = "Capitalize the titles in this blog post draft."
model_to_use = route_query(user_ask)  # Will likely return 'gpt-4o-mini'
print(f"Routing to: {model_to_use}")

Real Error & Fix: You will hit rate limits when you centralize traffic to a single model.

• Error: RateLimitError: Rate limit reached for gpt-4o in organization org-xxx on requests per min. Limit: 10000 / min.
• Fix: Implement exponential backoff with a library like tenacity, use tier-appropriate rate limits, and use your routing function as an overflow valve. If GPT-4o is saturated, route some 'moderate' queries to Claude 3.5 Sonnet instead.

Semantic Caching: Never Pay for the Same Answer Twice

A shocking amount of production traffic is redundant. Users ask the same questions. Systems generate the same summaries. Your embedding model is cheaper than your LLM. Semantic caching computes the embedding of a new query, checks for similar cached queries (using cosine similarity), and returns the cached LLM response if a match is found. Hit rates of 40–60% on real production traffic are common, which directly translates to a 40–60% reduction in calls to paid APIs.

Here’s a basic implementation using LangChain and OpenAI's embeddings:

from langchain.globals import set_llm_cache
from langchain.cache import SQLiteCache, SemanticCache
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
import numpy as np

# Use a semantic cache backed by embeddings
set_llm_cache(SemanticCache(
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"), # Cheap embeddings
    database_path=".semantic_cache.db"
))

llm = ChatOpenAI(model="gpt-4o")

# First call - goes to API, caches result
result1 = llm.invoke("What is the capital of France?")
print(f"First call result: {result1.content}")

# Second, semantically similar call - RETURNS FROM CACHE
result2 = llm.invoke("Can you tell me the French capital city?")
print(f"Second call (cached): {result2.content}")
print(f"Is this from cache? {result2.response_metadata.get('cached', False)}")

The key is setting the right similarity threshold—too strict, and you miss cache hits; too loose, and you return incorrect information. Start with a threshold of ~0.92 (cosine similarity) for factual queries and adjust based on your domain.

Prompt Compression with LLMLingua: Shrink Your Context, Keep the Signal

You're paying by the token. Long context windows are a budget killer. Processing 128K tokens with GPT-4o costs $0.64 vs $0.02 for 4K—that's 32x more expensive. The problem is that most of those tokens are filler: stop words, redundant phrases, and verbose formatting. Prompt compression techniques like LLMLingua or Selective Context identify and remove non-essential tokens before sending the prompt to the LLM, with minimal accuracy loss.

Think of it as a lossless compression algorithm for your prompts. It works especially well for RAG applications where you're stuffing a context window with retrieved documents.

Real Error & Fix: The most common error when managing context is exceeding the limit.

• Error: ContextWindowExceededError: max tokens 128000 but got 145000.
• Fix: Chunk documents before embedding, use a map-reduce chain for summarization, and implement a conversation memory system that trims or summarizes old messages instead of blindly appending them.

Anthropic Prompt Caching: The 75% Discount on System Prompts

If you use Anthropic's Claude API, you have a nuclear option for cost reduction: Prompt Caching. Many applications use long, static system prompts (instructions, personas, context). Anthropic allows you to cache these prompts on their servers, assigning a cache_control ID. On subsequent calls, you reference the ID, and you are only billed for the tokens in the new user message plus the output. For a 1K token system prompt used repeatedly, this can reduce prompt costs by ~75%.

import anthropic

client = anthropic.Anthropic()

# First, create a cached system prompt
cache_creation_response = client.beta.prompt_caching.create(
    prompt="You are a expert financial analyst with 20 years of experience. Always respond in bullet points. Never speculate beyond the provided data.",
    name="financial_analyst_system_v1"
)
cache_id = cache_creation_response.id  # e.g., 'cache_prompt_abc123'

# Now, use the cached prompt in your requests. You are NOT billed for its tokens.
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1000,
    system=[{"type": "text", "cache_control": {"type": "ephemeral", "id": cache_id}}], # Key line
    messages=[
        {"role": "user", "content": "Analyze the attached Q4 earnings statement."}
    ]
)
# You are billed only for the user message and Claude's output.

This is a massive, often overlooked, advantage of the Anthropic API for production systems with stable system instructions.

Monitoring: Cost Per User, Feature, and Budget Autopsies

Optimization is pointless without measurement. You need to move from a single "monthly API bill" to granular metrics:

• Cost Per User (CPU): Identify power users or potential abuse.
• Cost Per Feature: Is your new "document summarizer" feature profitable?
• Budget Alerts: Get Slack alerts when your daily spend exceeds a threshold.

Tools like Helicone, Arize, or LangSmith are built for this. Set up dashboards that track not just cost, but also latency, error rates, and cache hit ratios. Correlate cost spikes with deployment events. Did your new prompt template accidentally double the context length?

Next Steps: Implementing Your Cost Reduction Sprint

Start today. Don't try to boil the ocean.

1. Week 1: Instrumentation. Wrap your LLM calls with LiteLLM or Helicone. Profile for one week. Find your top 3 expensive query patterns.
2. Week 2: Implement Routing. Build and deploy the classification-based router. Start by routing only obvious "simple" queries to cheaper models. Monitor quality closely (use human eval or LLM-as-a-judge).
3. Week 3: Roll Out Caching. Implement semantic caching for a single, high-volume endpoint. Measure the hit rate. A 40% hit rate on a $3,000/month endpoint saves $1,200 immediately.
4. Week 4: Compress and Specialize. Apply prompt compression to your most expensive RAG pipelines. If you use Claude, migrate system prompts to use prompt caching.

The goal isn't just to cut costs; it's to build a sustainable, efficient LLM infrastructure that lets you scale features without fearing an exponential bill. That $12,000 bill can become a $3,600 bill, and the $8,400 you save? That's your budget for fine-tuning a specialized model that makes your product truly unique. Stop burning cash on redundancy and start investing in differentiation.