How to Reduce OpenAI API Costs With a Redis Cache

Why LLM costs pile up fast

OpenAI charges per token, and real apps repeat the same prompts constantly. A customer-support bot gets asked "what's your refund policy?" dozens of times per hour. A documentation assistant re-embeds the same query. Each call pays the same price whether it's the first or the five-hundredth time.

The fix is a prompt-level cache: before calling the LLM, check Redis for a stored response keyed on the prompt. Return it directly on a hit. Call the LLM only on a miss, then store the result for next time.

The cache-aside pattern

import redis
import requests

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
LLM_URL = "https://api.openai.com/v1/chat/completions"

def ask(prompt: str) -> str:
    key = f"llm:{prompt}"

    cached = r.get(key)
    if cached:
        return cached                   # cache hit - no LLM call

    response = requests.post(
        LLM_URL,
        headers={"Authorization": f"Bearer {OPENAI_API_KEY}"},
        json={
            "model": "gpt-4o-mini",
            "messages": [{"role": "user", "content": prompt}],
        },
        timeout=60,
    )
    response.raise_for_status()
    answer = response.json()["choices"][0]["message"]["content"]

    r.set(key, answer, ex=86400)        # store with 24h TTL
    return answer

r.get(key) returns None on a miss, so the check is a single condition. On a miss, r.set(key, answer, ex=86400) stores the response for 24 hours - after which Redis evicts it automatically, so stale model answers don't linger forever.

Picking the right TTL

• FAQ / static content - 24-48h is safe; the answers rarely change.
• Dynamic queries - short TTLs (minutes) or skip caching entirely.
• Personalised responses - include a user-scoping prefix in the key (llm:{user_id}:{prompt}) or don't cache at all.

Measuring the hit rate

Track a counter per request to see how often the cache is saving you a call:

llm_calls = 0

def ask(prompt: str) -> str:
    key = f"llm:{prompt}"
    cached = r.get(key)
    if cached:
        return cached
    global llm_calls
    llm_calls += 1
    # ... LLM call and store ...

Expose it on a /metrics or /counter endpoint. If hit rate is low, the prompts are too varied - semantic caching (embed the prompt, look up the nearest stored vector) handles that case at the cost of more infrastructure.

Going further

Exact-string keys are a good start but won't catch near-duplicates ("what's your return policy?" vs "what is your return policy?"). Production systems often layer:

1. Exact match in Redis - zero-latency, free.
2. Semantic similarity lookup - embed the prompt, search a vector store for a cached embedding within a cosine-similarity threshold.
3. Provider-level prompt caching - OpenAI and Anthropic both cache long system prompts at the API layer automatically, reducing token cost for the static prefix of every request.

Start with exact-match Redis caching - it is free, simple, and often sufficient for high-traffic question-answering endpoints.