How to Cache LLM Responses in Redis

Why cache LLM calls

Identical prompts produce (effectively) identical answers, but each call costs tokens and seconds. For an FAQ or Q&A endpoint where users repeat questions, caching turns a repeat question into an instant, free lookup. This is cache-aside: check cache, miss -> call LLM -> store.

Hash the prompt into a key

Use a hash of the normalized prompt so the key is fixed-length and identical inputs map to the same key:

import hashlib

def cache_key(question: str) -> str:
    norm = question.strip().lower()
    return "llm:" + hashlib.sha256(norm.encode()).hexdigest()

Cache-aside with a TTL

import redis, json
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
TTL = 300   # 5 minutes

def ask(question: str) -> str:
    key = cache_key(question)
    hit = r.get(key)
    if hit is not None:
        return json.loads(hit)          # cache hit - no LLM call

    answer = call_llm(question)          # miss - do the work
    r.setex(key, TTL, json.dumps(answer))
    return answer

First request for a question hits the LLM and caches; repeats within the TTL are served from Redis.

The details that matter

• Normalize before hashing (trim, lowercase) so "Hello" and "hello " share a cache entry
• tune this to how much you want near-duplicate questions to collide.
• TTL balances freshness vs hit rate - short if answers can change, long for stable FAQs. Use setex so entries expire on their own.
• Include model + params in the key if you serve multiple models or temperatures, so a cached answer from one config isn't returned for another.
• Don't cache errors - only store successful responses, or you'll serve a failure for the whole TTL.
• For fuzzy matching of similar (not identical) questions, use a semantic cache (embed the question, match by vector similarity) instead of an exact hash.