Why Your S3 Cache Never Hits: Debugging boto3 NoSuchBucket

The symptom: X-Cache is always "miss"

A Flask service stores LLM answers in S3, one object per question. A repeated question should come from the cache, but even two identical requests in a row both miss:

curl -si -X POST localhost:8000/ask -H 'Content-Type: application/json' \
  -d '{"query":"hi"}' | grep -i x-cache
# X-Cache: miss
curl -si -X POST localhost:8000/ask -H 'Content-Type: application/json' \
  -d '{"query":"hi"}' | grep -i x-cache
# X-Cache: miss

The service still answers, so nothing looks broken. But every request pays for a full model call, and the cache does nothing.

Where the error hides

Here is the cache code:

BUCKET = "cache"

def cache_get(query: str) -> str | None:
    try:
        obj = s3.get_object(Bucket=BUCKET, Key=_key(query))
        return obj["Body"].read().decode()
    except (s3.exceptions.NoSuchKey, botocore.exceptions.ClientError):
        return None

def cache_put(query: str, answer: str) -> None:
    try:
        s3.put_object(Bucket=BUCKET, Key=_key(query), Body=answer.encode())
    except botocore.exceptions.ClientError as e:
        app.logger.warning(f"cache put failed: {e}")

Catching NoSuchKey is correct - a missing key is a normal cache miss. But ClientError is the base class for almost every S3 error: NoSuchBucket, AccessDenied, an expired token, a wrong region. Catching it in cache_get turns all of those into "miss". On the write side the error is only a warning in the log.

Find the real error

Compare the bucket that exists with the bucket the code uses, and read the log:

aws s3 ls
# 2026-10-26 10:00:00 llm-cache
grep -n BUCKET llm_cache.py
# 11:BUCKET = "cache"
tail -n 5 /tmp/app.log
# cache put failed: An error occurred (NoSuchBucket) when calling the
# PutObject operation: The specified bucket does not exist

The code writes to cache, but the bucket is called llm-cache. Point the code at the bucket that exists:

BUCKET = "llm-cache"

Do not create a new bucket to match the code. The real bucket usually has the right policy, encryption, and lifecycle rules, and a second bucket splits your data.

Confirm the cache works

Send a new question twice. The first request should miss and write an object, and the second should hit:

curl -si -X POST localhost:8000/ask -H 'Content-Type: application/json' \
  -d '{"query":"hello"}' | grep -i x-cache     # miss
curl -si -X POST localhost:8000/ask -H 'Content-Type: application/json' \
  -d '{"query":"hello"}' | grep -i x-cache     # hit
aws s3 ls s3://llm-cache/

Only catch the errors that mean "miss"

To make sure a configuration error cannot hide again, check the error code and only treat a missing key as a miss:

from botocore.exceptions import ClientError

def cache_get(query: str) -> str | None:
    try:
        obj = s3.get_object(Bucket=BUCKET, Key=_key(query))
        return obj["Body"].read().decode()
    except ClientError as e:
        code = e.response["Error"]["Code"]
        if code in ("NoSuchKey", "404"):
            return None
        app.logger.error("cache read failed (%s): %s", code, e)
        return None   # still serve the request, but log loudly

The request still works when S3 has a problem - a cache should never take the service down - but the logs now show a different message for "not cached yet" and "the cache is broken".

Check the bucket at startup

A cheap way to catch a wrong name early is to call head_bucket when the app starts:

try:
    s3.head_bucket(Bucket=BUCKET)
except ClientError as e:
    raise SystemExit(f"cache bucket {BUCKET!r} is not usable: {e}")

You can also read the name from an environment variable (BUCKET = os.environ["CACHE_BUCKET"]) set by the same Terraform or deploy config that creates the bucket, so the two names cannot drift apart.

How the cache key works

The service keys each object on a hash of the model name and the question:

def _key(query: str) -> str:
    return hashlib.sha256(f"{MODEL}::{query}".encode()).hexdigest()

Including the model name means a model change does not return answers from the old model. Add an S3 lifecycle rule to expire old objects, so the cache does not grow forever. Track the hit rate (hits divided by total requests) as a metric - a hit rate of zero is the clearest sign of the problem on this page.