How to Cache Embeddings in AWS DynamoDB With boto3

Why cache embeddings

An embedding for a piece of text never changes as long as you use the same model. Yet many services compute it again every time the text comes in - the same product description, the same support question, the same document chunk. Each call costs model time and, with a hosted API, money.

A cache in DynamoDB keeps each embedding after the first call. It survives restarts, every app server shares it, and you do not run a cache server yourself.

The table

One table with a string partition key is enough:

aws dynamodb create-table \
  --table-name embedding-cache \
  --attribute-definitions AttributeName=text_hash,AttributeType=S \
  --key-schema AttributeName=text_hash,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

The key is a SHA-256 hash of the text, not the text itself. A hash has a fixed length, no character problems, and it stays far below the DynamoDB key size limit even for long documents.

The cache-aside function

import hashlib
import boto3

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("embedding-cache")

def get_embedding(text: str) -> list[float]:
    key = hashlib.sha256(text.encode()).hexdigest()

    item = table.get_item(Key={"text_hash": key}).get("Item")
    if item is not None:
        return [float(x) for x in item["embedding"]]        # hit

    emb = compute_embedding(text)                            # miss: call the model
    table.put_item(Item={
        "text_hash": key,
        "embedding": [str(x) for x in emb],
    })
    return emb

The order is the whole pattern: read the cache, return on a hit, and only on a miss call the model and write the result back. The next call with the same text returns from DynamoDB without touching the model.

The float trap

If you store the vector as a normal list of floats, boto3 refuses:

TypeError: Float types are not supported. Use Decimal types instead.

The DynamoDB resource API does not accept Python float, because floats can lose precision. You have two working options:

from decimal import Decimal

# Option 1: Decimal values (stored as a DynamoDB number list)
{"embedding": [Decimal(str(x)) for x in emb]}

# Option 2: string values (stored as a list of strings)
{"embedding": [str(x) for x in emb]}

Use Decimal(str(x)), not Decimal(x) - converting the float directly carries unwanted binary digits. Either way, convert back with float(x) when you read, because DynamoDB returns Decimal or str values and most vector code expects floats.

A third option for large vectors is to pack them into bytes and store a single binary attribute:

import struct
blob = struct.pack(f"{len(emb)}f", *emb)                     # write
vec = list(struct.unpack(f"{len(blob) // 4}f", item["embedding"].value))   # read

This is much smaller than a list of numbers, which matters because one DynamoDB item can be at most 400 KB.

Check that it works

Call the function twice with the same text and look at the table:

python3 -c "from embed import get_embedding as g; g('hello'); g('hello')"
aws dynamodb scan --table-name embedding-cache --select COUNT

The first call is a miss and writes one item. The second is a hit: it returns the same numbers, and the model is not called. A simple way to prove the second part in a test is to replace compute_embedding with a function that raises, then call get_embedding again with the cached text.

Production notes

• Put the model name in the key - for example sha256(f"{model}::{text}"). A new model creates new vectors, and you never want to mix them with old ones.
• Add a TTL attribute if old entries should expire. DynamoDB deletes expired items for you at no extra cost.
• Use batch_get_item when you embed many texts at once, to read up to 100 keys in one request instead of one by one.
• Keep the cache optional. If DynamoDB is slow or down, log the error and compute the embedding anyway. A cache should make the service faster, never unavailable.