Semantic Search Design: When to Compute Embeddings

The cost that matters: the embedding call

Semantic search turns text into vectors and finds the documents whose vectors are closest to the query vector. Comparing vectors is cheap. Calling the embedding model is expensive - often tens to hundreds of milliseconds per call, plus network time if the model runs behind an API.

So the design question is not "which similarity function?" It is "how many embedding calls happen on each search?" Take an example workload:

• 5,000 documents that change about once a day
• 100 queries per second
• an embedding call that takes about 150ms
• a latency budget of 50ms for a repeated query

Approach A: embed everything at query time

def search(corpus, query, k=3):
    qv = embed(query)
    scored = []
    for doc in corpus:
        dv = embed(doc)                 # re-embeds every document, every search
        scored.append((dot(qv, dv), doc))
    scored.sort(reverse=True)
    return [doc for _, doc in scored[:k]]

This returns correct results, and it is the simplest code to write. But each search makes one embedding call per document plus one for the query. With 5,000 documents, a single search makes 5,001 model calls. It cannot meet any real latency budget.

Approach B: embed documents once, embed the query every time

_doc_vectors = []

def init(corpus):
    global _doc_vectors
    _doc_vectors = [(embed(d), d) for d in corpus]   # once, at write time

def search(corpus, query, k=3):
    qv = embed(query)                                 # still one call per search
    scored = sorted(((dot(qv, dv), d) for dv, d in _doc_vectors), reverse=True)
    return [d for _, d in scored[:k]]

This is the standard shape of a vector index: documents are embedded when they are added, and the stored vectors are reused. It removes the biggest cost. But each search still waits for one embedding call, so every query takes at least 150ms. At 100 queries per second, that is 15 seconds of model time every second - one host cannot keep up.

Approach C: embed at write time and cache query embeddings

from functools import lru_cache

@lru_cache(maxsize=1024)
def cached_embed(text: str) -> tuple[float, ...]:
    return tuple(embed(text))

def init(corpus):
    global _doc_vectors
    _doc_vectors = [(cached_embed(d), d) for d in corpus]

def search(corpus, query, k=3):
    qv = cached_embed(query)            # cache hit = no model call
    scored = sorted(((dot(qv, dv), d) for dv, d in _doc_vectors), reverse=True)
    return [d for _, d in scored[:k]]

Real search traffic repeats itself: many users ask the same or very similar questions. With a query cache, a repeated query skips the model completely, and the hot path is only the vector comparison - well under a millisecond for a few thousand short vectors. A new query still pays the embedding cost once, and then it is cached.

Note that lru_cache needs a hashable return value, so the vector is stored as a tuple. A cache keyed on the exact text only helps exact repeats. Normalize the query first (trim spaces, lowercase) to raise the hit rate.

Measure the hot path, do not guess

Write a small benchmark that runs the same query twice and times the second run:

import time
init(corpus)
search(corpus, "reset my password")          # warm up
start = time.perf_counter()
results = search(corpus, "reset my password")
print(f"{(time.perf_counter() - start) * 1000:.2f} ms", results)

Run it against each design. The numbers make the choice obvious, and they give you something concrete to put in a design review.

How the design scales

The contract stays the same as the system grows: init(corpus) builds the index, and search(corpus, query, k) reads it. Behind that contract you can change the parts:

• More documents - replace the Python loop with a vector index such as FAISS, pgvector, or a hosted vector database. They use approximate nearest-neighbor search, so a search does not compare against every vector.
• Documents that change - embed each new or edited document when it is saved, instead of rebuilding the whole index.
• Several app servers - move the query cache to Redis, keyed on a hash of the normalized query and the embedding model name, so all servers share it.
• A new embedding model - re-embed all documents. Vectors from different models cannot be compared with each other.