How to Power RAG Search With a Vector Database

The two-step retrieval loop

RAG (Retrieval-Augmented Generation) retrieval works the same way regardless of which vector store you use:

1. Index - call an embeddings model for each document and store the resulting vector alongside the document text.
2. Query - embed the incoming query with the same model, then ask the vector store for the top-k documents whose vectors are closest (by cosine similarity) to the query vector.

The LLM that generates the answer is downstream of this retrieval step - retrieval quality is what makes or breaks the feature.

Indexing documents into ChromaDB

import json, urllib.request, chromadb

EMBED_URL = "http://ollama:11434/api/embeddings"
EMBED_MODEL = "nomic-embed-text"

_client = chromadb.EphemeralClient()
_collection = _client.get_or_create_collection("docs")

def _embed(text: str) -> list[float]:
    req = urllib.request.Request(
        EMBED_URL,
        data=json.dumps({"model": EMBED_MODEL, "prompt": text}).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["embedding"]

def index_docs(docs: list[str]) -> None:
    for i, doc in enumerate(docs):
        vec = _embed(doc)
        _collection.add(ids=[str(i)], embeddings=[vec], documents=[doc])

chromadb.EphemeralClient() keeps the index in memory - swap it for chromadb.PersistentClient(path="/data/chroma") when you need the index to survive a restart. For production scale, the same pattern works with OpenSearch, pgvector, or Pinecone - only the client and add/query calls change.

Querying for the top-k results

def search(query: str, k: int = 3) -> list[str]:
    qv = _embed(query)
    res = _collection.query(query_embeddings=[qv], n_results=k)
    return res["documents"][0]   # list of doc strings, ranked by similarity

collection.query returns results ranked by cosine distance - the closest vectors first. res["documents"][0] is the ranked list for the first (and only) query vector.

Putting it together

DOCS = [
    "Kubernetes orchestrates containers across a cluster of nodes.",
    "PostgreSQL is a powerful open-source relational database engine.",
    "Redis is an in-memory key-value store often used for caching.",
]

index_docs(DOCS)

for q in ("container orchestration", "in-memory caching"):
    print(f"Q: {q}")
    for doc in search(q):
        print(f"  -> {doc}")

The query "in-memory caching" surfaces the Redis doc first, even though the word "caching" doesn't appear in every candidate - that's semantic similarity at work.

What to tune in production

• Chunk size - long documents should be split (512-1024 tokens per chunk) so retrieved passages are actually relevant to the query.
• Hybrid search - combine vector search with BM25 keyword search and merge the ranked lists (reciprocal rank fusion) to catch exact-match terms that embeddings miss.
• Re-ranking - pass the top-20 candidates through a cross-encoder to reorder them before sending the top-3 to the LLM.
• Eval - measure hit rate at k and mean reciprocal rank against a labeled query set so retrieval regressions are caught before they affect answer quality.