RAG Reranking: How to Add a Second-Stage Reranker to Retrieval

Why retrieval order matters

When you pass retrieved passages to an LLM, it weights the top context most. If the most relevant document is ranked third, answer quality drops - even though it was fetched. Retrieval and ranking are two separate jobs:

• First stage (retriever) - fast, coarse; bi-encoders or TF-IDF scan millions of documents for recall. They bring the right answer somewhere in the top-k.
• Second stage (reranker) - slower, more precise; rescores only the shortlisted candidates with a richer (query, doc) relevance signal and reorders them.

The two-stage shape is the industry standard pattern because applying a heavy cross-encoder to the full corpus is too slow; running it on just the top-k is fast.

Implementing rerank()

Given a list of candidate passages from the retriever, score each one against the query and sort by score descending:

def relevance(query: str, doc: str) -> float:
    """Score one (query, doc) pair. Higher = more relevant."""
    q_terms = set(query.lower().split())
    d = doc.lower()
    # exact word match scores higher than substring match
    return sum(3 if t in d.split() else (1 if t in d else 0) for t in q_terms)


def rerank(query: str, candidates: list[str]) -> list[str]:
    """Return candidates sorted by relevance score, highest first."""
    return sorted(candidates, key=lambda doc: relevance(query, doc), reverse=True)

Then wire it into your search() function as a second stage:

def search(query: str, k: int = 2) -> list[str]:
    candidates = retrieve(query, k=10)   # broad first-stage fetch
    ranked = rerank(query, candidates)   # precision reorder
    return ranked[:k]                    # return top k

Fetching more candidates at the first stage (e.g. 10) and trimming after reranking (to 2) is the right shape - more recall, then higher precision.

Production rerankers

In production, the relevance() function is typically a cross-encoder - a model that reads both the query and the document together before scoring, which gives far better precision than a bi-encoder or lexical score:

• Cohere Rerank API - cloud, one API call for the whole shortlist
• BGE-Reranker / MiniLM cross-encoder - self-hosted via sentence-transformers
• ColBERT late-interaction models - good accuracy/speed tradeoff

All of them plug into the same rerank(query, candidates) slot; swapping the relevance function does not change the pipeline shape.

What to watch for

• Fetch more, rerank to fewer - first stage k should be 3-10x the final k so the reranker has a real shortlist to work with.
• Don't rerank the whole corpus - reranking is O(k), not O(N); keep first-stage retrieval as the broad filter.
• Score ties - use a stable sort (Python's sorted is stable) and consider a tiebreak on retriever rank to preserve original ordering within equal scores.