How to Build an LLM Evaluation Harness

Why you need one

LLM output is non-deterministic, so "it looked fine when I tried it" isn't a test. The moment you change a prompt or swap a model, quality can silently regress. An eval harness turns "looks fine" into a measurable score you can track and gate releases on.

1. A golden set

A list of representative cases with an expected signal. The simplest useful version is (question, expected_keywords):

GOLDEN = [
    {"q": "How do I reset my password?",
     "expected": ["reset", "email", "link"]},
    {"q": "What are your business hours?",
     "expected": ["9", "5", "monday"]},
]

Start with 10-20 real cases (including known failure modes). It grows every time a bug ships - add the case that broke.

2. A scoring function

Turn one response into a number in [0, 1]. Keyword coverage is a cheap, deterministic start:

def score(response: str, expected_keywords: list[str]) -> float:
    if not expected_keywords:
        return 0.0
    low = response.lower()
    hits = sum(1 for k in expected_keywords if k.lower() in low)
    return hits / len(expected_keywords)

Later you can add stronger scorers (regex, JSON-schema validity, or LLM-as-judge for open-ended answers) - the harness shape stays the same.

3. A runner that aggregates

Run every case, score it, and report the aggregate so you get one number to watch:

def run(chat_fn) -> float:
    scores = [score(chat_fn(c["q"]), c["expected"]) for c in GOLDEN]
    avg = sum(scores) / len(scores)
    print(f"avg score: {avg:.2f} over {len(scores)} cases")
    return avg

How to actually use it

• Gate changes: run the harness before/after a prompt or model change; block the change if the score drops.
• In CI: fail the build if the average falls below a threshold.
• Grow the set: every production miss becomes a new golden case, so the same bug can never ship twice.