How to Reduce LLM Moderation False Positives

Why the moderator flags everything

An LLM moderation step usually has one prompt and one rule: reply APPROPRIATE or INAPPROPRIATE. When every comment comes back flagged, people often blame the model. In most cases the prompt asks for it. A typical over-strict prompt looks like this:

You are a strict content moderator. Flag ANY content that could be considered
negative, controversial or informal, including:
- Negative sentiment or complaints
- Opinions or subjective statements
- Informal language or slang
- Topics like food, health or politics
- Constructive criticism
Respond with ONLY one word: APPROPRIATE or INAPPROPRIATE

A good model obeys each line. "I disagree with this approach" is an opinion, so it is flagged. "This recipe was too salty" is negative and about food, so it is flagged. The result is a moderator that blocks normal users and teaches them to stop commenting.

Measure before you change the prompt

Collect a small labeled set of comments - some clearly fine, some clearly harmful - and score the moderator against it. Ten to fifty examples are enough to see the problem:

import json

tests = json.load(open("test_data.json"))   # [{"comment": ..., "expected": ...}]
correct = 0
for t in tests:
    label = moderate_comment(t["comment"])
    ok = label == t["expected"]
    correct += ok
    if not ok:
        print(f"WRONG: {t['comment']!r} -> {label} (expected {t['expected']})")
print(f"Accuracy: {correct}/{len(tests)} ({100 * correct // len(tests)}%)")

If almost every "WRONG" line is a harmless comment marked INAPPROPRIATE, you have a false-positive problem, and the prompt is the place to work.

Rewrite the rules around real harm

A better prompt does three things:

1. It lists only the content you really want to block - threats of violence, personal attacks and harassment, hate speech or slurs.
2. It says clearly what is allowed - opinions, criticism, disagreement, casual language and everyday topics.
3. It removes every blanket rule from the old prompt. If a word like "opinions" or "informal" stays in the prompt, even inside a sentence that allows it, the model can still read it as a reason to flag.

MODERATION_PROMPT = """You are a content moderator. Mark a comment as INAPPROPRIATE
only if it contains:
- Threats of violence or harm
- Personal attacks, insults, or harassment
- Hate speech or slurs

All other comments are APPROPRIATE. People may disagree, complain, criticize, use
casual words, or talk about any normal subject.

Examples:
Comment: "Great article, thanks for sharing!"
Answer: APPROPRIATE

Comment: "I disagree with this approach but respect your view"
Answer: APPROPRIATE

Comment: "You are an absolute idiot and should be banned"
Answer: INAPPROPRIATE

Comment: "I will find where you live and hurt you"
Answer: INAPPROPRIATE

Respond with ONLY one word: APPROPRIATE or INAPPROPRIATE"""

Why few-shot examples help

Rules describe the boundary in words. Examples show the boundary. A model that sees a rude-but-allowed comment next to a real insult learns where the line is much faster than from a list of adjectives. Good few-shot examples:

• Cover both labels - at least one APPROPRIATE and one INAPPROPRIATE.
• Include the hard cases - criticism, disagreement, and strong but harmless language.
• Use the exact output format you want, so the model copies it.
• Do not reuse comments from your test set, or your accuracy number will look better than it really is.

Keep the output easy to parse

The last line of the prompt matters: one word, one of two labels. Then parse defensively, because models sometimes add punctuation or a short sentence:

def moderate_comment(comment: str) -> str:
    reply = call_llm(MODERATION_PROMPT, f'Comment: "{comment}"')
    text = reply.strip().upper()
    if "INAPPROPRIATE" in text:
        return "INAPPROPRIATE"
    if "APPROPRIATE" in text:
        return "APPROPRIATE"
    return "REVIEW"   # unknown reply -> send to a human

Check for INAPPROPRIATE first, because the word APPROPRIATE is inside it. Set temperature to 0 so the same comment gets the same label every time.

In production

• Track both error types. False positives annoy users. False negatives let harm through. Report precision and recall, not only accuracy.
• Re-run the test set on every prompt change. A prompt is code. Keep the labeled comments in the repo and run them in CI.
• Use a human review queue for uncertain cases instead of blocking them.
• Use categories (threat, harassment, hate) when you need more detail, so you can tune each rule on its own.