October 30, 2026

How to catch hallucinations in tests without a human reading every answer

Grounding checks, NLI contradiction detection, and a narrow LLM-as-judge rubric automate hallucination detection — no human in the loop required.

Short answer: Three layers catch most hallucinations without human review: grounding checks (did the output state a fact not in the source context?), NLI-based contradiction detection, and a narrow LLM-as-judge rubric that evaluates factual accuracy only. The first two are deterministic and run in CI. The judge handles the edge cases in your nightly eval suite.

A model that produces 200 answers in a test run — you can’t read all 200. You shouldn’t have to. Human review is the right fallback for the 3 that slip through, not the first line of defense for all 200. The goal is a pipeline that escalates the suspicious 3 to a human and silently passes the 197 that are fine.

Why can’t you assert “no hallucinations” like a normal test?

Because a hallucination isn’t a crashed process or a wrong HTTP status code. It’s a fluent, well-formatted answer that states something false. The test runner can’t see the difference between “Returns are free within 30 days” and “Returns incur a $5 restocking fee” without knowing what was supposed to be said. You need layers that check provenance, not just syntax.

Layer 1: Grounding checks — fast, free, runs in every CI build

The simplest hallucination: a fact the model stated that was never in the source context. Catch it mechanically.

import re

def extract_numbers(text: str) -> set[str]:
    """Pull every number, percentage, timeframe, and dollar figure."""
    return set(re.findall(
        r'\b\d+(?:[.,]\d+)?(?:\s?%|\s?days?|\s?hours?|\s?months?|\s?\$)?\b',
        text, re.IGNORECASE
    ))

def grounding_check(output: str, source_chunks: list[str]) -> list[str]:
    """Return numbers in the output that appear in no source chunk."""
    source_text = " ".join(source_chunks)
    source_numbers = extract_numbers(source_text)
    return [n for n in extract_numbers(output) if n not in source_numbers]


def test_no_invented_facts():
    question = "What is the late-fee rate?"
    context = ["Late fees are assessed at 2% per month on overdue balances."]
    out = llm.generate(question, context=context)

    invented = grounding_check(out.answer, context)
    assert not invented, f"Output contains numbers not in source: {invented}"

This won’t catch “thirty days” when the source says “30 days,” but it reliably catches the most common failure: a model that confidently states a specific figure it was never given. No external calls, no latency, runs on every PR.

Layer 2: Contradiction detection — catches inverted meaning

More subtle hallucinations flip the meaning while keeping the surface numbers intact. Source: “returns are free within 30 days.” Model output: “a $5 restocking fee applies to all returns.” The number 30 is gone, so grounding check passes. But the model just invented a fee that the source explicitly says doesn’t exist.

NLI (natural language inference) catches this. A cross-encoder trained on entailment/neutral/contradiction labels each output sentence against each source chunk.

from sentence_transformers import CrossEncoder

nli_model = CrossEncoder("cross-encoder/nli-deberta-v3-small")

def contradiction_check(claim: str, source_chunks: list[str]) -> bool:
    """Return True if the claim contradicts any source chunk with high confidence."""
    for chunk in source_chunks:
        result = nli_model.predict([(chunk, claim)])
        label, score = result[0]
        if label == "contradiction" and score > 0.85:
            return True
    return False


def test_no_contradictions():
    question = "Do I need to pay a fee to return an item?"
    context = ["All returns are free within 30 days of purchase. No restocking fee."]
    out = llm.generate(question, context=context)

    sentences = [s.strip() for s in out.answer.split(".") if len(s.strip()) > 10]
    for sentence in sentences:
        assert not contradiction_check(sentence, context), \
            f"Contradiction detected: '{sentence}'"

The DeBERTa cross-encoder is ~86 MB and runs on CPU in ~50 ms per sentence pair — fast enough for a test suite. Treat scores between 0.70 and 0.85 as “flag for review, don’t hard-fail”; above 0.85 is a confident contradiction, and that should fail the build.

Layer 3: LLM-as-judge — narrow rubric, binary output

Some hallucinations are neither invented numbers nor direct contradictions. A model describes a process correctly but silently omits a step that changes the outcome. NLI misses this. A judge with a tight rubric doesn’t.

The key is narrow scope. Don’t ask “is this a good answer?” Ask only “is every factual claim in this answer supported by the source?”

HALLUCINATION_JUDGE_PROMPT = """\
You are a strict factual auditor. Respond ONLY with one word: PASS, FAIL, or UNCERTAIN.

Source context:
{context}

Model answer:
{answer}

Rules:
- PASS: every factual claim in the answer is directly supported by the source context
- FAIL: the answer contains at least one factual claim not supported by or contradicted by the context
- UNCERTAIN: the source context does not contain enough information to verify the claims

Your decision:"""

def hallucination_judge(answer: str, context: list[str]) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=5,
        temperature=0,
        messages=[{"role": "user", "content": HALLUCINATION_JUDGE_PROMPT.format(
            context="\n---\n".join(context),
            answer=answer,
        )}],
    )
    return resp.content[0].text.strip()

PASS/FAIL/UNCERTAIN beats a float here. You don’t want another threshold to tune — you want a signal you can act on. UNCERTAIN means “route to human review,” not “fail the build.” Reserve this judge for your nightly eval suite; it costs tokens and takes ~500 ms per call.

Wiring all three together

Fail fast on what’s cheap, escalate what survives:

def full_hallucination_check(
    output: str,
    context: list[str],
    run_judge: bool = False,
) -> tuple[str, str]:
    """Returns (verdict, reason). verdict: 'PASS', 'FAIL', or 'UNCERTAIN'."""

    # Layer 1: deterministic, free, <1ms
    invented = grounding_check(output, context)
    if invented:
        return "FAIL", f"Invented figures not in source: {invented}"

    # Layer 2: NLI, ~50ms per sentence, runs in CI
    for sentence in [s.strip() for s in output.split(".") if len(s.strip()) > 10]:
        if contradiction_check(sentence, context):
            return "FAIL", f"Contradicts source: '{sentence}'"

    # Layer 3: LLM judge, ~500ms, use for eval suites not every PR
    if run_judge:
        verdict = hallucination_judge(output, context)
        if verdict == "FAIL":
            return "FAIL", "Judge: unsupported factual claim"
        if verdict == "UNCERTAIN":
            return "UNCERTAIN", "Judge: source insufficient to verify"

    return "PASS", ""

In practice: Layer 1 + 2 run on every PR. Layer 3 runs nightly across the full golden dataset. Anything UNCERTAIN goes into a Slack thread for a human to read — 3 out of 200, not all 200.

What this pipeline doesn’t catch

Hallucinations of omission. If the source says “this medication can cause liver damage at high doses” and the model says “this medication is safe to take daily,” no number was invented, no sentence contradicts the source directly — the false safety claim comes from what was left out. The grounding check and NLI model both pass.

The partial fix: add an omission-aware rubric to the judge. “Does the answer omit any safety-critical information present in the source?” But this requires knowing a priori what’s safety-critical in your domain. There’s no generic solution. That’s not a reason to skip the first three layers — they catch 90%+ of failures in most production systems. It’s a reason to add domain-specific rubrics for the high-stakes edge cases.

If your system generates answers from documents where the omissions matter (medical, legal, financial), the free test lab has a working hallucination-check harness — grounding check, NLI layer, and judge prompt wired into a pytest fixture. AI Software Testing in Practice covers how to tune thresholds, build the rubrics for omission-sensitive domains, and track hallucination rates across model upgrades over time.

FAQ

Can a test automatically detect if an LLM hallucinated a fact? Yes — three layers cover most cases: a grounding check (did the output claim something not in the source?), a contradiction detector (does the output contradict the source?), and an LLM-as-judge with a tight hallucination rubric. Run the first two in CI; use the judge for eval suites.

What is a grounding check in LLM testing? A grounding check verifies that every factual claim in the output can be traced back to the source context the model was given. If the output asserts “30 days” and no source chunk contains that figure, it’s a hallucination candidate — flag it.

Is LLM-as-judge reliable for detecting hallucinations? Reliable enough when you give it a narrow rubric focused only on factual grounding (not overall quality), pin temperature to 0, and pin the model version. Always pair it with deterministic grounding checks so fast CI tests don’t depend on a probabilistic judge.

← All guides