Why LLMs Hallucinate — and What Actually Helps

Ask a language model for citations on a niche topic and it may cheerfully produce five papers: plausible titles, credible-sounding authors, journal names, volume and page numbers. None of them exist. Nothing about the output looks broken — it is fluent, well-formatted, confident. That's precisely the problem: hallucination doesn't look like failure. It looks like success with different contents.

If you ship LLM features, treating hallucination as an embarrassment to be prompted away is the wrong mental model. Treat it the way a reliability engineer treats packet loss or clock drift: an expected property of the substrate, something to detect, bound, and design around. This article is a field guide to doing that — what hallucination actually is mechanistically, how to catch it in production pipelines, which mitigations measurably help, and which popular fixes are folklore.

Definitions that matter

Factuality vs. faithfulness

Two properties get lumped under "hallucination," and separating them changes what you build:

These axes are independent. A summarizer can be perfectly faithful to a document that is itself wrong (faithful, not factual). Worse, a RAG system can emit a factually correct answer that isn't supported anywhere in its retrieved context (factual, not faithful) — which sounds harmless until you notice nobody, including the system, can verify it. In production, faithfulness is usually the property you can actually measure and enforce, because your sources are in hand and the ground truth isn't. Enforce faithfulness first; let source quality drive factuality.

Intrinsic vs. extrinsic hallucination

Intrinsic errors are easier to catch, because contradiction is detectable. Extrinsic errors are the sneaky kind: plausible, often harmless-looking additions that slip past both human reviewers and automated checks.

"The model lied" is the wrong frame

Lying requires knowing the truth and choosing to state something else. A transformer does neither. It generates the most plausible continuation of the text so far, conditioned on context and weights. Fluent interpolation over imperfect knowledge isn't a malfunction — it's the default operation of a next-token predictor asked a question it only half-knows.

There is no internal module that tags output as "recalled" versus "improvised." "The study found X" and "the study found Y" come off the same assembly line with identical fluency. The engineering consequence is blunt: there is no bug to locate and patch. You are managing a statistical property of a generative system, permanently. Systems must be designed around it, not purified of it.

Why it happens

Four forces stack up, and each points at a different countermeasure.

The training objective rewards plausibility, not truth

Pretraining maximizes the likelihood of the next token. Truth is correlated with plausibility — much of what's likely is true — but the correlation is loosest exactly where hallucinations live: rare entities, obscure claims, unusual phrasings. The model has also absorbed the style of confident assertion from its corpus, including millions of confidently wrong sentences. Genre-appropriate confidence is learnable from text; ground truth isn't directly observable by the objective.

Knowledge has boundaries and cutoffs

Weights are a lossy compression of a training corpus. Frequently documented entities compress well; the long tail stays fuzzy. Everything after the cutoff date is absent entirely. When prompted past these boundaries, generation doesn't return an empty result — it continues, filling a biography-shaped question with biography-shaped text. The gap gets filled with something statistically appropriate, never with nothing.

Sampling adds variance

At temperature above zero, decoding deliberately explores lower-probability tokens, so a fraction of runs land in the tail. Real, but smaller than most people assume: plenty of hallucinations survive greedy decoding untouched, because the confabulated version genuinely is the highest-likelihood continuation. Randomness trims one variance source, not the phenomenon.

Feedback pipelines historically rewarded confidence

Models tuned on human preference data inherit rater biases, and raters have tended to prefer confident, complete answers over hedged ones. A crisp wrong answer outscores "I'm not sure." Preference optimization therefore pushes toward confident-sounding output and away from calibrated uncertainty — newer training runs push back, but the deployed ecosystem is mixed and the pressure is structural.

The classic probe makes this vivid: ask an unguarded model "Who was [invented person]?" and it may produce a full biography — birthplace, career, notable works — rather than objecting to the premise. Unknown-entity prompts remain the cheapest smoke test for calibrated refusal, and they belong in your eval set (see below).

Detecting hallucination in practice

No single detector covers the space. Each common signal catches a different slice and fails differently:

Technique How it works What it catches Blind spots
Self-consistency sampling Sample the same prompt several times; disagreement signals uncertainty (the basis of SelfCheckGPT-style detectors) Cases where the model is genuinely unsure Systematic errors: consistently wrong answers agree with themselves perfectly; cost scales with sample count
Token-level confidence Logprobs/entropy as an uncertainty proxy; semantic entropy clusters samples by meaning Local uncertainty worth hedging Probability measures belief in the wording, not truth — a fluent falsehood is high-probability by construction; logprobs are hidden or degraded on many hosted APIs
NLI/entailment checks Split output into atomic claims; classify each entailed/contradicted/neutral against the sources Unfaithful summaries, unsupported additions in RAG Only as good as the NLI model and chunk quality; how you treat "neutral" silently decides your precision/recall tradeoff
Retrieval cross-checking After drafting, search independently for support or contradiction of key claims Freshness-sensitive, checkable factual claims Latency and cost; retrieval quality becomes the ceiling; judgment calls resist lookup
Judge models An LLM grades groundedness or factuality of outputs Nuance classifiers miss; scalable review Judges hallucinate too, and show position, verbosity, and self-preference biases — calibrate scores against human labels; never treat judge output as ground truth

Two practices make these usable together:

  1. Layer them. Run cheap deterministic signals (NLI checks, logprob thresholds) on everything; spend expensive ones (multi-sample consistency, judge review, web cross-checks) on high-risk traffic or sampled audits.
  2. Own the thresholds. Whether NLI "neutral" verdicts count as failures, how much disagreement blocks a response — these trade customer friction against error leakage. They're product decisions; don't let defaults decide them.

Mitigations that actually move the needle

Grounding with retrieval, done properly

Retrieval-augmented generation trades "remember" for "look up," converting tail-knowledge questions into reading-comprehension questions — strictly easier and auditable. The details decide whether it actually helps:

Let the model abstain

A system that says "I don't know" reliably beats one that's right slightly more often. In practice: give the model explicit permission and a protocol to refuse; pair retrieval with a relevance bar so questions without adequate support get declined; measure abstention rate as a health metric, not a defect. Zero abstentions doesn't mean omniscience — it means unaudited.

Abstention only works if the product honors it. If your funnel punishes empty answers, no amount of prompting matters; the system will learn to fill space.

Constrain decoding for verifiable fields

Some output components need conformance, not judgment: dates, currencies, order IDs, category enums, machine-readable records. For those, structured outputs and schema-constrained decoding make malformed values physically unrepresentable. Know the limit: constraints guarantee syntactic validity, not semantic truth — the model can still pick the wrong-but-well-formed SKU. Constrain the format and validate values against your system of record.

Verify after generating: the claim-check loop

Treat the model's draft as untrusted input and review it like a junior engineer's pull request: decompose, then check every atom.

python
async def answer_with_verification(query: str) -> Answer:
    # Retrieve and sharpen evidence
    docs = retrieve(query, top_k=20)
    docs = rerank(query, docs)[:5]

    # Draft with mandatory citations
    draft = generate(query, docs, require_citation_per_claim=True)

    # Decompose into atomic, self-contained claims
    claims = extract_atomic_claims(draft)

    # Check each claim against the evidence
    verdicts = await gather([
        check_entailment(claim, evidence=docs)  # SUPPORTED | CONTRADICTED | UNSUPPORTED
        for claim in claims
    ])

    unsupported = sum(v != "SUPPORTED" for v in verdicts) / max(len(claims), 1)
    if unsupported > ABSTAIN_THRESHOLD:
        return abstain(query, reason="insufficient support", verdicts)

    # Strip or flag sentences behind failed claims; attach citations
    return annotate_and_trim(draft, claims, verdicts)

Every stage is independently swappable — better claim extractor, stronger NLI model, tighter threshold — and every stage emits telemetry you can route to humans. Prompt-only approximations exist too (chain-of-verification style: draft, generate verification questions, answer them independently, revise), and they help — but the pipeline form is auditable and tunable in a way prompt gymnastics aren't.

Prefer tools over memory

Every fact fetched by a tool call replaces a probabilistic guess with a verifiable read:

Draw the boundary explicitly in design docs: what the model may know versus what the system must look up. Anything on the look-up side stops being a hallucination risk and becomes an ordinary integration-reliability problem — the kind your organization already knows how to run.

Product-level defenses

Mitigations reduce the rate; none reach zero. Ship accordingly:

And the folk remedies, examined honestly:

If you're mapping these controls onto formal frameworks, our earlier articles cover where measurement, incident response, and human-oversight obligations fit into the NIST AI RMF, EU AI Act, and ISO/IEC 42001, plus the retrieval-and-tooling plumbing of LLM application engineering in depth.

Measure before you ship

You cannot operate a reliability number you've never measured. Before launch:

Key takeaways