Hallucinations and How to Verify

AI models sometimes state things that are simply false, in the same fluent, confident tone they use for true things. Knowing why this happens, and when it is most likely, lets you use AI for real work without getting caught out.

What a hallucination is

A hallucination is an output that sounds plausible but is wrong or made up: a citation to a paper that does not exist, a statistic with no source, a quote nobody said, a software function that was never written. The word is a little misleading, since the model is not seeing things. It is producing text that fits the pattern of a good answer without that text being anchored to a real fact.

Why confident wrong answers happen

A language model is trained to predict likely next words based on patterns in enormous amounts of text. It is not looking facts up in a database, and during its basic training it is never told which sentences are true and which are false. That has a few consequences (this is a simplification, but the core idea is sound):

• Patterns are learned well; one-off facts are not. Grammar, common knowledge and widely repeated facts appear thousands of times in training data, so the model gets them right. A specific person's birthday or the page number of a court ruling may appear rarely or never, so the model has no pattern to follow and produces a plausible-looking guess instead.
• Fluency is not a confidence signal. The model writes a made-up citation in exactly the same style as a real one. Its tone tells you nothing about whether it is right.
• Guessing has been rewarded. OpenAI researchers argued in 2025 that common training and testing methods reward models for guessing rather than saying "I don't know", much as a multiple-choice exam rewards a lucky guess over a blank answer. Developers are working on this, but no current model is free of the problem.

An analogy: imagine a well-read friend at a dinner party who hates admitting they do not know something. Most of what they say is right. But when you ask about something obscure, they will still give you a smooth, specific answer, and you cannot tell from their delivery which answers were invented.

A real example

In 2023, lawyers in a US federal case, Mata v. Avianca, filed a brief that cited court decisions generated by ChatGPT. Several of the cases did not exist. When questioned, the lawyers asked the chatbot whether the cases were real, and it said they were. The judge sanctioned the lawyers and fined them $5,000. The lesson is not just "AI makes things up" but also "asking the AI to confirm its own answer is not verification."

When the risk is highest

Be most careful with:

• Citations and references. Paper titles, authors, case names, URLs, page numbers and quotes are classic hallucination territory.
• Specific numbers. Statistics, prices, dates, dosages, measurements and percentages. A number that is slightly off still looks right.
• Niche facts. Small companies, local regulations, less-famous people, specialised technical details, anything rarely written about.
• Recent events. A model's built-in knowledge stops at its training cutoff, the date its training data ends. Without a search tool it may not know about anything after that, and may confidently describe an outdated situation as current.
• Long chains of reasoning. An early mistake in a calculation or argument gets carried forward.
• Questions with a false premise. Ask "Why did the 2019 law ban X?" and a model may explain a law that never existed instead of correcting you.

Grounding: give it something real to work from

Grounding means giving the model trustworthy source material and asking it to answer from that material rather than from memory. Two common ways:

• Documents. Paste or upload the contract, report or manual and say: "Answer only using the document below. If the answer is not in it, say so."
• Search. Many AI tools can search the web and show links. This helps a lot with recent events and niche facts, but the model can still misread a page, or rely on a poor-quality one.

Grounding reduces hallucinations; it does not eliminate them. A model can still misquote a document it was given.

How to verify

Worked example. Suppose you ask for statistics for a presentation and get: "According to a 2024 survey by the National Retail Institute, 62% of shoppers..." (an invented example, which is exactly the point). Here is how to check it:

1. Find the source yourself. Search for the organisation and survey by name. Does it exist? Is it a real, reputable body?
2. Find the exact figure. Open the original report and locate the number. Is it 62%? Is it about the same group of people and the same year?
3. Check the context. Does the source say what the AI claimed, or something narrower?
4. If you cannot find it in two or three minutes, drop it or replace it with a figure you have confirmed.

Other useful habits:

• Ask for sources, then open them. A link that is real but does not support the claim is as bad as no link.
• Ask the model to flag uncertainty: "Mark anything you are not sure about." This helps, but treat the result as a hint, not proof.
• Cross-check important answers against an authoritative source: an official website, the original document, a professional.
• Match effort to stakes. Brainstorming dinner ideas needs no checking. Medical, legal, financial and safety decisions always do.

Going deeper

Probabilities, not look-ups

At every step a language model assigns a probability to each possible next token and then picks one. For a widely repeated fact ("Paris is the capital of France") almost all the probability sits on the right answer. For a fact the model saw once or never, such as an obscure person's birthday, the probability is spread across many plausible candidates. The model still has to emit something, and whichever date it picks will be written just as fluently. Kalai, Nachum, Vempala and Zhang (2025) make this precise: they argue that for arbitrary facts with no learnable pattern, a base model's error rate is at least roughly the fraction of such facts that appeared only once in its training data. Rare facts are where hallucinations concentrate, and this is a statistical consequence of the training objective, not a bug that one patch can remove.

Calibration is the property that stated confidence matches real accuracy: among answers given with 80% confidence, about 80% should be right. Guo et al. (2017) showed that modern neural networks are often overconfident, and you should assume a chatbot's verbal confidence ("I'm certain") is a weak signal at best.

Why guessing pays under binary grading

Suppose a test awards 1 point for a correct answer and 0 for a wrong answer or "I don't know". If a model's chance of being right is $q$, guessing earns $q$ points on average and abstaining earns 0, so guessing always wins, however small $q$ is. Kalai and colleagues argue that most widely used benchmarks grade this way, so training and evaluation keep rewarding confident guesses.

Their proposed fix is an explicit confidence target $t$: tell the model to answer only if it is more than $t$ confident, and deduct $\frac{t}{1-t}$ points for a wrong answer. The expected score of answering is then

$E = q \cdot 1 - (1-q)\cdot\frac{t}{1-t},$

which is positive exactly when $q > t$ (multiply through by $1-t$: $q(1-t) - (1-q)t = q - t$). So the best strategy becomes "answer when your real confidence exceeds $t$, otherwise abstain".

Worked example. With $t = 0.75$, a wrong answer costs $0.75/0.25 = 3$ points. A model that is 60% sure: $E = 0.6 - 0.4 \times 3 = -0.6$, so it should abstain. A model that is 90% sure: $E = 0.9 - 0.1 \times 3 = 0.6$, so it should answer. You can apply the same logic to yourself: the higher the cost of acting on a wrong answer, the higher your confidence bar before you rely on it.

Errors compound

Small error rates become large across many claims. If each citation in a list is independently correct with probability 0.95, the chance that all 20 are correct is

$0.95^{20} \approx 0.358,$

so about two times in three at least one is wrong, and on average one is fake ($20 \times 0.05 = 1$). The same arithmetic applies to reasoning: ten steps that are each 98% reliable give $0.98^{10} \approx 0.817$, so nearly one chain in five contains an error somewhere. This is why long, citation-heavy outputs need checking item by item.

Consistency checks

Asking the same model "are you sure?" is weak, but asking the same question several times independently is informative. Wang et al. (2022) showed that sampling several reasoning paths and taking the majority answer ("self-consistency") improves accuracy on reasoning tasks. Used as a check, the logic is: if five fresh attempts give five different birth dates, none of them should be trusted. Agreement is not proof, because a model can repeat the same mistake consistently, but disagreement is a cheap, useful warning sign.

Grounding, and checking the grounding

Retrieval-augmented generation (RAG), introduced by Lewis et al. (2020), retrieves relevant passages and puts them in the prompt so the model answers from them. It reduces hallucination on knowledge questions, but the model can still misquote or over-generalise what it retrieved. When you ask for quotes from a document, you can check them mechanically:

import re

def normalise(s):
    return re.sub(r"\s+", " ", s.lower()).strip()

def check_quotes(answer, source, min_len=10):
    """List each quoted passage in `answer` and whether it appears in `source`."""
    src = normalise(source)
    quotes = re.findall(r'"([^"]{%d,})"' % min_len, answer)
    return [(q, normalise(q) in src) for q in quotes]

source = "The tenant must give 60 days written notice before leaving the property."
answer = ('The lease says "the tenant must give 60 days written notice" and '
          '"the landlord pays all repair costs".')
for quote, found in check_quotes(answer, source):
    print("OK  " if found else "MISSING", quote)

The first quote is found; the second is flagged as missing because it is not in the source at all. Normalising case and whitespace avoids false alarms from line breaks. The check catches invented quotes, but not a real quote used to support the wrong conclusion; that still needs a human reader.

Common pitfalls

• Asking the model to verify itself in the same conversation, as in Mata v. Avianca. Verify against an independent source.
• Treating a real link as support. Open it and find the claim; many hallucinated citations point to real pages that say something else.
• Assuming search or RAG makes answers correct. Grounding lowers the error rate; it does not remove the need to check what matters.
• Trusting precise-looking numbers. False precision ("62.4%") is not evidence of a source.
• Uniform effort. Checking everything equally wastes time on low-stakes text and leaves too little for high-stakes claims; triage by consequence.
• Leading questions. A question with a false premise invites an answer that goes along with it; ask neutrally ("Did a 2019 law ban X?").

In practice

• Treat AI output as a first draft from a fast, well-read assistant who sometimes makes things up.
• Never paste an AI-generated citation, quote or statistic into anything important without opening the original source.
• Give the model the documents you want it to rely on, and tell it to say when the answer is not there.
• Use a search-enabled tool for anything recent, and still check the links.
• Do not ask the model to verify itself; verify against an independent source.