Why AI Chatbots Make Things Up, and How to Catch It Before It Costs You

An AI chatbot can write a flawless paragraph about a court case that never happened. It can quote a statistic with a decimal point and a source, and both can be fiction. The industry calls this hallucination, and it is the single most important limitation for anyone using AI for real work. The good news is that hallucinations are not random. They happen for understandable reasons, cluster around predictable kinds of questions and can be caught with a handful of habits.

What a hallucination actually is

A hallucination is output that is fluent and confident but not supported by reality or by the material the model was given. It comes in a few common forms:

• Invented facts: a wrong date, a nonexistent product feature, a made-up company policy.
• Fabricated sources: academic papers, news articles or court decisions with plausible titles and authors that do not exist.
• Misattributed quotes: real people credited with words they never said.
• Unsupported summaries: a summary of a document that includes a point the document never makes.
• Confident math errors: a calculation presented as precise that is simply wrong.

The defining feature is not the error itself; people make errors too. It is that the error arrives with exactly the same tone and polish as a correct answer, so there is no natural signal to slow down.

Why it happens

A large language model generates text one small chunk at a time, choosing what is most plausible given everything before it. During training it learned what good answers look like across a vast amount of writing. It did not build a database of verified facts it can look up.

When a fact appeared often and consistently in training, plausible and true line up. When it did not, the model has no internal alarm that says "I don't know this." The pattern of the conversation still calls for a specific, confident answer, so the model produces one. Ask for "three peer-reviewed studies on X" and the most plausible-looking response is three citations formatted like real studies, whether or not the model can recall any.

Several factors make the problem worse:

• Rare or niche topics where the training data was thin.
• Recent events after the model's knowledge cutoff.
• Precise details such as page numbers, figures, case numbers and URLs.
• Leading questions that assume something false ("Why did the Supreme Court rule X in 2019?"), which the model may go along with.
• Long conversations where earlier instructions or documents have scrolled out of the model's working memory.

Where the risk is highest

American courts have already seen the consequences. Lawyers in several cases have been sanctioned or reprimanded by judges after filing briefs that cited nonexistent decisions generated by chatbots. The lesson applies to any professional setting: the person who submits the work owns its accuracy.

How product makers reduce it

Most serious AI products now add machinery around the model:

• Grounding with search or documents. Retrieval-augmented generation pulls relevant passages from the web or from your files and asks the model to answer from them. This sharply reduces invented facts, though the model can still misread a source.
• Citations. Showing links lets you verify claims, provided you actually click them.
• Tools for exact work. Routing math to a calculator or code environment avoids arithmetic slips.
• Training for honesty. Developers increasingly train models to say when they are unsure or lack information.

These features vary between products and even between modes of the same product. If accuracy matters, find out whether the tool you are using searches, reads your documents or relies on memory alone.

Warning signs in an answer

Train yourself to slow down when you see:

• Specific figures, percentages or dates without a source.
• Citations you have not seen before, especially with very neat titles.
• Quotes attributed to named people.
• An answer that perfectly confirms the assumption in your question.
• Details about a niche subject that seem more precise than anyone would reasonably know.
• A summary point you do not remember reading in the original.

None of these prove an answer is wrong. They mark the claims worth checking first.

A practical checking routine

You do not need to verify every sentence. Match the effort to the stakes:

1. Decide what would hurt if wrong. A brainstorm needs no checking; a client email with a price, a legal requirement or a health claim does.
2. Give the model the source. Paste the policy, contract or report and tell it to answer only from that text and to say "not stated" when something is missing.
3. Ask for quotes, not paraphrases. Requesting the exact sentence that supports each claim makes unsupported claims easier to spot.
4. Open every citation you rely on. Confirm the source exists and says what the answer claims.
5. Check numbers at the origin. Government figures belong to the agency that publishes them, such as the Bureau of Labor Statistics or the Census Bureau. Go there, not to a summary.
6. Ask the question a second way. If two phrasings produce different specific answers, treat both with suspicion.
7. Use a second source for anything public. A quick search on an official site often settles it in under a minute.

For teams, the federal NIST AI Risk Management Framework describes this as part of measuring and managing the "validity and reliability" of AI systems, and it is a useful reference when writing internal rules.

Prompts that lower the odds

Wording helps at the margin:

• "If you are not sure, say so rather than guessing."
• "Answer only using the document below. If the answer isn't in it, say that."
• "List the claims in your answer that I should verify."
• "Do not provide citations unless you can link to them."

These instructions do not make a model infallible, but they make it more likely to flag gaps instead of filling them with invention.

Living with an imperfect tool

Hallucinations do not make AI useless. They make it a tool that is excellent at drafting, explaining, restructuring and brainstorming, and unreliable as a final authority on facts. Use it to get to a strong first version quickly, then spend some of the time you saved confirming the handful of details that matter. That division of labor captures most of the benefit while keeping your name off the next embarrassing correction.