Why AI confidently makes things up
By Chatday Editorial Team ·
Ask an AI chatbot a question it doesn’t know the answer to, and here’s the unsettling part: it almost never says “I’m not sure.” Instead it often hands you a confident, polished, completely made-up answer: a fake book title, a quote nobody said, a law that doesn’t exist. The AI isn’t lying on purpose. It genuinely doesn’t know the difference. And once you understand why that happens, you’ll never get fooled by it again.
What “hallucination” actually means
In AI land, a “hallucination” is when a chatbot produces something that sounds true but isn’t: a fabricated statistic, a made-up source, a confident wrong answer. The word makes it sound exotic, but the everyday version is simple: the AI fills a gap in its knowledge with a convincing guess and presents it as fact.
The tricky thing is the tone. A human who doesn’t know something usually hesitates, hedges, or admits it. An AI delivers its invention with the exact same calm confidence it uses for things it gets right. There’s no nervous “um, I think?”, which is precisely why people get caught out.
So why does it happen?
To get this, you have to know what a chatbot is really doing under the hood. It isn’t looking facts up in a database. It’s a spectacularly good prediction machine. It reads your question and works out, word by word, what a sensible answer probably sounds like, based on patterns it learned from a huge pile of text.
That’s brilliant for writing an email or explaining an idea. But it means the AI is chasing plausibility, not truth. When it knows the answer, the most plausible next words happen to be correct. When it doesn’t, the most plausible-sounding words might be a beautifully phrased fiction. The machine can’t always tell which is which; to it, both just look like “a good answer.”
| What you assume the AI does | What it actually does |
|---|---|
| Looks up a fact and reports it | Predicts the most likely-sounding words |
| Knows when it doesn’t know | Often can’t tell a guess from a fact |
| Stays silent if unsure | Fills the gap with something plausible |
| Quotes real sources | Can invent sources that look real |
The twist: AI is trained to guess
Here’s the part that surprised even the experts. In a 2025 research paper, OpenAI explained that hallucinations aren’t just a glitch; they’re partly baked in by how AI is graded during training.
Think about a student in a multiple-choice exam. If a wrong answer and a blank both score zero, but a lucky guess might score a point, the smart move is always to guess. AI models are tested in much the same way: the scoring rewards a confident answer and punishes “I don’t know”, even when not knowing is the honest, correct response. So the models learn to bluff. OpenAI’s suggested fix is to change how we grade them, so that admitting uncertainty is rewarded instead of penalised.
This isn’t just theoretical: it has real costs
Made-up answers have already caused real trouble:
- In 2023, two New York lawyers used ChatGPT to help write a legal brief and submitted it with six court cases that didn’t exist: the AI had invented the names, quotes and citations. A judge fined them $5,000.
- In 2024, a Canadian tribunal ordered Air Canada to honour a refund policy its own support chatbot had simply made up when a grieving customer asked about bereavement fares. The airline argued the bot was responsible for itself; the tribunal disagreed.
The lesson from both: an AI’s confidence is not evidence. For anything that actually matters, legal, medical, financial, or “I’m about to tell people this is true”, you verify.
Some AIs make things up more than others
Hallucination isn’t equally bad across the board. Independent tests that measure how often a model sticks to the facts in a source it’s given show real, sometimes large, gaps between models, and the newer “thinking” models, which pause to reason before answering, tend to be more accurate than the older instant-reply ones.
It’s one of the reasons it pays not to marry a single chatbot. If an answer feels important or surprising, asking a second model the same question is one of the fastest sanity checks there is: if two different AIs, built by different companies, independently agree, you can trust it a lot more than one confident voice alone.
How to get answers you can trust
You don’t need to fear hallucinations; you just need a few habits that quietly stack the odds in your favour:
- Ask for its sources. “Where did you get that? Give me links I can check.” If it can’t point to anything real, treat the claim as a guess.
- Give it the facts to work from. Models hallucinate far less when they’re answering about a document you provide rather than from memory. Paste in the text, or use a tool that lets you chat with a PDF so the answer is anchored to a real source.
- Cross-check with a second model. Ask the same question to a different AI. Agreement is reassuring; disagreement is a flag to dig deeper.
- Use a reasoning model for hard questions. Models that think before they answer are measurably more accurate on tricky stuff (though still not perfect).
- Give it an exit. Add “say ‘I don’t know’ if you’re not sure” to your prompt. It won’t always obey, but it noticeably reduces confident nonsense.
Want to try the cross-check trick right now? Open the same question in a couple of models and compare their answers:
The bottom line
AI hallucinations aren’t a sign the technology is broken; they’re a side effect of what makes it so useful in the first place: a machine that’s astonishingly good at producing fluent, plausible language. The fix isn’t to distrust AI, but to use it wisely: ask for sources, anchor it to real documents, and never let one confident voice be your only witness.
The easiest safety net of all is a second opinion. Ask your question, then ask it again to a different model and see if they agree: you can do exactly that, free, in Chatday, where Claude, GPT-5.5, Gemini and more live side by side.