OpenAI admits AI hallucinations are mathematically inevitable, not just engineering flaws

[usonian]

Oops, no “Well, that’s a tiny bug that we’ll fix tonight”

https://www.computerworld.com/article/4059383/openai-admits-ai-hallucinations-are-mathematically-inevitable-not-just-engineering-flaws.html

OpenAI, the creator of ChatGPT, acknowledged in its own research that large language models will always produce hallucinations due to fundamental mathematical constraints that cannot be solved through better engineering, marking a significant admission from one of the AI industry’s leading companies.

The study, published on September 4 and led by OpenAI researchers Adam Tauman Kalai, Edwin Zhang, and Ofir Nachum alongside Georgia Tech’s Santosh S. Vempala, provided a comprehensive mathematical framework explaining why AI systems must generate plausible but false information even when trained on perfect data.

“Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty,” the researchers wrote in the paper. “Such ‘hallucinations’ persist even in state-of-the-art systems and undermine trust.”

The admission carried particular weight given OpenAI’s position as the creator of ChatGPT, which sparked the current AI boom and convinced millions of users and enterprises to adopt generative AI technology.

Paper here: https://arxiv.org/pdf/2509.04664

Opinion:

So, “Let’s use this fundamentally flawed system for critical work.”

A Harvard report found a few obstacles to filtering hallucinations, trivial things like “ budget, volume, ambiguity, and context sensitivity”

They had me at “budget”

💰

[patphil]

2. Once an AI creates an hallucination, will other AI programs see it and use it as if it was actual truth?

Could we slowly see a trend toward an AI created electronic reality that is clearly distinct from the one we live in?
How could this affect the world we actually live in, in terms of our being able to trust the information the AI gathers?
Could this eventually lead to a self-conscious AI?

Just a few thoughts on how strange this new world of AI could be.

[Ilsa]

3. Infallibility

"...large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty,"

I might be more inclined to check a statement coming from a person vs second guessing AI results

[AZJonnie]

4. Do they mean like multi-purpose AI like ChatGPT, or does he mean this is true of

ANY AI, no matter how specifically and thoroughly trained? Like an AI drone? Or an AI watching over life support monitoring at a hospital?

If true in both cases, and he has solid evidence he's mathematically irrefutably correct? The fallout from this being factual could be massive to the industry.

But it probably won't be because it's probably still statistically making decisions better than people would on average, so people will be convinced it's all fine.

[usonian]

5. Check the paper.

I think the problem will boil down to:

No matter how good the results are, they always need checking, and that costs money, and that means some hallucinations will creep through.

This is not like a mathematical library where results have been tested like crazy and can be relied upon.

They don't guess.

[Hugin]

7. Just like any other technological endeavor...

Generative AI suffers from the potential for so-called gear clash. In fact, they’ve enshrined it at the board room level.

Entropy is a cruel mistress.

[Mr. Sparkle]

6. My "unqualified" opinion is that it will take another technology leap to fix hallucinations and make AI mainstream

right now you would not trust it to make important decisions that can have a big financial impact on your life.