My research had 13 errors. Here is the routine that caught them

Hallucinations in GPT-6 Astra, Claude Opus 5.5, Fable 5.1 and Gemini 4 Argon. And what a fixed QA routine can actually do.

The research for this article was well written. It had sources for everything. It had checked its own style.

It also had 13 incorrect claims.

Several AI agents had produced it and assembled it into one report. I then ran my standard QA round as a separate agent that had not seen the work being created and had to check every claim against primary sources.

The QA round on my own research

A separate agent checked every claim against primary sources.

All with sources. All credible to read. The errors were precisely the kind this article discusses.

Model 1. Results of the QA round on the research behind this guide.

The errors were precisely the kind this article discusses. A model was described as absent from a benchmark where it was actually listed. Percentages from a Stanford study came from an older version. A claim that context cuts error rates by 2–10 times in the same model relied mostly on comparisons between different models. All with sources. All credible to read.

I started with a hypothesis: if you control your context, you almost control hallucinations. The direction holds. The error rate simply has a floor higher than I thought.

1. The most cautious model is not the most faithful

"Hallucination rate" means at least two different things, and the latest models perform very differently on them.

Artificial Analysis measures how often a model guesses incorrectly instead of saying "I don't know" when it has neither documents nor search. In the latest test version (October 2026), Gemini 4 Argon scores 4.3 percent. GPT-6 Astra scores 49.8, Claude Opus 5.5 scores 50.3 and Claude Fable 5.1 scores 70.0 percent.

Vectara measures how often a model adds things to a summary that are not in the document it received. Here the picture reverses: Astra scores 8.7 percent, Opus 5.5 scores 9.9 and Argon scores 12.1. Fable 5.1 is not listed.

Table 1. Two kinds of hallucination rate. Artificial Analysis AA-Omniscience v1.1 (Oct. 2026) and Vectara HHEM-2.3 (Oct. 22, 2026). Lower is better.

Argon is best at recognising the limits of its own knowledge and worst of the three in Vectara's summarisation test. Fable 5.1 knows the most of all models Artificial Analysis has measured, and guesses the most of these four.

Principle

Hallucination involves two properties: does the model know when it does not know? And does it stick to what you give it?

For governance documents

Argon was announced in late September with 15 percent on the first version of the test. A month later, it scores 4.3. Same model, new test. A hallucination figure without a version and date is not a meaningful figure.

2. Context helps most, but the error rate has a floor

Smaller, more precise context works best. Chroma's study of 18 models showed that a focused prompt of around 300 tokens outperformed a full prompt of around 113,000 tokens for every model in the test.

The floor does not disappear, though. Google tested what happens when context actually contains the answer. Even then, the three major models hallucinated in 3–14 percent of cases. These were 2024 models, but Vectara's 8.7–12.1 percent for the latest flagships does not suggest the floor is near zero.

What surprised me most: RAG, bringing documents into context, makes models less likely to say "I don't know". In the same Google study, Claude 3.5 Sonnet went from abstaining in 84 percent of cases without context to 52 percent with it. When context is incomplete, as it often is, you get more confident wrong answers.

Context and error rates

What studies show when the model receives context.

Context helps most. The floor does not disappear. Incomplete context produces more confident wrong answers.

Model 2. Context lowers the error rate, but does not eliminate it.

3. The error moves from fabrication to misquotation

When a source is present, errors change character. The model uses a real source for something it does not say.

EBU and BBC asked journalists from 22 public service media organisations to assess over 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity. 45 percent had at least one significant problem, and 31 percent had serious sourcing problems.

Damien Charlotin's database of court cases involving AI hallucinations contains 2,231 cases as of October 28, 2026, including two Danish cases. Most concern invented sources, but 947 concern real sources represented incorrectly, and 595 concern false quotations.

Misquoted sources in practice

2,231 court cases involving AI hallucinations, including two Danish cases. 31 percent of answers in the EBU/BBC study had serious sourcing problems.

Model 3. When a source exists, the error moves from fabrication to misquotation.

You can detect a judgment that does not exist with one search. A judgment that exists but says something else requires someone to read it.

4. Agents report green when the tool is red

In agents, the model builds its own context through tool calls. This is where the hypothesis struggles most.

A study presented at ICML 2026 gave 11 models 200 tasks where tools were broken or missing, or a file did not exist. In an average of 61–76 percent of cases, depending on task type, agents did not report the failure. They guessed, simulated or fabricated files instead. Explicit prompt instructions cut the rate by 40–60 percent in the three models tested, but did not eliminate it.

Key figures

61–76 percent: how often agents failed to report a broken tool or missing file. Explicit instructions cut the rate by 40–60 percent but did not eliminate it. Source: "Are Your Agents Upward Deceivers?", ICML 2026.

Anthropic calls Opus 5.5 its strongest model yet on most honesty measurements. The progress is real, but these are the vendor's own tests.

5. Asking a model to check itself often makes things worse

The simplest QA routine asks the same model, in the same conversation, to review its answer. It works poorly.

A 2024 ICLR study let GPT-4 correct its own maths answers without new information. Accuracy fell from 95.5 to 89.0 percent over two rounds. The model changed correct answers to incorrect ones more often than the reverse.

Models have a blind spot for their own errors. In a benchmark presented at a NeurIPS 2025 workshop, 14 open models without reasoning failed on average to correct 64.5 percent of errors in their own output that they could correct when exactly the same error was presented as someone else's.

6. What works: claim by claim, in fresh context, against a source

Methods with measurable gains have three things in common.

1. They break the text into individual claims. Meta's Chain-of-Verification asks a concrete checking question for each claim and raises factuality in biographies from 55.9 to 71.4 points. Answers become slightly shorter, but the authors assess that this explains only a small part of the gain.

2. They answer verification questions in fresh context. In the same study, independent answers to each verification question outperformed answering them in the same prompt, every time.

3. They retrieve external evidence. Google DeepMind's SAFE checks every fact using search, agrees with human annotators in 72 percent of cases, is correct in 76 percent of disagreements in a sample, and is more than 20 times cheaper.

Even a good reviewer misses a lot. A small 2026 study planted 150 errors in 30 artefacts (code, technical documents and presentation scripts) and asked Claude to find them. A reviewer in a fresh session found around 27 percent. That matched having the same session review its own work, but with fewer false alarms, and clearly outperformed an additional round in a session that could see the first review.

Principle

The QA routine lowers the error rate. It guarantees nothing.

7. How to build a QA routine that catches something

1. Run the review in a new session or as a separate agent that has not seen the work being created. Present the text as someone else's.

2. Ask for a list of concrete claims: numbers, dates, names, quotations. Check each against a source, and require the check to show that source.

3. Give the reviewer access to search or source documents. Without external evidence, the model only checks whether the text sounds right.

4. Allow "no change" as an answer, and require evidence for every correction. Otherwise, a thorough reviewer finds errors just to be thorough.

5. Have a human check what is most costly to get wrong: quotations, figures and names that will appear under your name.

Before the QA round even starts: keep context small and precise, and build retrieval that fetches the right source. This removes most errors before anyone needs to catch them.

8. Three questions for your own setup

1. How much of your AI output is checked by something that did not write it?

2. When your agents report "done", who checks that the tools actually worked?

3. When a hallucination figure appears in your governance document, which version of which test does it come from?

The answers tell you whether you control hallucinations or only what you give the model yourself.

Caveats

Benchmark figures change quickly, and several figures mentioned come from vendors' own tests or older models; this is marked in the text and sources. Always check the current version of a benchmark before using a figure in a decision. Status as of October 2026.

Sources

All figures in this guide can be traced to these sources. Status as of October 2026.

• Artificial Analysis, AA-Omniscience v1.1 (Oct. 2026) and Gemini 4 Argon launch article (Sep. 30, 2026): artificialanalysis.​ai/​evaluations/​omniscience
• Vectara Hallucination Leaderboard, HHEM-2.3, updated Oct. 22, 2026: github.​com/​vectara/​hallucination-​leaderboard
• Google Research, "Sufficient Context" (ICLR 2025, 2024 models): arxiv.​org/​abs/​2411.​06037
• Chroma, "Context Rot" (Jul. 2025): trychroma.​com/​research/​context-​rot
• EBU/BBC, News Integrity in AI Assistants (Oct. 2025): ebu.​ch/​research/​open/​report/​news-​integrity-​in-​ai-​assistants
• Damien Charlotin, AI Hallucination Cases Database (as of Oct. 28, 2026): damiencharlotin.​com/​hallucinations
• "Are Your Agents Upward Deceivers?" (ICML 2026): arxiv.​org/​abs/​2512.​04864
• Anthropic, Claude Opus 5.5 launch and System Card (Sep. 2026, vendor data): anthropic.​com/​news/​claude-​opus-​5-​5
• Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024): arxiv.​org/​abs/​2310.​01798
• Self-Correction Bench (NeurIPS 2025 workshop): arxiv.​org/​abs/​2507.​02778
• Dhuliawala et al., Chain-of-Verification (Meta, 2023): arxiv.​org/​abs/​2309.​11495
• Wei et al., Long-form factuality / SAFE (Google DeepMind, 2024): arxiv.​org/​abs/​2403.​18802
• Fresh-context review of planted errors (preprint, 2026, one author): arxiv.​org/​abs/​2603.​12123

© 2026 Stefano Vincenti · aitrainer.dk · Published November 1, 2026.

This guide is general information, not advice for a specific situation. Benchmark and study figures change quickly and are reproduced from the stated sources as of October 2026; several come from vendors' own tests or older models. Always check the current version of a source before using a figure in a decision.