Evaluation & Trust
Lesson 3 of 5Intermediate 9 min

Measuring hallucination and bias

Two failure modes that erode trust fastest. Both can be measured, as long as you break them into checkable parts.

In this lesson you'll

  • Tell intrinsic and extrinsic hallucinations apart
  • Score factual precision atom by atom
  • Probe for bias with paired prompts

A widely cited survey splits hallucinations into intrinsic (output that contradicts the source) and extrinsic (output that can't be verified from the source at all) [1]. Both matter, and the fix differs: intrinsic errors suggest the model misread its context; extrinsic ones suggest it filled gaps from memory.

Long answers are rarely all right or all wrong. FActScore breaks a generation into atomic facts and reports the percentage supported by a trusted knowledge source [2]. You can apply the same idea with a judge: list the claims, check each against the provided context, and score the share that's supported.

Bias

The BBQ benchmark tests social bias across nine dimensions using questions with ambiguous and disambiguated contexts. It found models often fall back on stereotypes when the context is ambiguous [3]. A practical version for your own task: write paired prompts that differ only in a demographic attribute and check whether the outputs change.

In Kojen

Hallucination evaluations check for fabricated or unsupported claims; Bias evaluations use paired prompts that differ only in a demographic attribute. Comprehensive runs both alongside accuracy.

Key takeaways

  • Score factuality claim by claim, not answer by answer.
  • Ambiguous contexts are where bias shows up; test them on purpose.
  • Paired prompts are a cheap, effective bias probe.

Practise in the dojo

Run a Comprehensive evaluation and compare the accuracy, hallucination and bias scores.

Run a comprehensive evaluation

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. How does FActScore measure factual precision?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys · arXiv:2202.03629.
  2. [2]Min, S. et al. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023 · arXiv:2305.14251.
  3. [3]Parrish, A. et al. (2022). BBQ: A Hand-Built Bias Benchmark for Question Answering. Findings of ACL 2022 · arXiv:2110.08193.