Build an AI Answer Check That Catches Unsupported Details

A fluent answer can mix supported information with a detail that the source never supplied. Instead of asking whether the whole answer sounds right, test the claims that would change your decision. This exercise uses a fictional event notice, so no private data or live booking is needed.

Make a tiny reference

Write: “The workshop starts at 10:00 on Saturday. Bring a notebook. Registration closes Thursday.” Ask an assistant to make a preparation checklist using only that text. Require it to mark missing information as unknown. Keep the reference beside the answer rather than relying on memory.

Underline every concrete statement in the output. Time, date, location, price, equipment and eligibility are separate claims. A sentence can contain both a supported instruction and an invented venue. Split that sentence before judging it.

Use three evidence labels

Label a claim Supported when the reference directly establishes it, Inference when it is a reasonable interpretation, and Unknown when evidence is absent. “Bring a notebook” is supported. “Arrive ten minutes early” is an optional planning suggestion, not a published event requirement. A venue address is unknown.

Ask for a correction with this prompt: “Remove unsupported factual details. Keep useful suggestions in a separate optional section. List questions the organiser still needs to answer.” This gives the revision a specific job.

Check the revision independently

Compare the new checklist with the original notice again. Do not treat the model's explanation of its own accuracy as verification. If a critical detail remains unknown, obtain it from the organiser before acting.

For real work, retain the source date and the checked answer together. Repeat the check when the source changes. This method tests faithfulness to a reference; it does not establish that the reference itself is accurate or current.