Build an AI Answer Check That Catches Unsupported Details
A fluent answer can mix supported information with a detail that the source never supplied. Instead of asking whether the whole answer sounds right, test the claims that would change your decision. This exercise uses a fictional event notice, so no private data or live booking is needed.
Make a tiny reference
Write: “The workshop starts at 10:00 on Saturday. Bring a notebook. Registration closes Thursday.” Ask an assistant to make a preparation checklist using only that text. Require it to mark missing information as unknown. Keep the reference beside the answer rather than relying on memory.
Underline every concrete statement in the output. Time, date, location, price, equipment and eligibility are separate claims. A sentence can contain both a supported instruction and an invented venue. Split that sentence before judging it.
Use three evidence labels
Label a claim Supported when the reference directly establishes it, Inference when it is a reasonable interpretation, and Unknown when evidence is absent. “Bring a notebook” is supported. “Arrive ten minutes early” is an optional planning suggestion, not a published event requirement. A venue address is unknown.
Ask for a correction with this prompt: “Remove unsupported factual details. Keep useful suggestions in a separate optional section. List questions the organiser still needs to answer.” This gives the revision a specific job.
Check the revision independently
Compare the new checklist with the original notice again. Do not treat the model's explanation of its own accuracy as verification. If a critical detail remains unknown, obtain it from the organiser before acting.
For real work, retain the source date and the checked answer together. Repeat the check when the source changes. This method tests faithfulness to a reference; it does not establish that the reference itself is accurate or current.
A sentence-by-sentence answer check
Use the fictional notice from this exercise: “The workshop starts at 10:00 on Saturday. Bring a notebook. Registration closes Thursday.” Now inspect this invented assistant reply: “Arrive at West Hall at 09:45 on Saturday for the free workshop. Bring a notebook and register by Thursday.”
| Output detail | Assessment | Action |
|---|---|---|
| Saturday | Supported by the notice | Keep |
| West Hall | Not supplied | Remove or ask the organiser |
| 09:45 arrival | Suggestion, not the stated start time | Label as optional advice or remove |
| Free | Not supplied | Remove |
| Notebook | Supported | Keep |
| Thursday registration deadline | Supported | Keep |
Write a corrected version
“The workshop starts at 10:00 on Saturday. Bring a notebook and register by Thursday. The notice does not provide a venue or price.” This keeps the supplied facts and exposes the gaps. It does not quietly turn a suggested arrival time into an official requirement.
Distinguish an error from an open question
“The venue is unknown from this notice” is different from “the venue does not exist.” Missing evidence supports the first statement, not the second. If a detail is necessary for action, obtain a source rather than asking the assistant to guess more confidently.
For a real notice, compare each concrete statement with the latest authorised source. If the source changes, update the checklist and repeat the comparison. Keep the original answer so that the correction remains traceable.
Use a release decision
Mark the answer ready only when required facts match, missing details remain marked, and suggestions are visibly separate from source facts. Otherwise keep it in review. This is a repeatable editorial exercise, not a guarantee that every unsupported statement will be detected or that a model is reliable in unrelated tasks.
Comments