Compare AI Assistants With a Rubric Before Picking a Winner

A single impressive reply is a weak basis for choosing an assistant for recurring work. First decide what a usable answer must contain. Then compare outputs against the same rubric, using tasks that resemble your actual workload. Define a small evaluation set Choose three different tasks: a reference-based summary, a rewrite with firm constraints, and a structured extraction. Use fictional or public material that you are authorised to share. Include one awkward case, such as a missing field, rather than testing only clean examples. For a fictional meeting note, ask every assistant to extract decisions, owners and dates, marking absent values as unknown. Provide identical instructions and inputs. Record product settings you can observe, the date, and whether browsing or other tools were enabled. Score observable outcomes A simple rubric can award one point each for complete required fields, faithful use of the source, correct handling of unknowns, and usable formatting. Add a separate c...
Post a Comment

Keep a Source Ledger for an AI-Assisted Research Note

A research note becomes difficult to audit when links are scattered through chat history. A source ledger is a small table that connects each important claim to the evidence you actually opened. It is useful even if the final note is only one page long.

Start with claims, not a pile of links

Create columns for Claim, Source title, Exact location, Publication or update date, Checked date, and Status. Add one row for each statement that a reader might reasonably challenge. Keep a separate list for interesting links that have not been inspected.

For a fictional software release, one row might read “Export is available in the desktop app,” followed by the vendor's release note and the paragraph containing that statement. A search snippet or an assistant-generated link is a lead until you open it and confirm its contents.

Capture the limits of the evidence

Record whether a statement applies to a particular version, platform, country or account tier. A source about one edition does not establish the same feature everywhere. If a page has no visible update date, record that limitation instead of guessing one.

Use a short note such as “Direct statement,” “My interpretation,” or “Not resolved.” When two sources conflict, preserve both rows and explain the difference. Do not silently select the wording that makes the article easier to finish.

Give AI a bounded editing task

Supply a non-sensitive excerpt and ask: “Identify which claims in this draft are not supported by the attached ledger. Do not add sources or complete missing facts.” Review the resulting flags yourself. An assistant can miss a mismatch or flag a harmless paraphrase.

Before publication, open the links again and check that the final wording stays within their scope. The ledger supports traceability; it is not a substitute for judging the quality and independence of the sources.

Comments

Post a Comment