Compare AI Assistants With a Rubric Before Picking a Winner
A single impressive reply is a weak basis for choosing an assistant for recurring work. First decide what a usable answer must contain. Then compare outputs against the same rubric, using tasks that resemble your actual workload.
Define a small evaluation set
Choose three different tasks: a reference-based summary, a rewrite with firm constraints, and a structured extraction. Use fictional or public material that you are authorised to share. Include one awkward case, such as a missing field, rather than testing only clean examples.
For a fictional meeting note, ask every assistant to extract decisions, owners and dates, marking absent values as unknown. Provide identical instructions and inputs. Record product settings you can observe, the date, and whether browsing or other tools were enabled.
Score observable outcomes
A simple rubric can award one point each for complete required fields, faithful use of the source, correct handling of unknowns, and usable formatting. Add a separate count for corrections you had to make. Do not invent timing measurements from how fast a response felt.
Keep the original outputs. A score of three is more informative when the missing point can be traced to an invented deadline than when it appears as an unexplained total. Do not combine factual errors and cosmetic preferences into one vague quality rating.
Interpret the result narrowly
Repeat a failed task after improving an unclear instruction, while retaining the first result. That distinguishes a prompt problem from a persistent limitation. If you change the rubric after seeing the outputs, disclose the change and rescore all candidates.
Choose based on your tested workflow, correction burden and practical constraints. These small trials are not a universal model ranking, a controlled benchmark, or proof that the same product will behave identically next month.
Worked scoring example: two fictional outputs
Suppose the source says: “Mira will draft the agenda by Tuesday. The team approved a weekly check-in. No owner was assigned to the room booking.” These are invented meeting details used only to demonstrate scoring.
Output A says: “Decision: weekly check-in. Agenda owner: Mira. Agenda deadline: Tuesday. Room-booking owner: not assigned.” Output B says: “Decision: monthly check-in. Agenda owner: Mira. Agenda deadline: Tuesday. Room-booking owner: Leo.”
Score observable evidence, not confidence
| Criterion | A | B |
|---|---|---|
| Required fields present | 1 | 1 |
| Decision matches the source | 1 | 0 |
| Owner and deadline match | 1 | 1 |
| Unassigned work remains unknown | 1 | 0 |
A scores 4 out of 4 and B scores 2 out of 4 under this deliberately small rubric. Both are readable. B fails because it changes weekly to monthly and invents Leo. These scores describe fictional answers, not measured performance of a named product.
Add a critical-failure rule
A total can hide an unacceptable error. Before testing, decide whether invented owners, leaked private data or unsupported commitments automatically disqualify an answer from your workflow. Record that rule alongside the points rather than changing it after seeing which product you prefer.
Next test a different source with a missing deadline and another with two owners. Keep inputs and instructions identical across the candidates. Record each response separately; a good average must not erase a failure that matters in practice.
Make the decision proportional to the test
A few examples can identify issues to investigate, but cannot establish a universal winner or a statistical reliability rate. For an actual recurring workflow, extend the set with representative cases, repeat after material product changes and keep human review for consequential decisions. Retain the date, settings, raw outputs and scoring notes so another reader can understand your choice.
Comments