Compare AI Assistants With a Rubric Before Picking a Winner

A single impressive reply is a weak basis for choosing an assistant for recurring work. First decide what a usable answer must contain. Then compare outputs against the same rubric, using tasks that resemble your actual workload.

Define a small evaluation set

Choose three different tasks: a reference-based summary, a rewrite with firm constraints, and a structured extraction. Use fictional or public material that you are authorised to share. Include one awkward case, such as a missing field, rather than testing only clean examples.

For a fictional meeting note, ask every assistant to extract decisions, owners and dates, marking absent values as unknown. Provide identical instructions and inputs. Record product settings you can observe, the date, and whether browsing or other tools were enabled.

Score observable outcomes

A simple rubric can award one point each for complete required fields, faithful use of the source, correct handling of unknowns, and usable formatting. Add a separate count for corrections you had to make. Do not invent timing measurements from how fast a response felt.

Keep the original outputs. A score of three is more informative when the missing point can be traced to an invented deadline than when it appears as an unexplained total. Do not combine factual errors and cosmetic preferences into one vague quality rating.

Interpret the result narrowly

Repeat a failed task after improving an unclear instruction, while retaining the first result. That distinguishes a prompt problem from a persistent limitation. If you change the rubric after seeing the outputs, disclose the change and rescore all candidates.

Choose based on your tested workflow, correction burden and practical constraints. These small trials are not a universal model ranking, a controlled benchmark, or proof that the same product will behave identically next month.