AI Evals for Knowledge Work: Test Sets, Rubrics, and Failure Thresholds
Build a local AI evaluation from representative cases, anchored rubrics, critical blockers, and versioned results rather than anecdotal demonstrations.
Build an AI eval by sampling real work, stratifying routine and difficult cases, defining observable criteria with anchored examples, and setting critical failure thresholds before testing. Preserve raw outputs and error categories. Use aggregate scores for comparison, but never let them average away a release blocker. A finite test set cannot establish safety for every novel input, and model graders can reproduce biases or miss subtle domain errors.
A benchmark answers only the question it was built to ask
Public benchmarks are useful for research and model comparison, but they rarely reproduce your documents, users, exceptions, or consequences. A high score on a general exam does not establish that a system can extract clauses from your contracts without inventing one.
HELM argues for evaluation across scenarios and multiple metrics including accuracy, calibration, robustness, fairness, bias, and efficiency. helm, nist, openai NIST treats testing, evaluation, verification, and validation as part of operational risk management. nist Provider guidance similarly recommends task-specific evals and iterative measurement. openai
Start with the decision
Write:
We will use this eval to decide whether system X, under configuration Y, may assist user Z with task T, subject to release rule R.
That sentence prevents a prototype score from becoming a general endorsement. Define the baseline too: current human process, earlier model, deterministic method, or no intervention.
Construct a representative test set
Sample from real input categories:
- ordinary, frequent cases;
- rare but consequential cases;
- messy or incomplete inputs;
- languages and populations in scope;
- known historical failures;
- ambiguous cases that should trigger escalation;
- hostile or policy-violating inputs where relevant.
Start small enough to inspect deeply—perhaps several dozen cases—but do not pretend that convenience sampling is representative. Add cases when production reveals a new failure, preserving a locked subset for honest comparison.
Write anchored criteria
A rubric should describe observable output properties:
| Criterion | Fail | Acceptable | Strong | |---|---|---|---| | Source fidelity | Invents or contradicts source | Main claims supported | Claims linked to exact passages | | Coverage | Omits mandatory field | All mandatory fields present | Includes useful, permitted context | | Uncertainty | Hides missing evidence | Marks uncertainty | Explains decision impact and escalation | | Format | Cannot be consumed | Valid required structure | Clear without changing substance |
Give graders examples at each anchor. If experts disagree, examine why before averaging their scores.
Set blockers and thresholds
Some failures are not merely “low quality.” Define blockers such as:
- any fabricated authority;
- disclosure of prohibited data;
- external action without approval;
- discriminatory treatment above a defined limit;
- failure to escalate a safety-critical ambiguity.
Then set comparative thresholds for noncritical criteria: accuracy, time, correction burden, cost, or user comprehension. Thresholds should follow consequence and baseline—not the score the new system happens to achieve.
Evidence for the diagnosis: the Local AI Eval Builder
Choose graders with care
Deterministic checks are strongest for schemas, calculations, required fields, and source IDs. Expert humans are necessary for nuanced truth, context, and professional judgment. Model graders can scale comparison when calibrated against human judgments, but they should not silently grade the same blind spots as the system under test.
For a subset, use multiple qualified graders. Record disagreements, adjudication, and rubric revisions. Never overwrite earlier results after changing criteria.
Build the first credible local eval
- Name the use decision, system boundary, and baseline.
- Collect representative cases with source-of-truth material.
- Tag strata before seeing system results.
- Write criteria, anchors, blockers, and escalation outcomes.
- Run the baseline and candidate without cherry-picking.
- Preserve raw output, configuration, grader, and timestamp.
- Analyze failures by type and stratum, not only total score.
- Decide: reject, redesign, pilot with controls, or approve within scope.
Define quality with the Four-Layer Model, preserve state with Why AI Answers Change, and extend the suite with attacks from Prompt Injection Explained.
Give the result an accountable reader
An eval has no operational meaning until a decision owner states what the score permits. Preserve the item-level results, representative test set, grader disagreements, blocker failures, and system configuration in a review packet. A human checkpoint must be able to inspect the cases behind an average and reject release when a high-consequence failure appears.
Run a shadow decision: ask the owner what they would do from the aggregate score alone, then reveal the worst failures and ask again. If the decision reverses, the dashboard hid material evidence. Repair the reporting layer before changing the model. This exercise also distinguishes evaluation uncertainty from model error; sometimes graders disagree because the criterion or reference answer is underspecified. The accountable response is to narrow the claim, clarify the rubric, or collect expert adjudication—not to select the most favorable number.
How evals become confidence machines
- Building the test set from examples the prompt already solved.
- Reporting one average without subgroup or failure analysis.
- Changing the rubric after seeing a disappointing score.
- Using a model grader without human calibration.
- Removing hard cases as “unfair.”
- Testing content quality but not tools, permissions, and action traces.
- Treating a passing pre-release set as permanent proof.
Passing is evidence not immunity
A finite eval cannot cover an open world. Ground truth may be disputed, expert grading costly, and production inputs may shift. Repeated tuning can overfit the test set. Security, privacy, legal, and societal impacts may require methods beyond output evaluation. A pass supports only the stated system, task, population, configuration, and release rule.
The point of an eval is not to manufacture confidence. It is to make a bounded deployment decision easier to challenge with evidence.
Named sources
Evidence and further reading
Published July 29, 2026. No substantive revision has been recorded. Evidence last verified July 28, 2026.