An AI demonstration usually begins with an example that works. An evaluation set should do something different: reveal the circumstances in which the workflow stops being useful, trustworthy or safe enough for its intended role. The aim is not to embarrass the system. It is to discover its limits before ordinary users discover them during important work.
A useful test collection can start small, but it should reflect the decisions the application actually makes. Twenty carefully chosen tasks with clear review criteria may teach a team more than a large collection of generic questions. The number is a planning choice, not a universal standard. Coverage, evidence and repeatability matter more than a round total.
Define the unit of success
Describe one completed task in terms that a process owner can recognise. For a quotation assistant, success might mean extracting the requested items, preserving uncertainty about quantities and identifying missing delivery details. A pleasant paragraph is not enough if the estimator still has to reread the request to find the essential information.
Separate the quality of the answer from the quality of the action. An assistant may explain a customer issue correctly but update the wrong record. It may also produce an acceptable result after several unnecessary retries. Those are different behaviours and should have different checks rather than being hidden inside one overall rating.
Write down failures that are unacceptable regardless of other strengths. Examples include exposing restricted evidence, inventing an approval or taking an action outside the user's permissions. A test that passes on average while allowing one of these failures should not be treated as a successful release.
Collect examples from the actual workflow
Ask the people doing the work for recent examples that were easy, difficult and misleading. Include the inputs they really receive, not cleaned-up versions that remove the ambiguity. Preserve the relevant structure of forms, email threads and attachments while removing or replacing information that should not enter the test environment.
Use authorised, appropriately protected records. A convenient testing exercise is not a reason to copy customer or employee information into an uncontrolled tool. Where real records are unsuitable, construct synthetic examples that reproduce the important features and label them clearly as synthetic.
Record why each example exists. A case may test a missing field, a conflict between sources, a revised instruction or a duplicate submission. That reason helps the team maintain coverage when cases are replaced later. Without it, a test collection can gradually become another selection of easy examples.
Include the work that should stop
Some tasks should result in a clarification, a refusal to act or a transfer to a person. Include those outcomes in the evaluation set. Otherwise, a system can appear more capable merely by producing an answer to everything, including questions for which it has no supporting information.
For an internal assistant, a valuable test might ask about a procedure that is absent from the approved knowledge base. The desired behaviour could be to identify the missing evidence and point to the appropriate owner. A plausible invented procedure would fail even if it is well written and broadly sensible.
Include permission boundaries too. The same question from two users may legitimately produce different results because they can access different documents. Make those expectations explicit and test that citations, summaries and cached responses do not bypass the underlying access rules.
Write review criteria before running the test
For each case, define the required facts, unacceptable claims and allowed actions. Where several answers could be valid, specify the properties they must share instead of requiring an exact wording match. This keeps the evaluation focused on business usefulness rather than stylistic similarity.
Create a simple scoring guide with examples of a pass, a repairable issue and a blocking failure. Ask two reviewers to score a small sample independently. If they disagree frequently, clarify the guide before using their scores to compare models or releases. Ambiguous criteria create noisy results that look more objective than they are.
Keep human judgement visible. Automated checks are useful for required fields, valid formats and forbidden actions. They are less reliable as the only authority on nuanced interpretation. If another model helps score answers, treat its assessments as evidence to examine, not as a replacement for a meaningful review process.
Separate development examples from final checks
Developers will naturally improve the system against examples they can see. That is useful, but it can make the same examples a weak measure of generalisation. Reserve a separate set for release checks and avoid repeatedly tuning the workflow around its exact contents.
The separation does not need to be elaborate at first. Maintain one collection for diagnosis and another for independent checks, with ownership and access appropriate to the team. Add new examples from real failures over time, while retaining enough stable cases to compare changes consistently.
Do not treat one successful run as proof of stable performance. Where the system can vary between runs, repeat important cases and inspect the spread of outcomes. The right repetition count depends on the risk and operating context; the key is to recognise variability rather than hide it behind one convenient result.
Keep a record of the system under test
An evaluation result needs context. Record the model configuration, prompt version, tool permissions, retrieval settings and source snapshot used for the run. If those details change, a score without its configuration becomes difficult to interpret or reproduce.
Capture intermediate evidence where practical. A weak final answer may come from poor retrieval, an incorrect tool result or a drafting error. Microsoft distinguishes retrieval assessment from answer groundedness and relevance in its RAG evaluator documentation. That distinction is useful because different faults need different repairs.
Keep logs proportionate and protected. Store what is necessary to investigate the behaviour, not every sensitive field by default. Reviewers should be able to understand a failure without receiving broader access than their role requires.
Use the results to make a release decision
Summarise outcomes by meaningful task group. If ordinary requests pass but revised instructions fail, that is more useful than a blended average. Show which failures remain, what controls reduce their impact and what users must do when the system reaches its boundary.
Agree on a release decision before looking for a flattering score. The process owner should know whether the evidence supports a limited pilot, a broader rollout or more work. A smaller release with explicit limits is often easier to operate than a broad release supported by a vague confidence statement.
An evaluation set is a maintained operational asset. Add real failures, retire obsolete assumptions and rerun the relevant checks when prompts, tools or models change. A practical AI implementation becomes more dependable when the team can explain not only what works, but also how it knows and where the evidence stops.