Define the work the sample represents
Name the staff roles, task types and source collection included in the pilot. In a fictional equipment business, the first scope is internal questions about product-care documentation, not every operational policy. Ask staff for examples of questions they actually need answered. Keep the scope explicit so a good result cannot later be described as proof for the whole company. Avoid choosing questions only because someone already knows the assistant will answer them well.
Include different ways people ask
Collect direct questions, short follow-ups and requests using the language staff normally use. Include a question with missing context that should prompt clarification. Keep fictional or approved examples free of unnecessary personal details. In the fictional equipment pilot, one person may name a product precisely while another describes it informally. Record whether those versions ask the same thing. A sample should test the intended staff task without relying entirely on terminology copied from the document headings.
Choose sources with meaningful variation
Include the document formats and structures the intended collection contains, such as short instructions, longer manuals and relevant tables. Do not add unrelated files simply to increase the sample size. Have the responsible owner confirm that each selected source is suitable for the pilot’s purpose. Note preparation problems separately, including unreadable sections or missing context. The sample should make those limitations visible before testing rather than silently replacing difficult material with cleaner files and reporting only the easier result.
Write expected outcomes independently
Ask an appropriate reviewer to describe the answer or next action supported by the selected source before inspecting the system’s output. Include any qualification necessary to avoid a misleading answer. Some cases should result in clarification or referral, not a confident response. Keep the expected outcome tied to the original question. Do not let the generated answer redefine what the test was asking. Where reviewers disagree, resolve or document that uncertainty before using the case to judge the service.
Keep evaluation cases separate from practice examples
Distinguish examples used to shape the workflow from those reserved for evaluating it. If a test case is repeatedly used to improve the configuration, record that exposure rather than treating its later success as fresh evidence. Keep a stable record of selected questions and sources so omissions can be explained. Include a question the collection cannot answer and a request outside scope. Those cases help show whether the handover behaviour works, without assuming that every pilot item should produce an answer.
Report coverage and choose the next sample
Summarise the task types, source types and unresolved cases actually tested. Show the number of cases and any excluded items with reasons. A small selected sample supports conclusions about that sample, not an accuracy guarantee for all future questions. Use the observed gaps to choose the next bounded set of cases. Do not replace failed cases and present the new set as the original result. Keep the business owner involved in deciding whether the evidence supports preparation work, another pilot or a limited expansion.