Define quality in terms of the task
Choose criteria that reflect the intended work. For a directory summary, quality might mean preserving required facts, avoiding unsupported additions, following the requested structure, and remaining readable. For a category suggestion, the important question may be whether the record belongs in the proposed category and when the workflow should request review.
The LLM prompt and context guide helps define the input and output contract. Use that contract as the starting point for evaluation, rather than accepting a response because it sounds confident.
Build a representative evaluation collection
Prepare a small collection of realistic tasks with expected characteristics. Include ordinary examples, incomplete records, conflicting notes, long inputs, and cases that should remain unresolved. Keep the authoritative source material available so reviewers can explain why an output is acceptable.
- Write an understandable rubric with examples of acceptable results.
- Record the model identifier, relevant settings, prompt, and source version.
- Review factual accuracy and formatting as separate criteria.
- Keep some examples outside routine prompt editing for later checks.
- Record the reason for a failure and the action it requires.
Compare environments using the same tasks. The local model guide and frontier model guide identify practical questions to include alongside output quality.
Control changes so the results are interpretable
When investigating a problem, change one meaningful part of the workflow at a time. A prompt revision, different context selection, and model replacement can each alter the result. If you change all three together, record that the comparison evaluates the combined revision rather than attributing the difference to one cause.
Review difficult cases after each accepted change. Preserve the previous working configuration and note why the new one was selected. This record helps a later maintainer understand decisions without repeating the entire investigation.
If reviewers disagree, examine the rubric before collecting more examples. Ask each person to explain which source detail or output requirement informed the judgment. Refine an ambiguous criterion and revisit the affected cases. A comparison is easier to interpret when reviewers share a clear standard.
Common mistake: measuring only the easiest examples
Examples used to demonstrate a workflow may omit its awkward boundaries. Add cases that expose missing information, ambiguous categories, and conflicting evidence. Define a useful fallback for each. An appropriate request for human review can be a better outcome than a forced answer.
Assign responsibility after the experiment
Decide who reviews reported problems, updates source material, evaluates new configurations, and restores a previous version if needed. Keep the evaluation collection current when the audience or task changes. A quality standard built for short directory summaries may not fit a later workflow that drafts detailed guidance.
The local and frontier model comparison article offers a practical way to connect quality, deployment conditions, and ongoing maintenance in one decision.
