When you need a number you can track across releases, you need an evaluation task. An expert scores a single model output against a rubric, either pass/fail or on a scale. Unlike a preference task, there is no second answer to compare against: the standard is the rubric, and the job is to apply it consistently.
The rubric is the whole game
An evaluation is only as good as its rubric. A vague instruction (rate the quality from 1 to 5) produces five graders and five private definitions of quality. A good rubric names the dimensions, defines each level with an example, and tells the grader how to handle partial credit and edge cases. The aim is that two qualified experts, grading the same output, land in the same place.
Pass/fail or a scale?
Pass/fail is best for gates: does this output meet the bar to ship, yes or no? It is fast, hard to game, and easy to act on. A scale (say 1 to 5, or a set of labelled bands) carries more information and is better for tracking gradual improvement, but it demands a tighter rubric so the bands mean the same thing to everyone. Pick the one that matches the decision you will make with the result.
Measuring the graders, not just the model
Because evaluation produces a headline number, it is worth checking that the number is trustworthy. Send some items to more than one grader and watch inter-rater agreement. High agreement means your rubric is doing its job. Low agreement is a signal that the rubric is ambiguous, not that your experts are careless, and it is far cheaper to catch before the number reaches a slide.
Where evaluation fits
Evaluation is how you benchmark a release, run a regression check, and set a quality gate. Preference tells you which of two answers is better; evaluation tells you how good a single answer is against a fixed standard. In a data programme, evaluation is the measurement stage: the part that tells you whether the fine-tuning and preference work actually moved the model.
On Pathwize
An evaluation task on Pathwize sends each output to credential-verified experts, records the score and the rubric it was graded against, and keeps a signed trail of who graded what. You get a benchmark you can defend and reproduce, not a number whose origin you cannot reconstruct. Create one from the AI tasks console and set how many experts grade each item.