The preference task is the workhorse of RLHF. An expert is shown two model outputs for the same prompt and asked a deceptively simple question: which is better, and why? Collect enough of these pairwise judgments and you can train a reward model that captures what people actually prefer, then use it to steer the model itself.
Why pairwise, not a score
You could ask an expert to score each answer out of ten, but people are inconsistent at absolute scores and drift over a shift. Comparison is far more stable: humans are good at saying which of two things is better even when they struggle to put a number on either. Pairwise data is the signal reward models are built to learn from.
The rationale is not optional
A bare choice (A over B) tells you what an expert preferred but not why. The rationale is where the value is. It surfaces the tie-breaker the expert applied, catches the case where they picked correctly for the wrong reason, and gives your team something to audit when two experts disagree. A preference task without a rationale is throwing away half the signal.
Designing a good preference task
Keep the two outputs genuinely comparable: same prompt, similar length, no giveaway formatting that biases the choice. Tell experts what dimension to weigh (helpfulness, safety, factual accuracy) so different raters optimise the same thing. And allow a considered tie, because forcing a choice between two equal answers manufactures noise.
Use more than one rater per item where the judgment is subtle, and treat disagreement as information: a pair that splits your experts is often a pair worth a second look.
Where preference data goes wrong
The classic failure is contaminated feedback: a contributor quietly routes the task through an LLM and submits the model's preference as their own. You then train your model to imitate another model while believing you captured human judgment. The defence is provenance and real expertise, knowing who judged each pair, under what instructions, and that they were qualified to.
On Pathwize
A preference task on Pathwize routes each pair to credential-verified experts and records the choice, the rationale, and who made it, as a signed trail. You get comparison data you can defend, not just a spreadsheet of A-or-B. Create one from the AI tasks console and set how many experts should judge each item.