There is a failure mode in RLHF that almost no one puts on a slide: some of the human feedback you paid for was produced by a machine. A contributor pastes the task into an LLM, lightly edits the output, and submits it as their own judgment. It looks human, passes a quick review, and quietly poisons the reward model you are training.
Why it happens
It is not usually malice. Contributors are paid per task, deadlines are tight, and a capable model is one tab away. When the incentive is speed and the oversight is thin, routing tasks through an LLM is the path of least resistance. The more a platform pushes volume over quality, the worse it gets.
Why it is so damaging
The whole point of RLHF is to teach a model what humans prefer. If the 'human' preferences are actually a model's preferences, you are training your model to imitate another model, and inheriting its blind spots and errors, while believing you captured genuine human judgment.
It is also self-concealing. The contaminated answers are fluent and confident, so they sail through light review. The damage only surfaces later, as unexplained quality drift in evaluations, by which point it is baked into the training set.
Why the usual defences fail
Spot checks sample randomly and assume the rest is fine, but contamination clusters around specific contributors, task types and deadlines. Attestations ('I did not use AI') are unenforceable. And blunt AI-text detectors are unreliable on short, edited answers. None of these give you something you could show a customer.
Design against it, do not just detect it
The durable fix is structural: verify contributors, capture the human step as work happens, score trust per batch rather than per person, and monitor live agreement so drift shows up immediately. Combined with behavioural signals (implausible timing, boilerplate phrasing, characteristic model errors), this lets you quarantine a suspect batch instead of distrusting a whole dataset.
The goal is not to win an arms race with detectors. It is to make the authentic human signal the path of least resistance, and to be able to prove it.
Keep the human signal you paid for
Pathwize pairs credential-verified experts with recorded human oversight and per-batch trust, so the feedback is genuinely human and you can evidence it. Book a demo to pressure-test it on a sample of your pipeline.