Most data vendors demo well. The gap shows up later, when a batch fails review, an auditor asks where a label came from, or a scarce expert you were promised turns out to be an anonymous contractor. Due diligence is about surfacing that gap before you sign, not after. Here is the checklist we would use.
Who actually does the work?
Start with the people. Ask how contributors are sourced, how their credentials are verified, and whether you are getting the specialists you were shown in the pitch or a general pool behind them. For hard domains, generalist throughput is the wrong thing to optimise for.
Ask how the vendor prevents undisclosed AI use, where a contributor quietly pastes model output instead of applying their own judgment. If the answer is a policy rather than a detection method, treat the human-labelled claim as unproven.
Can they prove provenance, not just assert quality?
A quality score is a summary. Provenance is a record. For any item in a delivered batch, can the vendor show who produced it, under what instructions, who reviewed it, and when? That trace is what survives a dispute or an audit, and it is the single most useful thing to test during a pilot.
This matters even more under the EU AI Act, where high-risk systems need documented data governance. A vendor that captures provenance as work happens saves you from reconstructing history under deadline.
What quality signals do they expose?
Look for signals you can see in flight, not just a final accuracy claim. Overlapping assignments and live inter-rater agreement tell you whether a task is well defined before a batch is delivered. Batch-level trust indicators tell you where to focus review. If quality is a black box, you are buying trust rather than evidence.
Where does the data live?
For EU teams, data residency and lawful processing are not optional extras. Ask where data is stored and processed, who has access, how contributors are classified, and whether the vendor can produce a data processing agreement without a fight. EU-native by default is very different from EU-available on request.
Run a paid pilot with a pass or fail bar
Do not decide on a deck. Run a small paid pilot on a representative slice of real work, define what pass looks like up front (agreement thresholds, turnaround, a clean provenance export), and check the trace yourself. A vendor confident in their process will welcome it.
Pathwize is built so provenance, expert qualification and quality signals are visible from the first pilot batch. Book a demo to run one against your own tasks.