Vendor shortlist
Independent model evaluation services compared
Independent evaluation means someone who did not build the model grades it, with rubrics the buyer can inspect. Three kinds of provider do this: benchmark bodies, evaluation software, and expert graders.
Benchmark bodies publish scores on fixed tasks. Evaluation software scores your own traffic automatically with model-based judges. Expert graders apply a rubric by hand and write down why. For regulated deployments where correctness has to be objectively verified, the third kind is what auditors accept, and the rubric has to come from someone who holds the credential.
The providers below are the ones named in 2026 shortlists for each kind.
kinds of evaluation provider
providers compared
based in the EU or Switzerland
that grades with credentialed experts from the EU
Vendor facts come from the vendors' own material and widely cited public sources as of October 2026; pricing is per quote unless the vendor publishes it. Pathwize is EU-native: all data stays within the EU.
How we compared
01
Who grades
A benchmark body, a model acting as judge, or a human with the relevant credential. Each answers a different question.
02
Rubric ownership
Whether you can read, change and audit the rubric, and whether experts validated it.
03
Evidence per grade
Rationale, grader identity, time spent, agreement between graders. Without this a score is a number.
04
EU data residency
Evaluation data is often production data. Where it goes matters.
The shortlist
| Vendor | Model | Best for | Pricing | EU residency |
|---|---|---|---|---|
Vals AI US | Independent benchmarks on real-world work (finance, law, software), built in-house Runs its own evaluations rather than selling a platform. | Independent published scores on professional tasks | Published benchmarks; custom evaluations per quote | No |
METR Berkeley, US (nonprofit) | Frontier model risk and capability evaluations Labs provide model access; relevant to the independence question. | Safety and capability assessments of frontier models | Nonprofit, engagement-based | No |
Patronus AI US | Evaluation and guardrail software for LLM apps, RAG and agents; Lynx hallucination judge Model-based judging; tracing and monitoring need separate tools. | Automated scoring of agent traces in regulated apps | Per quote; demo tier only | No |
LatticeFlow AI Zurich, CH | Model assessment software, COMPL-AI benchmark mapped to the EU AI Act ETH Zurich spinout; public AI Atlas registry. | EU AI Act technical compliance checks | Per quote | Yes |
Giskard Paris, FR | Open-source testing plus commercial hub Strong on hallucination and safety test suites. | Continuous automated evaluation with EU alignment | Open source free; enterprise per quote | Yes |
Surge AI San Francisco, US | Curated human graders Positioned at the top of the price range. | Premium human evaluation for frontier labs | Per quote | Partial |
Scale AI San Francisco, US | Managed human evaluation plus automated leaderboards Also publishes its own leaderboards. | Large evaluation programs | Per quote | Partial |
Pathwize Cologne, Germany | Credentialed experts and students from a 200k+ network, qualification test on your own data, paid per hour Graders hold the credential; each grade carries rationale, grader, time and agreement on a sample. | Expert-graded evaluation with inspectable rubrics in medicine, engineering, law, finance | Per hour, quoted per task; free pilot on 50 items | Yes |
Vals AI
US
Independent benchmarks on real-world work (finance, law, software), built in-house
Best for: Independent published scores on professional tasks
Pricing: Published benchmarks; custom evaluations per quote
Runs its own evaluations rather than selling a platform.
METR
Berkeley, US (nonprofit)
Frontier model risk and capability evaluations
Best for: Safety and capability assessments of frontier models
Pricing: Nonprofit, engagement-based
Labs provide model access; relevant to the independence question.
Patronus AI
US
Evaluation and guardrail software for LLM apps, RAG and agents; Lynx hallucination judge
Best for: Automated scoring of agent traces in regulated apps
Pricing: Per quote; demo tier only
Model-based judging; tracing and monitoring need separate tools.
LatticeFlow AI
Zurich, CH
Model assessment software, COMPL-AI benchmark mapped to the EU AI Act
Best for: EU AI Act technical compliance checks
Pricing: Per quote
ETH Zurich spinout; public AI Atlas registry.
Giskard
Paris, FR
Open-source testing plus commercial hub
Best for: Continuous automated evaluation with EU alignment
Pricing: Open source free; enterprise per quote
Strong on hallucination and safety test suites.
Surge AI
San Francisco, US
Curated human graders
Best for: Premium human evaluation for frontier labs
Pricing: Per quote
Positioned at the top of the price range.
Scale AI
San Francisco, US
Managed human evaluation plus automated leaderboards
Best for: Large evaluation programs
Pricing: Per quote
Also publishes its own leaderboards.
Pathwize
Cologne, Germany
Credentialed experts and students from a 200k+ network, qualification test on your own data, paid per hour
Best for: Expert-graded evaluation with inspectable rubrics in medicine, engineering, law, finance
Pricing: Per hour, quoted per task; free pilot on 50 items
Graders hold the credential; each grade carries rationale, grader, time and agreement on a sample.
Where Pathwize fits
Pathwize provides the human expert layer: credentialed graders apply your rubric, or validate it first, and write the rationale for every grade. You see who graded, how long it took, and agreement between two graders on a sample.
Experts qualify on your own items before they grade anything, so the rubric is tested against real cases before the run. Disagreements go to a reviewer; the audit trail is exported with the grades.
For automated coverage of your traffic, pair it with evaluation software. Pathwize is the part that has to be defensible in front of a regulator or a customer.
FAQs
Yes. Pathwize matches credentialed experts (physicians, engineers, lawyers, scientists) to your evaluation task, tests them on your items first, and lets you approve who grades. Surge, Scale and Mercor offer expert graders in the US.
For published independent benchmarks: Vals AI and METR. For your own models graded by people who did not build them, with an inspectable rubric: expert evaluation services like Pathwize, Surge or Scale.
Expert-validated rubrics need humans with the credential: Pathwize does this as part of the qualification round. Automated platforms (Patronus, Giskard, LatticeFlow) then apply the rubric at scale.
Usually within a few days. You send the task, the classes or rubric and a few examples; qualified experts take the test on your data; you approve who continues. A 50-item pilot is free.
Per hour of expert work, quoted per task depending on the domain and the complexity. Time is tracked automatically per item, so you see cost per item and per class in the dashboard and in the export.
In the EU. Pathwize is based in Cologne, Germany; storage and processing run on EU infrastructure, and experts work from Europe. Sensitive data can be worked in a no-download environment.
Related