Inter-Annotator Agreement
Definition
Inter-annotator agreement is a measure of how consistently two or more people assign the same labels to the same data, used to judge whether a labelling task is reliable.
If qualified people disagree a lot on the same items, either the task is genuinely ambiguous or the guidelines are unclear. Measuring agreement, often with statistics such as Cohen's or Fleiss' kappa, surfaces that problem before it is baked into a dataset.
High agreement among qualified raters is evidence that the labels are reliable. Low agreement is a signal to refine the guidelines, add examples, or route disagreements to a reviewer. Agreement is therefore both a quality metric and a diagnostic tool.
Agreement is usually measured on a sample where more than one person labels the same items, running alongside the main annotation work.
How Pathwize relates
Pathwize uses a sampled second rater and disagreement review as standard, so inter-annotator agreement is measured and acted on rather than assumed.
Frequently asked
A measure of how consistently different people assign the same labels to the same data.
Low agreement warns that a task is ambiguous or the guidelines are unclear, before unreliable labels enter the dataset.
By having more than one person label the same sample and computing an agreement statistic such as kappa.
Related terms