Inter-Annotator Agreement
A statistical concept measuring the degree to which multiple annotators produce consistent results when independently labeling the same data. Common metrics include Cohen's Kappa and Krippendorff's Alpha, and it serves as a foundation for assessing human annotation quality.
Timeline (last 90 days)
Human annotators themselves are inconsistent, which sets a 'ceiling' for evaluating models; if even agreement among professional linguists can only reach a moderate level, then human-annotated data itself carries noise.
GPT-4 as a judge can achieve over 80% agreement with human evaluations, but exhibits systematic issues such as position bias, verbosity bias, and self-preference.
数据标注评估考核中常见的质量指标包括标注者间一致性（IAA）和与黄金标准的偏差
All Facts (3)
Human annotators themselves are inconsistent, which sets a 'ceiling' for evaluating models; if even agreement among professional linguists can only reach a moderate level, then human-annotated data itself carries noise.
50%UnverifiedGPT-4 as a judge can achieve over 80% agreement with human evaluations, but exhibits systematic issues such as position bias, verbosity bias, and self-preference.
50%Unverified数据标注评估考核中常见的质量指标包括标注者间一致性（IAA）和与黄金标准的偏差
50%