Fleiss' kappa
A statistical measure of agreement among multiple annotators, applicable to settings involving more than two annotators.
Timeline (last 90 days)
Three annotators, following a written guideline, reached a Fleiss' kappa of 0.788 agreement on 816 data items.
Fleiss' κ values typically range from 0 to 1, where 0 indicates agreement equivalent to random guessing and 1 indicates complete agreement; academia generally regards 0.61–0.80 as strong agreement and above 0.81 as very strong agreement.
For annotation consistency, Cohen's kappa was 0.781 (range 0.753-0.807) and Fleiss' kappa was 0.77.
PGT-FG uses Fleiss' kappa to measure inter-annotator agreement on the shared subset, with a kappa value of 0.6521 for the overall toxicity label.
Fleiss' kappa is a statistical metric used to measure the degree of agreement among multiple annotators, applicable to scenarios where three or more annotators assign categories to the same set of samples.
All Facts (5)
Three annotators, following a written guideline, reached a Fleiss' kappa of 0.788 agreement on 816 data items.
50%UnverifiedFleiss' κ values typically range from 0 to 1, where 0 indicates agreement equivalent to random guessing and 1 indicates complete agreement; academia generally regards 0.61–0.80 as strong agreement and above 0.81 as very strong agreement.
50%UnverifiedFor annotation consistency, Cohen's kappa was 0.781 (range 0.753-0.807) and Fleiss' kappa was 0.77.
50%UnverifiedPGT-FG uses Fleiss' kappa to measure inter-annotator agreement on the shared subset, with a kappa value of 0.6521 for the overall toxicity label.
50%UnverifiedFleiss' kappa is a statistical metric used to measure the degree of agreement among multiple annotators, applicable to scenarios where three or more annotators assign categories to the same set of samples.
50%