Human Annotation
The stage in a machine learning pipeline in which trained personnel read, classify, score or correct data samples to produce high-quality labels for training and evaluating models or for diagnosing defects. In the era of large models, it extends to human scoring of model outputs for usefulness, safety and factuality.
Timeline (last 90 days)
Human annotation is a common step in machine learning pipelines, in which trained personnel read, classify, score or correct data samples, producing high-quality labels used to train models, evaluate model outputs, or investigate model defects.
In the era of large models, human annotation also includes manual scoring of the helpfulness, safety and factuality of model responses (often referred to as human preference data in RLHF).
Existing methods for obtaining process labels rely on manual annotation, rollout-based estimation, or auxiliary verification, which are costly and difficult to scale.
The research team first manually annotated a portion of the parliamentary records as ground truth, then had the LLM perform event extraction on the same batch of texts, and calibrated the model outputs by computing metrics such as precision and recall to verify extraction quality.
When reviewing samples that mix real and synthetic text, human annotators anchor on the surface fluency of synthetic text, thereby systematically favoring synthetic samples.
The project combines four types of evaluation methods: rule-based deterministic checks, LLM-based judge evaluation, sampling evaluation based on manual annotation, and online evaluation based on user feedback.
All Facts (6)
Human annotation is a common step in machine learning pipelines, in which trained personnel read, classify, score or correct data samples, producing high-quality labels used to train models, evaluate model outputs, or investigate model defects.
50%UnverifiedIn the era of large models, human annotation also includes manual scoring of the helpfulness, safety and factuality of model responses (often referred to as human preference data in RLHF).
50%UnverifiedExisting methods for obtaining process labels rely on manual annotation, rollout-based estimation, or auxiliary verification, which are costly and difficult to scale.
50%UnverifiedThe research team first manually annotated a portion of the parliamentary records as ground truth, then had the LLM perform event extraction on the same batch of texts, and calibrated the model outputs by computing metrics such as precision and recall to verify extraction quality.
50%UnverifiedWhen reviewing samples that mix real and synthetic text, human annotators anchor on the surface fluency of synthetic text, thereby systematically favoring synthetic samples.
50%UnverifiedThe project combines four types of evaluation methods: rule-based deterministic checks, LLM-based judge evaluation, sampling evaluation based on manual annotation, and online evaluation based on user feedback.
50%