Rationale Annotation
A form of expert feedback in which annotators provide justifications alongside labels when coding cases, combining correct answers with reasoning paths. Rooted in the think-aloud methodology from behavioral psychology, it is used to help large language models improve codebook comprehension.
Timeline (last 90 days)
Rationale annotation requires experts to directly label disagreement cases and provide explanatory annotations, a method rooted in the think-aloud methodology from behavioral psychology.
In experiments on thousands of tutoring session transcripts, the rationale annotation approach achieved the highest LLM annotation accuracy at 64.9%.
The rationale annotation approach outperforms the traditional expert codebook revision approach by approximately 7 percentage points.
Having experts provide labeled annotations with rationales for specific disagreement cases is more effective than having experts directly rewrite the entire codebook.
All Facts (4)
Rationale annotation requires experts to directly label disagreement cases and provide explanatory annotations, a method rooted in the think-aloud methodology from behavioral psychology.
50%UnverifiedIn experiments on thousands of tutoring session transcripts, the rationale annotation approach achieved the highest LLM annotation accuracy at 64.9%.
50%UnverifiedThe rationale annotation approach outperforms the traditional expert codebook revision approach by approximately 7 percentage points.
50%UnverifiedHaving experts provide labeled annotations with rationales for specific disagreement cases is more effective than having experts directly rewrite the entire codebook.
50%