LLM-in-the-loop
A semi-automated paradigm for constructing evaluation data: large language models handle initial screening, rewriting, candidate option generation, and difficulty stratification, while human experts are responsible for final annotation, correction, and quality control, balancing scale and quality.
Timeline (last 90 days)
The PerSo research team built an LLM-in-the-loop evaluation pipeline to generate evaluation tasks and validated it through human annotation.
LLM-in-the-loop is a semi-automated paradigm for constructing evaluation data: large language models handle stages such as initial screening, rewriting, candidate option generation, and difficulty stratification, while human experts are responsible for final annotation, correction, and quality control.
All Facts (2)
The PerSo research team built an LLM-in-the-loop evaluation pipeline to generate evaluation tasks and validated it through human annotation.
50%UnverifiedLLM-in-the-loop is a semi-automated paradigm for constructing evaluation data: large language models handle stages such as initial screening, rewriting, candidate option generation, and difficulty stratification, while human experts are responsible for final annotation, correction, and quality control.
50%