ConceptExpert-in-the-loop
Domain Expert Co-construction
A methodology in AI agent evaluation that involves domain experts in judgment, having experts score agent execution trajectories segment by segment or perform pairwise comparisons, and structuring the experts' decision processes to build high-quality evaluation sets.
Timeline (last 90 days)
Oct 31
Expert-in-the-loop in Agent evaluation requires judging whether a series of actions is reasonable; such judgments are hard for non-experts to make, so the decision process of domain experts needs to be structured.
Unverified50%