ExLabel
An LLM-based, self-evolving, experience-guided evaluation assistant used to assess task clarity and test coverage of SWE datasets. Its core mechanism is an experience bank.
Timeline (last 90 days)
ExLabel is an LLM-based, self-evolving, experience-guided assistant specifically designed for task clarity and test coverage evaluation in SWE datasets.
At the core of ExLabel is an experience bank that distills knowledge across instances into stable and adaptive judgment criteria and reuses relevant experience as guidance when evaluating new instances.
Experiments on three backbone LLMs show that ExLabel achieves the best F1 score under all settings.
On the Python and multilingual subsets, ExLabel's task clarity evaluation F1 improved by 44.2% and 49.9% relative to the best baseline, respectively.
Fine-tuning a coding model on SWE-rebench data filtered with ExLabel, the model solved 66.2% of problems on SWE-bench Verified, 14.5% higher than the model trained on data filtered with the best baseline.
All Facts (5)
ExLabel is an LLM-based, self-evolving, experience-guided assistant specifically designed for task clarity and test coverage evaluation in SWE datasets.
50%UnverifiedAt the core of ExLabel is an experience bank that distills knowledge across instances into stable and adaptive judgment criteria and reuses relevant experience as guidance when evaluating new instances.
50%UnverifiedExperiments on three backbone LLMs show that ExLabel achieves the best F1 score under all settings.
50%UnverifiedOn the Python and multilingual subsets, ExLabel's task clarity evaluation F1 improved by 44.2% and 49.9% relative to the best baseline, respectively.
50%UnverifiedFine-tuning a coding model on SWE-rebench data filtered with ExLabel, the model solved 66.2% of problems on SWE-bench Verified, 14.5% higher than the model trained on data filtered with the best baseline.
50%