FewHumans
An open-source statistical tool library for LLM evaluation. It determines which samples require human annotation and uses prediction-powered inference to combine human labels with judge labels, providing confidence intervals with statistical guarantees.
Timeline (last 90 days)
FewHumans is an open-source library for deciding which samples should go to human annotation, fusing human labels with judge labels using prediction-powered inference (PPI), and determining when to stop annotating.
In practice FewHumans saves only about 9% of human labels (interval estimate 0.91 [0.87, 0.95]), not the 10x claimed.
The FewHumans library can be installed via pip install fewhumans, includes 47 tests, and provides an example script examples/toy_end_to_end.py.
The limitations the authors list include: the 9% saving is not an order-of-magnitude improvement, the 7 tasks are skewed toward English, there are only three mid-tier judge models, each configuration was run only once, ratio-type estimators such as F1 are not yet supported, and active routing requires the workflow to log probabilities while some production systems lack such logs.
All Facts (4)
FewHumans is an open-source library for deciding which samples should go to human annotation, fusing human labels with judge labels using prediction-powered inference (PPI), and determining when to stop annotating.
50%UnverifiedIn practice FewHumans saves only about 9% of human labels (interval estimate 0.91 [0.87, 0.95]), not the 10x claimed.
50%UnverifiedThe FewHumans library can be installed via pip install fewhumans, includes 47 tests, and provides an example script examples/toy_end_to_end.py.
50%UnverifiedThe limitations the authors list include: the 9% saving is not an order-of-magnitude improvement, the 7 tasks are skewed toward English, there are only three mid-tier judge models, each configuration was run only once, ratio-type estimators such as F1 are not yet supported, and active routing requires the workflow to log probabilities while some production systems lack such logs.
50%