Agent Micro-Task Benchmark
A benchmark designed to evaluate the capabilities of small language models in agent system micro-task scenarios. It comprises four representative micro-tasks and applies confidence-interval-aware qualification rules for rigorous assessment of model performance.
Timeline (last 90 days)
The study designed 4 agent micro-tasks, each with fixed prompts and automated evaluation metrics, and pre-set a threshold τ.
Among all 16 configurations across 4 tasks × 4 models, not a single one passed the qualification threshold.
The study categorized failures into two types: genuine capability deficits and issues that can be mitigated by adjusting the decoding threshold, further subdividing them into 4 failure mechanisms.
The core insight of the study is that the qualification gap is primarily determined by model size rather than precision.
Adding 3 neutral paraphrased versions per cell beyond the original prompt, 0 out of 112 configurations qualified (0/112).
All Facts (5)
The study designed 4 agent micro-tasks, each with fixed prompts and automated evaluation metrics, and pre-set a threshold τ.
50%UnverifiedAmong all 16 configurations across 4 tasks × 4 models, not a single one passed the qualification threshold.
50%UnverifiedThe study categorized failures into two types: genuine capability deficits and issues that can be mitigated by adjusting the decoding threshold, further subdividing them into 4 failure mechanisms.
50%UnverifiedThe core insight of the study is that the qualification gap is primarily determined by model size rather than precision.
50%UnverifiedAdding 3 neutral paraphrased versions per cell beyond the original prompt, 0 out of 112 configurations qualified (0/112).
50%