Are Small Models Good Enough? Benchmarking SLM Performance on Agent Micro-Tasks

When Agents Start Delegating Grunt Work to Small Models

In LLM-powered agent systems, there's a long tail of frequent but tedious "micro-tasks" to handle: auto-approving shell commands, writing to memory, selecting tools, ranking historical conversation turns. Routing all of these to frontier large models is expensive and introduces noticeable latency. A natural idea follows — can we use off-the-shelf small language models (SLMs) to absorb this low-level workload?

A study published on arXiv tackles this question head-on. The researchers built a benchmark around agent micro-tasks and systematically tested whether off-the-shelf SLMs are actually "good enough" — why they fail when they do, and whether quantization changes the picture. The verdict is unambiguous: under rigorous statistical criteria, off-the-shelf small models essentially fail across the board on these micro-tasks.

Benchmark Design: A Pass Threshold with Confidence Intervals

What makes this study rigorous is that it doesn't rely on vague impressions of model quality. The researchers designed 4 micro-tasks, each with a fixed prompt and automated evaluation metric, and pre-defined a threshold τ.

This threshold isn't arbitrary — it's anchored to a cheap, non-LLM baseline. In other words, a small model only adds value if it outperforms a traditional lightweight approach. More critically, the study uses a "confidence-interval-aware" pass rule: a configuration only passes if its confidence lower bound genuinely exceeds τ. This prevents random variation from falsely indicating a model has reached the bar — a methodologically sound safeguard.

What Was Tested and How

The researchers evaluated Qwen3 in four sizes — 0.6B, 1.7B, 4B, and 8B — under optimal conditions: FP16 precision, greedy decoding, a single frozen prompt, and no fine-tuning. This gave the small models the fairest possible environment, eliminating excuses like "it wasn't tuned properly."

Qwen3 is Alibaba's open-source language model family, spanning from 0.6B to 235B parameters and optimized for tasks like reasoning and instruction following. Greedy decoding selects the highest-probability token at each step — more deterministic than sampling, and ideal for reproducible evaluations. FP16 (half-precision floating point) is the standard inference format that halves memory usage compared to FP32 while preserving most model capability. This setup represents the most typical "deploy as-is" configuration in engineering practice, making the results directly actionable rather than contingent on specific tuning conditions.

The confidence-interval-aware pass rule is one of the study's core methodological contributions. Traditional evaluations typically report only point estimates (e.g., 78% accuracy) while ignoring the uncertainty inherent in a finite test set. A model scoring 80 on 100 samples might have a 95% confidence interval of [71%, 87%] — and a lower bound of 71% may still fall below the threshold. This study requires the lower bound of the confidence interval to exceed τ, essentially asserting with 95% confidence that the model is "genuinely capable" rather than "happened to pass this time." This strict standard matters especially in production: agent micro-tasks fire at high frequency, and a model whose true capability slightly undershoots the threshold will accumulate meaningful errors at scale. BM25, a classic term-frequency–inverse document frequency ranking algorithm from information retrieval, serves as the lightweight non-neural baseline — requiring no GPU and offering extremely low latency, making it a natural reference point for assessing whether using an LLM is even worthwhile.

The Eligibility Gap: 0 Out of 16 Configurations Pass

The results are sobering. Across 4 tasks × 4 models — 16 configurations in total — not a single one passed the eligibility threshold. The researchers manually verified raw outputs and parser behavior, confirming this wasn't a pipeline bug but a genuine capability shortfall.

This "eligibility gap" is a wake-up call for anyone hoping to slot small models directly into agent micro-tasks. At least out of the box and without fine-tuning, off-the-shelf SLMs don't meet the practical bar.

Dissecting the Failures

The study doesn't stop at "they failed." Using a log-probability-based decision threshold diagnostic (labeled T1/T3/T4, with T2 tested via context-length and cascade probes), failures were categorized into two types: genuine capability deficits, and issues that can be mitigated by adjusting decoding thresholds. The study further identified 4 distinct failure regimes, pointing the way toward future improvements — some failures aren't a matter of the model being "dumb," but of decoding strategy being misaligned.

Quantization Isn't the Answer: The Gap Tracks Model Size

A practical question that can't be ignored: if you quantize to 4-bit to save memory, does that make things worse? The study tested three 4-bit quantization schemes — RTN, GPTQ, and AWQ.

The conclusion is that quantization-induced degradation correlates with model size, but no configuration crossed the eligibility threshold in either direction due to quantization. In other words, models that were ineligible stayed ineligible; quantization didn't make things qualitatively worse or better. The researchers validated this using reconstructable hard-label tasks (T1/T3) and applied diagnostic and windowed robustness checks for T2/T4.

The core insight: the eligibility gap is primarily driven by model size, not precision. This is worth sitting with for resource-constrained deployments — blindly tweaking precision yields little benefit.

RTN (Round-To-Nearest), GPTQ, and AWQ are the three mainstream post-training 4-bit quantization schemes, each with a different approach: RTN is the simplest, rounding weights directly; GPTQ minimizes quantization error layer-by-layer using second-order gradient information; AWQ uses activation-aware weight scaling to protect precision-sensitive channels. The three offer similar compression ratios but increasingly better precision retention. Testing all three rules out "wrong quantization method" as an explanation — if even AWQ, which minimizes accuracy loss, can't help models pass the threshold, the bottleneck is clearly parameter count, not quantization strategy. The practical implication: without fine-tuning, hoping to "rescue" a small model by switching to a more sophisticated quantization method is not realistic.

Robustness Checks and Cross-Model Replication

To ensure the findings aren't a fluke, the study ran multiple robustness checks:

• Cross-model replication: Re-ran on the Llama-3.x family — all 12 configurations ineligible (12/12).
• Anchor robustness: Swept across values of threshold τ — conclusions held regardless of baseline anchor choice.
• Prompt robustness: Added 3 neutral paraphrases of each prompt per cell — 0 out of 112 configurations passed (0/112).

This level of cross-validation makes the eligibility gap finding quite credible. It's unlikely to be an artifact of poor prompt engineering or evaluation bias.

Practical Takeaway: Put Small Models Behind a Baseline

The most valuable part of this study isn't the pessimistic finding that "small models don't work" — it's the pragmatic architectural advice it offers:

Place SLMs behind a baseline that can reach the confidence-interval-supported threshold, and only invoke SLMs where the baseline falls short.

The paper gives a concrete example: using a 4B model to re-rank a shortlist of BM25 candidates outperforms BM25 alone (+0.047, CI [0.020, 0.073]) — even though the 4B re-ranker didn't pass eligibility on its own. This points to a layered collaboration approach: small models don't need to operate independently; they can serve as an incremental optimization layer on top of traditional baselines, adding marginal value where the baseline breaks down.

This "baseline as safety net + SLM as booster" design philosophy may be the right way to integrate small models into agent systems today. Rather than asking "can a small model replace a frontier model?", the better question is: "at which step can a small model deliver a verifiable, measurable improvement?"

One-line Summary

Off-the-shelf small models fail across the board on agent micro-tasks out of the box — the eligibility gap tracks model size, not quantization precision.

Paragraph Summary

This arXiv study systematically tests whether off-the-shelf small language models (SLMs) can handle the high-frequency micro-tasks in agent systems — command approval, memory writes, tool selection, and conversation turn ranking. Using "confidence interval lower bound exceeding a non-LLM baseline threshold" as a strict pass criterion, the researchers evaluated Qwen3 from 0.6B to 8B across four sizes. All 16 configurations failed. Three 4-bit quantization schemes (RTN/GPTQ/AWQ) changed no configuration's eligibility status, confirming the gap is driven by model size rather than precision. Findings replicated on Llama-3.x (12/12 ineligible) and across 112 prompt variants (0/112 passing). Log-probability diagnostics identified 4 failure regimes, distinguishing true capability deficits from issues fixable via decoding strategy. The practical recommendation: place SLMs behind an eligible baseline as an incremental optimization layer, rather than expecting them to independently replace frontier models.