Qwen3-8B Fine-Tuning Experiment: Does LLM Routing Need More Data?

Sending every AI request to the most expensive model is a common way to waste money. Extracting a field from JSON or formatting a block of text requires a completely different level of reasoning than debugging a complex algorithm or analyzing system design problems. LLM routing exists to solve this problem: before processing a request, determine how difficult it is, then decide which model should handle it.

A developer built an experimental project called Complexity Scorer using Qwen3-8B—fine-tuning the model to assign prompts a complexity score from 0.00 to 1.00, then routing based on that score to either a cheap model or an advanced reasoning model. The experiment produced a counterintuitive finding: a model fine-tuned on just 50 examples achieved higher accuracy than a version fine-tuned on 2,500 examples.

What Is Complexity Scorer?

The core idea is straightforward: the model doesn't answer the user's question—it only evaluates the question's complexity. The score mapping looks roughly like this:

Score	Difficulty	Typical Tasks
0.00	Trivial	Extract a value, format text
0.25	Easy	Execute a few straightforward steps
0.50	Medium	Combine information with constraints
0.75	Hard	Analyze trade-offs and complex requirements
1.00	Very Hard	Deep reasoning across complex systems

The full pipeline is: user prompt → Complexity Scorer assigns a score → Model Router → cheap model or advanced reasoning model. The goal is to cut unnecessary API costs without sacrificing answer quality.

From an engineering perspective, the value of this design is that it decouples "judging difficulty" from "completing the task." The router can be iterated and evaluated independently, without touching the downstream generation model.

Open-weight models refer to models whose parameters can be publicly downloaded and that allow users to deploy and fine-tune them themselves, as opposed to closed-source API models (such as GPT-4 and Claude). Qwen3-8B falls into this category—its weights are freely available, which is why the author could perform supervised fine-tuning on it via Tinker. The advantages of open weights are greater freedom in task customization, data staying local, and strong reproducibility; the cost is that you must bear the inference hardware and operational overhead yourself. It's worth noting that open weights are not the same as open source—some models only release weights while withholding training data and code, so license terms deserve attention.

Fine-Tuning Results: 50 Examples Beat 2,500

The project used Tinker to fine-tune Qwen3-8B and compared it against the original model. The evaluation set was fixed at 50 samples, and all experiments shared the same evaluation data.

Experiment	Training Samples	Accuracy
Original Qwen3-8B	0	50%
Fine-tuned model	50	86%
Fine-tuned model	500	74%
Fine-tuned model	2,500	76%

The final model (trained on 2,500 examples) reached 76% accuracy—a 26-percentage-point improvement over the original model's 50%. But the model trained on only 50 samples scored the highest at 86% on this evaluation set.

This result shouldn't be simplified into "less is more." The author offered several explanations, each worth unpacking.

Label Consistency in Synthetic Data

Training samples were AI-generated, and AI-generated complexity labels can be inconsistent: if two similar prompts receive different scores, the model struggles to learn clear category boundaries. The larger the dataset, the more opportunities there are for this noise to be amplified.

Synthetic data refers to training samples automatically generated by large models rather than labeled by humans. In this project, AI was responsible for generating a 0.00–1.00 complexity score as the label for each prompt. This approach costs far less than human labeling but introduces label noise: the generating model's judgment about the boundary between "medium" and "hard" may be self-contradictory, causing similar prompts to receive different scores. Supervised fine-tuning relies on labels to provide a stable gradient signal; once labels are inconsistent, the decision boundary the model learns becomes fuzzy. As data volume grows, the absolute number of contradictory samples rises, and the interference from noise during training may be amplified—this is one of the core mechanisms behind the author's explanation of why "more data isn't necessarily better."

The 50-Sample Evaluation Set Is Too Small

The 86% figure rests on 50 evaluation samples. With such a small sample size, the gain or loss of a single sample can shift accuracy by 2 percentage points. The result likely reflects the model being "coincidentally well-suited to these 50 samples" rather than genuine generalization ability.

Statistical significance refers to whether an observed difference could be caused by random fluctuation. On an evaluation set of 50 samples, each sample carries a weight of 2 percentage points; a 2% accuracy difference between two models amounts to just one sample's worth of gain or loss, so chance factors cannot be ruled out. To reliably distinguish gaps like 74% vs. 76%, you typically need hundreds to thousands of independent evaluation samples, along with confidence intervals or hypothesis testing. A more rigorous approach is to use a held-out dataset (data the model never encountered during training); otherwise the model may inflate its scores by memorizing the training distribution. The author lists "larger held-out evaluation" as the top follow-up direction in the conclusion—a methodological response to exactly this problem.

The 500-to-2,500 Improvement Amounts to One Sample

Going from 74% to 76% accuracy on a 50-sample evaluation set corresponds to getting just one more question right. This difference is statistically almost meaningless; a larger and more diverse evaluation set is needed to determine whether it's real.

The author also explicitly notes at the end that the small evaluation set limits how generalizable these accuracy results are. This deserves credit—many similar experiments would skip over this limitation.

API Engineering: From Model to Usable Service

After fine-tuning, the project wrapped the model into a service using .NET Minimal API, including API key authentication, rate limiting, exception handling, integration with the Tinker inference API, and a browser-based playground for testing prompts. The model adapter was published to Hugging Face for other developers to reuse.

The significance of this engineering layer is that complexity evaluation becomes an independent service, separated from the application that ultimately answers user questions, allowing each to evolve on its own.

Distinguishing proxy metrics from the true objective is key to evaluating a routing system. Classification accuracy is only a proxy metric—it measures whether scores are close to reference answers, but cannot reflect whether routing decisions actually save money without degrading quality. The true objective is the joint optimization of cost, latency, and quality: a hard problem misclassified as easy may produce a wrong answer because it was handed to a weak model, and that cost far exceeds the few API dollars saved. Effective routing evaluation should therefore incorporate a cost-weighted version of the confusion matrix, or directly measure end-to-end cost per thousand requests and user satisfaction, rather than looking in isolation at a classifier's score on a test set.

Why Choose an Open-Weight Model

Using a closed-source model for every classification request is also an option, but it introduces additional vendor dependency and inference costs. Open-weight models offer a different kind of flexibility: fine-tuning for a specific task, experimenting with different training sets, sharing the model with other developers, and exploring local deployment on compatible runtimes.

The author also remained restrained: local deployment still requires suitable hardware, and open weights don't inherently guarantee lower total cost. This framing is closer to reality than claims that "open source is always cheaper."

Four Lessons from This Experiment

Optimize the true objective, not proxy metrics. A router must balance inference cost, latency, and the risk of misclassifying hard problems to weak models. Chasing complexity classification accuracy alone doesn't mean the routing system is effective.

Treat evaluation results with caution. Small test sets make performance differences look more significant than they are. This is the most direct lesson from this experiment.

Put guardrails around AI tools. Automated agents that can access paid APIs or launch training jobs should have explicit spending limits and permission boundaries.

Don't assume piling on data solves everything. Better data generation, consistent labels, and stronger evaluation methods may be more valuable than increasing sample count.

Worthwhile Next Steps for Validation

The author lists several directions: evaluate on a larger held-out dataset, check whether complexity scores correlate with actual task difficulty, measure routing decisions by real inference cost and answer quality, compare against simpler classification methods, and experiment with confidence thresholds and fallback mechanisms.

The most valuable one is validating the correlation between complexity scores and actual task difficulty. If the model assigns 0.3 to a certain type of prompt, but that type of prompt often actually requires an advanced model to answer well, then no matter how "accurate" the classifier is, it's meaningless. The ultimate metric for a routing system should be the joint optimization of cost and quality, not classification accuracy itself.

FAQ

What scale of application is LLM routing suitable for? When request volume and model price differences are large enough that the savings from routing exceed the inference overhead of routing itself, it's worth doing. For small-scale applications, simple rules or lightweight classifiers may be more cost-effective.

Why did accuracy decrease as data volume increased after fine-tuning? The reasonable explanation in this experiment is inconsistent synthetic data labels, compounded by fluctuation caused by an overly small evaluation set. This cannot be used to draw a general conclusion that "more data is worse."

Is routing with open-weight models always cheaper than closed-source? Not necessarily. The author explicitly states that open weights don't guarantee lower total cost; hardware and operational overhead must be calculated separately.