[KongchangAI]
· 3 min read· 1,858 words

Small Model Fine-Tuning Upset: 2B/4B Parameters Beat GPT-6 Luna

Small Model Fine-Tuning Upset: 2B/4B Parameters Beat GPT-6 Luna

Fine-tuned Qwen 3.5 small models beat GPT-6 Luna on e-commerce classification at 45 cents per run.

Through a Shopify product classification experiment, the article shows fine-tuned small models can beat general large models on vertical tasks. Using ~209 categories, 5,000 training and 1,000 evaluation samples, it compares un-fine-tuned GPT-6 Luna (68.2%), Qwen 3.5 2B (20.4%), and 4B (43.5%). After LoRA fine-tuning on Crusoe, 2B reached 69.8% and 4B 73%, both surpassing GPT-6 Luna at ~45 cents per training run. The author details preprocessing, hyperparameters (rank 16, alpha 32, lr 0.0002), and checkpoint selection, noting the method suits closed-label, clearly defined high-frequency tasks but should not be overgeneralized.

Why Small Model Fine-Tuning Deserves Attention

Amid the arms race of large models, a counterintuitive fact is being repeatedly validated: on vertical tasks, fine-tuned small open-source models can outperform general-purpose large models. A YouTube tech creator conducted a complete experiment using an e-commerce product classification task—fine-tuning 2-billion and 4-billion parameter Qwen 3.5 models and pitting them against GPT-6 Luna, with convincing results.

The starting point was simple: GPT-6 Luna is ready out of the box and should logically be the strongest model overall; the two small models perform poorly without fine-tuning due to limited contextual understanding. But the real question is—can fine-tuning turn a "poor student" into an overachiever?

Experiment Design: Shopify Product Classification Task

Dataset and Preprocessing

The author chose Shopify product catalog data as a proof of concept, with the task of assigning product titles and descriptions to the correct category. The dataset contains approximately 209 hierarchical categories, such as "Sporting Goods > Indoor Games".

The preparation script converts raw data into the JSONL format required for fine-tuning: each line is a message history containing a system prompt, user input (title + description), and assistant response (correct category label). For reproducibility, the random seed was fixed at 42, using 5,000 products for training and 1,000 for evaluation. The author also admits that using the full dataset might yield better results.

The entire pipeline consists of two scripts: prepare and evaluate. The code is open-sourced and can be reused or modified directly.

Three-Way Comparison Baseline

The evaluation script compares three groups:

  • GPT-6 Luna (not fine-tuned, serving as the baseline to beat)
  • Qwen 3.5 2B (not fine-tuned)
  • Qwen 3.5 4B (not fine-tuned)

There are two versions of the system prompt: for the general large model, all candidate labels must be listed; for the fine-tuned models, the label list can be omitted and classification requested directly—this itself reflects a capability difference brought by fine-tuning. The only evaluation metric is accuracy.

Baseline Results: How Big Is the Gap?

Without fine-tuning, the three models show a clear gradient. GPT-6 Luna achieves 68.2% accuracy on 1,000 evaluation samples, the threshold to cross. Qwen 3.5 2B reaches only 20.4%, almost useless in practice; Qwen 3.5 4B reaches 43.5%, better than the 2B version but still significantly behind GPT-6 Luna.

These results are expected: small models lack understanding of the product category system and sufficient instruction-following ability. The real suspense lies after fine-tuning.

Fine-Tuning in Practice: LoRA Training on a Serverless Platform

Cloud Training Workflow

The author chose serverless fine-tuning on Crusoe's Intelligence Foundry (a sponsor of the video). This choice has practical considerations: consumer-grade hardware can handle 2B/4B models, but scaling to 8B, 16B, or 32B parameters means local devices are either too slow or simply cannot fit the model.

The path is straightforward: go to Model Shaping → Serverless Fine-Tuning → create a fine-tuning job, select the Qwen 3.5 2B model, upload train.jsonl, and the platform shows the estimated training cost—only about 45 cents this time.

Hyperparameter Settings

The author manually set a custom set of hyperparameters (rather than using defaults) to reproduce previously obtained results:

  • batch size: 16
  • LoRA rank: 16
  • LoRA alpha: 32 (twice the rank)
  • dropout: 0.05
  • learning rate: 0.0002
  • epochs: 1
  • warm-up ratio: 0.05
  • weight decay: 0

After training starts, the training loss curve can be monitored in real time, with a checkpoint generated approximately every 100 steps. The author specifically warns: the last checkpoint is not necessarily the best—in a previous run, the checkpoint at step 300 was optimal. Whether to use the best checkpoint must be judged based on training loss, but training loss does not necessarily map to validation set accuracy.

After training, there are two ways to use the model: deploy directly on the Crusoe platform (suitable for scenarios without local compute or with larger models), or download the checkpoint zip for local inference. The author chose the latter, completing evaluation on a Dell Pro Max with GB10.

The fine-tuning process for the 4B model is identical, just selecting Qwen 3.5 4B and reusing the uploaded dataset with the same hyperparameters.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Its core idea is to freeze all original weights of the pre-trained model and inject two low-rank matrices A and B beside each layer (usually the projection matrices of attention layers), updating only these small matrices during training. If the original weight matrix has dimensions d×k, LoRA decomposes its update into the product of two matrices of dimensions d×r and r×k, where rank r is much smaller than d and k (r=16 in this article). This reduces trainable parameters from hundreds of millions to millions, greatly lowering memory usage and training time, and because the original weights remain unchanged, the model does not forget pre-trained knowledge. LoRA alpha is a scaling factor; the actual update is (alpha/r)·BA. Here alpha=32 gives a scaling factor of 2, commonly used to make the magnitude of low-rank updates better match the original weights.

Serverless fine-tuning means users do not need to manage servers or GPU clusters; they simply upload datasets and configure hyperparameters, and the platform charges based on actual compute resources consumed. It is suitable for small-to-medium-scale experiments, with costs as low as a few dimes per run, but it also means limited control over underlying hardware and parallelization strategies.

Batch size refers to the number of samples used per parameter update. Here it is set to 16, meaning gradients are computed and LoRA weights updated every 16 product records. Batch size affects training stability and speed: too small means noisy gradients, too large may hurt generalization and increase memory usage.

Dropout is a regularization method that randomly zeros neuron outputs with a certain probability (0.05 here) during training to prevent the model from over-relying on certain features. In LoRA it typically applies to the low-rank branch; a small value (0.05) indicates weak regularization.

Learning rate (0.0002) controls the step size of each parameter update. For LoRA fine-tuning, 1e-4 to 3e-4 is a common range; this value is at the mid-to-high end, and combined with only 1 epoch of training, it can complete adaptation in a relatively short time without overfitting.

Warm-up ratio (0.05) means that during the initial 5% of total steps, the learning rate linearly increases from 0 to the set value, avoiding large updates at the start that could damage pre-trained weights. Weight decay (0) means no weight decay regularization is applied, since LoRA itself has very few parameters and low overfitting risk.

Results: The Small Models' Upset

The fine-tuned report card is surprising. Qwen 3.5 2B jumps from 20.4% to 69.8%, slightly surpassing GPT-6 Luna's 68.2%—the author admits this gap might be considered noise or a tie, but the trend is clear.

Qwen 3.5 4B improves even more, reaching 73% accuracy after fine-tuning, significantly exceeding GPT-6 Luna.

The significance of this result must be understood in a comparative framework: GPT-6 Luna is a closed-source frontier model (the author mentions it is the smallest in the GPT-6 series, though the exact parameter count is unknown), while the two open-source small models surpassed it on a vertical task with a few hundred training samples and less than a dollar of training cost.

Checkpoint is a snapshot of model weights saved periodically during training. Here, one is saved approximately every 100 steps, so 5,000 records in 1 epoch with batch size 16 produce about 313 steps, corresponding to roughly 3 checkpoints. Choosing which checkpoint to deploy is a model selection problem: the checkpoint with the lowest training loss may overfit the training set and perform worse on unseen validation data. Therefore, the author emphasizes selecting based on validation set accuracy rather than just the loss curve. This also explains why "the last checkpoint is not necessarily the best"—loss may still be decreasing late in training, but generalization performance may have already peaked.

Where This Methodology Can Be Replicated

The author repeatedly emphasizes in the video that the Shopify dataset is just a proof of concept; the same process can be transferred to any task with "text input and limited output categories": customer service ticket routing, content moderation grading, product tag system maintenance, intent recognition, etc.

Several practical takeaways worth noting:

First, establish a baseline before fine-tuning. Without measuring the accuracy of the un-fine-tuned model, you cannot judge the true gain from fine-tuning.

Second, choose small models with a gradient. The performance gap between 2B and 4B shows that bigger is not always better, but too small a model has a limited ceiling even after fine-tuning; it is worth trying several sizes on the same task.

Third, training loss is not the same as business metric. Checkpoint selection should incorporate validation set evaluation, not just the lowest point of the loss curve.

Fourth, evaluation must use unseen data. The 1,000 evaluation samples in this experiment were never used in training—this is the precondition for credible conclusions.

Fifth, fine-tuned models depend less on prompts. No longer needing to stuff all candidate labels into the prompt is itself an engineering simplification.

The Real Boundaries of Small Model Fine-Tuning

The conclusion of this experiment should not be overinterpreted as "small models comprehensively surpass large models". It holds under very specific conditions: clearly defined task, closed label system, and high-quality labeled data. In such scenarios, the "breadth" of general large models is useless, while a specially trained small model can hit the target precisely.

Conversely, if the task requires open-ended reasoning, multi-step planning, or cross-domain knowledge integration, the advantage of fine-tuned small models quickly fades. The truly pragmatic strategy is choosing models by task: high-frequency, structured tasks with limited labels go to fine-tuned small models—low cost, low latency, strong controllability; complex reasoning tasks continue to go to frontier large models.

From a cost perspective, the training cost for both fine-tuning runs here is on the order of a few dimes. Compared with the ongoing cost of calling closed-source APIs, the economic gap at scale will be very considerable. For developers wanting to validate this path, the barrier is now so low that there is almost no reason not to try.

One-line Summary

Fine-tuned Qwen 3.5 small models surpass GPT-6 Luna in accuracy on an e-commerce classification task, at a cost of only 45 cents.

Paragraph Summary

Through a Shopify product classification experiment, the article validates the feasibility of fine-tuned small models beating general large models on vertical tasks. The experiment used about 209 categories, 5,000 training samples, and 1,000 evaluation samples, comparing un-fine-tuned GPT-6 Luna (68.2%), Qwen 3.5 2B (20.4%), and 4B (43.5%). After LoRA fine-tuning on the Crusoe platform, the 2B model's accuracy jumped to 69.8% and the 4B reached 73%, both exceeding GPT-6 Luna, with a single training cost of only about 45 cents. The author details data preprocessing, hyperparameter settings (rank 16, alpha 32, learning rate 0.0002, etc.), and checkpoint selection tips, and notes that the method applies to high-frequency scenarios with closed label systems and clearly defined tasks, but should not be overinterpreted as small models comprehensively surpassing large models.

#small model fine-tuning#LoRA#Qwen 3.5#GPT-6 Luna#serverless fine-tuning#product classification#model comparison
Share:

Related articles