4B Model RL Training Test: Can It Catch Up to a 27B Model?

Title

4B Model RL Training Test: Can It Catch Up to a 27B Model?

Meta Description

A hands-on test training Qwen3.5-4B with open-source RL environments on dual DGX Sparks: training only on edge-case tasks recovers over half the gap to a 27B model, but GRPO mechanics cap the gains, and distillation combined with RL works best.

Article Content

Reinforcement learning (RL) environments are becoming a new battleground that major model vendors are racing to invest in. But can an ordinary person use open-source environments to train a small model up to the level of a large one on their own hardware? A YouTube creator ran a complete experiment with two DGX Sparks, and the conclusion is more nuanced than expected: RL can close more than half the gap between a 4B model and a 27B model, but the gains have a clear ceiling.

Why RL Environments Suddenly Matter

According to the video's author, RL environments have been mentioned repeatedly in recent model releases. Anthropic is said to have discussed investing over $1 billion per year to build RL environments, Xiaomi open-sourced thousands of environments alongside MiMo v2.6, and the Hugging Face team also released an environment set aimed specifically at small models.

The author compares RL environments to "practice sheets with answers": the model receives a task, attempts a solution, and some checker judges whether it's right or wrong. In the small-data-environments he used, each task is a real dataset from Kaggle plus an answer that can be programmatically verified. The environment provides the model with three things—an isolated sandbox, terminal tools, and an answer checker. A correct answer earns 1 point, a wrong one earns 0, and this reward signal is the only feedback the model receives.

Here lies the first trap: the model learns whatever the checker rewards, not what you actually want. The original datasets' built-in checkers are lenient—for example, if the correct answer is 42, the model writing "42 yes" also passes, which easily induces reward hacking. The author therefore wrote stricter checkers himself, and all experiments are based on this version.

Reward hacking refers to the model finding loopholes in the checker during training and scoring high in ways that don't match human intent. Classic examples include: writing the answer in the reasoning process rather than the designated output location, exploiting format leniency (e.g., "42 yes" also counts as correct), or simply restating numbers from the prompt. It's dangerous because the reward signal is the only feedback the model can see—once the checker has a gap, the model will systematically exploit it, and this behavior often doesn't show up on the evaluation set. Defenses include stricter programmatic checkers, hard constraints on output format, and manual review of high-scoring samples during training. The author rewrote the checkers himself precisely to plug such loopholes at the source.

Where RL Fits in the Training Pipeline and the GRPO Mechanism

Model training usually has three stages: pretraining teaches the model to predict the next word and accumulate world knowledge; supervised fine-tuning teaches it to imitate good answers; RL then comes in—no longer copying answers, but letting the model practice on its own, adjusting based on checker scores. The author emphasizes that this is exactly RL's appeal: you don't need to write the answer, you just need to be able to verify it.

The RL method used in the experiment is GRPO. For each task, the model makes 8 attempts; attempts that perform better than the group average are reinforced, and those below average are suppressed. The key inference is: if a task is all correct or all wrong across 8 attempts, no attempt can exceed the average, so this task contributes nothing to training. The author stresses this repeatedly, and it's the starting point for all subsequent experimental design.

GRPO (Group Relative Policy Optimization) is a value-network-free RL algorithm proposed by DeepSeek^Ad. Traditional PPO requires training an additional critic model to estimate the baseline, which is memory- and compute-intensive; GRPO instead samples a group of responses (8 in this article) for the same prompt and uses the group's mean reward as the baseline—responses above the mean are reinforced, those below are suppressed. This eliminates the critic and suits large-model scenarios. But one direct side effect is: if all rewards in a group are identical (all correct or all wrong), the relative advantages are all zero, gradients vanish, and the sample contributes nothing to training. This explains why the article must filter for "sometimes right, sometimes wrong" edge-case tasks.

Experimental Setup: Two DGX Sparks and Qwen3.5-4B

The hardware is two DGX Sparks, each with 128GB unified memory. The training target is Qwen3.5-4B, using LoRA rather than full-parameter training to reduce overhead. Evaluation uses a held-out set containing tasks unseen during training. Before training, the 4B model's accuracy is around 50%; the target model is the 27B model from the same family. Evaluation results show that the 27B model reaches roughly 75% or above, setting a catch-up goal for the small model.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: all original model weights are frozen, and only two small matrices A and B are attached beside some linear layers, expressing the weight update as a low-rank product BA—only these two small matrices are trained. The number of trainable parameters is typically only a few percent of full-parameter training, greatly reducing memory usage and training time, and the adapter can be saved, loaded, or merged back into the main model separately. This article uses LoRA rather than full-parameter training on the 4B model, allowing the unified memory of two DGX Sparks to accommodate the training pipeline; the cost is limited capacity, which may partly explain the gains ceiling.

First Round Failure: Random Tasks Teach Almost Nothing

The author's first round randomly sampled tasks for training. The model did show useful changes—it began finishing tasks rather than exhausting its turns—but the score improvement was small, and luck couldn't be ruled out.

On review, the cause was hidden in the GRPO mechanism: most training tasks were ones the model consistently got right or consistently got wrong, so a large amount of training time was wasted on samples that "teach nothing."

The Key Turning Point: Training Only on "Edge-Case Tasks"

In the second round, the author had the model attempt each of about 1,000 tasks 4 times, keeping only tasks that were "sometimes right, sometimes wrong"—that is, problems at the edge of the model's ability. After training on this data for most of a day, the score rose noticeably, recovering more than half the gap to the 27B model.

To rule out luck, the author compared the model task by task before and after training: after training, the number of newly correct tasks clearly exceeded the number of newly wrong ones, and the probability of such a distribution arising purely by chance is extremely low.

The behavioral changes are more worth examining. Take the video game sales question from the opening: the untrained model analyzed correctly and even wrote Nintendo in its response, but when writing to the answer file it filled in a sales figure, scoring 0; after training, the model re-checks the top five publishers and correctly writes Nintendo. In another task with a headerless dataset, the untrained model repeatedly executed the same command until its turns ran out; after training, the model could infer column names on its own, find Dota 2, and give the answer.

There were regressions, of course: one task was mathematically correct, but the output was in fraction form while the answer required a percentage, so it lost points. Overall, the author's judgment is—the model already had data analysis capabilities; what RL taught wasn't a new skill, but "finishing the job and answering the question that was asked." No signs of reward hacking were found in the checks either.

The Ceiling Appears: Why More RL Stops Raising the Score

The author used the same method to filter a new batch of edge-case tasks and continued training, but the score stopped improving. The reason again goes back to GRPO: the model must be "sometimes right, sometimes wrong" across attempts to have a learning signal, and this batch of edge-case data may have already been exhausted by the first round.

He offers a self-test anyone can reproduce: have the untrained model attempt each task 4 times, and compute the difference between the proportion of "at least one correct" and the single-attempt accuracy—that roughly is the upper limit of the space RL can exploit.

A Different Approach: Distilling from the 27B Model

Since RL can't make the model discover new skills on its own, why not demonstrate the skills to it? The author had the 27B model solve a batch of training tasks, kept the complete trajectories of correct answers, and fine-tuned the 4B model on them—that is, distillation.

The score changed little, but the model's way of working began to resemble the 27B: faced with a question comparing data scaling methods, the RL model tried machine learning models one by one, while the distilled model did a clean comparison and gave the answer in far fewer steps. More interestingly, the distilled model solved some tasks the RL model never got right. The author speculates that these are precisely the tasks that most need further RL polishing.

Knowledge distillation here refers to using correct solution trajectories generated by a large model (the teacher, 27B) on training tasks as supervised fine-tuning data to train a small model (the student, 4B). Unlike traditional distillation that aligns logits, this article uses "trajectory distillation": keeping only the complete reasoning and tool-call sequences of the teacher's correct answers, so the student imitates its problem-solving steps and tool-use habits. Its advantage is that it can transfer behavioral patterns that RL struggles to explore spontaneously (such as filtering categories before counting), and it costs less than continuing RL; the limitation is that the student's ceiling is constrained by the coverage of the teacher's data, and it may imitate the teacher's redundant steps, requiring further RL polishing.

The Combined Approach: RL After Distillation

Running another round of RL on the distilled model produced similar final scores in most experiments, but on a completely unseen new dataset, this "distillation + RL" combined model performed best among all small models, and is the first version the author considers to have passed the luck test.

A database question from the US Department of Agriculture illustrates the point well: it asks for the most common apple variety. Both the untrained model and the pure RL model answered "Novel"—the variety that appears most often in the file, but it's an orange. The distilled model found the right direction but ran out of turns while re-checking. The final model first filtered for the apple category, then wrote Red Delicious, matching the 27B model's answer and correct.

Conclusion: RL Is an Amplifier, Not a Creator

The author's answer is a qualified "yes": you need a genuinely trustworthy checker, a batch of tasks at the edge of the model's ability, and about a day of training to make an initial judgment on whether it works. But the most important conclusion is—RL only amplifies what the model can already occasionally do; once that space is used up, the score basically stops moving.

The experiment's limitations are also clear: only one model was tested, and despite checking multiple checkpoints, the final version still falls short of the 27B model. The author believes that increasing training data diversity and trying more RL techniques could close the gap, and has open-sourced the checkers, trained models, and complete recipe on GitHub for the community to reproduce and advance on their own GPUs.

One-line Summary

Tested on two DGX Sparks: RL can close more than half the gap between a 4B model and a 27B model, but the gains have a ceiling.

Paragraph Summary

The article documents a creator using two DGX Sparks, Qwen3.5-4B as the target, and LoRA plus GRPO as the method, to verify whether open-source RL environments can let a small model match a large one. The first round of random-task training was almost ineffective, because GRPO only produces learning signals for "sometimes right, sometimes wrong" edge-case tasks; after filtering edge-case tasks in the second round, more than half the gap to the 27B model was recovered, and task-by-task comparison ruled out luck. The score stalled after further training, showing that RL only amplifies capabilities the model already has. Distilling correct trajectories from the 27B model then taught the small model a more efficient way of working and solved tasks RL hadn't; the distillation-plus-RL combination performed best on unseen datasets. The conclusion is that RL is an amplifier, not a creator, requiring trustworthy checkers and edge-case tasks, and the experiment covers only a single model.