Fine-Tuning a 4B Small Model: Browser Task Accuracy Soars from 22% to 63%

Can Small Models Handle Browser Automation?

For a long time, browser automation tasks were considered the exclusive domain of large models. When faced with scenarios requiring open-ended research or deep reasoning, small-parameter models often fell short. However, an experiment shared by a developer on Reddit tells a different story: by fine-tuning Qwen3.5-4B on approximately 3,000 browser operation trajectories, they successfully increased benchmark accuracy from 22% to 63%.

The core insight of this experiment is: not all browser tasks require complex reasoning. Many real-world browser automation workflows are actually highly procedural—such as extracting structured data, uploading and downloading documents, filling out and submitting forms. These tasks can typically be precisely defined and executed repeatedly, so theoretically a small model paired with an appropriate browser harness should perform quite well.

Experimental Design and Data Sources

The author chose Qwen3.5-4B as the base model, a lightweight model with only 4 billion parameters. Qwen3.5-4B is the lightweight version in Alibaba's Qwen3.5 series, which employs a hybrid inference architecture supporting switching between "thinking mode" and "non-thinking mode," allowing the model to dynamically allocate computational resources based on task complexity. The 4B parameter count means this model can run efficiently on a single consumer-grade GPU (such as an RTX 4090 or even lower specs), and can even be deployed on edge devices after quantization. While models of this scale typically lag far behind those with tens of billions of parameters in general benchmarks, their fast inference speed and low memory footprint make them ideal base models for fine-tuning in specific vertical scenarios.

The training data came from approximately 3,000 browser agent operation trajectories generated by GPT-5.6 Luna. GPT-5.6 Luna itself is a strong baseline model in the browser task domain, so using high-quality trajectories generated by it as supervised fine-tuning (SFT) teacher data is a typical knowledge distillation approach. Knowledge Distillation, originally proposed by Geoffrey Hinton and colleagues in 2015, centers on the idea of having a small model (student model) learn from the outputs of a large model (teacher model), thereby achieving performance close to the large model while maintaining a smaller parameter count. In this experiment, this distillation took a fairly direct approach: using a strong model to generate high-quality operation trajectories, then having the small model learn the behavioral patterns through SFT. SFT essentially involves further training a pretrained model with labeled input-output pairs. Compared to more complex training paradigms like reinforcement learning, this method has lower implementation costs and a more stable training process.

You may not have noticed, but the author made tradeoffs in framework selection. He chose Browser Agent rather than BrowserCode as the execution framework for fine-tuning. These two represent different design philosophies for browser automation: Browser Agent frameworks typically use high-level semantic actions (like "click element X" or "input text Z in input field Y") to describe operations, with each action being a structured instruction; whereas BrowserCode frameworks have the model directly generate executable code (usually Python or JavaScript) that controls the browser through automation libraries like Playwright or Selenium. The advantage of code frameworks is stronger expressiveness, enabling complex conditional logic and loop operations, but the tradeoff is significantly increased token consumption per operation—a Python code snippet is far longer than a structured instruction. Longer operation trajectories mean dramatically higher costs for training and trajectory generation. For a small model experiment pursuing cost-effectiveness, the more compact Browser Agent framework is clearly more economically efficient.

What is a Browser Harness?

For readers unfamiliar with this concept, a browser harness is essentially the "middleware" connecting the language model with the actual browser. The model's output intentions (like clicking a button or entering text) need to be translated by the framework into actual browser operations, while the framework also feeds webpage state back to the model. Specifically, the framework needs to handle several key aspects: converting webpage DOM structure or screenshots into text/visual inputs the model can understand, parsing the model's output action instructions, executing operations through automation tools, handling wait logic for page loading and state changes, and error recovery mechanisms. The design quality of the framework often directly determines the performance ceiling of the same model on browser tasks. A well-designed framework can effectively reduce the amount of information the model needs to process and lower decision complexity, enabling even less capable small models to complete seemingly difficult tasks.

Interpreting Benchmark Results

The experimental quantitative results are quite compelling, validated on two benchmarks:

• BrowseWebApp Bench: This benchmark tests more complex but well-defined web application workflows, rather than open-ended web research. The fine-tuned model's accuracy jumped from 22% to 63%, nearly tripling.
• BU Bench V1: This is a more balanced Browser Use benchmark. Post-fine-tuning accuracy increased from 15% to 53%.

In the AI Agent field, benchmark design is crucial because different benchmarks emphasize vastly different capability dimensions. WebArena, MiniWoB++, Mind2Web, and others are early representative benchmarks in this domain, respectively focusing on real website interaction, simplified web operations, and cross-site generalization capabilities. BrowseWebApp Bench focuses on well-defined web application operation flows (like automating SaaS tool operations), while BU Bench more evenly covers diverse tasks in Browser Use scenarios. Understanding a benchmark's design bias is critical for correctly interpreting experimental results—the small model's excellent performance in this experiment largely benefits from the target benchmarks emphasizing procedural tasks rather than open-ended reasoning.

Even more surprising, on BU Bench V1, this SFT model with only 4B parameters even surpassed the 45% score reported by DeepSeek V4 Pro and Kimi K2.6 in the open-source Browser Use framework. In other words, a small model with targeted fine-tuning can beat general-purpose large models in specific task domains. This once again confirms the practical principle that "there's no best model, only the most suitable model for a specific task."

What This Means

Task Specialization vs General Capability

The greatest value of this experiment is confirming an increasingly important trend: in well-defined vertical tasks, small models with targeted fine-tuning can match or even exceed large models. This isn't just about cost—4B models can run on consumer-grade hardware with low inference latency and flexible deployment, making them highly suitable for automation scenarios requiring large-scale, high-frequency execution.

For enterprises, this means not needing to call expensive flagship model APIs for every browser automation workflow. Assigning common, procedural workflows to fine-tuned small models while reserving complex reasoning tasks for large models is a more reasonable resource allocation strategy. This "tiered scheduling" architectural design is becoming the mainstream paradigm for AI application deployment: deploying lightweight specialized models on the front end to handle high-frequency simple tasks, while maintaining large models on the back end to handle long-tail complex requests, controlling overall costs while ensuring user experience.

Large Models Still Have Irreplaceable Value

The author also maintained appropriate clarity. He explicitly noted that achieving SOTA (State-of-the-Art) performance still requires pairing stronger models with frameworks like Browser Agent or BrowserCode. The advantage of small models lies in being "good enough and efficient," not comprehensive superiority. For complex tasks requiring open-ended research and cross-page deep reasoning—such as comparing information across multiple websites, understanding complex webpage interaction logic, or handling never-before-seen website layouts—the comprehensive capabilities of large models remain indispensable. These scenarios require models to possess strong generalization capabilities, long-context understanding, and multi-step reasoning abilities, which are precisely the domains that small models struggle to reach even after fine-tuning.

Insights for Developers

This work provides a clear practical path for developers looking to build browser automation systems:

1. First assess task nature: If your workflow is highly procedural and precisely definable, small model fine-tuning is likely the better choice.
2. Leverage strong models to generate training data: Using strong baseline models like GPT-5.6 Luna to generate high-quality trajectories, then distilling to small models, is a validated low-cost path. Notably, the data scale of 3,000 trajectories isn't large, indicating that in scenarios with concentrated task distributions, small-scale high-quality data can bring significant performance improvements.
3. Prioritize framework selection: Framework design significantly impacts training costs and final performance, requiring tradeoffs between performance and cost.

The author also open-sourced related resources, including the fine-tuned Qwen3.5-4B Browser Agent SFT FP8 model, the Browser Agent framework, and both BrowseWebApp Bench and BU Bench V1 benchmarks. FP8 (8-bit Floating Point) is a low-precision numerical representation format designed specifically for deep learning inference acceleration. Compared to traditional FP16, it can halve model memory footprint while achieving significant inference speedup on FP8-supporting hardware (like NVIDIA H100, RTX 40 series GPUs). The author directly providing an FP8 version means developers can run this model with extremely low resource consumption, further lowering the deployment barrier. These open-source resources provide a solid foundation for community reproduction and further research.

Conclusion

From 22% to 63%, the message behind these numbers is clear: in the wave of AI application deployment, "bigger" isn't always the answer. For specific task scenarios, fine-tuning a small model with appropriate data and frameworks can often achieve a better balance between cost, speed, and effectiveness. As open-source toolchains mature, this "small but specialized" approach may become the preferred solution for more and more practical automation projects.