GPT-2 Fine-Tuning Experiment: 88% Function Calling Success Rate with a 355M Small Model

A Counter-Intuitive Experiment

In an era where everyone chases hundred-billion-parameter models, one developer did something strikingly different: they dusted off GPT-2 (the 355M parameter version, released in 2019) and fine-tuned it to learn Function Calling — the core mechanism that enables large language models to act as agents and invoke external tools.

Even more surprisingly, the entire training process ran on free Kaggle GPUs, and the final model achieved an 88% success rate for producing correct, parseable function call outputs. This result challenges a widely held assumption: achieving agent-level capabilities doesn't necessarily require frontier models with hundreds of billions of parameters or expensive compute.

Why Choose a 6-Year-Old Model?

GPT-2 was released by OpenAI in 2019 as the second generation of the GPT series, built on a Transformer decoder-only architecture. The 355M parameter version sits second among its four sizes (117M, 345M, 762M, and 1.5B). At launch, GPT-2 was so capable that OpenAI famously delayed its full release over concerns about misuse — a moment that became a landmark in the history of AI safety discussions. While it's tiny compared to today's GPT-4 (estimated to exceed 1 trillion parameters), GPT-2's architectural design is the direct ancestor of virtually every modern LLM. Understanding GPT-2 is essentially understanding the structural DNA of modern language models.

From an engineering perspective, choosing GPT-2 (355M) wasn't about raw performance — it was chosen precisely because it's small and transparent. The developer's goal wasn't to chase state-of-the-art results, but to genuinely understand how large language models work under the hood.

The learning journey is highly representative: starting from basic text processing, through attention mechanisms and causal modeling, all the way to instruction fine-tuning — building a working model from scratch, step by step. The entire process followed a systematic tutorial book on LLM fundamentals, implementing every chapter end-to-end.

A brief note on attention and causal modeling: The attention mechanism is the core of the Transformer architecture, introduced by Vaswani et al. in the 2017 paper Attention Is All You Need. The key idea is that when processing each token, the model dynamically decides which positions in the sequence to "attend to" by computing similarity between Queries and all Keys — rather than relying on fixed-window convolutions or recurrent structures. Causal modeling adds a masking constraint on top of this, ensuring that predictions at position i can only depend on positions 0 through i-1, preventing information leakage and enabling autoregressive token-by-token generation. Manually implementing the dot-product computation, scaling, Softmax normalization, and masking logic of an attention matrix is the key step to truly understanding why LLMs can understand context.

Active Reconstruction, Not Passive Following

The developer's study method is worth emulating: read a section, write the code, understand the logic — then close the book and rebuild it from memory. This approach feels slow, but it leads to genuine understanding rather than superficially copying code.

For data scientists and ML engineers who want to deeply understand how LLMs work at the architectural level — not just call APIs — this "build the wheel yourself" approach is especially valuable.

From Understanding to Practice: Teaching GPT-2 Function Calling

After mastering GPT-2's complete implementation, the developer took a critical next step: fine-tuning it directly to support function calling.

Function Calling was formally introduced by OpenAI in the GPT-3.5/GPT-4 APIs in 2023. Its core idea is that instead of only outputting natural language, the model can produce structured JSON-format instructions specifying which external function to call and what parameters to pass. This mechanism is essentially a form of targeted instruction fine-tuning: by training on large numbers of paired samples in the format of "user intent → function call format," the model learns to map natural language to machine-parseable invocation signals — such as calling a weather API, querying a database, or performing a calculation. Without reliable function calling, there's no real agent application.

How instruction fine-tuning works: Instruction fine-tuning is the core technique that transforms a pretrained language model from a "text completion machine" into an "instruction follower." During pretraining, the model learns statistical patterns of language from massive unlabeled text. During instruction fine-tuning, the model's parameters are adjusted using supervised data in the format of "instruction → expected response." This technique was systematized by InstructGPT (2022), combined with Reinforcement Learning from Human Feedback (RLHF) to better align models with human intent. For function calling scenarios, training data typically describes available tools using JSON Schema alongside user queries, with the target output being a standardized function call JSON.

What 88% Success Rate Actually Means

Achieving 88% correct, parseable function call output on a 355M parameter model from 2019 is a result worth paying attention to. Small models can achieve relatively high success rates on this type of task precisely because the output space for function calling is limited and pattern-consistent — the fine-tuning signal is sufficient to teach the model to "memorize" this structured output template, without requiring deep world knowledge. This tells us:

• Function calling capability depends more on targeted instruction fine-tuning than on simply scaling model size;
• For relatively "pattern-based" tasks like structured output, small models can handle the majority of scenarios after alignment;
• The remaining 12% failure rate primarily stems from complex parameter combinations, edge cases, or format deviations — in engineering terms, this means 12 out of every 100 calls still require retries or fallback handling, which is a non-trivial challenge in production but highly respectable for a teaching experiment. This is also where the real gap between small models and frontier models lies.

A Reproducible Experiment on Free GPUs

The entire project ran on free GPU resources provided by Kaggle. Kaggle is Google's data science competition platform that offers free GPU compute to registered users — currently providing NVIDIA Tesla P100 (16GB VRAM) and Tesla T4 x2 (16GB each), with approximately 30 hours of free quota per week. For fine-tuning a model at the scale of GPT-2 (355M), memory usage is roughly 2–4GB under mixed precision (FP16), well within what a single T4 GPU can handle from data loading through to model training. This stands in stark contrast to the thousands of A100s required to train GPT-3 (175B), vividly illustrating the exponential gap between model scale and hardware requirements.

Understanding and practicing core LLM mechanisms doesn't require expensive hardware. Beyond Kaggle, Google Colab (free T4) and Hugging Face Spaces are similarly accessible free compute options, together forming a "democratized compute ecosystem" for learners. A cloud GPU freely accessible to any developer is sufficient to run the complete pipeline from model construction to instruction fine-tuning.

This low-cost, reproducible experimental path makes for an excellent hands-on project for getting started with LLM fine-tuning. The developer also shared a complete technical write-up (with animation demos and results) and a GitHub repository in the comments, making it easy for others to replicate.

Key Takeaways for Developers

This is a small project, but it reflects several ideas worth thinking deeply about.

Scale Isn't the Only Answer

In the rush toward ever-larger models, it's easy to overlook that many real-world tasks — especially structured, pattern-based ones — don't require top-tier models. Using a small model for a well-matched problem often offers significant advantages in cost and inference latency.

Implementing from Scratch Is the Deepest Form of Learning

Calling ready-made APIs is certainly efficient, but if the goal is to truly understand how LLMs work, implementing the attention mechanism, causal modeling, and fine-tuning pipeline by hand yields a fundamentally different kind of understanding. This "knowing the why" becomes a core competitive advantage when debugging and optimizing.

Low Barriers Make It Accessible to Everyone

Free compute resources combined with open-source code mean that core LLM technology is no longer the exclusive domain of large companies. Any motivated developer can walk through the complete model fine-tuning journey on a free cloud GPU.

Conclusion

Achieving 88% function calling success rate on a 6-year-old model running on free GPUs — the value of this project isn't in topping any leaderboard. It's in clearly demonstrating a low-cost, understandable, and reproducible path for practicing LLM development. For developers who want to genuinely master the underlying principles of large language models, this may be far more meaningful than blindly chasing the latest and largest models.