$9 to Have Gemini Train Its Own Replacement: A Practical Look at Model Distillation

An Experiment Title Worth Pondering

"I spent $9 to have Gemini train its own replacement" — this headline, which surfaced on Hacker News, hit squarely on one of the hottest topics in AI engineering today: using large models to train small ones. The original post was modest (19 upvotes, 7 comments), but the technical trend it reflects is worth unpacking.

The core idea isn't complicated: have a powerful, general-purpose model like Gemini act as the "teacher," generating high-quality training data or annotations for a specific task, then use that data to train a smaller, cheaper, self-hostable "student" model. When the student model reaches near-teacher performance on that specific task, you've effectively "replaced" expensive large-model API calls at a fraction of the cost.

Why $9 Is Enough to Pull This Off

The key lies in task convergence. General-purpose large models are expensive because they're designed to handle open-ended questions across every domain. Real-world production tasks, however, tend to be highly vertical — classification, extraction, formatting, text generation in a specific style, and so on. For these narrow tasks, a model with hundreds of millions to a few billion parameters, trained with targeted fine-tuning, can absolutely match or even outperform a general-purpose large model.

The $9 cost typically breaks down into two parts: API fees for calling Gemini to generate a dataset, and compute costs for fine-tuning a small model (or applying LoRA fine-tuning to an open-source base model). When the training data is kept to a few thousand to tens of thousands of examples and parameter-efficient fine-tuning methods are used, getting the total cost down to single-digit dollars is completely realistic. This is precisely the cost appeal of Knowledge Distillation.

The Technical Logic of Model Distillation

Transferring Knowledge from Teacher to Student

Traditional distillation methods have the student model learn from the teacher model's output probability distributions (soft labels), which carry richer information than hard labels alone. In the large language model era, however, the more common approach is data distillation: directly using the teacher model to batch-generate input-output pairs, then treating this synthetic data as supervised fine-tuning corpus.

This approach has a low engineering barrier — you don't need access to the teacher model's internal weights or logits; you just need API access. This is also why ordinary developers can accomplish something like "having Gemini train its own replacement" on a tiny budget.

It's worth noting that data distillation differs fundamentally from classical distillation in its technical approach. Classical distillation (proposed by Hinton et al. in 2015) relies on the teacher model's "soft labels" — probability distributions across all classes rather than simple 0/1 hard labels. These distributions carry implicit information about inter-class similarity (e.g., the teacher thinks an image is "80% cat, 15% dog"), so the student learns not just the right answer but a kind of "knowledge structure." In the large language model context, however, researchers rarely have direct access to complete output probabilities (logits), especially from closed-source models accessed via API. The industry has therefore shifted toward a more engineering-friendly synthetic data distillation approach: having the teacher model act as a data annotator, batch-producing high-quality input-output pairs for standard supervised fine-tuning (SFT) of the student. This evolution significantly lowered the technical barrier, but it also means the student model cannot learn the teacher's "confidence distribution," limiting its generalization on ambiguous samples.

Choosing the Student Model

In practice, student models are typically based on open-source foundations (such as Llama, Qwen, Gemma, etc.). There's an interesting closed loop here: Google's Gemini and its open-source Gemma share the same lineage, so using Gemini to generate data for fine-tuning Gemma-series small models is in some ways the ideal "big model teaches small model" pairing.

LoRA (Low-Rank Adaptation) is the most mainstream parameter-efficient fine-tuning (PEFT) method today, and it's what makes the "$9" cost achievable. The core idea: instead of updating all model parameters during fine-tuning, insert a pair of low-rank matrices (typically rank 4 to 64) alongside each Transformer layer's weight matrix, and only train those small matrices. For a 7-billion-parameter model, full fine-tuning requires updating roughly 7 billion parameters, while LoRA might only update 0.1–1% of them. This slashes the required VRAM from hundreds of GB down to something runnable on a consumer GPU (like an RTX 4090), and compresses training time from days to hours. After training, LoRA weights can either be merged back into the base model or saved separately as "adapters" for flexible switching. It's precisely the maturity of techniques like LoRA that empowers individual developers to customize models for vertical tasks at minimal compute cost.

The Real-World Significance of This Trend

For enterprises and independent developers alike, these practices are changing the deployment economics of AI applications. Long-term reliance on third-party large-model APIs means ongoing costs that grow linearly with usage volume, plus concerns around data privacy, latency, and vendor lock-in. A distilled small model, by contrast, can be deployed locally, run offline, trained once and used indefinitely — with near-zero marginal cost.

In other words, large models are shifting from "a permanent fixture in production" to "a one-time investment in training." You use them to "crystallize" knowledge into a cheap small model, then free yourself from dependence on them. This is why the headline's framing — "training its own replacement" — is both clever and accurate.

Limitations That Deserve Sober Assessment

Distillation isn't a silver bullet. The student model's capability ceiling is constrained by the teacher model and the quality of training data, and it still struggles to match frontier large models on open-domain reasoning and complex multi-step tasks. Beyond that, using the output of one commercial model to train another may run afoul of terms of service — many API agreements explicitly prohibit using outputs to train competing models. This is a real compliance risk that can't be ignored in practice.

Data diversity is also a common pitfall. If the synthetic data generated by the teacher model lacks coverage, the student model will easily fail on edge cases outside the training distribution. High-quality distillation therefore often still requires human review, data augmentation, and iterative evaluation.

On the compliance risk front, the major API providers take notably different stances. OpenAI's early terms explicitly prohibited using its outputs to train competing models; that position has softened somewhat but restrictions remain. Google's Gemini API terms similarly include prohibitions against "using the service outputs to develop models that compete with Google's products." However, the actual enforceability and legal validity of such clauses varies across jurisdictions, and parts of the open-source community take a more permissive interpretation. In practice, the safer path is to prioritize models with more open licensing (such as Meta's Llama series or Mistral) as data generation sources, or to carefully review the current version of any API's terms before embarking on a commercial project. Compliance risk isn't a technical problem — but it may be the biggest potential obstacle to an entire distillation project.

Conclusion

This $9 experiment is, in essence, a microcosm of the "cost reduction and efficiency gains" logic driving AI industrialization. When the capabilities of powerful models can be cheaply "distilled" and "transferred," the economics of AI applications are being rewritten. For developers, mastering the methodology of model distillation may offer more practical value than simply chasing the latest and largest models.

One caveat: since the original post was fairly brief, the discussion of specific implementation details in this article is based on general practices in the field. Actual results will depend on task type and engineering quality.