Small Models or Frontier Models? Choosing the Right AI for Your Feature

When a SaaS product decides to embed an AI feature, the question founders ask most often is usually "which model is the smartest?" But the question they should really be asking is: which model passes my evaluation set at an acceptable accuracy level while costing the least to run? This shift in perspective determines whether an AI feature becomes a profit center or a cost sink.

Why "Smartest" Is the Wrong Question

The leaderboard-champion model is rarely the best choice for a feature that runs fifty thousand times a month in your product. Take ticket classification: with fifty thousand tickets a month, the leaderboard champion and some distilled "flash" model label clear-cut tickets identically. Differences only show up on ambiguous cases—and an output validator that checks whether the label is in an allowed list, plus retry-on-failure, catches a fair share of the small model's mistakes.

The key measurement is simple: take fifty to a few hundred real tickets, run them through both models, and compare price per call with accuracy side by side. The decision shifts from betting on vendor marketing to engineering measurement.

A distilled model is a smaller, cheaper-to-run model trained using a large model's outputs as training signal. The common approach is to have the small model imitate the large model's answer distribution over a large number of inputs, thereby "compressing" part of the large model's capability into the small one. Distilled models often approach the teacher model's performance on tasks with clear patterns and stable input distributions, but degrade noticeably when they need to reason on the fly or handle out-of-distribution inputs. This is precisely the technical root of the article's "identical labels on clear tickets, divergence on ambiguous cases": distillation preserves high-frequency patterns and loses long-tail judgment. So when evaluating a distilled model, you must cover edge cases with real business samples rather than looking only at average accuracy.

Four Questions That Decide the Model Tier

1. Is the task bounded?

Bounded tasks have a known input shape, output shape, and set of acceptable answers. "Extract the invoice number, due date, and total amount from this PDF" is bounded; "read the customer's last six emails and decide what to do" is not. Small models perform strongly on bounded tasks because the prompt can carry most of the intelligence—schema definitions, rules, three examples. They degrade on unbounded tasks because those require improvisation, and the fewer parameters a model has, the worse its improvisation quality.

2. Can outputs be checked cheaply?

If deterministic checks can catch most bad outputs, you can run a smaller model and let the validator carry the weight. JSON schema validation, regex for dates and amounts, database lookups, length caps—these cost zero per call and turn the model's occasional error into a retry rather than a bug the customer sees. Conversely, if outputs can only be checked by a human reading them, the model has to get it right far more often on the first try, and the price naturally goes up.

An output validator is a deterministic program decoupled from the model that judges whether model output is acceptable before it reaches downstream systems. Common forms include JSON Schema validation, field type and value-range checks, regex matching, enum whitelists, and queries against external databases or fact sources. The value of a validator is turning "the model occasionally errs" into "a detectable and retryable failure," thereby reducing reliance on the model's own accuracy. A well-designed validator should satisfy three things: deterministic judgment logic, near-zero cost per execution, and, on failure, a specific reason usable for retry. A validator cannot judge the semantic level of "picked the right fact"—that is precisely where its applicability ends.

3. How costly is a wrong answer?

A mislabeled support ticket costs an agent ten extra seconds. A wrong dosage summary in a medical tool, or a wrong refund decision in an e-commerce agent, costs far more. Measure the cost of failure, not the cost of the call. Where failure is cheap, optimize per-call cost; where failure is expensive, the frontier model's per-token premium is negligible relative to the risk—and you should add human review regardless of which model you use.

4. Does the input fit? Does it need memory?

Small models usually have shorter usable context and "get lost" earlier in long documents. If a feature needs to examine a 200-page contract at once, or carry state across a 20-step workflow, you either chunk aggressively (more engineering) or pay for a model that can fit it. Multi-step tool-calling agents are the clearest use case for frontier models.

The context window refers to the maximum number of tokens a model can receive in a single call, including the prompt, examples, input documents, and accumulated conversation history. When the window is insufficient, there are usually two responses: chunking, which splits long documents into multiple segments processed separately, or summarization, which distills historical state into short text. Chunking severs cross-segment references and requires additional merge and deduplication logic; summarization may lose details. Multi-step agents also need to carry state between steps, and the state itself occupies the window. So "does the input fit" is really asking: can all the information the task requires plus intermediate state coexist within a single call's window—otherwise you pay for a larger-window model or take on extra engineering.

The Small Model's Real Strength: Our Outreach Engine

Our own outreach tool is a concrete example. It uses self-hosted Firecrawl to crawl each prospect's website, hands the page text to a local model, and writes a tailored email for each company. The task is bounded: the input is a few pages of text, the output is a short email following a template and tone, and the validator rejects content that is too long, mentions products that don't exist, or fails to cite facts from the website.

Surprisingly, having a single call do "extract facts + write email" beat the multi-step chain (extract, then plan, then write) on both cost and quality—even with a smaller model. The fewer the hops, the fewer chances the model has to drift from the source text. The takeaway for buyers: a well-constrained small model plus a validator is the right tool, and a frontier model would only add cost without changing reply rates.

When to Pay for a Frontier Model

Be honest about the cases where "cheap turns out expensive." The outreach engine above can use a small model for one reason only: single hop, text in and email out, with a validator behind it. Remove any one of those conditions and the picture changes.

• Open-ended reasoning over messy input. Triage that requires judgment rather than rules. A validator can reject an email that doesn't cite website facts, but it cannot judge whether the model picked the right facts—when that judgment is itself the feature, you need a stronger model.
• Multi-step tool use. When the task becomes "crawl the site, decide which pages matter, check the CRM, then write the email," the small model must carry state across hops, which is exactly where it drops threads.
• Low-frequency, high-value calls. For a feature that runs only a few hundred times a month and has every result reviewed by a senior person, the cost difference is noise. Buy accuracy.
• Code generation and complex transformations. The quality gap here remains large.
• The first version, before you have an evaluation set. It's often cheaper to launch with a strong model, collect real traffic, build an evaluation set of fifty to a few hundred real samples from it, and then downgrade where the evaluation set allows.

The last point matters. Starting from a frontier model and moving down is a measured decision backed by an evaluation set; starting from a small model and discovering in production that it isn't enough means rewriting prompts and guardrails under pressure, with no evaluation set to tell you when you're done.

How Model Choice Changes the Quote

Founders often assume a small model makes the project cheaper to build. Usually it doesn't. Build cost lies in prompt design, validators, evaluation sets, retry and fallback logic, and product integration—work that is the same regardless of the model behind it. A small model even needs a thicker validation layer, slightly increasing the engineering.

What changes is the running cost. The per-call price gap between a frontier model and a small hosted model is many times over, and across a year of real traffic the difference can exceed the original build budget. Before signing, ask for a cost projection at three tiers of usage. As for self-hosting a small model, vLLM and Ollama make it technically easy, but it only pays off with steady usage or data-residency requirements; at typical early-stage usage, a cheap hosted API model wins once you factor in engineering time.

The Hedge: Build a Switch, Not a Bet

Whichever way you choose, don't let it be welded into the code. Insist on three things in the requirements doc:

• The model hides behind a config value. Switching from a frontier model to a small one, or changing vendors, should be a deploy, not a sprint.
• Ship an evaluation set with the feature. Fifty to a few hundred real samples with expected outputs, run before every model change. Without it, "is the cheaper model good enough" is just a guess.
• A fallback path. When the small model fails validation twice in a row, escalate that call to a stronger model rather than showing the user a failure. That way you get the small model's economics plus the frontier model's worst-case quality.

Do this, and the model question becomes a knob you adjust gradually with traffic and evidence—not a one-shot decision made without data.

One-line Summary

When choosing a model, ask "which is cheapest and still passes the evaluation set," not "which is smartest."

Paragraph Summary

When embedding AI in SaaS products, the article argues, the real question isn't which model is smartest but which model passes your own evaluation set at acceptable accuracy with the lowest running cost. The author proposes four dimensions: whether the task is bounded, whether outputs can be checked cheaply, how costly errors are, and whether the input fits and requires memory. When a task is bounded, verifiable, cheap to fail, and short-input, a small model plus validator is often enough—a single call can even beat a multi-step chain. Conversely, open-ended reasoning, multi-step tool use, low-frequency high-value calls, and code generation still call for frontier models. The article recommends starting with a frontier model to collect an evaluation set before downgrading, hiding the model behind a config value, shipping an evaluation set with the feature, and setting up an escalation fallback path.