Building a Tiny Model Router in TypeScript: Architectural Lessons from the Gemini Agent

Title

Building a Tiny Model Router in TypeScript: Architectural Lessons from the Gemini Agent

Meta Description

Google's Gemini agent separates model selection from agent orchestration. This article builds a tiny model router in TypeScript with a three-step routing strategy: capability filtering, cost sorting, and budget control.

Article Content

At the Gemini at Work event, Google Cloud announced the Gemini agent, billed as a "general-purpose agent for work." It can plan tasks, invoke skills and tools, and connect to business systems. But one line in the announcement caught my attention:

It selects the most suitable model for each task, with built-in cost control and the security, management, and governance capabilities enterprise customers need.

In his keynote, Thomas Kurian added that the agent can orchestrate across Google's Gemini models and Anthropic's Claude models, emphasizing that "the model best suited to a task isn't necessarily the biggest one."

This means the agent and the model are now two independent choices. Every task needs someone — or some code — to make that choice.

Routing is a policy, not a feeling. If you can't explain why a model was chosen, you don't have a router at all.

The Nature of the Problem: Three Questions Decide Everything

A router must answer three questions for every task.

Which models are capable of completing this task? Of those qualified models, which is the cheapest? Is there any budget left to run it?

Most code only answers the first question — hardcode a model name and call it a day. When the bill arrives, the third question answers itself for you.

The order is itself the design: capability first, price second, budget last. This order cannot be reversed, because a "cheap but incapable" model is essentially a retry with extra steps. It looks like it saves money, but it's actually generating rework.

Model Routing is not a concept Google invented; it has a clear technical lineage in the large-model engineering space. Early implementations appeared in multi-model API gateways (such as OpenRouter and LiteLLM), where the core motivation was cost and latency — the same task can differ in expense by a hundredfold depending on the model used. After 2024, as the Claude, Gemini, and GPT series gradually diverged in capability, routing upgraded from a "money-saving trick" to an "architectural decision": different models have their own weaknesses in code, reasoning, long context, and tool calling, and a single model struggles to cover the entire workload. Academic work like RouteLLM and FrugalGPT approaches this from another direction, using trained classifiers to predict "task difficulty" and dynamically switching between weak and strong models to cut costs while preserving quality. Enterprise-side routing also layers in a compliance dimension: data residency, audit logs, and vendor lock-in all require the orchestration layer to be decoupled from the model. This is why the article emphasizes that "the agent and the model are two independent choices" — it's fundamentally about splitting "what to do" and "who does it" into two architectural layers, so models can be replaced, measured, and governed just like backend services.

Step One: Describe the Models

To the router, a model is just a set of comparable facts. Each model has a tier (small/medium/large), a price (cents per thousand tokens), a maximum context window, and whether it supports tool calling.

Each task, meanwhile, declares what tier it needs, how big it is (token count), whether it needs tools, and whether there's a human-specified model override.

The override is kept for a reason — Google explicitly stated in the announcement that users can choose their own model. This design detail is worth keeping in the mini implementation because it reflects a real requirement: humans sometimes understand the special circumstances of a given scenario better than the policy does.

type Tier = "small" | "medium" | "large";

type Model = { name: string; tier: Tier; usdPer1kTokens: number; maxContext: number; tools: boolean };
type Task = { id: string; project: string; need: Tier; tokens: number; tools?: boolean; override?: string };

Step Two: Determine Fitness

Fitness is a hard filter, not a score. A model whose tier is below what the task requires doesn't fit. A model whose context window can't hold the task's token count doesn't fit. A model that doesn't support tools when the task requires them doesn't fit.

const RANK: Record<Tier, number> = { small: 0, medium: 1, large: 2 };

function fits(m: Model, t: Task): boolean {
  if (RANK[m.tier] < RANK[t.need]) return false;
  if (t.tokens > m.maxContext) return false;
  if (t.tools && !m.tools) return false;
  return true;
}

The key insight here is that fitness checks must happen before price sorting. Exclude the models that can't do the work first, then pick the cheapest among those that can. Doing it the other way around just wastes time — you might pick a cheap model first, discover its context is insufficient, switch, then discover it doesn't support tools, and switch again.

The Context Window is the most underappreciated constraint in routing decisions. It refers to the maximum number of tokens a model can receive in a single inference pass, including the input prompt and the output. Windows vary enormously across models, from the early 4K and 8K to today's 128K, 200K, and even million-token ranges. Bigger isn't simply better: a larger window means the quadratic complexity of the attention mechanism drives up inference cost and latency significantly, and the "lost in the middle" phenomenon dilutes key information buried in ultra-long contexts. In engineering practice, task token counts are usually hard-compared against the model's window — exceeding it means elimination, a non-negotiable physical limit. The summarize-thread example in the article is typical: the task is labeled at the small tier, but its 40,000-token input immediately rules out the smaller-window mini model, triggering an upgrade to mid. This kind of "capability label doesn't match actual input size" situation is extremely common in production, and it's why fitness filtering must run before price sorting — first check whether it can hold the load, then talk about whether it's expensive.

Step Three: Pick the Cheapest Qualified Model

Candidates that pass the fitness filter are sorted by estimated cost, and the top one is chosen.

const candidates = MODELS.filter((m) => fits(m, t))
  .sort((a, b) => cost(a, t) - cost(b, t));
if (candidates.length === 0) return { why: "no model fits" };
return { model: candidates[0], why: `cheapest model that fits (${candidates.length} fit)` };

Note a few design decisions. Overrides still have to pass the fitness check — if someone insists on assigning a small model to an oversized task, the router should refuse rather than execute. Unknown model names are rejected rather than guessed at. Every result carries a why string, and that string is where the entire value of the router lives.

Step Four: Enforce the Budget Ceiling

Each project has a budget cap, and a Ledger tracks how much has been spent. The check happens before the task runs, not after.

class Ledger {
  spent = new Map<string, number>();
  constructor(private caps: Record<string, number>) {}

  route(t: Task): Route {
    const { model, why } = pick(t);
    if (!model) return { kind: "rejected", task: t.id, why };
    const c = cost(model, t);
    const spent = this.spent.get(t.project) ?? 0;
    const cap = this.caps[t.project] ?? 0;
    if (spent + c > cap) {
      return { kind: "paused", task: t.id, model: model.name, cost: c,
        why: `cap $${cap} would be hit (spent $${spent.toFixed(4)})` };
    }
    this.spent.set(t.project, Math.round((spent + c) * 10_000) / 10_000);
    return { kind: "run", task: t.id, model: model.name, cost: c, why };
  }

  raise(project: string, cap: number) { this.caps[project] = cap; }
}

Paused is not the same as failed. A paused task waits for a human to raise the cap, then resumes. Google says its agent has built-in cost control, and this is a mini version of that concept.

When a human decides to raise a project's budget cap, paused tasks automatically resume. But the router never raises its own budget — that constraint is deliberate.

"Pause vs. fail" in cost control is a deliberate design pattern with precedents in distributed systems and cloud billing. The traditional approach is to throw an error when a task exceeds budget, letting the caller retry or give up; but in agent scenarios, the first half of a long-chain task may have already produced valid output, and discarding it means all the prior tokens were wasted. Pause semantics preserve task state and resume in place once external conditions (a manual cap increase) change, closer to a workflow engine's "wait for signal" than a "compensating transaction." Paired with this are pre-authorization and reconciliation: the article uses estimated cost to check and record before running, which is optimistic reservation; a more rigorous production approach is two-phase commit — freeze the quota before running, then settle against actual usage afterward and release the difference, otherwise estimation drift accumulates into a mismatch between books and reality. Moreover, a single-process Map ledger creates race conditions under multi-replica deployment; a real system needs Redis or a database's atomic increment or a distributed lock to ensure the cap can't be breached simultaneously.

How It Actually Runs

The whole flow is demonstrated with seven example tasks, with input and output as follows:

tiny-model-router demo
[RUN]    tag-email         mini $0.0008  cheapest model that fits (3 fit)
[RUN]    draft-reply       mid $0.0180  cheapest model that fits (2 fit)
[RUN]    contract-review   big $2.7000  cheapest model that fits (1 fit)
[PAUSED] summarize-thread  mid $0.1200  cap $0.1 would be hit (spent $0.0188)
[RUN]    quick-fix         big $0.0150  user override
[REJECT] translate         override "gpt-x" is not a known model
[PAUSED] second-review     big $2.7000  cap $3 would be hit (spent $2.7000)

raise legal cap to $6, resume paused work
[PAUSED] summarize-thread  mid $0.1200  cap $0.1 would be hit (spent $0.0338)
[RUN]    second-review     big $2.7000  cheapest model that fits (1 fit)

spent: { support: 0.0338, legal: 5.4 }

A few routing decisions worth noting: tag-email can be handled by all three models, so it's assigned to the cheapest, mini. Draft-reply needs tool calling, which mini doesn't support, so mid wins. Contract-review needs the large tier, and only big qualifies. Summarize-thread is labeled a small-tier task, but its 40,000 tokens exceed mini's context window, so it routes to mid — and that step would exceed the support project's budget cap, so it pauses. Translate requests a non-existent model name and is rejected. The second legal review (second-review) would exceed the legal project's $3 cap and pauses, waiting for a manual cap increase.

After a human raises the legal cap, second-review resumes. Summarize-thread stays paused because no one raised support's cap. That's correct — the router has no authority to raise its own budget.

The Limits of This Model

This is a teaching model; real routing is much harder.

The tier is a guess. Real tasks don't label themselves "I'm small tier." You need a classifier or a cheap heuristic to make that call.

Price is computed per token. Actual cost also includes tool calls, sandbox environments, retries, and cached tokens.

There's no quality check. A real router should monitor whether the cheap model's answer holds up and route to a more expensive model if it doesn't.

Estimates come before the run. Token counts are only known exactly after running, but the budget check uses estimates.

Single ledger, single process. Real spend tracking is shared, and two tasks could cross the cap at the same time.

None of these limitations change the overall shape of the architecture: filter by capability, sort by price, check the budget, print the reason.

The Bigger Picture

The entire routing logic can be summed up as one decision tree:

task → filter: tier, context, tools → sort: cheapest fit (or override) → ledger: under the cap? → run / pause / reject, with a reason attached at every step.

Tasks provide requirements, models provide options, filters provide the floor, price provides the ordering, budget provides the ceiling, and humans provide the cap increases.

I think the question "what model are you using?" is becoming the wrong question to ask about an agent. Models change every few months. Google said almost exactly that in its announcement.

The better question is: what's your routing policy, and can you read it?

Choosing a model is now part of the job. So it should be part of the logs, too.

One-line Summary

Google released the Gemini agent, decoupling model selection from agent orchestration and making routing an independent infrastructure layer that needs an explainable policy.

Paragraph Summary

At the Gemini at Work event, Google's Gemini agent turned model selection into a decision independent of agent orchestration: it orchestrates across Gemini and Claude models and picks the best-suited model per task. The article argues that routing is a policy, not a feeling, and must answer three questions in a fixed order — which models are capable of the task, which of the qualified ones is cheapest, and whether the project budget still has room. The author implements a tiny router in TypeScript: capability filtering, price sorting, budget checking, with a why string at every step. In the demo, a task labeled small gets upgraded to mid because its 40,000 tokens exceed the window, over-budget tasks enter a paused state and resume after a manual cap increase, and unknown model names are rejected outright. The article concludes that the gap between the teaching model and a production system lies in needing a classifier for task tiering, including tools and retries in cost, and the lack of quality feedback — but the overall architectural shape stays the same.