Local-First AI Router GreenRoute: Should Your Prompts Leave Your Laptop?

A Question Mainstream Routing Tools Overlook

Every time you use an AI feature—whether it's a chatbot, a writing assistant, or a "summarize this" button—the request goes to a data center. Even if you're just asking "What's the capital of Australia," even if a small model on your laptop could answer it perfectly well.

There are already plenty of LLM routing tools out there: OpenRouter, NotDiamond, Martian, plus the built-in model routers from major cloud platforms. They solve the question of "which cloud model gives the best value," but nobody before this had asked: does this request actually need to leave the device at all?

Developer ironman2cool built GreenRoute to answer that question. It decides whether each request should be handled locally, explains the reasoning, and estimates how many resources were saved. The project started as a weekend hackathon prototype, then was rewritten into a complete version with a real router, local inference, benchmarks, and verifiable data.

LLM routing refers to the process of dynamically choosing among multiple available models to execute a given request, with the goal of balancing quality, cost, latency, and other metrics. Mainstream tools like OpenRouter, NotDiamond, and Martian all focus on horizontal comparisons between cloud models—for example, using a cheap small model for simple questions and handing hard problems to a flagship model. They assume by default that requests must leave the device, so the optimization space is limited to "which cloud to send it to." GreenRoute proposes an orthogonal split: first determine whether the request needs to leave the device at all, then make a binary choice between "local" and "cloud." This effectively expands the routing decision space from "a set of cloud models" to a hybrid structure of "local + cloud," and brings energy consumption, water usage, privacy, and offline availability into the optimization objectives.

Core Mechanism: A Deliberately "Boring" Router

GreenRoute is a lightweight Python service built on FastAPI, with a web chat interface as the frontend. For each prompt, it performs four steps: classify the request (task type and complexity), check device status (local model size, available memory, battery level), route to the local model (Llama 3.2 1B running on Ollama) or the cloud (Google Gemini), and finally estimate the resource footprint of that answer relative to an "all-cloud" baseline.

If the local model is missing or crashes mid-generation, the request automatically falls back to the cloud and the user is notified. No path ends with an empty reply.

Rule-Driven, Not a Trained Classifier

Instead of training a classifier, the author uses a set of rules that can be read in one sitting. This is a deliberate choice: every decision must be explainable in a single sentence, and it must complete in well under 1 millisecond, never slowing down the request.

The classification step determines task type through pattern matching (Q&A, summarization, rewriting, translation, code, analysis, long-form writing, image generation, etc.). Each type has a base complexity score, which is then adjusted based on evidence: input length (a 3,000-token document is much harder than a single sentence), expected output length ("1,500 words," "5 paragraphs," "one sentence"), and cue words like "detailed," "step-by-step," "brief."

The author shares two lessons from building this. First, keywords should only be counted in the instruction, not in the material you paste: in Make this email more polite: "Send me the report now.", the word "report" should not turn the request into a report-writing task. The first version made exactly this mistake. Second, when multiple rules trigger simultaneously, the most complex task wins, so errors lean toward the cloud—the expensive direction rather than the wrong-answer direction.

Seven Checks Executed in Order

The router's second step is seven checks; the first one that fails sends the request to the cloud and becomes the reason the user sees:

Check	Sent to Cloud When
Task type	Image requests (local text models can't draw)
Device mode	Device is flagged as low-spec
Battery	Below 20% and not plugged in
Input length	Prompt exceeds roughly 1,500 tokens
Confidence	Request can't be classified
Complexity	Score exceeds the local model's ceiling
Memory	Not enough free RAM to load the model

The complexity ceiling depends on local model size: around 0.35 for a ~1B model, 0.5 for ~3B, 0.6 for ~7B. Bigger local models keep more work on-device. Every answer has a "Why?" panel listing all checks, so users can see exactly which rule sent the request away.

Ollama is an open-source local LLM runtime that provides an OpenAI-like HTTP interface and can load and run open-weight models like Llama, Mistral, and Gemma on consumer hardware. It handles model downloads, quantized weight management, memory allocation, and inference scheduling, so developers don't have to assemble the inference stack themselves. GreenRoute chose Ollama as its local execution layer, so "local availability" actually depends on two things: whether Ollama has already pulled the corresponding model, and whether the current device has enough RAM to load that model entirely. The "memory check," "device mode," and other rules later in the article are essentially determining whether Ollama can reliably complete this inference.

The Honest Math: Is Local AI Really More Efficient?

"Local AI is greener" is easy to say but hard to get right. The author used traceable public data sources.

The cloud side uses Google's own published figures (August 2025): a median text prompt sent to Gemini consumes 0.24 Wh of energy, 0.26 mL of water, and 0.03 g CO₂e. This covers accelerators, hosts, idle capacity reserved for reliability, and data center cooling. This is much lower than the scary estimates circulating in 2023, and the author uses it as the baseline.

The local side isn't free. The laptop also draws power during generation. GreenRoute estimates it using additional power draw × measured generation time, with a default of 25W for laptop CPU and carbon emissions based on grid intensity (458 gCO₂/kWh, 2025 world average, from Ember via Our World in Data). Local inference doesn't use data center cooling water. Neither side counts power plant water, keeping the comparison fair.

Savings = cloud baseline − local cost. If a slow local answer on a high-power machine costs more than the cloud, savings show up as negative. The author wants the tool to be able to say "this one isn't worth it."

Wh (watt-hours) is a unit of energy; 1 Wh represents the energy consumed by running at 1 watt for 1 hour, commonly used to compare per-task energy consumption across computing devices. CO₂e (carbon dioxide equivalent) converts different greenhouse gases into equivalent amounts of carbon dioxide based on global warming potential, making it easier to measure carbon footprints uniformly. The comparison in this article is built at the "single prompt" scale: the cloud side uses Google's published median (0.24 Wh, 0.03 g CO₂e), while the local side multiplies additional power draw by measured generation time. Understanding these two units helps you read the break-even point later in the article: when local additional power draw reaches about 136W, energy per unit time multiplied by the longer generation time exactly cancels out the cloud baseline, and above that it becomes negative savings.

Experimental Results: Routing Accuracy and the Quality Gap

Test Design

Before running any tests, the author wrote 60 everyday prompts and labeled each one with where it should run—the criterion being "can a ~1B on-device model give a good enough answer that I don't need to ask again?" Of these, 38 were labeled local and 22 cloud. Then the real system was run on the author's laptop (Intel Core i5-1135G7, 4 cores, 12GB RAM, no discrete GPU), while for comparison, the local model was also run on all prompts, including those the router sent away. A larger cloud model (Gemma 4 26B via the Gemini API) wrote reference answers and scored each local answer on a 1-5 scale.

Routing Performance

GreenRoute kept 35 of the 60 prompts on-device (58%), agreeing with human labels on 57/60. All three disagreements leaned toward the safe side: questions the author thought could be handled locally (a question about Python len(), a one-line SQL query, a short email to a landlord) were sent to the cloud. No prompt labeled cloud was kept local.

Some cloud decisions had less flattering correct reasons: a proof problem, a travel itinerary, and a system design question didn't match any rule and were sent away by the "unclassifiable, handle conservatively" check rather than being identified as hard tasks.

Quality: The Most Critical Finding

This is the result the author cared about most. On the 35 prompts the router kept local, the 1B model's answers were judged acceptable 30 times (86%, averaging 4.4/5). On the 23 text prompts sent to the cloud, the same model's answers were acceptable only 4 times (17%, averaging 2.0). The line drawn by the router roughly falls where the small model's capability drops off.

The five weak answers that slipped through are instructive: a grammar correction that didn't fix "Me and him," two translations, a date extraction that lost "June 3rd," and a TL;DR that changed the reason for a project delay. For a 1B model, "simple" doesn't equal "safe."

Resource Footprint

Compared to sending all 60 prompts to the cloud, GreenRoute's estimated footprint is 48% lower in energy (7.5 vs 14.4 Wh), 58% lower in water (6.5 vs 15.6 mL), and 19% lower in CO₂e (1.46 vs 1.80 g). Carbon savings are smallest because Google's per-prompt figures already reflect its clean energy procurement, while the author's laptop runs on the average grid. If every weak local answer were re-asked in the cloud, savings shrink to 39%, 50%, and 11%.

The per-prompt savings of about 0.1 Wh are small, but they matter at scale: the same savings applied to a billion prompts a day would be roughly 100 MWh per day.

Energy results are highly device-dependent. Local routing stops saving energy at about 136W of additional generation power. A laptop CPU (~25W) or Apple Silicon Mac (~10W) is far below that; but a gaming desktop GPU pushed to 150W is not, and would use about 6% more energy than the all-cloud baseline.

Speed

On the author's laptop, the 1B model took a median of 4.4 seconds (90th percentile 12.8 seconds) on the prompts it handled, about 12 tokens/sec. The free Gemma 4 endpoint used for reference answers had a median of 5.8 seconds, but fast commercial models are quicker: Gemini 2.5 Flash took about 1-2 seconds on short prompts in the author's spot check. On a laptop CPU, local is not the fast option. Its advantages are energy, water, privacy, and offline availability.

Limitations

Sixty prompts is a small sample, and the only annotator was the author. The judge itself is an LLM that also wrote the reference answers. Local power draw is an assumed value (25W) rather than measured, and Google's numbers are medians for its own traffic and models. All prompts, labels, judging prompts, and raw answers are public in the repository, so anyone can inspect or rerun them.

Benchmarking here is not a standardized public evaluation set, but rather the author's own 60 everyday prompts with manual "should be local / should be cloud" labels. Its value lies in reproducibility: all prompts, labels, judging prompts, and raw answers are public in the repository, so anyone can rerun or challenge them. It's worth noting that conclusions from such small-sample benchmarks are highly conditional—device model, model version, and even the day's grid carbon intensity can affect results. So figures like "48% energy savings" in the article should be understood as estimates for that configuration, not universal laws.

Directions for Next Steps

The author believes the decision-making itself should stay as is: a handful of cheap rules, running in under 1 millisecond, explainable in one sentence. The focus of work is making the rules more accurate without adding weight.

Set ceilings by task type. Currently all tasks of the same model size share one complexity ceiling. Benchmarks show the 1B model is reliable on facts and rewriting but unstable on translation and extraction. A per-task-type ceiling table is still a lookup operation and could catch several weak answers that slipped through.

Adjust ceilings from feedback. A "thumbs down" button on local answers plus benchmark judgment scores is enough to fine-tune each task's ceiling over time. This is a running count per task type, not model training.

Do cheap checks on local answers. Some failures only show up in the output: the answer is far shorter than requested, repeats itself, or omits part of the request. Simple checks on answers (length vs. request, whether prompt keywords appear) can flag suspicious local answers and silently retry them in the cloud. The routing decision stays cheap, and only suspicious cases pay the cost of a second call.

Fix known misses. All three routing mistakes were short, harmless requests: a Python function question, a one-line SQL query, and a short message. A "short code question" rule and a "short message" rule would cover them.

Only introduce a small learned scorer if rules stop scaling. If the rule list becomes hard to maintain, logistic regression on signals the rules already compute (task keywords, input length, requested length, detail cues) would replace hand-tuned weights with fitted ones. It's still a weighted sum running in microseconds, and the weights can still explain every decision.

Worth Thinking About

The code is open-sourced on GitHub (ironman2cool/greenroute), supporting Windows, macOS, and Linux, requiring only a free Gemini key, with Ollama optional. The benchmarks, prompts, judging prompts, and all results are in the repository.

If your application sends every request to the cloud by default, it's worth asking: how many of them actually need to leave the device?