SpecJudge: A Locally-Run AI Model Selection CLI Tool

An Overlooked Pain Point: We're Constantly Over-Using Frontier Models

In AI application development, there's a pervasive problem that few people confront head-on: developers habitually default to the most powerful frontier models, often with the sole justification of "just in case." Frontier models typically refer to the flagship large language models released by major AI labs—OpenAI's GPT-4o, Anthropic's Claude Sonnet/Opus, Google's Gemini Pro, and so on. These models represent the current state of the art in terms of parameter scale, training data volume, and overall capabilities, but correspondingly, their API costs are far higher than mid-tier or smaller models. Taking GPT-4-class models as an example, their per-million-token input cost is typically 10-30x that of GPT-3.5-class models, yet many everyday development tasks (such as code formatting, simple text extraction, template generation) don't remotely require that level of reasoning capability.

Behind this "better safe than sorry" mentality lies significant cost waste and resource misallocation—many tasks simply don't need GPT-4-level capabilities, and a local small model would be perfectly adequate. According to industry observations, many startups see their API bills balloon rapidly during the early prototyping phase due to indiscriminate use of frontier models, with some teams wasting 30%-60% of their AI infrastructure budget each month on "over-provisioned" model calls.

Recently, a developer shared their solution on Reddit: a command-line tool called SpecJudge. Its core philosophy is simple yet precise—rather than guessing which model your project needs based on gut feeling, let a tool read your project's spec documents and provide a recommendation based on actual requirements.

How SpecJudge Works

Analysis Starting from Spec Documents

SpecJudge is designed for projects that follow Spec-Driven Development. Spec-Driven Development is a methodology where project requirements, constraints, and task breakdowns are pre-defined in structured document form. It's highly aligned with the "Context Engineering" concept that has gained traction in AI-assisted programming. In traditional software development, we have requirements documents and technical specifications; in AI coding tool workflows (such as Cursor, Aider, Claude Code, etc.), spec documents play a similar role—they provide AI with clear contextual boundaries, enabling it to work efficiently within constraints rather than repeatedly guessing developer intent in open-ended conversations. The core assumption of this methodology is: if you can clearly describe a task, then you can assess what level of intelligence is needed to complete it.

SpecJudge reads spec artifacts from your project—including constitutions (constraint documents defining the AI assistant's behavioral boundaries and principles), specs (specification documents describing functional requirements and architectural design), and tasks (task lists breaking work into executable atomic units). These documents essentially describe what the project needs to do, what constraints exist, and what specific tasks need to be completed.

The tool calls a model in your local Ollama environment (the "judge," chosen by you) to read these tasks and evaluate the "rigor level" of the work across several dimensions—essentially how demanding these tasks are in terms of model capabilities. Ollama is an open-source framework designed specifically for running large language models locally, simplifying the complex model deployment process into an experience similar to Docker managing containers. Users can download and run various open-source models through simple commands (like ollama pull llama3), including Meta's Llama series, Mistral, Phi, Gemma, and more. Under the hood, Ollama handles model quantization, GPU/CPU scheduling, memory management, and other technical details, making it possible to run 7-billion or even 13-billion parameter models on consumer-grade hardware. In SpecJudge's architecture, Ollama serves as the local inference engine—the "judge" model itself runs on your machine, which is the technical foundation enabling the entire tool to achieve "zero cloud calls."

Matching Against a Model Capability Catalog to Output Recommendations

The evaluation results are cross-referenced against a declarative model capability catalog. The term "declarative" here is an important design choice—it means the catalog only describes "what is" (what capabilities each model has, what domains it excels in), not "how to" (the specific evaluation algorithm logic). This shares the same design philosophy as Kubernetes resource configurations and Terraform infrastructure definitions in the DevOps world—using human-readable configuration files to declare desired state, separating complex implementation logic from user-visible configuration. The choice of YAML (YAML Ain't Markup Language) as the configuration format is also no accident: compared to JSON, YAML supports comments, has more intuitive hierarchical structure, and better readability, having become the de facto standard configuration language in the cloud-native and developer tool ecosystem.

This catalog is maintained in human-readable YAML format, recording various models and their capability profiles. Ultimately, SpecJudge outputs a ranking "podium," giving each candidate model a rating:

• good — The model's capabilities best match the task requirements
• overkill — The model's capabilities far exceed what's needed, meaning you're paying for capabilities you won't use
• fair — The model can basically complete the task, but may have shortcomings in certain dimensions
• poor — The model's capabilities clearly cannot meet task requirements

Price information is also included. It's worth emphasizing that rankings are based on "fit" rather than price—price only serves as a tiebreaker when multiple models have comparable fit scores. This design logic embodies an important principle: the cheapest isn't necessarily the best, the most expensive isn't necessarily the most suitable—what matters is "just right."

Fully Local Execution: Privacy and Security Guaranteed

One of the tool's most attractive features is its fully local execution. Throughout the entire evaluation process, no data leaves your machine—no API keys required, no account registration, no cloud calls whatsoever. For teams and individual developers who value code privacy and data security, this is extremely valuable. In enterprise environments, codebases and project spec documents often contain sensitive business logic, architectural designs, or even trade secrets, and sending this content to third-party API services could violate security compliance requirements (such as SOC 2, GDPR, etc.). SpecJudge's local execution mode completely eliminates this risk.

Even more interesting is the author's own observation: in most of his projects, the top of the podium is typically occupied by a local model, while frontier models rank below, marked as overkill. This indirectly confirms the problem raised at the beginning—we are indeed often "using a sledgehammer to crack a nut."

Honest Limitations: This Is Not a Benchmark

The author maintains a rare clarity and candor about the tool's positioning. He explicitly states that SpecJudge is not a benchmark, but rather an "opinion made inspectable."

This positioning choice deserves deeper understanding. In the AI field, benchmarks (such as MMLU, HumanEval, GPQA, etc.) typically refer to quantifiable, reproducible performance evaluations of models on standardized datasets, with results presented as definitive scores, aiming to provide "objective facts." However, benchmarks have well-known limitations in practical applications: leaderboard scores often diverge significantly from real-task performance (the "leaderboard effect"); models may have been optimized for specific benchmarks (benchmark gaming); and generic benchmarks cannot reflect the specific needs of particular projects. SpecJudge's choice to position itself as "opinion" rather than "fact" actually acknowledges a deeper reality: for the question "is this model suitable for my project," there is no universally applicable objective standard—only reasonable judgments based on specific context.

This statement highlights two key design philosophies of the tool:

Every Conclusion Is Traceable

Every rating prints its reasoning process, so you can see why the tool made a particular judgment. This transparency means developers don't have to blindly trust results—they can understand and challenge them. In AI tool design, this is a paradigm known as "explainability first"—when a system cannot guarantee 100% correctness, enabling users to audit the reasoning process is more valuable than pursuing superficial certainty.

The Model Catalog Itself Can Be Modified

The model capability catalog is a human-readable YAML file, and you're fully empowered to disagree with and modify the judgments within it. This "debatable" open attitude is far more pragmatic than a black-box "authoritative answer."

Additionally, the tool has a responsible design feature: when spec documents are too thin to make a judgment, it refuses to give a recommendation rather than guessing. As the author puts it, "vague specs only get vague answers"—this holds true for any context-dependent AI tool. This "rather not answer than answer poorly" design principle is known in AI applications as "calibrated uncertainty" (knowing what you don't know), and is an important practice in building trustworthy AI systems.

Open Source Installation and Community Co-Creation

SpecJudge is open-sourced under the MIT License, hosted on GitHub (github.com/JoaquinRuiz/SpecJudge), with an extremely simple installation:

pip install specjudge

The MIT License is one of the most permissive licenses in the open-source world, allowing anyone to freely use, copy, modify, and distribute the software, with the only requirement being to retain the copyright notice. This choice lowers the legal barrier for enterprise adoption—whether you're an individual developer or a large corporation, you can use the tool in commercial projects without concern.

The author specifically mentioned the direction where he'd like community help: the local model catalog is not yet comprehensive enough. The good news is that adding a new model only requires writing a YAML configuration snippet—no Python code needed whatsoever. This design greatly expands the range of potential contributors—you don't need to be a Python developer; as long as you have hands-on experience with a particular model, you can contribute your knowledge to the catalog in a structured way. This is similar to Homebrew's Formula mechanism or Docker Hub's image descriptions: reducing the core contribution action from "writing code" to "writing configuration," thereby activating broader community participation.

At the same time, the author warmly welcomes users pointing out incorrect rating judgments. This low-barrier contribution approach, combined with the debatable design philosophy, gives SpecJudge the potential to grow into a community-maintained model selection reference that stays close to real project needs.

Summary: Turning AI Model Selection from Intuition into Explainable Decisions

What SpecJudge actually solves is a methodological problem: transforming model selection from "defaulting to the strongest based on gut feeling" into "explainable decisions based on actual project requirements."

Its value lies not in providing an absolutely correct answer (the author himself admits this is just an "opinion"), but in offering an inspectable, debatable, locally-run thinking framework. For developers practicing spec-driven development who also want to optimize model costs and resource allocation, this is a lightweight tool worth trying. In cost-sensitive AI deployment scenarios, tools like this that "help you choose right rather than choose expensive" may be more practical than yet another new model. As the open-source model ecosystem continues to flourish and local inference capabilities keep improving, "matching the right model to each task" is evolving from an idealistic pursuit into a viable engineering practice—and SpecJudge represents an emblematic early example of this trend.