llm-typesafe: Integrating TypeSafe AI's Jev Judgment Model into the LLM CLI Tool

Simon Willison has released a new LLM plugin called llm-typesafe (current version 0.1a0), which connects TypeSafe AI's new Jev model to his widely used command-line tool LLM. The plugin has a clear purpose: rather than providing yet another interface to a general-purpose conversational model, it's purpose-built for structured judgment tasks — treating the LLM as a classifier and scorer that produces deterministic, well-typed outputs.

What Problem Does the Jev Model Solve?

In real engineering workflows, developers often don't need a model to "chat" — they need it to make a clear judgment about a piece of text: Is this a refund request? Which team should handle this ticket? How reproducible is this bug report? The traditional approach involves writing prompts for a general-purpose model and hoping it returns properly formatted JSON, but output stability and type safety have always been pain points.

The core value of the Jev model (and the llm-typesafe plugin) lies in standardizing these types of tasks. It reduces judgment problems to a set of fixed answer types, giving the model's output an inherently parseable and reliable structure — which is exactly what the "typesafe" in the plugin name refers to.

This pain point is commonly known in the industry as the "Structured Output" problem. Major providers like OpenAI and Google have introduced mechanisms like JSON Mode and Function Calling to constrain output format, but these solutions still rely on general-purpose models applying format constraints during generation. For judgment tasks with a fixed set of classification labels and a limited answer space, these approaches suffer from high latency, high token consumption, and occasional formatting errors. Jev's differentiating approach is to optimize for judgment tasks at the model training level, rather than patching a general-purpose model with prompts or parsing layers — making its output determinism and latency theoretically superior to the "general model + format constraints" combination.

Installation and Configuration

Installing the plugin follows the streamlined convention of the LLM ecosystem:

llm install llm-typesafe

You'll then need to set an API key. TypeSafe AI provides a console where you can apply for one, and according to the author, the waitlist moves quite quickly:

llm keys set typesafe
# Paste key

Once configured, you can invoke the model using -m jev. The entire process is fully consistent with other LLM model plugins, so users already familiar with the tool face virtually no learning curve.

Three Judgment Types: noul, choice, and score

The most distinctive feature of the Jev model is how it breaks judgment tasks down into three clearly defined answer types.

Yes/No Judgment (noul)

The most basic type is the yes/no "noul" question, where the model returns a numerical value with a confidence score. For example, determining whether a message explicitly requests a refund:

llm -m jev 'Please refund my last payment.' \
  -s 'Does this message explicitly request a refund?'

The output is structured JSON:

{"type": "noul", "noul": 0.99}

The 0.99 here represents the model's confidence in a "yes" judgment — more informative than a simple true/false.

The name "noul" comes from a blend of "no / null / yes," representing a continuous three-state semantic space: close to 0 means a clear negative, close to 0.5 means uncertain or insufficient information (the null semantic), and close to 1 means a clear positive. Compared to a traditional binary boolean, this preserves the model's uncertainty information, enabling downstream systems to apply thresholds — for example, only triggering automated processing when confidence exceeds 0.9, and routing to human review otherwise — allowing flexible trade-offs between automation coverage and accuracy.

Multi-Class Classification (choice)

When you need to choose among multiple categories, you can use the choice type and pass in category definitions via criteria. The author's example is ticket routing — determining which team should handle a message, with an explicit rule to prioritize billing when both billing and technical issues are present:

cat message.txt | llm -m jev \
  -s 'Which team should handle this message? If billing and technical issues both occur, choose billing.' \
  -o answer_type choice \
  -o criteria '{
    "billing":"Charges, invoices, payments, or refunds",
    "technical":"Problems installing or using the product",
    "other":"Neither category fits"
  }'

Explicitly writing category definitions into criteria reduces ambiguity and makes the model's decision criteria transparent and controllable.

Graded Scoring (score)

The third type is score, suitable for measuring a property on a graduated scale. For example, assessing the reproducibility of a bug report using an ordered list of criteria to define each score level:

cat report.txt | llm -m jev \
  -s 'How reproducible is the problem described in this report?' \
  -o answer_type score \
  -o criteria '[
    "No reproduction instructions",
    "Some instructions, but important steps are missing",
    "Complete steps with expected and actual results"
  ]'

This design transforms vague subjective judgments into an aligned, leveled rubric — particularly useful in quality assessment, content moderation, and similar scenarios.

What This Means for Developers

llm-typesafe reflects a growing trend: embedding large language models as composable judgment components within automated pipelines, rather than using them solely as conversational interfaces. Command-line support for standard input (cat file | llm ...) means it can be easily integrated into scripts, batch jobs, and CI pipelines for high-frequency use cases like ticket routing, content filtering, and report quality checks.

It's worth noting that the current version is 0.1a0 — an early alpha stage — and the API and behavior may change. Interested developers can refer to the project's README for more details. For teams already using Simon Willison's LLM toolchain, this is a new structured judgment option worth keeping an eye on.

Simon Willison's LLM is an open-source command-line tool that supports dozens of different model providers via a plugin system (including OpenAI, Anthropic, local Ollama models, and more), providing a unified CLI, conversation history storage (based on SQLite), and a Python API. Its plugin ecosystem is designed to minimize the cost of integrating new models — developers only need to implement a specific interface to reuse the entire toolchain — which is why llm-typesafe can seamlessly support pipe operations and script integration. For teams that have already incorporated the LLM tool into their daily workflows, new model plugins introduce virtually no additional operational overhead.