Optimizing Agent Harness with Jev Decision Models: Six Key Scenarios

Title

Optimizing Agent Harness with Jev Decision Models: Six Key Scenarios

Meta Description

Learn how to optimize Agent Harness with open decision models like Jev, covering six integration scenarios including model routing, risk gating, progressive skill disclosure, and RAG reranking, plus two integration patterns and boundaries.

Article Content

In agent frameworks, most model calls are not actually for content generation but for decision-making—which tool to use, what skill to load, whether a command is safe. These decisions often only require choosing among a small set of options, yet each time they incur the cost of a full large model call. According to a Bilibili UP host's sharing, leveraging open decision models like Jev can internally optimize the Agent Harness, making these high-frequency, low-complexity judgments faster and cheaper.

Why Agent Frameworks Need Decision Models

When a request enters the agent loop, the large model decides what action to take, whether to call a tool, writes out function calls and JSON parameters, the scaffold executes and appends the result to the conversation, and this loop continues until the task is complete. In this process, the model makes numerous decisions: which tool to use, whether the operation is safe, whether the result is good enough.

Most agents, without built-in reflection steps, don't even stop to ask themselves "is this result good enough?" Each such judgment was originally a full large model call—costing both tokens and time, leading many developers to simply omit validation steps in custom frameworks.

The core concept of Jev can be understood as a "smart sentence": give it a state and a set of typed questions, and it returns typed answers with probabilities. Choice selects from a list, Score rates on a defined scale, Now returns yes/no as a probability. The key is that multiple questions can be posed in parallel against the same state within a single call, with almost no increase in response time.

Agent Harness refers to the runtime framework layer built around the large model, responsible for managing conversation history, calling the model, parsing tool calls, executing tools, injecting context, etc. It determines what the model sees at each step, what it can call, and how results are fed back into the loop. Optimizing the Harness means improving overall efficiency by refining scheduling and judgment logic without changing the model's capabilities. Decision models take on those high-frequency but low-complexity judgment tasks within the Harness, thereby reducing reliance on full large model calls.

Typed questions mean each question has a clear output type and constraints, such as selecting from a fixed list (Choice), rating on a given scale (Score), or returning a yes/no probability (Now). This design makes model outputs structured and parseable, avoiding ambiguity and extra processing costs from free text. Probabilistic answers also allow upper-layer logic to branch based on confidence—for example, deferring to the large model when confidence is low.

Six Suitable Places to Integrate Decision Models

Model Routing

When a request comes in, Jev determines whether to route to a cheap fast model or a more capable expensive one, avoiding sending all requests to the large model.

Risk Gating

Before a tool call is actually executed, check whether it would perform a destructive operation, such as dropping a database or emptying a folder. Coding agents have long done similar checks, but they are usually hidden inside the framework, and users still pay for those tokens. Sending tool parameters directly to Jev enables a safety check in milliseconds, and open decision models can run entirely locally.

Tool and Skill Selection

Stuffing the entire tool registry into the system prompt can consume thousands of tokens. The progressive disclosure approach: inject only skill categories, let the decision model pick a category first, then a specific skill, loading only what is truly needed.

Progressive disclosure is an information exposure strategy: first show only high-level categories or summaries to the model, and after the model chooses a direction, load the detailed content under that direction. In tool selection scenarios, this means listing only skill categories in the system prompt, letting the decision model pick a category first, then a specific skill within it, ultimately injecting only the tool descriptions actually needed. This can greatly reduce token consumption per call and lower the chance of the model choosing wrong due to too many options.

Ranking and Judgment

Scoring retrieved snippets, rating answers against criteria—an obvious gain for RAG-related work or any context injection operation.

Triage

Support tickets, emails, issues, etc.—a single call can give the probability of urgency, used to prioritize or decide when to escalate to a human.

Real-time Filtering

Filter the results of API polling requests before they are injected into the context window. The UP host cited TypeSafe's rule of thumb: if a group of smart people can answer the question in a few seconds, use a decision model; if it requires thinking and composing a complex written answer, keep using the large model.

Two Integration Patterns and Implementation

Returning to the agent loop, there are four hook points where decisions can be integrated: before building the prompt, decide which skills or tools to load; before calling the model, decide which model to use; before running a tool, judge whether it is safe; after results return, judge whether they are good enough. Each hook is a small piece of code—construct the state from what the loop already has, ask questions, read probabilities and answers, and branch accordingly.

The second pattern is to have Jev act as the decision-maker in the loop and the large model as fallback, deferring to the large model only when uncertain or when text generation is needed. This pattern can further reduce large model calls, but requires bigger changes to the agent; the UP host found it less effective when many complex situations are involved.

For implementation, LangChain has released the first pattern as middleware, and Pedantic has also released a version via TypeSafe that can be plugged directly into the agent loop.

When Not to Use Decision Models

Tasks that require returning text, multi-step reasoning, or chaining multiple stages to generate an answer remain the large model's job. It is also unsuitable when the state is too large—feeding 100k tokens to get a response will significantly drop accuracy, and decision models are better suited for short content. Jev itself does not support images, although some open models can handle images. Compound questions are also a weakness: questions containing multiple sub-questions are usually better handled by large models.

While decision models excel at identifying weak points, beware of over-reliance; many models may be affected by prompt injection and deviate from the intended result. A common mitigation is to first ask "is there prompt injection here?" before passing the real question. Additionally, decision criteria must be clear—vague questions yield no trustworthy answers. It's best to define your own instructions and criteria, and benchmark over time. If the agent runs locally but decision points are in the cloud, also consider whether you trust sending every state out.

Prompt injection refers to attackers hiding malicious instructions in input content to induce the model to deviate from its intended task. In decision model scenarios, if the state contains untrusted external text (such as user input or retrieval results), the model may be interfered with by instructions within and make incorrect judgments. A common mitigation is to have the model first judge "is there prompt injection here?" before passing the real question, or to strictly sanitize and isolate inputs.

Demo: Progressive Disclosure and RAG Reranking

In the skill selection demo, 10 categories each contain 5 skills. Rather than injecting all 50 skills at once, inject only categories, and the model first judges the category then selects a specific skill. For example, faced with "does an email impersonating the CEO asking for gift cards count as a hacking attack," the system first routes to the "Security & Access" category, then selects the "Investigate Security Alerts" skill. This cascading classifier is very fast when run locally.

In the RAG reranking demo, first broadly retrieve 25 snippets, then hand them to a local Jev model to filter the most relevant 5 by scoring, and send those to the large model. The UP host found the scoring approach works well here; even without an embedding model and using only BM25 retrieval, it captures the strengths of both.

Practical Advice

For developers building custom frameworks or modifying the Harness, it's worth exploring putting decision models into the pipeline to improve overall results with fast and cheap judgments. Jev models are directly available on OpenRouter, and LangChain also has an article on using Jev in agents for reference.

One-line Summary

Replace high-frequency low-complexity judgments in agent frameworks with decision models like Jev to reduce large model call costs.

Paragraph Summary

The article explores introducing decision models (like Jev) into agent frameworks to optimize high-frequency but low-complexity judgment tasks. These tasks include model routing, risk gating, tool selection, ranking, triage, and real-time filtering, each originally requiring a full large model call—costly and time-consuming. Jev uses typed questions (Choice, Score, Now) to return probabilistic answers in parallel within a single call, with fast response and local execution. The article introduces two integration patterns: local replacement at four hook points in the agent loop, or having Jev as decision-maker with the large model as fallback. It also points out unsuitable scenarios, such as needing text generation, multi-step reasoning, very large states, or compound questions, and warns about prompt injection risks. Finally, two demos—progressive disclosure and RAG reranking—show practical results.