jevals: A New Approach to LLM Evaluation Using Typed Jev Decisions Instead of LLM Judges

Introduction: Why LLM Evaluation Needs a New Paradigm

Evaluating the quality of large language model (LLM) outputs has always been a thorny challenge in AI application development. The dominant approach today is "LLM as Judge" — using one large model to evaluate the output of another. While flexible, this method comes with real pain points: vague evaluation criteria, poor reproducibility, and high costs.

A recently surfaced open-source project on Hacker News, jevals, proposes a fresh alternative: replacing traditional LLM judges with typed Jev decisions. The goal is to bring stronger structure and determinism to LLM evaluation.

In practice, the "LLM as Judge" pattern works like this: during testing, the output of the model under evaluation — along with a scoring prompt — is sent to a separate "judge" model (such as GPT-4 or Claude), which returns a score or a pass/fail verdict. The upside is that it can handle open-ended semantic questions without requiring manual annotation. But the downsides are equally significant: the judge model's behavior is influenced by temperature, prompt wording, and model version updates, meaning the same test cases run at different times can yield different conclusions. This fails to meet the basic engineering requirement of determinism in regression testing. On top of that, every evaluation requires additional LLM API calls, with costs scaling linearly as test suites grow. These problems have driven the industry to explore more structured evaluation approaches.

Core Concept: From Fuzzy Judgments to Typed Decisions

The biggest problem with traditional LLM judges is output non-determinism. When you ask a large model to score or judge something, the result often lacks strict structural constraints — the same input can yield different verdicts at different times. For engineering teams that depend on stable regression testing, this is a serious obstacle.

jevals' central proposition is the introduction of typed decisions. "Typed" here means that evaluation outputs are no longer free-form text or arbitrary scores, but are constrained within a predefined, verifiable type system. This brings several immediate benefits:

• Predictability: Results fall within a well-defined type space, making programmatic processing straightforward
• Reproducibility: Structured decision logic reduces random variation
• Composability: Typed decision units are easier to assemble and reuse in complex evaluation pipelines

The Significance of the Jev Decision Unit

The "Jev" in the project name represents its unique decision abstraction unit. Rather than handing an entire evaluation task to a black-box LLM, the idea is to decompose evaluation into a series of judgment steps with explicit type signatures. This approach draws on the reliability benefits that strong typing brings to software engineering, attempting to migrate the concept of type safety into the AI evaluation domain.

The value of strong typing for software reliability can be understood by analogy: in statically typed languages like TypeScript or Rust, a function's inputs and outputs are declared as specific types upfront, allowing the compiler to catch many errors before runtime rather than during execution. Applying this thinking to AI evaluation means that the "verdict" of each evaluation step is no longer a chunk of natural language with unpredictable formatting — it's an enum value, a boolean, or a struct. Code can directly assert, aggregate, and compare these results just like any ordinary data type. This makes the evaluation logic itself testable and version-controllable, fundamentally changing how AI evaluation integrates with engineering infrastructure.

Comparison with Mainstream Evaluation Approaches

In the current LLM evaluation tool ecosystem, mainstream solutions like LangChain's evaluation module and OpenAI Evals still largely rely on LLMs or human annotation to make quality judgments. jevals takes a differentiated path:

Dimension	Traditional LLM Judge	jevals Typed Decisions
Output format	Free text / scores	Type-constrained structured results
Reproducibility	Weak	Strong
Runtime cost	Requires additional LLM calls	Can reduce model dependency
Debuggability	Black box	Clear structure, easy to trace

This design is particularly appealing to teams looking to incorporate LLM applications into CI/CD pipelines for automated regression testing.

CI/CD (Continuous Integration/Continuous Delivery) pipelines are the standard practice in modern software engineering for automating build, test, and deployment workflows. Integrating LLM applications into CI/CD means that every code or prompt change automatically triggers an evaluation run, blocking releases if key metrics regress — exactly the same purpose as unit tests and integration tests in traditional software. However, LLM Judge-based evaluation approaches face a natural barrier in this context: non-deterministic verdicts cause pipelines to pass sometimes and fail other times, undermining the value of automation. Add in the cost of model API calls on every CI run, and expenses accumulate quickly for high-velocity teams. A typed decision approach that can run evaluation logic locally and deterministically would dramatically lower this integration barrier.

Notable Concerns and Limitations

As an early-stage project that just appeared on Hacker News (only 7 upvotes at launch, with no discussion comments), jevals has limited publicly available information. Several key questions remain to be answered:

The expressiveness trade-off. Typed decisions improve determinism, but may sacrifice some flexibility. For evaluation scenarios requiring open-ended, semantic-level judgment — such as "does this response show empathy?" — whether a purely typed approach can handle the task requires more real-world validation.

The design barrier of the type system. For typed decisions to be genuinely useful, the design of the type system itself is critical. Too complex, and it increases the learning curve for users; too simple, and it fails to cover real evaluation needs.

Insufficient community validation. The project currently has low visibility and lacks third-party real-world feedback. Its actual effectiveness and stability will need to be tested over time.

Conclusion and Outlook

jevals represents a noteworthy direction of exploration in LLM evaluation — applying the mature concept of type safety from software engineering to combat the inherent uncertainty of AI evaluation. As LLM evaluation increasingly becomes a production necessity, making evaluations reproducible, debuggable, and automatable is a shared challenge across the industry.

For developers struggling with LLM application quality evaluation, jevals offers a perspective that differs from the "use a model to judge a model" paradigm. That said, given the project's early stage, it's advisable for interested teams to keep an eye on its development, pilot it in limited scenarios, and validate the results before committing to deeper adoption.