Laya: An Open-Source Decision Model with 421M Parameters That Surpasses the Jev Benchmark Using RLCD

The Birth of an Open-Source Decision Model

With sustained interest from the reinforcement learning community, developer NandhaKishorM has released an open-source decision model called Laya. It represents an evolution of the earlier Jev architecture — according to the developer's Reddit post, the community provided tremendous support, with dozens of users asking him to build a general-purpose model, run comprehensive benchmarks, and set up a publicly accessible Hugging Face Space. These requests ultimately brought Laya to life.

Unlike today's large language models that routinely boast billions or even hundreds of billions of parameters, Laya takes a fundamentally different approach. With just 421 million parameters, it claims to surpass the original Jev architecture across all benchmark tests. This "small but specialized" philosophy is particularly worth discussing at a time when decision-making tasks are receiving increasing attention.

Non-Autoregressive Architecture: Built for Decision Tasks

Laya's most defining technical characteristic is that it is a non-autoregressive decision model — a fundamental departure from mainstream autoregressive generative models.

Specifically, it pairs a bidirectional ModernBERT-large encoder with a Transformer head trained from scratch. This head is dedicated to scoring [MASK] option tokens, resolving typed schemas in a single forward pass of approximately 35 milliseconds.

The significance of this design is clear: for tasks like intent routing, content moderation, and fact-checking — where the goal is to "make a decision" rather than "write text" — token-by-token autoregressive generation is overkill. Laya outputs a decision result directly in a single forward pass, and the 35ms latency is well-suited for real-time routing and moderation in production environments. Choosing ModernBERT-large as the encoder backbone also reflects the author's emphasis on bidirectional contextual understanding — decision tasks typically require considering both preceding and following context to make accurate judgments.

Autoregressive models (like the GPT series) generate text serially, one token at a time, with each step depending on the previous result. This approach is naturally suited for "writing" tasks, but for decision tasks that only need to output a classification label or probability distribution, it introduces unnecessary computational overhead and latency. Non-autoregressive models break this sequential dependency and can compute all output positions in parallel during a single forward pass, dramatically reducing inference latency. BERT-series models are classic non-autoregressive bidirectional encoders — their [MASK] filling mechanism is itself non-autoregressive. Laya draws on this paradigm, applying the [MASK] token scoring mechanism directly to option selection rather than streaming text generation. ModernBERT, released in late 2024, is a modernized improvement on the BERT architecture that retains the advantages of bidirectional attention while incorporating Rotary Position Embeddings (RoPE), alternating local/global attention, and other modern design choices. Compared to the original BERT, it offers significant improvements in long-text understanding and inference efficiency, making it one of the leading lightweight encoder backbones today.

25,000+ Human-Annotated Examples: No Synthetic Data Shortcuts

On the data side, the author emphasizes a practice that is relatively rare in today's AI training landscape: 100% human annotation.

Laya's training corpus contains over 25,000 real-world samples spanning multiple decision scenarios:

• Intent routing
• Fact-checking
• Moderation consensus
• Prompt guardrails
• Rubric scoring
• Multi-turn conversation trajectories

The author explicitly states there were "no synthetic data shortcuts." In an era when synthetic data is widespread, committing to full human annotation is a double-edged sword: on one hand, it ensures data quality and authenticity while avoiding the distributional shift that synthetic data can introduce; on the other hand, 25,000 samples is a relatively limited dataset, which is one reason the author himself acknowledges the model "still needs more refinement."

RLCD: Using Reinforcement Learning to Pursue Calibrated Probabilities

Another technical highlight of Laya is the author's RLCD method (which he specifically notes is an "unofficial" name).

This is a policy-gradient-based reinforcement learning approach for optimizing decision models against strictly proper scoring rules. The core idea is: the model only receives maximum reward when it outputs genuinely and mathematically calibrated probabilities.

This has significant practical value. Many classification and decision models achieve decent accuracy, but their confidence probability outputs tend to be "overconfident" or "miscalibrated." In high-stakes scenarios like content moderation and guardrails, a model that produces trustworthy probabilities is far more useful than one that only outputs hard labels — you can set thresholds based on those probabilities and decide whether human review is needed. By directly optimizing for calibration through reinforcement learning, Laya attempts to address this long-standing pain point at the training objective level.

Strictly proper scoring rules are a mathematical concept from the field of probabilistic prediction. Their core property is that expected score is maximized if and only if the forecaster reports the probability they truly believe. Common examples include log loss and the Brier score. "Strictly proper" means any attempt to "game" the score by distorting probabilities will lead to a lower expected score, which theoretically forces the model to output honest, calibrated probabilities. Policy-gradient methods are a class of reinforcement learning approaches that directly update policy parameters using gradient estimates of the reward signal, without needing to build a value function as in Q-learning. They are well-suited for discrete decision spaces and non-differentiable rewards. RLCD combines this mechanism with strictly proper scoring rules, meaning the model's reward signal is itself a mathematical function that penalizes probability miscalibration — transforming calibration from a post-processing problem into a training objective. This is its core innovation over traditional cross-entropy training.

Engineering a Single-GPU Training Run

Also worth noting is Laya's training cost. The entire model was trained on a single RTX 6000 Pro (96GB VRAM).

For a 421M-parameter model pursuing specialized decision-making capabilities, this hardware requirement is relatively accessible, and it validates the feasibility of the "small model + high-quality data + targeted training objective" approach under resource-constrained conditions. By contrast, training general-purpose large models typically requires clusters of hundreds or thousands of GPUs — the two approaches have entirely different scopes and use cases.

Try It Yourself

The author has fully open-sourced Laya and provided multiple entry points for the community to evaluate and use it:

• Hugging Face Space demo: Test the model directly in your browser
• GitHub repository: Full source code
• Hugging Face model repository: Download model weights

This approach of releasing the model, code, benchmarks, and demo all at once reflects the author's commitment to community feedback, and allows interested developers to quickly verify the claim of "surpassing the Jev benchmark."

Summary

Laya represents a philosophy distinct from mainstream large models: rather than pursuing general generative capability, it focuses squarely on decision tasks themselves — trading autoregressive architecture for low latency, using human-annotated data to ensure quality, and applying reinforcement learning to optimize probability calibration. For teams deploying intent routing, content moderation, or guardrail systems in production, this kind of "small but specialized" open-source decision model deserves serious consideration in technology selection.

That said, the author himself acknowledges the model "still needs more refinement," and both the 25,000-sample dataset size and the unofficial RLCD method leave room for further validation and iteration. But as a project built by an individual developer on a single GPU and fully open-sourced, Laya's technical choices alone are enough to spark meaningful reflection.