Jev in Practice: A Decision Model That Replaces Token-by-Token Decoding with a Single Prefill

When large language models make structured decisions, they typically have to decode JSON token by token—latency is unpredictable and the output format has to be forced through prompting. An approach called Jev does the opposite: it skips the entire decode phase, runs only a single prefill forward pass, and directly outputs bounded, typed decisions—choices, ordinal scores, and boolean judgments. This article, based on a complete tutorial from a YouTube tech channel, breaks down Jev's inference paradigm, its two implementation paths (encoder and decoder), and the RLCD training workflow.

What Problem Does Jev Solve

Standard LLM inference has two stages: prefill consumes the entire prompt, is compute-intensive, and prepares to generate the first token; decode then emits subsequent tokens one by one, feeding the already-generated sequence back into the model at each step. The tutorial uses a credit card fraud detection case to illustrate the pain point: determining whether a transaction is fraudulent, its priority level, and whether it needs escalation—these are judgments among three finite options, yet the model must generate an entire JSON string token by token.

Jev's core claim is just two points: one forward pass, no decode; and predictable output. You declare in advance what you want—options, ordinal scores, or booleans—and the model returns exactly that, instead of making you debug uncertainty through syntax and phrasing.

The tutorial divides Jev's interface into three categories, corresponding to three probability distribution shapes:

• choice (nominal): mutually exclusive options such as east/west/south/north, with no ordering relationship
• score (ordinal): options with order, such as calm/frustrated/very angry, which can be ranked by level of dissatisfaction
• null (binary): yes/no questions, returning the corresponding probability

The APIs differ for the three, but the tutorial chooses to first build a unified softmax skeleton and then distinguish their nuances on top of it.

Prefill and decode have completely different compute characteristics: prefill processes the entire prompt in parallel at once, is matrix-multiplication intensive, and achieves high GPU utilization; decode is serial token by token, processing only one token per step, is limited by memory bandwidth, and often leaves the GPU underutilized. Jev only does prefill, effectively pulling inference from a "memory-bandwidth-limited serial loop" back to a "compute-intensive parallel forward pass"—this is the underlying reason its latency is controllable. However, this also means output length is strictly limited to the predefined option set and cannot generate long text.

Prerequisite Knowledge: This Is Not an Introductory Course

The author sets a hard barrier: you need to understand the attention mechanism, KV cache principles, logits and softmax, and be proficient with PyTorch. Missing any one of these will make the course difficult to follow. Hugging Face basics, the VS Code debugger, and modern transformer encoder/decoder structures are bonuses.

Counterintuitively, no GPU is required—the course is deliberately made inference-intensive, and the implementation part only runs inference. No training experience is needed either, and the math only requires understanding softmax. This trade-off allows Jev's engineering ideas to be validated at low cost by more developers.

KV cache is a key optimization in autoregressive generation: during decoding, each new token generated requires recomputing the Key/Value vectors of all previous tokens, with cost growing quadratically with sequence length. KV cache stores these intermediate results and reuses them directly in subsequent steps, avoiding redundant computation. Jev's decoder path leverages exactly this—the context is prefilled only once, and afterward each question only computes the KV of the newly added tokens, allowing a large number of questions to be mounted on a single cache. Understanding KV cache capacity and lifecycle is a prerequisite for judging whether Jev suits your scenario.

Baseline: Where Exactly Vanilla Decoding Is Slow

The first phase of the tutorial runs traditional decoding to establish a control group. It uses a small model on the order of Qwen 3 6B, which is enough to fit in most laptops. The prompt asks the model to output JSON for a suspicious credit card transaction, and the schema is designed with exactly three fields corresponding to Jev's three interfaces: priority (high/low, ordinal), decision (fraud/fraud review/clear, choice), and escalate (boolean).

In implementation, the input is about 130 tokens, and the model returns predictions for 130 positions, each a probability distribution over the full vocabulary. The conventional approach discards all predictions except the last position, takes the argmax to get the next token, updates the KV cache, and loops. The tutorial also demonstrates a best practice: if the same context is to answer multiple downstream questions, you can cache the context state first, and afterward each question only needs to be appended to the cache.

After running this baseline, the author's assessment is blunt: from the first quotation mark to priority and then to generating the full JSON token by token, the time could have been entirely avoided—as long as the question is framed convergently enough that the model doesn't need to enter this vicious decoding loop.

Path One: Encoder Implementation, Scoring Options with Mask Tokens

The first alternative uses a BERT-like encoder backbone without any training, to see whether it can be adapted directly.

There's a detail in token ordering: question first, context after. The author explains that attention typically assigns higher weight to the beginning of the sequence, and experiments show this order performs better. A special mask token is inserted before each option, and its hidden state is used to measure the option's propensity.

The encoder has bidirectional attention, which brings a structural limitation: every time the question changes, the attention weights at all positions change entirely, and even placing the context first doesn't allow caching and reuse. The more questions and the longer the context, the heavier the redundant computation. What you get in return is that every model parameter participates in bidirectional understanding of the context.

At the implementation level, the build function concatenates the class token, question, separator, mask+option, another separator, and context in order. During inference, it takes the last hidden state, extracts tokens from the question span and applies mean pooling to get the query vector, then extracts the vectors at each option's mask position, takes their dot product and applies softmax to get the probability of each option.

The zero-training results are not accurate—a question that should be "yes" only gives about 30%. But the tutorial emphasizes that the 768-dimensional embeddings themselves are high quality; what's missing is only the layer that makes "output conform to probability semantics." It therefore offers three progressive paths: train only a linear head to map 768 dimensions to a scalar; do LoRA fine-tuning; or fully retrain the backbone. Which to choose depends on task complexity and available data.

Path Two: Decoder Implementation, Prefill Once and Reuse the Cache

The key difference in the decoder path is that the context can be placed first and cached. Under causal masking, the context only needs to be prefilled once to obtain the KV cache, and each subsequent question reuses it directly, computing only the newly added question tokens. The tutorial mentions that Jev officially supports mounting up to 5000 questions on a single state, so it speculates that its underlying implementation may be the decoder path—but the official implementation has not been disclosed, and the author reminds readers not to jump to conclusions.

The specific approach organizes the question as a completion task: context + question + colon, and after the model emits the first token, only the probabilities of the option-corresponding tokens are looked up in that token's probability distribution, ignoring everything else. Two training-free variants: one rewrites the options as A/B/C/D letters for the model to choose; the other adds a linear pointer head to turn probabilities into an interpretable distribution. The tutorial also mentions two open-source references—Kev (an open-source alternative to Jev) and Cloudflare's Clef (which uses pooled span to cover the full option and then adds a linear head).

The Pitfall of Duplicate First Tokens

If two options share the same first token, such as fraud and fraud review, the simple implementation fails outright—the model can only give 50/50. The author's engineering handling is to throw an error upon detecting a duplicate first token and prompt the user to rewrite the option (e.g., change fraud review to review). Clef's pooled span approach avoids this problem at the root, because it pools over the entire option and can distinguish even completely duplicate options.

Mask Design for Packing Multiple Questions

When stuffing multiple questions into a single forward pass, the causal mask needs modification: each question can see the full context, but questions must be isolated from each other. If question two can see the content of question one, the order of asking affects the answer, and reproducibility is broken. The author specifically points out that Cloudflare's Clef does not do this isolation, which in his view causes output drift when the order changes.

In implementation, segments are used to mark which question each token belongs to, a same-question mask is constructed and logically ANDed with the standard causal mask, and then the context portion is set entirely visible to obtain the final attention mask. Taking 73 tokens (38 context + 35 question tokens) as an example, after slicing you can see a clear block structure: the lower-left corner is all false (question two cannot see question one), and causality is preserved within each question block.

An unexpected benefit of the decoder path: it works out of the box. In the tutorial, the first question "Is the customer requesting a refund" gives 95% confidence, and the second question "Which team is responsible" gives 97% attribution to billing, without any fine-tuning. After the context is cached, new questions can be continuously appended and options changed, while the state remains unchanged.

Causal mask is a fundamental mechanism of decoder models: each position can only see itself and preceding tokens, ensuring generation does not "peek" into the future. Jev modifies this mask when packing multiple questions—the context is visible to every question, but questions are masked from each other. In implementation, a block-diagonal matrix is constructed: question i can only attend to the context and its own tokens, not to other questions. This way, even if dozens of questions are stuffed into a single forward pass, each question's answer is exactly the same as asking it alone—order-independent and reproducible.

RLCD Training: Using GRPO to Calibrate Probabilities to the Teacher Distribution

The training section only covers principles without running code; the core is RLCD (Reinforcement Learning with Calibrated Decisions). The starting point is teacher-student distillation: the teacher gives a probability distribution (e.g., 0.7/0.2/0.1), and the student's raw scores after softmax preserve the ordering but deviate in distribution; the goal is to bring the student distribution close to the teacher's.

Since Jev constrains output to a limited set of tokens, there are no natural multiple trajectories, so it uses the GRPO idea to artificially create trajectories: add random perturbations to the raw scores to generate several candidate distributions. Scoring consists of three parts—log score measures whether probability mass falls on the teacher's high-probability positions, spherical score measures whether the overall direction is consistent, and both are weighted by teacher probabilities. The ordinal score interface additionally adds RPS (rank probability score), which captures "how badly wrong" an ordinal prediction is: when the true answer is "very angry," predicting "frustrated" should score higher than predicting "calm," whereas ordinary log score gives the same score to both errors.

The trajectory score is subtracted by the mean and divided by the standard deviation to get the advantage; trajectories with positive advantage receive lower loss. The final loss is the full cross-entropy plus the RL loss (spherical+log, plus RPS for the score interface), and backpropagation can train only the linear head, train LoRA, or fully fine-tune.

GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm proposed by DeepSeek^Ad. Unlike PPO, it does not rely on an independent value network; instead, it samples a group of trajectories for the same prompt and uses the group average score as the baseline to compute advantage. Jev borrows this idea: since output is constrained to a limited set of tokens and there are no natural multiple trajectories, the author adds random perturbations to the raw scores to artificially create candidate distributions, then assigns advantage based on relative performance within the group. This completes policy optimization without a value function, making training lighter and fitting Jev's inference-intensive design orientation.

How to Choose Between Encoder and Decoder

The tutorial ends with comparison dimensions:

• Context reuse: the encoder rereads for every question, the decoder reads once and caches; choose the decoder when there are many questions and limited context
• Cost growth with questions: the encoder grows linearly, the decoder stays roughly flat (only question tokens are new)
• Long documents: the decoder is more suitable
• Bidirectional interaction between question and context: bidirectional attention gives the encoder a slight edge
• Multi-token options: the encoder's mask-prepend design naturally fits, the decoder is weaker
• Multi-label simultaneous output: the encoder can switch to a sigmoid head, the decoder is limited by softmax and requires architecture changes
• Parameter efficiency: for the same task, the encoder can usually use a smaller model, worth weighing in memory- and speed-sensitive scenarios

In engineering practice, the encoder writes one sequence per question, while the decoder must handle the combination of masks, positions, and cache. The tutorial's stance is clear: master the first principles first, and the differences and tuning of various implementations will naturally become understandable.

For software logic judgment scenarios that require low latency and enumerable outputs—risk control grading, ticket routing, customer service sentiment scoring—Jev's "prefill only, no decode" paradigm offers an engineering option worth serious evaluation.

One-line Summary

Jev skips token-by-token decoding, using a single prefill to directly output three types of bounded decisions—choice/score/null—with controllable latency and predictable output.

Paragraph Summary

Jev proposes an unconventional LLM inference paradigm: skip the entire decode phase, run only a single prefill forward pass, and directly output three types of bounded decisions—choice, score, and null—for structured judgment scenarios such as risk control grading and ticket routing. The article compares the encoder and decoder implementation paths: the encoder scores options with mask tokens, and under bidirectional attention the context cannot be cached; the decoder prefills and caches the context once, supports mounting many questions on a single state, but must handle duplicate first tokens and multi-question mask isolation. For training, it uses RLCD, borrowing GRPO ideas to create trajectories and scoring with a combination of log score, spherical score, and RPS for ordinal scenarios. Selection depends on context reuse, number of questions, and output type.