Open-Source Classification Model Showdown: Qwen-Based Kev Beats Llama-Based Laya

Two Open-Source Classification Models Go Head-to-Head

Following the recent buzz around a proprietary classification tool (referred to as Jev in the video), an overseas blogger set out to find locally deployable open-source alternatives and settled on two candidates: Laya, built on the Llama ecosystem, and Kev, based on Qwen 3.5. The goal of this evaluation wasn't to crown an absolute winner, but to explore where each model shines and how they differ under a consistent set of standards.

To be clear, both Kev and Laya are independent open-source solutions. The evaluation targets these two local models directly — not the proprietary Jev. Both models were run separately on the same machine equipped with a Spark CPU, limited to three threads, with identical inputs to ensure comparability.

Architecture Differences: Encoder vs. Pointer Head

The two models take fundamentally different architectural approaches. Laya's English checkpoint starts from modern BERT large, a bidirectional encoder that builds word representations using context from both sides simultaneously. It then passes through two additional Transformer refinement layers before a scoring network assigns scores to each candidate option.

Kev uses Qwen 3.5's 0.8B base model as its text backbone, with LoRA fine-tuning weights merged in. It reads context in a unidirectional, sequential manner and uses a pointer head to compare the final decision representation against each candidate answer's representation.

In short, the difference isn't just in the backbone — it's in how each model "reads." One encodes bidirectionally; the other reads through to a decision sequentially. Both ultimately convert scores to probabilities and select a label, and neither generates responses token by token during this test. The blogger notes that the architecture diagrams illustrate the processing pipeline, not the actual attention distributions measured inside the models.

modern BERT large is a classic example of an encoder-only architecture. Unlike GPT-series models that generate text left-to-right token by token, BERT uses a masked language modeling objective during pretraining — forcing the model to look at all context on both sides of a masked word to predict it. This makes it naturally strong at understanding the semantics of a full sentence. "Modern BERT" is an engineering-improved version of the original BERT, with updates including replacing absolute positional encodings with RoPE (Rotary Position Embedding) and improved normalization placement. It retains the bidirectional encoding advantage while improving training efficiency and downstream task transfer. This architecture is particularly well-suited for tasks like classification and named entity recognition that require "reading the whole sentence before deciding," rather than generating long text step by step.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Rather than directly modifying the original pretrained model's weights, it inserts a pair of low-rank matrices (typically rank 4 to 64) alongside each target layer, training only those small matrices. At inference time, the product of the low-rank matrices is added back to the original weights — functionally equivalent to full fine-tuning, but with tens to hundreds of times fewer trainable parameters. Kev "merging" the LoRA weights means no additional adapter-loading logic is needed at inference time, and latency is comparable to a standard fully fine-tuned model.

Resource Footprint and Response Speed

Three size metrics are worth examining separately:

• Parameter count: Laya ~421M, Kev ~753M
• Disk footprint: Laya packages to ~846MB; Kev, including base model, adapter, and pointer head, to ~1.82GB
• Peak memory: On CPU, Laya peaks at ~2.86GB, Kev at ~4.94GB

For speed, the blogger timed 90 decisions per model (30 messages × 3 option orderings). Laya's median call latency was 541ms; Kev's was 756ms. The 95th percentile was approximately 607ms and 802ms respectively. Timings cover tokenization, inference, and formatting, but exclude model loading and network calls. Since the software stacks differ and both share the same machine, these are measurements for this specific configuration — not general speed ceilings.

The Test Task: Customer Service Ticket Routing

The evaluation used a concrete scenario: given a customer message, the model must select one label from four destinations — "billing, technical support, sales, needs clarification" — which the application then uses to determine the subsequent workflow. This kind of classification routing is highly practical, but the blogger emphasizes that producing a valid label is only the first step — it can still be the wrong label, and a seemingly confident probability score doesn't fix that.

The test set contains 30 messages: 10 direct requests, 10 cases where policy dictates "request clarification," and 10 edge cases — including negations, already-resolved issues, explicitly stated priorities, and instructions embedded within the message body. Inputs and answers were frozen before scoring. The blogger acknowledges these are synthetic samples that don't represent real customer traffic or industry benchmarks; the goal is to make model behavior — especially on tricky cases — verifiable.

Key Cases: Where Contextual Understanding Makes the Difference

Both models performed similarly on straightforward cases. For example, when a customer reported a duplicate charge on a renewal and requested a refund, both correctly selected "billing" — Laya with ~88% confidence, Kev with ~73%. But the higher number only describes its score on that specific sample; it doesn't mean Laya is more reliable overall.

The real gap emerged in cases requiring contextual understanding. When a customer explicitly said they did not want a refund and only needed a password reset, Kev correctly chose technical support, while Laya chose billing — simply because billing-related keywords appeared in the message. In another message, a customer mentioned a login issue but noted it "was fixed yesterday" and was actually asking to correct an invoice. Kev selected billing; Laya selected technical support.

The blogger also honestly presented shared failures: one message where all exports were generating empty files — the correct answer was technical support — had Kev selecting "clarification" and Laya selecting billing. This honesty avoids a "perfect winner" narrative: a higher total score doesn't make individual errors disappear.

Overall Scores and the Hidden Risk of Option Ordering

With the original option ordering, Kev scored 26/30 and Laya scored 12/30. Breaking it down: on the 10 clear requests, Kev scored 8 and Laya scored 7 — a small gap. On clarification cases: 8 vs. 0. On edge cases: 10 vs. 5. Most of the gap comes from policy-related and context-dependent messages, so this cannot be taken to mean Laya is incapable of handling ordinary routing.

The blogger also ran an important robustness test: shuffling the four options for each message across three different orderings. Results showed that 4 of Kev's messages and 5 of Laya's changed their output depending on option order. Across all three orderings combined, Kev scored 74/90 and Laya scored 33/90. The blogger specifically cautions against treating these 90 trials as 90 independent test questions, as that would overstate the strength of the evidence.

The phenomenon of classification models being sensitive to candidate answer ordering is known as position bias, and has been systematically documented in LLM evaluation research. Unidirectional autoregressive models (like Kev's Qwen backbone) sometimes assign higher weight to candidates appearing later in the sequence during generation or scoring, because the attention mechanism under causal masking can only "look backward." Bidirectional encoders (like Laya's BERT backbone) are theoretically less susceptible to this, but in practice may still be affected by how options are arranged in the input. This finding suggests that before any production deployment, models should be tested repeatedly with multiple option orderings to observe whether labels remain consistent — exactly the robustness practice the blogger applied here by shuffling across three orderings to quantify instability.

Conclusion: Local Inspectability Is the Real Value

For this policy-heavy pilot task, Kev is the stronger starting point. Laya is faster and uses less memory, but under this ruleset needs improvement before it can be trusted. The blogger repeatedly emphasizes that no result can replace testing on your application's real messages.

What makes open-source classification models genuinely interesting isn't that they can return a label — it's that you can run them locally, inspect their behavior, and judge for yourself whether a given decision is useful. These 30 transparent cases taught specific lessons and leave room to improve configurations, expand the dataset, or even overturn the conclusions entirely — and that reproducibility and scrutability is the core value of open-source solutions.