LLM vs. Classical ML: Testing Jev Against 11 Models on 8 Classification Datasets

A Comparison Worth Paying Attention To

Large language models (LLMs) are generating enormous hype — some even claim they can replace traditional machine learning pipelines entirely. But in real classification tasks, can LLMs actually beat carefully trained classical models? A developer shared a rigorous comparative experiment on Reddit, offering a data-driven and refreshingly sober answer.

The LLM under test is called Jev. The researcher evaluated it under both zero-shot and few-shot settings against 11 classical machine learning models — including SVM, XGBoost, logistic regression, and other widely used algorithms. Performance was measured using balanced accuracy across 8 classification datasets. The full code is open-sourced, and anyone can reproduce the results or suggest improvements.

The Results Are Unsurprising: Classical Models Still Win When Labels Are Available

The findings align with the intuition of many experienced practitioners: if you already have labeled data, training a classical model is still the more sensible choice.

On intent classification datasets like Banking77, and across several commercial tabular datasets, classical models achieved notably higher accuracy. The researcher also highlighted a practical cost consideration — trained classical models can run locally, completely eliminating the per-call API fees associated with LLM inference. In production environments requiring high-frequency, large-scale prediction, this cost difference is often decisive.

In short, for structured tasks with sufficient labeled data, mature tools like XGBoost remain an extremely difficult baseline to beat.

Banking77 is a widely used benchmark for conversational intent classification, containing 77 banking-related intent categories with roughly balanced class distributions and approximately 13,000 user utterances in total. Its fine-grained, semantically similar categories make it a rigorous test of a model's discriminative ability. Tabular datasets refer to structured data organized in rows and columns, typically mixing numerical and categorical features — the traditional stronghold of tree-based models like XGBoost and Random Forest, which excel at capturing nonlinear feature interactions without relying on large pre-trained knowledge bases. Balanced accuracy is a corrected metric for multi-class imbalance problems, computed as the arithmetic mean of per-class recall rates. It prevents inflated accuracy scores caused by a model simply favoring the majority class, making it more meaningful for real-world business data with uneven class distributions.

Where Jev Shines — and Where It Falls Short

The experiment wasn't a complete shutout. Jev performed well on the IMDb sentiment classification dataset, demonstrating that LLMs do have genuine strengths in certain natural language understanding tasks. This reinforces a pragmatic view: model selection is highly task-dependent, and there is no universally optimal solution.

However, one counterintuitive finding stands out: providing Jev with a small number of examples (few-shot prompting) did not consistently improve performance. This runs counter to the common expectation that "more context and examples lead to better results," suggesting that the gains from few-shot prompt engineering may be far less reliable than commonly assumed.

The researcher's overall assessment was notably restrained: based on these results, he sees no compelling reason to replace a machine learning model that is already working well.

Zero-shot means providing no task examples at all, relying entirely on the model's pre-trained knowledge to make predictions. Few-shot involves including a small number of labeled examples in the prompt to guide the model by analogy. In theory, examples help the model align with the task format and label definitions — especially when label semantics are ambiguous. In practice, however, few-shot gains depend heavily on example quality, selection strategy (random sampling vs. similarity-based retrieval), and the model's ability to effectively utilize its context window. Poorly chosen examples can even introduce bias and degrade performance. This is why few-shot prompting failed to consistently outperform zero-shot in this experiment — it is not a plug-and-play improvement, but an engineering effort that requires careful, task-specific design.

Credit Where It's Due: Methodological Transparency

What makes this study particularly valuable is the author's candid disclosure of its limitations — a refreshing contrast to the overhyped AI benchmarks that typically dominate the conversation.

On the methodological side, the experiment used three training random seeds with a single fixed test split and a limited hyperparameter tuning budget. The author explicitly noted that zero-shot predictions were cached and reused across random seeds, meaning this portion does not constitute truly independent repeated trials. He also acknowledged that the experimental setup "still has plenty of room for improvement" and invited community feedback.

This kind of self-critical transparency actually makes the conclusions more credible — the goal isn't to "prove" a predetermined position, but to provide a reproducible, discussable body of evidence.

Key Takeaways for Practitioners

From this set of experiments, a few actionable lessons emerge:

• Assess your data first: For classification tasks with labeled, structured data, classical models are typically the more efficient and cost-effective starting point.
• Do the cost math: Locally deployed trained models carry no API call fees — a significant advantage at scale.
• Let the task guide model selection: Jev's strong IMDb performance shows that LLMs still have a place in specific text understanding tasks.
• Don't overestimate few-shot prompting: Feeding examples to an LLM doesn't reliably improve results; always validate empirically on your specific task.

For teams focused on deploying AI in production, small-scale but methodologically transparent comparison experiments like this one are often more practically useful than vendor-supplied benchmarks. Interested readers can access the author's publicly available results page and open-source code to reproduce and verify the findings independently.