Old-School Classifiers Still Beat Jev: Embeddings + sklearn Cuts Error by 37%

Why Does the Popular Decision Model Jev Lose to an Old-School Classifier?

An engineer from Voyage AI posted on Reddit that they tested a traditional classification and regression approach: use an open-source or low-cost embedding model to process the input, then fit a simple sklearn model on top of the embeddings. The result is cheaper than the popular decision model Jev and almost always significantly more accurate—with a 37% reduction in error rate.

The post sparked considerable discussion: in real-world scenarios where labeled data is available, is the zero-shot decision model overrated?

Experimental Design: The Minimalist Combination of Embeddings + sklearn

According to the poster (a Voyage employee who disclosed the conflict of interest), the tested approach is very straightforward:

• Use an open-source or cheap embedding model to convert input text into vectors
• Train a simple sklearn classifier or regressor using your own labeled data
• Compare effectiveness and cost against decision models like Jev

This "embeddings + shallow model" approach was the standard workflow for NLP classification tasks before the rise of deep learning, and is now often seen as "old-school." But the author emphasizes that in most real use cases, you always have some labeled data that can be used to personalize the model.

Embedding Model is a class of neural network models that map discrete inputs (text, images, etc.) into dense vectors, represented by OpenAI's text-embedding-ada-002, Voyage AI's voyage series, and open-source BGE, E5, GTE, etc. Their core characteristic is that semantically similar inputs are closer in vector space, so they can be directly used as input features for downstream models. Unlike large models that directly generate answers, embedding models are only responsible for "representation," have smaller parameter counts, much lower inference costs, and do not rely on prompt engineering. In this experiment, the role of the embedding model is to compress raw text into fixed-length vectors, allowing a shallow classifier to learn decision boundaries in a low-dimensional space.

sklearn (scikit-learn) is the most mainstream classical machine learning library in the Python ecosystem, providing a large number of ready-made algorithms such as logistic regression, support vector machines, random forests, and gradient boosting. Its characteristics are unified interfaces, extremely fast training, and strong interpretability; on sample sizes ranging from hundreds to tens of thousands, fitting usually takes only seconds. The so-called "fitting a simple sklearn model on top of embeddings" essentially means treating embedding vectors as feature inputs and training a supervised classifier or regressor, which is a typical two-stage paradigm of "feature extraction + shallow learning."

Win-Win on Cost and Accuracy: What Does a 37% Error Reduction Mean?

The core finding is: the old-school approach is not only cheaper but almost always significantly more accurate, with a 37% relative reduction in error rate compared to Jev. This means that under the same data conditions, the traditional method can bring fewer misjudgments while avoiding the cost of calling expensive decision models.

The author provided a complete write-up on the Voyage AI blog, and the code has also been open-sourced on GitHub (mongo-predict) for easy reproduction.

Zero-Shot vs Supervised: When to Use Which?

The poster finally raised a question: has anyone seen scenarios where zero-shot performs better? This touches on the key to method selection—

• When labeled data is available: Embeddings + sklearn is usually better, especially when the amount of labeled data is sufficient to cover the task distribution
• Zero-shot scenarios: When labeled data is lacking, decision models like Jev still have value
• Task complexity: For tasks requiring complex reasoning, decision models may have the advantage, but in simple classification and regression tasks, traditional methods are not inferior

Some users in the discussion also pointed out that the quality of the embedding model and its domain match significantly affect the final results.

Zero-shot refers to a model making predictions directly without having seen any labeled samples for the target task, relying on general knowledge and instruction-following capabilities accumulated during pretraining; in contrast, supervised learning requires a certain number of "input—correct output" paired samples for training. The two are not simply substitutes: the advantage of zero-shot is zero startup cost and quick coverage of long-tail tasks, but accuracy is constrained by how well the task matches the pretraining distribution; supervised methods usually have a higher accuracy ceiling when annotations are abundant, but require annotation investment and are difficult to generalize to out-of-distribution data.

An often overlooked middle ground in practice is few-shot fine-tuning and prompt learning: when the amount of labeled data is only tens to hundreds of examples, directly training a classifier can easily overfit; in this case, consider fine-tuning the embedding model with a small number of samples, or using k-nearest neighbor classification (finding the nearest neighbor labels in the embedding space). Therefore, method selection should be weighed along the three dimensions of "annotation volume—task complexity—cost budget," rather than choosing between zero-shot and full supervision.

Implications for Practitioners

This test reminds us that before chasing new models, first confirm whether you have labeled data on hand. If you do, a cheap embedding model plus sklearn may solve the problem, with better results and lower cost. This is not to deny the value of decision models, but to emphasize that method selection should be based on data conditions and task requirements, not on hype.

The author also invites the community to share counterexamples to more comprehensively understand the boundaries of the two paradigms.

One-line Summary

When labeled data is available, embeddings plus sklearn has a 37% lower error rate than the zero-shot decision model Jev and is cheaper.

Paragraph Summary

A test shared on Reddit by a Voyage AI engineer shows that using an open-source or low-cost embedding model to convert inputs into vectors, then training a simple sklearn classifier or regressor on top of them, is cheaper and almost always significantly more accurate than the popular zero-shot decision model Jev, with a relative error reduction of 37%. This approach is essentially the standard NLP workflow before the rise of deep learning, and the author emphasizes that in real scenarios there is usually always usable labeled data. The discussion raises the boundary question between zero-shot and supervised methods: when labeled data is available, embeddings plus shallow models are often better; when annotations are lacking or tasks require complex reasoning, decision models still have value, and embedding quality and domain match also significantly affect results. The author open-sourced the code and invites the community to provide counterexamples.