Ollama Decision Models Tested: Local Inference Far Outpaces Claude

Ollama recently added support for "decision models," letting developers run small-parameter models directly on consumer-grade hardware to handle lightweight inference tasks such as classification, scoring, and visual judgment. In hands-on testing, a Bilibili creator found that for scenarios with a fixed output structure, local models not only respond faster than cloud-based Claude but can even perform multimodal image classification on a Dell Pro Max equipped with a GB10.

Small Models Run Locally with a Clear Speed Advantage

The creator pulled the 0.8B-parameter TEF model via Ollama (the 4-bit quantization option must be explicitly specified) and ran three comparison tests on a Linux machine. The first was email classification—sorting emails into sponsored spam, fan support, and service work opportunities. After warm-up, the TEF model took 1.273 seconds total, while the same task called through Claude's official SDK took 2.568 seconds—"more than double."

The second was a scoring task: judging how "promising" a programming YouTube sponsorship inquiry was (weak/average/strong/excellent). TEF took 1.09 seconds, while Claude again took around 2.58 seconds. Note: these example questions were generated by Claude itself; the creator simply used them for classification.

Another model, Nimble, landed in the middle: 1.7 seconds after warm-up and 1.336 seconds on the scoring task. The creator emphasized that Claude requires no warm-up, whereas local small models incur startup costs on first run—"but if you're doing this at scale, it matters."

Quantization is a key concept in local inference. It compresses model weights from high-precision floating point (such as FP16) to low-precision integers (such as 4-bit or 8-bit), greatly reducing memory usage and compute so that small models like 0.8B can run on consumer-grade hardware. The trade-off is precision loss, which may affect output stability and accuracy. The 4-bit quantization option mentioned here must be explicitly specified when pulling the model via Ollama; otherwise the default configuration may differ. For tasks like decision models that only need to output structured labels, 4-bit quantization is usually sufficient, but deviations are more likely on fine-grained scores or edge-case samples.

Warm-up refers to the process of loading a model into memory/VRAM for the first time and completing initialization. Cold-starting a local small model requires reading weights from disk into VRAM, which can take several seconds; after warm-up, the weights stay resident and subsequent calls are nearly instantaneous. Cloud APIs shift this cost to the provider, so comparisons must distinguish cold-start from steady-state latency—otherwise local advantages will be overestimated.

Multimodal Capabilities: Image Classification Locally Too

What was truly surprising was the vision task. The creator connected a Dell Pro Max with a GB10 to the Ollama service and sent base64-encoded images via httpx, asking the model to judge three attributes of YouTube thumbnails: whether there is a face, whether it is clickbait, and whether the topic is programming/AI/gaming/Vlog or something else.

From the results, apart from the thumbnail featuring the creator himself (where the model gave a 0 probability of "has a face"), all other thumbnails were identified as containing faces with high confidence. On topic classification, most programming/AI content was correctly grouped into AI or programming, though a few—such as GCP and Boycott—were judged by the creator as "debatable." Clickbait judgments were also largely reasonable; for example, the "boycott non-Linux apps" and "Opus rocket" thumbnails were flagged as the most clickbait-y.

The creator admitted, "This is more powerful than Claude," but added that the faster the hardware, the better the results.

Base64 image encoding is a common practice of converting binary images into ASCII strings and sending them in the HTTP request body. Its advantages are that no separate file-upload endpoint is needed and it can be neatly packaged with the JSON request body; the downside is roughly 33% size inflation, which significantly increases transfer and parsing overhead for large images. In practice, it is often combined with image resizing and compression to balance recognition accuracy and latency.

The visual capabilities of small multimodal models typically come from a combination of a vision encoder and a language decoder: the encoder converts the image into feature vectors, which are then concatenated with the text prompt and passed to the model for judgment. Models at the 0.8B level perform reasonably well on coarse-grained attributes (whether there is a face, whether it is clickbait) but are prone to errors on fine-grained topic classification. The individual case in the article where the creator himself appeared but the model gave a 0 probability of a face shows that small models have blind spots for specific faces or out-of-distribution samples; in deployment, validation-set evaluation should be done for key categories.

What Exactly Are Decision Models?

Decision models are a systematic set of models introduced by Claude, specifically designed to solve problems that don't require complex reasoning—observing something and judging whether it is or isn't, i.e., binary classification (yes/no), or scoring (how good, how scam-like, how dangerous). Ollama now opens this capability to open-source models, and because they run locally, European users no longer have to endure API latency from the US.

The difference between decision models and generative large models lies in task shape: generative models output open-ended text, while decision models output labels or scores from a closed set, such as "yes/no" or "weak/average/strong/excellent." This constraint means the model doesn't need to generate long reasoning chains—it only needs to discriminate among given categories, resulting in low reasoning depth and low latency. In engineering practice, this is usually achieved through a fixed output structure (such as a JSON schema or enum constraints) so results can be parsed directly by programs.

It's important to distinguish that decision models are not a separate architecture but a way of using models tuned or prompt-constrained for classification/scoring tasks. Their reliability depends heavily on whether category definitions are clear and whether samples fall within the training distribution. For ambiguous samples (such as the topic categorization of the GCP and Boycott thumbnails mentioned in the article), judgments are prone to controversy, so confidence thresholds and manual spot checks are usually needed for large-scale deployment.

The Trade-off Between Quality and Speed

The creator was quite restrained in evaluating model quality: "I don't have particularly strong opinions; I haven't used these models in depth. Maybe Claude is more powerful in more complex scenarios—I'm not sure about that." This reminds us that decision models excel at structured output and low-reasoning-depth tasks, not general conversation or complex analysis.

For engineering scenarios requiring large-scale classification, scoring, and image filtering, the speed and cost advantages of local small models are already convincing enough. Ollama's support for decision models in such a convenient way is well worth developers trying out.

One-line Summary

Ollama supports local decision models that classify and score more than twice as fast as cloud-based Claude and can even do image classification.

Paragraph Summary

The article introduces Ollama's newly launched decision model support, which lets developers run 0.8B-parameter small models on consumer-grade hardware for classification, scoring, and visual judgment. In testing, the TEF model took 1.273 seconds for email classification versus 2.568 seconds for Claude; on a scoring task, local took 1.09 seconds versus about 2.58 seconds in the cloud, giving local a more-than-twofold speed advantage after warm-up. On the multimodal side, base64-encoded images let the local model judge thumbnails for faces, clickbait, and topic classification, with largely reasonable results but a few questionable judgments. The author notes that local models incur startup costs on first run, making them suitable for large-scale repeated calls; quality may still lag behind Claude in complex scenarios, but the speed and cost advantages are clear.