[KongchangAI]
· 2 min read· 1,362 words

How Fast Are Local LLMs at Classification Tasks? A Hands-On Comparison of Five Models

How Fast Are Local LLMs at Classification Tasks? A Hands-On Comparison of Five Models

Same email classification task tested: local models fastest after warm-up, but cold start and accuracy remain trade-offs.

The author compared end-to-end latency of local models and cloud APIs using the same email classification script on a Dell Pro Max with GB10 (Dell's DGX Spark implementation). The Ollama local script took 5.5s on first run and only 1.2s on the second, becoming the fastest overall; Nimble also dropped from 6.1s to 1.8s; cloud results: Gemini ~3.4s, GPT ~6.5s, Jeff ~2s. Cold-start model loading and memory initialization dominate first-run time, while cloud is affected by network round trips and API queuing. The author stresses local models are fastest but not necessarily best; accuracy should reference public benchmarks, selection should be scenario-based rather than second-based, and hardware differences matter.

The same email classification task, run on different decision models, can vary in speed by several times. A developer ran a set of hands-on tests on their own Dell Pro Max (equipped with GB10, the Dell implementation of DGX Spark): using the exact same email classification script, they called local models and cloud APIs respectively, focusing on latency rather than accuracy.

Model comparison runtime environment

Test Design: One Task, Five Approaches

The task itself is simple—a batch of emails, plus a classification instruction, needing to be sorted into categories like sponsorship, spam, support, fan mail, services, or job offer. The author admits this was distilled from their own real-world usage scenario.

The models compared fall into two camps. The local camp is Tef1 and Nimble, running on the Dell Pro Max's GB10 hardware; the cloud camp includes GPT (i.e., OpenAI's decisions API) called via the TypeSafe API, Gemini, and a local script going through Ollama. All scripts do exactly the same thing, just calling different backends, so the differences can largely be attributed to the models and call chains themselves.

The author explicitly states that this is not about scores—classification accuracy is better judged by public benchmarks; what they wanted to measure was end-to-end time consumption.

Email classification categories

About DGX Spark and GB10

GB10 mentioned in the article is a superchip from NVIDIA, integrating a Grace CPU and a Blackwell-architecture GPU, sharing unified memory through NVLink-C2C interconnect, with overall power consumption kept at desktop level. DGX Spark is a small AI development machine built by NVIDIA on GB10, positioned to let developers run large language models and prototype locally without relying on data center GPUs. Dell Pro Max is Dell's commercial implementation of the same platform, with essentially identical hardware specs. GB10's unified memory architecture allows model weights to reside directly in the shared CPU/GPU memory pool, which is the key to the extremely low latency of local models after warm-up; however, its memory bandwidth and compute power are still significantly lower than server-grade GPUs, so the cold-start overhead when first loading large models is quite noticeable.

Measured Data: Clear Gap Between Cold Start and Warm Start

The timing method is straightforward: wrapping the entire script with time uv run and recording total time.

Model/BackendTime
Gemini~3.4 seconds
GPT (decisions API)~6.5 seconds
Jeff~2 seconds
Ollama script (first run)~5.5 seconds
Ollama script (second run)~1.2 seconds
Nimble (first run)~6.1 seconds
Nimble (second run)~1.8 seconds

The most interesting part of these numbers is the difference between the two runs of local models. The Ollama script took 5.5 seconds on the first run and dropped to 1.2 seconds on the second, becoming the fastest of all; Nimble likewise went from 6.1 seconds to 1.8 seconds. The cold-start overhead (model loading, memory initialization) inflates the first-round results significantly, and once warm-up is complete, the advantage of local inference becomes apparent.

On the cloud side, GPT via the decisions API took 6.5 seconds, Gemini about 3.4 seconds, and Jeff only 2 seconds. Network round trips and API queuing are both counted in total time, which is why cloud solutions struggle to consistently reach the level of local warm starts.

Timing comparison process

Cold Start vs. Warm Start

Cold start refers to the first call when the model process is not yet running and weights are not loaded into memory; the time includes reading model files from disk, initializing the inference framework, allocating memory, and compiling or warming up the computation graph. Warm start means the model is already resident in memory, and subsequent calls only need to execute forward inference. The gap between the two is especially pronounced for local models in the multi-GB range, because weight loading often accounts for the bulk of first-round time. In the article, the Ollama script dropping from 5.5 seconds to 1.2 seconds and Nimble from 6.1 seconds to 1.8 seconds is exactly a manifestation of this mechanism. In engineering practice, cold start is often avoided through resident services, warm-up requests, or model caching—an important consideration in actual deployment of local inference solutions.

Beyond Speed: The Cost of Local Models

The author ends with a key judgment: local models are the fastest, but not necessarily the best.

This points out the limitation of latency comparisons. The real cost of classification tasks is "fast but wrong"—if the model misclassifies a customer support email as spam, the few seconds saved are meaningless. Cloud models are usually more reliable in inference quality, especially when category boundaries are fuzzy and instructions require nuanced understanding. Local small models have extremely low latency after warm-up and are suitable for high-frequency, low-risk, batch processing scenarios with clear categories.

Another easily overlooked variable is hardware. The GB10 used in this test is a desktop-level AI compute solution; on ordinary laptops or machines without dedicated acceleration cards, both cold-start and warm-start performance of local models will differ noticeably. In other words, the seconds above cannot be directly applied to your own device, but the pattern of "slow cold start, fast warm start" most likely holds.

Time comparison between Jeff and Ollama

About Ollama and API Call Chains

Ollama is a local LLM running and hosting tool that packages model download, quantized format loading, an inference engine (based on llama.cpp), and a local HTTP service; developers can launch a model with a single command and call it via API. The "local script going through Ollama" in the article means the script calls the model through Ollama's local service interface, so its latency includes inter-process communication and local HTTP round trips, but saves public network transmission. In contrast, the end-to-end time of cloud APIs consists of network round trips, server-side queuing, model inference, and streaming response return, among which network and server-side queuing are uncontrollable variables—which explains why cloud solutions struggle to consistently reach the level of local warm starts.

How to Choose: By Scenario, Not by Seconds

If your classification task is offline, batch, and has clearly defined categories, the low latency of local models after warm-up is very appealing, and you don't pay per call, nor does data leave your machine. If the task requires stronger semantic judgment, or call frequency is low and you're not sensitive to time-to-first-token, the stability and quality of cloud APIs are worth considering.

There's also a middle path: keep the local model resident and warmed up to avoid repeated cold starts. The 1.2 seconds of the second Ollama run in the test shows that as long as the model is already in memory, local inference can easily beat any cloud call. Conversely, if the script is a batch job run occasionally, then the first 5.5 seconds is the number you'll actually encounter.

The value of such comparisons is not to conclude a "fastest model," but to remind us: when evaluating inference speed, engineering details like cold start, warm-up state, and network links may have a greater impact on the final experience than the model's own parameter scale.

One-line Summary

Hands-on test of the same email classification task: local models are fastest after warm-up, but cold start and accuracy remain the trade-offs.

Paragraph Summary

The author compared end-to-end latency of local models and cloud APIs using the same email classification script on a Dell Pro Max equipped with GB10 (the Dell implementation of DGX Spark). Results show that the Ollama local script took 5.5 seconds on first run and only 1.2 seconds on the second, becoming the fastest overall; Nimble also dropped from 6.1 seconds to 1.8 seconds; on the cloud side, Gemini took about 3.4 seconds, GPT about 6.5 seconds, and Jeff about 2 seconds. Cold-start model loading and memory initialization are the main sources of first-round time, while cloud is affected by network round trips and API queuing. The author emphasizes that local models are fastest but not necessarily best; accuracy should be referenced to public benchmarks, selection should be by scenario rather than seconds, and attention should be paid to hardware differences in testing.

#local LLM#inference speed#cold start#warm start#Ollama#cloud API#email classification#DGX Spark#GB10#Gemini#GPT
Share:

Related articles