How Fast Are Local LLMs at Classification Tasks? A Hands-On Comparison of Five Models

Same email classification task tested: local models fastest after warm-up, but cold start and accuracy remain trade-offs.
The author compared end-to-end latency of local models and cloud APIs using the same email classification script on a Dell Pro Max with GB10 (Dell's DGX Spark implementation). The Ollama local script took 5.5s on first run and only 1.2s on the second, becoming the fastest overall; Nimble also dropped from 6.1s to 1.8s; cloud results: Gemini ~3.4s, GPT ~6.5s, Jeff ~2s. Cold-start model loading and memory initialization dominate first-run time, while cloud is affected by network round trips and API queuing. The author stresses local models are fastest but not necessarily best; accuracy should reference public benchmarks, selection should be scenario-based rather than second-based, and hardware differences matter.
The same email classification task, run on different decision models, can vary in speed by several times. A developer ran a set of hands-on tests on their own Dell Pro Max (equipped with GB10, the Dell implementation of DGX Spark): using the exact same email classification script, they called local models and cloud APIs respectively, focusing on latency rather than accuracy.

Test Design: One Task, Five Approaches
The task itself is simple—a batch of emails, plus a classification instruction, needing to be sorted into categories like sponsorship, spam, support, fan mail, services, or job offer. The author admits this was distilled from their own real-world usage scenario.
The models compared fall into two camps. The local camp is Tef1 and Nimble, running on the Dell Pro Max's GB10 hardware; the cloud camp includes GPT (i.e., OpenAI's decisions API) called via the TypeSafe API, Gemini, and a local script going through Ollama. All scripts do exactly the same thing, just calling different backends, so the differences can largely be attributed to the models and call chains themselves.
The author explicitly states that this is not about scores—classification accuracy is better judged by public benchmarks; what they wanted to measure was end-to-end time consumption.

About DGX Spark and GB10
GB10 mentioned in the article is a superchip from NVIDIA, integrating a Grace CPU and a Blackwell-architecture GPU, sharing unified memory through NVLink-C2C interconnect, with overall power consumption kept at desktop level. DGX Spark is a small AI development machine built by NVIDIA on GB10, positioned to let developers run large language models and prototype locally without relying on data center GPUs. Dell Pro Max is Dell's commercial implementation of the same platform, with essentially identical hardware specs. GB10's unified memory architecture allows model weights to reside directly in the shared CPU/GPU memory pool, which is the key to the extremely low latency of local models after warm-up; however, its memory bandwidth and compute power are still significantly lower than server-grade GPUs, so the cold-start overhead when first loading large models is quite noticeable.
Measured Data: Clear Gap Between Cold Start and Warm Start
The timing method is straightforward: wrapping the entire script with time uv run and recording total time.
| Model/Backend | Time |
|---|---|
| Gemini | ~3.4 seconds |
| GPT (decisions API) | ~6.5 seconds |
| Jeff | ~2 seconds |
| Ollama script (first run) | ~5.5 seconds |
| Ollama script (second run) | ~1.2 seconds |
| Nimble (first run) | ~6.1 seconds |
| Nimble (second run) | ~1.8 seconds |
The most interesting part of these numbers is the difference between the two runs of local models. The Ollama script took 5.5 seconds on the first run and dropped to 1.2 seconds on the second, becoming the fastest of all; Nimble likewise went from 6.1 seconds to 1.8 seconds. The cold-start overhead (model loading, memory initialization) inflates the first-round results significantly, and once warm-up is complete, the advantage of local inference becomes apparent.
On the cloud side, GPT via the decisions API took 6.5 seconds, Gemini about 3.4 seconds, and Jeff only 2 seconds. Network round trips and API queuing are both counted in total time, which is why cloud solutions struggle to consistently reach the level of local warm starts.

Cold Start vs. Warm Start
Cold start refers to the first call when the model process is not yet running and weights are not loaded into memory; the time includes reading model files from disk, initializing the inference framework, allocating memory, and compiling or warming up the computation graph. Warm start means the model is already resident in memory, and subsequent calls only need to execute forward inference. The gap between the two is especially pronounced for local models in the multi-GB range, because weight loading often accounts for the bulk of first-round time. In the article, the Ollama script dropping from 5.5 seconds to 1.2 seconds and Nimble from 6.1 seconds to 1.8 seconds is exactly a manifestation of this mechanism. In engineering practice, cold start is often avoided through resident services, warm-up requests, or model caching—an important consideration in actual deployment of local inference solutions.
Beyond Speed: The Cost of Local Models
The author ends with a key judgment: local models are the fastest, but not necessarily the best.
This points out the limitation of latency comparisons. The real cost of classification tasks is "fast but wrong"—if the model misclassifies a customer support email as spam, the few seconds saved are meaningless. Cloud models are usually more reliable in inference quality, especially when category boundaries are fuzzy and instructions require nuanced understanding. Local small models have extremely low latency after warm-up and are suitable for high-frequency, low-risk, batch processing scenarios with clear categories.
Another easily overlooked variable is hardware. The GB10 used in this test is a desktop-level AI compute solution; on ordinary laptops or machines without dedicated acceleration cards, both cold-start and warm-start performance of local models will differ noticeably. In other words, the seconds above cannot be directly applied to your own device, but the pattern of "slow cold start, fast warm start" most likely holds.

About Ollama and API Call Chains
Ollama is a local LLM running and hosting tool that packages model download, quantized format loading, an inference engine (based on llama.cpp), and a local HTTP service; developers can launch a model with a single command and call it via API. The "local script going through Ollama" in the article means the script calls the model through Ollama's local service interface, so its latency includes inter-process communication and local HTTP round trips, but saves public network transmission. In contrast, the end-to-end time of cloud APIs consists of network round trips, server-side queuing, model inference, and streaming response return, among which network and server-side queuing are uncontrollable variables—which explains why cloud solutions struggle to consistently reach the level of local warm starts.
How to Choose: By Scenario, Not by Seconds
If your classification task is offline, batch, and has clearly defined categories, the low latency of local models after warm-up is very appealing, and you don't pay per call, nor does data leave your machine. If the task requires stronger semantic judgment, or call frequency is low and you're not sensitive to time-to-first-token, the stability and quality of cloud APIs are worth considering.
There's also a middle path: keep the local model resident and warmed up to avoid repeated cold starts. The 1.2 seconds of the second Ollama run in the test shows that as long as the model is already in memory, local inference can easily beat any cloud call. Conversely, if the script is a batch job run occasionally, then the first 5.5 seconds is the number you'll actually encounter.
The value of such comparisons is not to conclude a "fastest model," but to remind us: when evaluating inference speed, engineering details like cold start, warm-up state, and network links may have a greater impact on the final experience than the model's own parameter scale.
One-line Summary
Hands-on test of the same email classification task: local models are fastest after warm-up, but cold start and accuracy remain the trade-offs.
Paragraph Summary
The author compared end-to-end latency of local models and cloud APIs using the same email classification script on a Dell Pro Max equipped with GB10 (the Dell implementation of DGX Spark). Results show that the Ollama local script took 5.5 seconds on first run and only 1.2 seconds on the second, becoming the fastest overall; Nimble also dropped from 6.1 seconds to 1.8 seconds; on the cloud side, Gemini took about 3.4 seconds, GPT about 6.5 seconds, and Jeff about 2 seconds. Cold-start model loading and memory initialization are the main sources of first-round time, while cloud is affected by network round trips and API queuing. The author emphasizes that local models are fastest but not necessarily best; accuracy should be referenced to public benchmarks, selection should be by scenario rather than seconds, and attention should be paid to hardware differences in testing.
Related articles

Harvard AI Safety Course CS 2881R Sparks Discussion
Harvard's CS 2881R AI Safety course was mentioned on Hacker News. This article covers the course background, core AI safety topics, and takeaways for learners, helping readers understand how top universities cultivate AI safety talent.

Expanding Your Social Circle: Friend-Making Apps Worth Trying
How to choose a friend-making app? This article reviews friendship apps like the friend-discovery platform BFF and activity-oriented Timeleft, analyzing which approach suits whom and offering practical tips on offline meeting safety and expectation management.

Why You Shouldn't Run the Hermes Agent on Your Personal Computer
Why shouldn't you run autonomous AI agents like Hermes on your personal computer? This article analyzes why agents should be deployed to a dedicated VPS instead of your primary device, from four angles: security risks, resource consumption, credential leakage, and autonomy requirements—plus the practical logic of isolated deployment.