4-Core CPU Benchmark: Choosing Small Models for Camera Agents — Why Do Large Models Fail?

Deploying AI on edge devices often means choosing models by guesswork. The OpenNVR maintainers ran a rigorous test: running every model a camera Agent depends on, on a 4-core CPU with no GPU. The results overturned a lot of conventional wisdom.

Test Background: When the Agent Has No GPU

OpenNVR is an open-source video recorder with a built-in camera Agent, supporting features like querying camera status, arming alarms, and retrieving historical recordings. Most users don't have a GPU — just a small 4-core PC. In the past, the team, like many open-source projects, chose models based on reputation. This time they decided to let the data speak: on a 4 vCPU, GPU-less machine, they ran standardized tests on every model the Agent involves, with all scripts and raw data published in the code repository.

The tests covered five task categories: language models (tool calling and multi-step investigation), vision models (annotated image Q&A), speech-to-text (word error rate), text-to-speech (real-time factor and intelligibility), and search (finding images from text descriptions). All timings were based on 4 CPU threads.

Language Models: Tool Calling Matters More Than Parameter Count

The team designed 20 investigation cases (e.g., "Someone in the loading dock at 1:14 AM"). The model had to check cameras, history, and on-site rules before making a judgment. Each case ran 3 times, for 60 runs total.

Model	Passes	Times tools checked before deciding
granite4:micro	44/60	56/60
granite4:tiny-h (7B MoE, 1B active)	35/60	49/60
qwen3:1.7b (old default)	21/60	36/60
LFM2-1.2B-Tool	19/60	37/60
qwen3:4b	9/60	9/60

The biggest surprise was qwen3:4b: despite a larger parameter count, its pass rate was less than half that of qwen3:1.7b. Looking at the traces, the reason was simple: it thought for a long time but almost never called tools, making decisions purely by guessing. In contrast, Granite 4 micro called about two tools on average before each answer, and that alone more than doubled its pass rate. Although its per-turn response is about twice as slow, investigation tasks run in the background, so the latency is acceptable — and the team made it the new default.

The tests also exposed a bug in the Agent itself: previously the model was required to look before describing, which meant that when a user said "Thanks, that's all," Granite would also go check the camera. After the fix, chit-chat no longer triggers tool calls, an improvement that benefits all models.

MoE (Mixture of Experts) and small active parameter counts: The "7B MoE, 1B active" label on granite4:tiny-h means the model has 7 billion total parameters but uses a mixture-of-experts architecture — each forward pass activates only about 1 billion of them (routing to a few "expert" subnetworks). This means its compute and VRAM footprint are close to a 1B-class dense model, while its knowledge capacity is close to a 7B-class model. In CPU scenarios, what determines latency is the number of active parameters actually involved in computation, not the total parameter count — which is exactly why MoE suits edge deployment. However, MoE is more sensitive to memory bandwidth, and all weights still need to be loaded, so loading and memory usage of 7B weights on a small 4-core machine cannot be ignored.

Tool Calling: This refers to an LLM outputting a structured request (e.g., JSON) in an agreed format before generating an answer, calling an external function (checking camera status, retrieving recordings), reading the returned result, and then continuing to reason. Whether an Agent can reliably "check first, then answer" depends on whether the model has been fine-tuned on tool-calling data and whether it supports function-calling templates. Parameter scale does not directly determine this ability — a small model, if specifically trained, can reliably trigger tools in multi-step investigations, while a large model not aligned to this behavior may tend to answer directly from internal knowledge. This also explains why parameter count and task pass rate were not positively correlated in this test.

Vision Models: The Trade-off Between Accuracy and Latency

Using 128 annotated images, each with 3 questions: one about something present in the image, one about something absent (answering "yes" counts as a hallucination), and one asking to count people.

Model	Wrong "yes" count (out of 126)	People count error ≤1 (out of 61)	CPU time per new frame
qwen3-vl:2b-instruct	0	50	18s (subsequent questions free)
gemma3:4b	8	48	17s
moondream 0.5b	22	42	7.6s

Qwen3-VL never said "yes" to something absent, and on a GPU it answers in about 3 seconds, so it became the default when a GPU is available. But on a 4-core CPU, reading a single frame takes 18 seconds — if someone appears at the door, an answer 18 seconds later is meaningless. So in CPU environments, Moondream is retained.

When using Ollama, note: the qwen3-vl:2b tag is the thinking version and ignores think: false; use the -instruct tag instead.

Word Error Rate (WER): The standard evaluation metric for speech recognition, calculated as (substitutions + deletions + insertions) ÷ total words in the reference text; lower is better. 1.4% vs. 3.3% may not look like a big gap, but in real experience it means roughly 1.4 vs. 3.3 errors per 100 words. For short commands (e.g., "turn on the porch light"), a doubled error rate significantly raises the chance of misunderstanding. Note that WER is sensitive to proper nouns, numbers, and accents, and the composition of the test set directly affects conclusions, so cross-model comparisons should use the same test set where possible.

Real-Time Factor (RTF): A speed metric for text-to-speech, where RTF = time required to generate audio ÷ audio duration. RTF < 1 means synthesis is faster than playback and can output in real time; RTF > 1 means generation can't keep up with playback, causing stutters. Kokoro's 1.9× real time means generating 10 seconds of speech takes about 19 seconds of compute — unacceptable in a conversational Agent; Piper's 0.23× leaves ample headroom. RTF is strongly hardware-dependent, and conclusions on a GPU may be completely reversed — which is why the article emphasizes testing on the target hardware.

Speech Recognition and Synthesis: Speed Determines the Choice

The speech-to-text results were unsurprising: Parakeet TDT 0.6B v3 (onnx-asr, int8) had a word error rate of just 1.4%, at 0.31 seconds per utterance, while faster-whisper base.en (int8) had a 3.3% error rate at 1.42 seconds. Parakeet halves the error rate and is more than 4× faster, turning speech recognition from the slowest step into one that takes only a third of a second. But Parakeet doesn't support translation or Hindi, so sites needing those features still keep Whisper.

For text-to-speech, Kokoro-82M sounds better than Piper, but on 4 cores it runs at 1.9× real time — generating audio slower than playback, causing replies to pause mid-way. Piper needs only 0.23× real time, so Piper is retained.

Search and Detection: Quality First

For description-based search like "red truck on the dock," SigLIP 2 B/16 scored 0.672 mAP, higher than CLIP ViT-B/32's 0.571. Although it takes 57ms per image (vs. 26ms for CLIP), search quality matters more, so it was made the default. After the upgrade, OpenNVR automatically re-embeds old recordings.

The always-on detector was upgraded from YOLOv8n to YOLO26n: 62ms per frame (down from 75ms), with a slight accuracy improvement and unchanged output format.

Key Findings: Hardware Determines Model Choice

This test revealed three lessons for edge AI deployment:

1. For Agents, tool calling matters more than model size. Whether tools are checked before deciding has a bigger impact than parameter count.
2. On cameras, a slow answer is a bad answer. A good answer in 8 seconds beats the best answer in 18 seconds.
3. Test on the hardware users actually have. Many rankings flip on a GPU, and most users don't have one.

The team also left one unsolved challenge: finding a vision model under 2B parameters that reads a frame within 5 seconds on a CPU without hallucinating. All test scripts and raw data are published in the OpenNVR repository, and results from other hardware are welcome as PRs.

The value of this test isn't just the replacement of four default models — it's a demonstration of methodology: open-source projects should make technical choices based on reproducible benchmarks rather than blindly following model reputation. For resource-constrained edge AI applications, this pragmatic attitude is especially valuable.

Background Notes

mAP (mean Average Precision): A ranking-quality metric commonly used in image retrieval. First, the area under the precision-recall curve (AP) is computed for each image, then averaged across all queries. It considers both whether the correct image is ranked near the top and whether recall is sufficient, making it more comprehensive than a simple Top-1 hit rate. The gap between SigLIP 2's 0.672 and CLIP's 0.571 is about 0.1, which is a clearly perceptible improvement in retrieval tasks — meaning when users search recordings with natural-language descriptions, the target segment appears higher up. In addition, after changing the embedding model, vectors for old recordings must be recomputed (the article mentions automatic re-embedding), because different models' vector spaces are incompatible and mixing them would scramble retrieval results.