[KongchangAI]
· 2 min read· 1,081 words

YOLO Latency Test Nearly 27x Wrong: 324 of 343 Layers Didn't Run on the NPU

YOLO Latency Test Nearly 27x Wrong: 324 of 343 Layers Didn't Run on the NPU

In YOLO NPU testing, 324/343 layers silently fell back to CPU, making latency data 27x wrong.

A developer shared a near-miss performance testing experience: while testing YOLO model NPU inference latency, he found that only 19 of 343 network layers actually executed on the NPU, with the remaining 324 silently falling back to the CPU, causing the measured latency to almost entirely reflect CPU performance—a 27x difference from the NPU's true capability. The article analyzes common fallback causes, including unsupported operators, data type and tensor shape constraints, and inference frameworks' silent behavior of neither erroring nor warning. The author notes that the most dangerous mistake in performance testing is measuring bad data that "looks normal," and recommends per-layer profiling to check each layer's actual execution device and mandating NPU-executed layer proportion statistics in benchmarks to avoid citing contaminated performance numbers.

A developer shared on Reddit a performance testing experience that almost led to a serious mistake: he nearly reported a YOLO inference latency that was off by 27x, because 324 out of 343 network layers were not actually running on the NPU. This story reveals a hidden yet widespread problem in edge AI deployment—you think you're using hardware acceleration, but most of the computation is still happening on the CPU.

Reddit original post screenshot

A Wrong Conclusion That Was Almost Published

According to the developer, while testing the inference performance of a YOLO model on an NPU, he obtained a latency number that looked "reasonable." But further investigation revealed that out of the model's 343 total layers, only 19 were actually executed by the NPU, while the remaining 324 all fell back to the CPU. This means the latency data he initially measured almost entirely reflected CPU performance, not the NPU's true capability. The gap between the two reached 27x.

If this result had been published directly, it would have misled other developers' judgment of NPU acceleration effectiveness. The most dangerous mistake in performance testing is not measuring bad data, but measuring bad data that "looks normal."

Why Most Layers Were Silently Dumped onto the CPU

NPU inference is not "one-click acceleration." The model compiler needs to partition the computational graph into parts supported by the NPU and parts that are not, handing the former to the NPU and falling back to the CPU for the latter. This fallback occurs when the model contains operators, data types, or tensor shapes that the NPU does not support.

YOLO-series models contain a large number of custom operators, post-processing logic (such as non-maximum suppression), and convolution operations in specific formats—all common fallback triggers. What's more troublesome is that the fallback behavior of many inference frameworks is silent—no errors, no warnings, just quietly moving the computation to the CPU and returning results as usual.

Operators and Computational Graph Partitioning

In deep learning compilers, a model is represented as a computational graph, where nodes are operators and edges are tensors. Common operators include convolution, matrix multiplication, activation functions, pooling, normalization, etc., while non-maximum suppression (NMS) in YOLO post-processing exists as a combination of multiple basic operators or as a custom operator. NPUs typically only have hardware acceleration support for specific operator sets. During compilation, the compiler traverses the computational graph, marks each operator as "can be offloaded to NPU" or "must remain on CPU," and splits the graph into several subgraphs along device boundaries. After partitioning, cross-device data must be transferred through host memory, which not only increases latency but may also offset acceleration gains due to multiple synchronizations. Therefore, layer fallback is not just "accelerating a few fewer layers"—it also introduces additional data transfer and synchronization overhead.

Common Technical Triggers for Silent Fallback

Fallback is typically triggered by several specific causes: first, unsupported operators, such as custom post-processing, non-standard activations, or dynamic shape operations in YOLO; second, data type mismatches—the NPU may only support specific combinations of INT8/FP16, while the model contains FP32 or mixed-precision nodes; third, tensor shape or layout constraints, such as channel counts not being an aligned multiple required by the hardware, or NHWC-to-NCHW layout conversions not being supported; fourth, dynamic control flow or data-dependent branches, which NPUs struggle to execute efficiently. Many inference frameworks default to fallback rather than erroring out in these situations to ensure the model "can run through"—this is precisely the root cause of the problem's hidden nature.

Static Analysis Saved You: How to Detect Layer Allocation Issues

The developer's lesson points to a practical troubleshooting method: don't just look at end-to-end latency—check the actual execution device of each layer. Mainstream NPU toolchains typically provide per-layer profiling capabilities that can output the execution unit of each operator.

Specifically, several signals are worth watching: whether NPU utilization is far below expectations, whether inference time is too close to the CPU baseline, and whether fallback records appear in compiler logs. If a large number of layers are found not to be allocated to the NPU, you need to go back and adjust the model structure or operator implementation, rather than accepting a contaminated performance number.

How Per-layer Profiling Works

Per-layer profiling works by inserting probes during inference or using hardware counters to record each operator's execution device, time consumption, input/output shapes, and data types. The toolchain typically outputs an operator allocation table indicating which nodes are executed by the NPU, which fall back to the CPU, and the number and time cost of cross-device memory copies. To judge whether acceleration is truly effective, you can compare the proportion of NPU operators, the cumulative time on the NPU side versus the CPU side, and check for frequent inter-device synchronization. If the NPU time proportion is very low while CPU time dominates, the end-to-end latency mainly reflects CPU performance and cannot serve as evidence of NPU acceleration capability.

Implications for Edge AI Deployment

The lesson from this case goes beyond YOLO. Any model that claims to "run on NPU" should undergo layer allocation verification. If a performance report lacks data on hardware execution distribution, its credibility should be discounted.

For developers using open-source toolchains, it is recommended to add a mandatory check to the benchmarking process: calculate the proportion of NPU-executed layers to total layers. When the proportion is too low, the latency data should not be cited as hardware acceleration performance. After all, a performance number that is 27x wrong is more dangerous than no number at all.

One-line Summary

In YOLO NPU testing, 324/343 layers silently fell back to CPU, making latency data 27x wrong.

Paragraph Summary

A developer shared a performance testing experience that nearly led to a false report: while testing YOLO model NPU inference latency, he found that only 19 out of 343 network layers were actually executed on the NPU, with the remaining 324 silently falling back to the CPU, causing the measured latency to almost entirely reflect CPU performance—a 27x difference from the NPU's true capability. The article analyzes common causes of fallback, including unsupported operators, data type and tensor shape constraints, and the silent behavior of inference frameworks that neither error nor warn. The author points out that the most dangerous mistake in performance testing is measuring bad data that "looks normal," and recommends using per-layer profiling to check each layer's actual execution device and mandating NPU-executed layer proportion statistics in benchmarks to avoid citing contaminated performance numbers.

#YOLO NPU latency#per-layer profiling#edge AI deployment#silent fallback#NPU hardware acceleration
Share:

Related articles