Hand-Written Tiny CNN Runs 3x Faster Than Inference Engines: A Practical Guide to Edge Optimization on Raspberry Pi

A Counterintuitive Experiment Result

In an era where edge computing and on-device AI are becoming increasingly prevalent, developers typically rely on mature inference engines (such as ONNX Runtime and ncnn) to deploy neural network models.

Background on Edge Computing and On-Device AI: Edge computing is a distributed architecture that performs computation near the data source at the network edge, rather than transmitting all data to the cloud for processing. On-device AI refers to running AI models directly on end devices (such as smartphones, IoT devices, and Raspberry Pi) for local inference. This architecture significantly reduces network latency, lowers bandwidth consumption, protects data privacy, and continues to function even when the network is disconnected. With the improving performance of low-power processors like ARM and RISC-V, and the maturation of model compression techniques (quantization, pruning, knowledge distillation), more and more AI applications are migrating from the cloud to the edge. Typical use cases include real-time object detection on smart cameras, voice wake-up on smart speakers, and anomaly detection on industrial sensors.

Mainstream Inference Engines: ONNX Runtime is Microsoft's open-source cross-platform inference engine that supports the ONNX (Open Neural Network Exchange) standard model format. It runs on CPUs, GPUs, NPUs, and other hardware, offering advanced features like automatic graph optimization, operator fusion, and memory planning. ncnn is a high-performance neural network forward computation framework open-sourced by Tencent's YouTu Lab, specifically optimized for mobile ARM platforms with zero third-party dependencies, and is widely used for computer vision tasks on mobile devices. Other popular frameworks include TensorFlow Lite, PyTorch Mobile, and MNN. These engines have years of engineering refinement behind them, containing thousands of operator implementations, multiple hardware backend adaptations, and comprehensive model conversion toolchains, making them the industry standard for deploying AI models.

These engines have undergone extensive engineering optimization and are considered the "gold standard" for performance. However, a Reddit developer recently shared an experiment that yielded a counterintuitive conclusion: by hand-writing a tiny CNN on a Raspberry Pi, they achieved performance 3x faster than these mainstream inference engines.

Raspberry Pi Hardware Characteristics: The Raspberry Pi is a series of low-cost, credit-card-sized single-board computers widely used in education, IoT, and embedded development. The device referenced here is likely a Raspberry Pi 4 or 5, featuring ARM Cortex-A72/A76 cores, 4 cores, clocked at 1.5–2.4 GHz, with 2–8 GB of RAM, and support for the ARM NEON SIMD instruction set. Compared to desktop CPUs, it has limited compute power (~10–50 GFLOPS floating-point performance), constrained memory bandwidth (4–8 GB/s), no dedicated GPU acceleration, and consumes only 5–15W. These resource-constrained characteristics make it an ideal platform for testing edge AI deployments: if an algorithm runs smoothly on a Raspberry Pi, it can be deployed to even more low-power scenarios. Its ARM architecture also represents the mainstream technology trajectory for mobile and embedded devices.

The project's code is only a few hundred lines, written entirely from scratch, and has been open-sourced on GitHub (optimize-cnn). The author's core argument is clear: when running small models on extremely resource-constrained devices, the "generality" of universal inference engines can actually become a performance burden, and customized implementations tailored to specific scenarios can achieve superior results.

Three Steps from Naive Implementation to Peak Optimization

The author didn't start with high-performance code. Instead, they followed a clear, incremental optimization path. The entire process was divided into three stages, each involving changes of only a few dozen lines of code while delivering significant performance gains. This "small steps, fast iterations" approach also kept the code highly readable.

Step 1: Naive Implementation

The starting point was the most straightforward CNN forward inference implementation—writing out the loops directly according to the mathematical definitions of convolution, activation, pooling, and other operators.

CNN Basics and Convolution Operations: Convolutional Neural Networks (CNNs) are the core architecture in deep learning for processing spatial data such as images and video. The central operation is convolution: a small filter (kernel) slides across the input feature map, performing element-wise multiplication and summation at each position to produce a single value in the output feature map. For example, a 3×3 convolution kernel sliding across an image involves 9 multiplications and 8 additions per position. A typical CNN layer might contain 64 such kernels, and for a 224×224 input image, a single layer requires hundreds of millions of multiply-accumulate operations. Subsequent activation functions (such as ReLU), pooling, batch normalization, and other operations further extract and normalize features. Understanding the mathematical essence of these operations is a prerequisite for optimizing inference performance.

This implementation offered mediocre performance but had the advantage of clear logic, serving as the baseline for all subsequent optimizations. Its value lies in giving the developer a complete understanding of every computational step, laying the foundation for targeted optimizations.

Step 2: SIMD Vectorized Acceleration

The second stage introduced SIMD (Single Instruction, Multiple Data) optimization.

SIMD Instruction Sets Explained: SIMD (Single Instruction Multiple Data) is a parallel computing technique that allows a single CPU instruction to process multiple data elements simultaneously. For example, the ARM NEON instruction set can process 4 32-bit floating-point numbers or 8 16-bit integers in a single instruction. In traditional scalar code, computing a[0]*b[0]+a[1]*b[1]+a[2]*b[2]+a[3]*b[3] requires 4 multiplications and 3 additions for a total of 7 instructions; with SIMD vector instructions, this can be accomplished with 1 multiplication instruction and 1 horizontal addition instruction, yielding a theoretical speedup of 3.5x. Intel x86 platforms have SSE and AVX instruction sets, ARM platforms have NEON, and RISC-V has RVV. Effective use of SIMD is a key technique for CPU performance optimization. Modern compilers can perform auto-vectorization to some extent, but manual optimization typically yields better results.

Modern CPUs (including the ARM architecture used in Raspberry Pi) support SIMD instruction sets (such as ARM NEON), which can process multiple data elements in parallel within a single instruction. Convolution operations are essentially massive multiply-accumulate operations, making them naturally suited for vectorization. By rewriting inner loops with SIMD instructions, computational throughput can be multiplied several times over.

Step 3: Operator Fusion to Reduce Memory Overhead

The final step was Operator Fusion. In a standard inference pipeline, operations like convolution, activation functions, and batch normalization are typically executed separately, with each step writing intermediate results back to memory before reading them again.

The Memory Access Bottleneck: Modern processors compute far faster than they can access memory. For example, the Cortex-A72 core in a Raspberry Pi 4 can perform 2 floating-point multiply-accumulate operations per cycle, and at 1.5 GHz, this translates to a theoretical peak of 12 GFLOPS. However, its memory bandwidth is only about 6 GB/s, meaning actual performance falls far short of the peak if every operation requires fetching data from main memory. To mitigate this disparity, CPUs employ multi-level caches: L1 cache is the fastest but smallest (32–64 KB), L2 cache is larger (256 KB–1 MB), L3 cache is even larger (2–8 MB) but somewhat slower, and main memory is the largest but slowest. The principle of data locality requires programs to reuse data in cache as much as possible. For CNN inference, if intermediate results are frequently written back to memory and then read again, it causes a large number of cache misses, severely limiting performance. This is the core motivation behind operator fusion optimization.

This frequent memory access becomes a serious bottleneck on bandwidth-constrained devices.

Operator Fusion Technical Principles: In the traditional graph execution model, a neural network is represented as a computation graph where each node is an operator (convolution, activation, normalization, etc.) connected by tensors. During execution, operators are computed sequentially in topological order, with each step requiring output memory allocation, computation, and input memory deallocation. Operator fusion (also known as kernel fusion) merges multiple consecutive operators into a single composite operator, completing them within the same compute kernel. For example, fusing "Convolution + BatchNorm + ReLU": each element of the convolution output is immediately normalized and activated, with the result written directly to the final output, while intermediate results exist only in registers. This optimization can reduce memory bandwidth consumption by 60%–80% while also reducing operator scheduling overhead. Compilers like TensorRT and XLA perform automatic fusion analysis, but hand-written code can achieve more aggressive fusion.

The idea behind operator fusion is to merge multiple consecutive operators into a single computational unit, allowing data to undergo multiple computation steps within registers or cache before being written back to memory, thereby drastically reducing memory access overhead.

Why Can Hand-Written Code Beat Mature Inference Engines?

This result seems surprising at first glance, but it makes perfect sense upon deeper analysis. To support all kinds of model architectures, hardware platforms, and data types, universal inference engines inevitably introduce extensive abstraction layers, operator scheduling logic, and runtime checks.

The Generality vs. Performance Tradeoff: There is a classic generality-performance tradeoff in software engineering. General-purpose frameworks need to support a wide range of scenarios: different model architectures (CNN, Transformer, RNN), different data types (FP32, FP16, INT8), different hardware (CPU, GPU, DSP, NPU), dynamic input sizes, and more. This necessitates abstraction layers, virtual function calls, runtime type checking, memory pool management, and other mechanisms—a single inference pass may involve thousands of function calls and conditional branches. For large models, this overhead is negligible as a fraction of total compute; but for tiny models with only 3–5 layers that complete in a few milliseconds, framework overhead can account for over 50% of total execution time. A specialized implementation, on the other hand, can determine all parameters at compile time, eliminate branches, inline all functions, and hand-write assembly for specific hardware—this is where the 3x performance gap comes from. This mirrors the "specialized systems beating general-purpose systems" phenomenon seen in the database field.

These generality-focused designs pay off when running large models, but when executing a tiny CNN with only a few layers, the framework overhead can exceed the actual computation itself.

Moreover, the optimization strategies of general-purpose engines target "average best performance" and cannot deliver extreme customization for a specific small model. Hand-written implementations, however, can perform "tailored" optimization for the specific network architecture, input dimensions, and hardware characteristics—such as pre-determining loop boundaries, eliminating dynamic branches, and precisely controlling memory layout. This is a vivid demonstration of "specialized beating general-purpose" in edge computing scenarios.

Practical Takeaways for On-Device AI Developers

The significance of this project is not to dismiss inference engines—for the vast majority of use cases, mature frameworks like ONNX Runtime and ncnn remain the most hassle-free and reliable choices. Its real value lies in revealing several engineering insights worth reflecting on:

First, understanding the fundamentals matters more than calling APIs. Only by truly understanding how convolution is computed, how memory is accessed, and how the CPU executes instructions can you make breakthrough optimizations in critical scenarios.

Second, for extreme scenarios, customization still offers enormous potential. On IoT devices, microcontrollers, sensor nodes, and other extremely resource-constrained platforms, a few hundred lines of custom code may be far more appropriate than a massive inference framework—saving resources while running much faster.

Third, incremental optimization is a reproducible methodology. The author's path from naive implementation to SIMD to operator fusion is a universal performance optimization paradigm worth adopting by any developer working in high-performance computing.

For developers looking to deeply learn the principles of inference optimization, an open-source project like this—only a few hundred lines long and highly readable—is an excellent learning resource. It presents optimization techniques usually hidden behind massive frameworks in the most concise form possible.

Key Takeaways