ncnn Vulkan Cross-Platform Edge Inference in Practice: Solving the GPU Fragmentation Deployment Challenge

The Real-World Dilemma of Edge Device Inference

For developers running machine learning models on end-user devices, the biggest pain point has never been model accuracy itself—it's hardware fragmentation. When you release a product targeting the consumer market, you can't make any assumptions about users' GPUs—they might have NVIDIA discrete graphics, AMD or Intel integrated graphics, or even Apple Silicon.

This fragmentation is not a new problem. Data from sources like the Steam Hardware Survey consistently shows that NVIDIA discrete GPUs hold roughly 75% of the PC gaming market—but that still means about a quarter of users are running AMD or Intel graphics hardware. In the broader consumer electronics market (including laptops and all-in-ones), Intel integrated graphics and Apple Silicon command an even higher share. For a desktop application that needs wide distribution, ignoring these users means voluntarily giving up market share.

The team behind PostSlate, a video editing tool, encountered exactly this classic challenge. They needed to run ML models for face detection and feature embedding on-device, which meant the inference backend had to run reliably across all major hardware. This real-world requirement immediately ruled out the industry's most popular choice.

Why CUDA Isn't Suitable for Cross-Platform Edge Inference

In the GPU-accelerated inference space, NVIDIA's CUDA is practically the de facto standard. Since its release in 2007, NVIDIA has built an extremely comprehensive deep learning acceleration ecosystem through CUDA and its companion tools (cuDNN for deep learning acceleration, TensorRT for inference optimization, and NCCL for multi-GPU communication). The vast majority of deep learning frameworks and optimization toolchains are built around CUDA, with a mature ecosystem and powerful performance. This ecosystem advantage creates strong path dependency—numerous academic papers default to CUDA as their experimental foundation, and mainstream industrial frameworks (PyTorch, TensorFlow) prioritize CUDA optimization, making it perpetually difficult for alternatives like AMD's ROCm and Intel's oneAPI to catch up in ecosystem maturity.

But CUDA has one fatal limitation—it's vendor-locked.

CUDA only runs on NVIDIA hardware. This means that if you choose CUDA as your inference backend, users with AMD, Intel integrated graphics, or Apple Silicon will be completely unable to benefit from GPU acceleration—they might not even be able to run the software at all. For a desktop application targeting the mass market, this is unacceptable.

Therefore, the team's core requirement was crystal clear: they needed a unified inference backend that runs on all devices. This wasn't just a performance issue—it was a usability issue that determined whether the product could reach its entire user base.

Choosing ncnn's Vulkan Backend for Cross-Platform GPU Acceleration

Ultimately, the PostSlate team chose ncnn's Vulkan backend as their solution. ncnn is a high-performance neural network forward computation framework open-sourced by Tencent's YouTu Lab in 2017, optimized specifically for mobile and edge devices. Unlike TensorFlow Lite, ONNX Runtime, and similar frameworks, ncnn adopts a zero-dependency design philosophy from the ground up—it doesn't rely on BLAS libraries or any third-party computation libraries, and the entire framework can be compiled as a purely static library. This makes deployment extremely clean across various platforms. ncnn supports operators covering mainstream CNN and Transformer architectures, and provides a complete model conversion toolchain from PyTorch/ONNX to ncnn format.

ncnn's Vulkan backend precisely solves the core challenge of cross-vendor GPU acceleration.

Vulkan is a cross-platform graphics and compute API released by the Khronos Group in 2016. As the successor to OpenGL, it serves not only graphics rendering but also provides Compute Shader capabilities for general-purpose GPU computing (GPGPU). Vulkan's design philosophy is a "thin driver layer"—giving more control to developers and reducing implicit driver behavior to achieve more predictable performance. In the general-purpose computing domain, Vulkan Compute competes with OpenCL, but Vulkan has a decisive advantage: it shares the same driver stack with graphics drivers, giving it far greater coverage on consumer devices than OpenCL. Apple platforms provide compatibility through MoltenVK (a Vulkan-to-Metal translation layer).

Its greatest advantage is that drivers are everywhere. Whether on NVIDIA, AMD, Intel, or Apple platforms, Vulkan drivers are already pre-installed on users' machines. This means developers don't need to force users to download specific runtimes, nor require any vendor-specific installation steps.

In the team's own words:

"Speed wasn't even the deciding factor—what really mattered is that Vulkan drivers already exist on every machine we ship to."

This statement captures the essence of production edge inference—deployment certainty is often more important than peak performance.

ncnn Vulkan Real-World Performance Data

Although cross-platform compatibility was the primary consideration, the performance gains from Vulkan were equally impressive. Here are the team's test results on an NVIDIA RTX 4070 using fp16 precision:

About fp16 Precision

fp16 (half-precision floating point) uses 16 bits to store a floating-point number, cutting memory footprint and bandwidth requirements in half compared to fp32 (single precision, 32 bits). On modern GPU architectures, fp16 compute unit throughput is typically twice that of fp32—for example, NVIDIA's Ada Lovelace architecture Tensor Cores have dedicated hardware acceleration paths for fp16 operations. For inference tasks, converting model weights from fp32 to fp16 typically introduces only minimal accuracy loss (usually <0.1% accuracy difference in face recognition scenarios) while significantly reducing VRAM usage and improving computation speed. This differs from more aggressive INT8/INT4 quantization approaches—which require calibration datasets and accuracy verification workflows—whereas fp16 is essentially a lossless precision-performance tradeoff.

Inference Latency Comparison

Model	ONNX CPU	ncnn Vulkan	Speedup
ArcFace R50 (face embedding)	30 ms	3 ms	10×
SCRFD (face detection)	25 ms	2.5 ms	10×

After migrating from CPU inference to Vulkan GPU acceleration, both models achieved approximately 10× latency improvement. ArcFace R50 dropped from 30 milliseconds to 3 milliseconds, and SCRFD from 25 milliseconds to 2.5 milliseconds.

About these two models: ArcFace (Additive Angular Margin Loss) is a face recognition approach proposed by the InsightFace team. Its R50 variant is based on a ResNet-50 backbone network and outputs a 512-dimensional face feature vector for identity comparison—the closer the cosine distance between two faces' feature vectors, the more likely they belong to the same identity. SCRFD (Sample and Computation Redistribution for Face Detection), also from the InsightFace team, is an efficient face detection model that redistributes computation and positive/negative sample ratios across different network stages, dramatically reducing computational overhead while maintaining high accuracy. These two models are used in combination, forming the classic "detect face location first, then extract identity features" processing pipeline.

For scenarios like video editing that require real-time processing of large numbers of frames, this speedup directly determines the fluidity of user experience. Taking 30fps video as an example, within the ~33ms budget per frame, 2.5ms for detection + 3ms for feature extraction totals less than 6ms, leaving ample headroom for other processing logic.

Model Size Optimization

Beyond inference speed, model storage size also saw significant compression:

ArcFace model: 174 MB (ONNX fp32) → 87 MB (ncnn fp16 weight storage)

With fp16 weight storage, model size was nearly halved. For edge models that need to be distributed alongside the application, smaller size means faster downloads and lower distribution costs. In desktop application scenarios, package size directly impacts users' willingness to download—shrinking from 174MB to 87MB could mean several minutes of download time difference in regions with poor network conditions.

Engineering Insights for Production Edge Inference

This case study offers several thought-provoking insights for edge AI developers.

The Mainstream Solution Isn't Always the Optimal Solution

While CUDA is the industry standard, for consumer products targeting heterogeneous hardware, its vendor lock-in actually becomes an obstacle. Technology selection must return to actual deployment scenarios rather than blindly following the ecosystem with the largest footprint. In data center and cloud training scenarios, CUDA's ecosystem advantage is overwhelming; but when the deployment target shifts from controlled server clusters to uncontrolled consumer devices, the evaluation dimensions must expand from "peak performance" to "compatibility coverage" and "deployment complexity."

Deployment Friction Is an Invisible Product Cost

Requiring users to install specific runtimes significantly increases churn rates and support burden. Vulkan's "zero additional installation" characteristic essentially lowers the product's barrier to entry, reducing friction at every step between installation and usage. In desktop software distribution, every additional installation step (whether downloading the CUDA Toolkit or installing a specific version of .NET Runtime) causes measurable user drop-off. For scenarios requiring technical support, the troubleshooting cost of issues like "user's machine lacks CUDA drivers" far exceeds intuitive expectations.

Performance and Compatibility Are Not a Zero-Sum Game

ncnn's Vulkan backend delivers acceleration close to that of dedicated backends while providing cross-platform capability. This proves that mature open-source inference frameworks can fully support production-grade edge inference needs. Of course, it should be noted that on NVIDIA hardware, dedicated optimization tools like TensorRT may be 20-50% faster than the Vulkan backend. But PostSlate's case clearly demonstrates that for most edge application scenarios, 10× acceleration over CPU is more than sufficient—the engineering cost of chasing that last bit of performance difference is far less valuable than ensuring all users can have a smooth experience.

For teams struggling with on-device ML deployment, the ncnn + Vulkan combination deserves serious evaluation. It may not achieve the absolute fastest results on any single piece of hardware, but "running well on all hardware" is precisely the quality production environments need most.