NPU
A Neural Processing Unit (NPU) is a specialized processor chip designed to accelerate artificial intelligence and machine learning tasks. Unlike general-purpose CPUs or parallel-computing GPUs, NPUs apply hardware-level optimization to matrix operations and convolution computations characteristic of neural networks, achieving higher energy efficiency for deep learning model training and inference. NPUs are widely deployed in smart terminals, data centers, and edge computing devices.
Core Facts
Local AI (on-device AI) runs model inference directly on end-user devices, offering advantages such as privacy protection, low latency, offline availability, and cost reduction.
The NPU is a hardware accelerator designed specifically for AI inference computation; the Hexagon NPU in Qualcomm's Snapdragon 8 series, Apple's Neural Engine, and MediaTek's APU have all been integrated into mobile chips.
NPUs are specifically optimized for low-precision (INT8/FP16) matrix multiplication and can achieve similar or higher throughput at a fraction of a GPU's power consumption when running quantized lightweight inference models.
Timeline (last 90 days)
VIDRAFT's further vision is to build a unified NPU data center that simultaneously performs serving and continuous learning on the same set of hardware, no longer relying on GPUs.
The hardware architecture of inference-only NPUs is optimized around forward operators such as convolution, GEMM, and activation functions, and lacks the instructions and memory pathways needed for backpropagation, such as element-wise residual gradient backflow, LayerNorm backward statistics, and optimizer momentum state storage.
VKORE's implementation approach is to recombine, in software, the training compute primitives missing from inference-only NPUs (such as gradient computation and weight updates) based on the low-level operations the NPU itself supports, and to maximize hardware utilization through scheduling.
VKORE's functional positioning is analogous to what CUDA is to NVIDIA GPUs, sitting between application-layer ML frameworks and NPU hardware, but the specific implementation details (whether it includes a compiler, kernel libraries, runtime scheduler) have not been disclosed.
VIDRAFT released VKORE, an AI acceleration software layer for NPUs, which through software means gives NPUs originally capable only of inference the ability to train, without replacing hardware.
According to ZDNet Korea, VIDRAFT officially announced: on the same NPU, after enabling VKORE, single-user inference latency improved 3.8x, concurrent server throughput improved 4.1x, and training throughput was about 170x compared with a CPU baseline.
Under VKORE acceleration, the single-request response latency of the same NPU is 3.8 times faster than the baseline NPU, and the server-level concurrent throughput is 4.1 times higher.
VKORE enabled LoRA Transformer training to be successfully completed on an NPU, which was previously infeasible on an NPU.
An NPU equipped with VKORE reaches about 75% of a single B200's pretraining throughput.
An NPU equipped with VKORE reaches 170% of a single H100's per-device pretraining throughput.
40 more timeline events
All Facts (20)
Local AI (on-device AI) runs model inference directly on end-user devices, offering advantages such as privacy protection, low latency, offline availability, and cost reduction.
85%VerifiedThe NPU is a hardware accelerator designed specifically for AI inference computation; the Hexagon NPU in Qualcomm's Snapdragon 8 series, Apple's Neural Engine, and MediaTek's APU have all been integrated into mobile chips.
80%VerifiedNPUs are specifically optimized for low-precision (INT8/FP16) matrix multiplication and can achieve similar or higher throughput at a fraction of a GPU's power consumption when running quantized lightweight inference models.
75%VerifiedIntel began integrating NPU units starting with the Core Ultra series processors.
70%VerifiedChip manufacturers such as Qualcomm and MediaTek have integrated dedicated NPUs (Neural Processing Units) into mobile SoCs.
70%VerifiedCopilot+ PCs require the device's NPU computing power to reach more than 40 TOPS.
65%VerifiedThe Neural Engine in Apple's custom chips is an NPU optimized specifically for matrix multiplication and convolution operations, and offers higher energy efficiency than general-purpose CPUs and GPUs on specific AI inference tasks.
65%VerifiedThe NPU built into Huawei's Kirin chip is optimized specifically for AI tasks, achieving several times better energy efficiency compared to CPU/GPU solutions.
65%Verified苹果M系列芯片中的Neural Engine本质上属于NPU
65%Unverified高通和联发科等芯片厂商最新旗舰芯片的NPU算力已足以运行数十亿参数的模型，为手机AI Agent本地化运行提供硬件基础
80%UnverifiedYOLOv8-nano模型在中端NPU上可实现30FPS以上的推理帧率
75%UnverifiedAI推理阶段依赖专用AI加速芯片，包括云端的NVIDIA GPU、Google TPU和终端的手机NPU
70%Partially VerifiedAn NPU (neural processing unit) is a processor designed specifically for neural network inference and training tasks, and its architecture typically includes large multiply-accumulate arrays (MAC arrays), on-chip caches, and dedicated data scheduling units.
65%UnverifiedVKORE is a software acceleration layer for NPUs, designed to be hardware-agnostic, enabling data centers to mix GPUs and NPUs and still train and serve inference from the same stack.
60%UnverifiedNPUs are designed specifically for tensor operations in deep learning inference, achieving high TOPS/W efficiency through systolic array architecture, on-chip SRAM caching, and low-precision computation.
60%UnverifiedNPUs are several times more energy-efficient than general-purpose CPUs when executing matrix operations.
60%UnverifiedQualcomm Snapdragon X series laptop chips have a built-in NPU.
60%UnverifiedIntel's Meteor Lake and subsequent architectures, AMD's Ryzen AI series, and Apple's Neural Engine all integrate NPU modules.
60%UnverifiedVKORE's functional positioning is analogous to what CUDA is to NVIDIA GPUs, sitting between application-layer ML frameworks and NPU hardware, but the specific implementation details (whether it includes a compiler, kernel libraries, runtime scheduler) have not been disclosed.
50%UnverifiedVKORE's implementation approach is to recombine, in software, the training compute primitives missing from inference-only NPUs (such as gradient computation and weight updates) based on the low-level operations the NPU itself supports, and to maximize hardware utilization through scheduling.
50%