Training from Scratch on an Inference-Only NPU: VKORE-46M-NPU Achieves Real Backpropagation

In the world of AI hardware, there is a long-standing technical boundary that has been taken for granted: NPUs (Neural Processing Units) are designed exclusively for inference and cannot handle training. The latest experiment from the VIDRAFT team directly challenges this assumption—they completed full pretraining of a language model from scratch on an inference-only NPU, without using any GPU.

Why Training on an Inference-Only NPU Is So Difficult

The design logic of an inference NPU is almost the exact opposite of what training requires. Its forward graph is statically compiled, weights are treated as constants, and computation precision is kept very low—all optimizations point toward the same goal: making a single forward pass as fast and as cheap as possible.

Training requires a different set of things: a backward graph that saves activation values, optimizer states (such as momentum buffers), safeguards for numerical stability with small gradients, and gradient synchronization across multiple chips. None of these requirements are natively supported in an inference NPU's architecture. For this reason, "NPUs can't do backpropagation" once became an industry consensus.

VIDRAFT's work proves that this consensus is not unbreakable. They executed backpropagation directly on the NPU, training from randomly initialized weights rather than fine-tuning based on any pretrained weights.

Static compilation refers to the process where, before model deployment, the compiler freezes the entire forward computation graph into an immutable execution plan. It determines the operator implementation, memory addresses, and parallel scheduling strategy for every layer, and deeply optimizes them for the target chip, thereby eliminating runtime scheduling overhead. The cost is that the graph structure cannot be dynamically adjusted during runtime—this is an advantage for inference that only runs forward passes, but during training, backpropagation needs to regenerate a gradient computation graph based on the forward graph, and a statically compiled graph cannot support this dynamic reconstruction. VIDRAFT's approach is to define an additional compilation path for the backward graph at the software layer for the NPU, so that gradient computation can also be compiled into a static program executable by the NPU.

Optimizer states refer to tensors that must be additionally saved and updated during training beyond the model weights, most typically the first-order momentum (exponential moving average of gradients) and second-order momentum (exponential moving average of squared gradients) in Adam. They correspond one-to-one with weights, so their memory footprint is typically 2 to 3 times the number of model parameters. The on-chip and external memory layout of an inference NPU was originally designed only for loading a single copy of weights, without reserving space and bandwidth for these additional states. To train on an inference-only NPU, memory must be re-planned at the software layer so that optimizer states are persistently stored and read/written each round within the training loop.

Evidence: Same Model, Two Hardware Platforms

The experimental design itself is clean. The same model, same initialization, same data, and same training schedule were run on a 32-chip NPU cluster and a reference GPU respectively.

The results contain two key numbers:

• Loss trajectory correlation reaches 0.99—the training loss curve on the NPU almost matches the GPU reference point by point
• Throughput converges to a comparable level—the gap in tokens/second gradually narrows as training progresses

Looking at a single declining loss curve alone does not prove that training is effective. But a 0.99 correlation means that backpropagation and optimizer updates on the NPU are doing numerically the same work as on the GPU. This is not approximate simulation—it is equivalent computation.

A Single NPU Covers the Entire Pipeline

VIDRAFT encapsulates the entire set of capabilities in a software layer called VKORE, positioned as "CUDA for NPUs"—a hardware-agnostic acceleration layer aimed at reducing dependence on specific GPUs and their ecosystems.

The stages VKORE covers include:

• Pretraining from scratch
• LoRA fine-tuning
• Inference acceleration (3.8x for single-user, 4.1x for server throughput)
• Running quantized models

The significance of this full-pipeline coverage is that enterprises do not need to purchase different hardware separately for training and inference. A single NPU handles both roles, and the logic of hardware procurement changes accordingly.

LoRA fine-tuning, short for Low-Rank Adaptation, is a parameter-efficient fine-tuning method. It freezes all the original weights of a pretrained model and only injects two small matrices (A and B) into the attention layers of the Transformer, using their product to approximate the weight update. The number of trainable parameters is usually only 0.1% to 1% of the original model, so both memory and compute overhead are extremely low. LoRA is suitable for domain adaptation on resource-constrained hardware, but it does not solve the problem of pretraining from scratch—and what VIDRAFT verified is precisely the more difficult step: completing backpropagation and optimizer updates directly on the NPU without relying on any pretrained weights.

Quantization refers to the process of converting model weights and activations from high-precision floating point (such as FP32 or BF16) to low-bit integers (such as INT8 or INT4), with the goal of reducing memory usage, improving throughput, and lowering power consumption. Inference NPUs are inherently designed for low precision, so running quantized models is almost a default capability for them. But training is extremely sensitive to numerical precision; gradients themselves have very small magnitudes, and low precision can easily cause underflow or accumulated rounding errors. VIDRAFT's ability to achieve a 0.99 correlation between the loss trajectory on the NPU and the GPU indicates that it applied additional handling for numerical stability along the training path, rather than simply reusing the low-precision format used during inference.

Why This Matters

The problems with high-end GPU servers are no longer news: high prices, long delivery cycles, and staggering power consumption. For enterprises that need to develop their own models, these costs ultimately translate into competitive disadvantage.

A cheap, low-power NPU that can also train offers another path—enterprises can complete model development in their own data centers, without waiting for GPUs to arrive or sending sensitive data to external clusters. This is especially valuable in industries with strict data compliance requirements (healthcare, finance, government).

It should be made clear that the boundaries of this release are very clear. The publicly released VKORE-46M-NPU is a 46M-parameter, character-level model trained on the TinyStories dataset. Its role is to verify the feasibility of the training path, not to make claims about model quality itself. Scaling to larger models and real data is the next step. The core technology is in the process of patent application, and the training code has not yet been open-sourced.

The model weights and model card have been published on Hugging Face: https://huggingface.co/FINAL-Bench/VKORE-46M-NPU

One-line Summary

VIDRAFT completed language model pretraining from scratch on an inference-only NPU, proving that NPUs can also train.

Paragraph Summary

The VIDRAFT team completed language model pretraining from scratch on a 32-chip inference-only NPU cluster, without using any GPU throughout the process, breaking the industry consensus that "NPUs can only do inference." In the experiment, the same model, same initialization, and same data were trained separately on NPU and GPU. The result was that the NPU loss trajectory reached a 0.99 correlation with the GPU, and throughput also converged to a comparable level, proving that backpropagation and optimizer updates are numerically equivalent. This capability is encapsulated in the VKORE software layer, covering the full pipeline of pretraining from scratch, LoRA fine-tuning, inference acceleration (3.8x for single-user, 4.1x for server), and running quantized models. The public model VKORE-46M-NPU has only 46M parameters and is used to verify the training path rather than to claim model quality; scaling to larger models is the next step.