Inference-Only NPUs Can Now Train Transformers: VKORE Achieves 0.9999 Precision Alignment

NPUs have long been positioned by the industry as "inference-only" accelerators, with training tasks by default handed off to GPUs. VIDRAFT's VKORE software acceleration layer challenges this division of labor—without modifying the chip itself, it not only boosts NPU inference performance several-fold but also enables LoRA training of Transformers on this class of "inference-only" hardware, with numerical precision reaching 0.9999 alignment with standard PyTorch.

Why Do Training on NPUs

The economics of data centers are shifting. The old thinking was "buy more GPUs"; now it's turning to "extract more intelligence from the same power budget and the same chips." NPUs have a clear energy-efficiency advantage in inference scenarios, so they are rapidly proliferating across inference clusters. But the industry generally views them as hardware that can only do inference, with training considered the exclusive domain of GPUs.

What VKORE aims to break is precisely this dividing line. If NPUs can also take on training tasks, operators won't need to build a separate GPU cluster just for fine-tuning. VIDRAFT's approach is straightforward: since the inference cluster already exists, why not let it work during off-hours too?

Architectural Differences Between NPUs and GPUs

An NPU (Neural Processing Unit) is an application-specific integrated circuit designed specifically for neural network tensor operations, with typical examples including Google TPU, Huawei Ascend, and Apple Neural Engine. Compared to GPUs, NPUs typically sacrifice general programmability in exchange for higher compute-per-watt: they embed large-scale systolic arrays or similar structures that can perform matrix multiply-accumulate at extremely low power, but they lack flexible instruction scheduling and dynamic branching capabilities, and unlike GPUs they don't have a mature CUDA ecosystem, large memory capacity, or high-speed interconnects (such as NVLink). For this reason, NPUs excel at inference—especially forward computation with fixed shapes and low-precision quantization—but have long been considered "outmatched" on training tasks that require backpropagation, gradient accumulation, and optimizer state maintenance. The significance of VKORE is that it attempts to compensate for NPUs' programmability shortcomings through a software layer, rather than waiting for the hardware itself to evolve.

Inference Speedup Measured on the Same NPU

The test platform is a Korean NPU server (8 cards), with Google's open-source Gemma 4 26B as the model. No hardware was modified; only the software layer was replaced.

• Single-user response: improved from 14.1 tokens/s to 53.9 tokens/s, roughly 3.8x.
• Server throughput: improved from 214 tokens/s to 881 tokens/s, roughly 4.1x.
• Under an overload scenario with 64 concurrent users, per-token generation time stabilized at 29ms, with zero failed requests.

The last point matters a lot for operators: when demand exceeds capacity, every user still experiences consistent speed, making service-level commitments easier to fulfill. Interestingly, these gains don't come from hardware upgrades but purely from the software acceleration layer—meaning already-deployed NPU clusters can benefit immediately.

Can Inference-Only NPUs Really Train?

The answer is yes. VIDRAFT performed LoRA training on standard Transformer blocks (attention layers plus feedforward networks) across all 32 chips of a single server, with the following results:

• Precision: 0.9999 agreement with standard PyTorch training.
• Speed: approximately 170x that of the same server's CPU.
• Versus a single NVIDIA B200: approximately 54% of its training throughput, at comparable hardware price.

LoRA works by attaching a small correction layer to an existing model and tuning it for a specific task—this is the most common training approach in enterprise practice. Getting LoRA to run correctly on NPUs means inference clusters can do fine-tuning during off-peak hours instead of sitting idle. From a cost perspective, if an NPU server already purchased for inference can also take on training tasks, its ROI improves markedly.

What LoRA and 0.9999 Training Precision Alignment Mean

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: it freezes the pretrained model's main weights and attaches a pair of low-rank matrices (A and B) alongside each Transformer layer, updating only this small subset of parameters during training, thereby reducing memory and compute requirements by orders of magnitude. The "0.9999 agreement" in the article refers to the numerical correlation coefficient or cosine similarity between the LoRA weights trained on the NPU and those trained with standard PyTorch under the same data and hyperparameters—not the model's final task accuracy. 0.9999 is a very high level of alignment, indicating that the NPU introduces no significant error in backpropagation, gradient computation, mixed-precision accumulation, and similar steps. Note, however, that what's aligned is this small LoRA adapter layer, which is not equivalent to full-parameter pretraining achieving the same precision—full training would impose far more demanding requirements on numerical stability and memory bandwidth.

The Natural Fit Between the ZTC Decision Engine and NPUs

The decision engine returns a judgment result rather than generated text. VIDRAFT's Darwin ZTC completes its judgment in a single forward pass, with zero output tokens. This means a large volume of short, deterministic computations rather than long token-by-token sequence generation—a compute pattern that happens to suit energy-efficient NPUs.

On the public S1MB leaderboard, Darwin-27B-ZTC-v5 ranks first among 152 models. Deploying a top-tier decision engine on NPUs allows the same power budget to serve more judgment requests. In other words, the NPU's energy-efficiency advantage is further amplified in this scenario.

What Is a "Zero Output Token" Decision Engine?

Traditional large-model inference is autoregressive: each generated token requires a complete forward pass, which is then appended to the context before computing the next one, with longer sequences incurring greater overhead. A ZTC (Zero-Token Classification, tentatively named) decision engine, by contrast, reads a certain aggregated vector directly after a single forward pass (such as the [CLS] representation or pooled output of the final layer) and outputs a classification judgment without producing any text tokens. This makes it a discriminative rather than generative model, with constant computation and predictable latency—ideal for scenarios like risk control, routing, and content moderation where "just give me a conclusion" is enough. Precisely because it needs no KV cache and no token-by-token decoding, the NPU's energy-efficiency advantage is fully unleashed, unencumbered by the memory bandwidth bottleneck of the decoding phase.

The Night-Shift Loop: A Data Center's Self-Evolving Closed Loop

Putting the above capabilities together, a data center can run a loop like this:

1. Serve inference requests during the day.
2. Use the decision engine to score the day's traffic and select data worth learning from.
3. Let the same batch of NPUs perform training at night.

The decision engine selects the data, self-improvement drives the model training, and the acceleration layer makes it all run on NPUs. VIDRAFT says the method has been patented, with internal details not yet disclosed, and results will be published as a paper.

Hardware-Agnostic Design

VKORE is positioned as a hardware-agnostic software layer, targeting data centers with mixed GPU and NPU deployments. It isn't tied to any specific chip vendor, preserving flexibility for operators in procurement and scheduling. For enterprises that have already invested heavily in inference clusters, this hardware-agnostic acceleration layer lowers the barrier to experimentation.

That said, the publicly available benchmark data so far comes from a single configuration—a Korean NPU server with an 8-card (32-chip) setup, running Gemma 4 26B. VIDRAFT plans to publish a full measurement methodology paper, at which point the reproducibility of these results on other hardware combinations can be verified.

How a Hardware-Agnostic Layer Works—and Its Limits

A so-called "hardware-agnostic" software layer is typically implemented through an intermediate representation (IR) plus dispatch-style backends: the upper layer describes the model with a unified computation graph, while the lower layer generates corresponding kernels or calls vendor runtimes (such as CUDA, CANN, and various NPU SDKs) for different chips. This lets the same codebase run on GPUs and NPUs, but there are limits: different hardware supports different operator sets, numerical precisions (FP16/BF16/INT8), and memory layouts, and certain optimizations may only take effect on specific architectures. So "hardware-agnostic" refers more to unified interfaces and deployment workflows than to fully portable performance. The fact that the article's benchmarks cover only a single configuration also shows that cross-hardware performance consistency still needs more independent verification.

FAQ

Is VKORE tied to a specific chip vendor? No, it is designed as a hardware-agnostic layer for data centers with mixed GPUs and NPUs.

Does training on NPUs sacrifice precision? In the benchmarks, LoRA training achieved 0.9999 agreement with standard PyTorch.

What model and hardware were used? Google's open-source Gemma 4 26B, running on a Korean NPU server with 8 cards totaling 32 chips.

How does the cost compare to B200? The NPU server achieves approximately 54% of a single B200's training throughput at comparable hardware price, while also being a cluster already purchased for inference.

Where can I find more information? VIDRAFT plans to publish its measurement methodology as a paper.