VKORE: Running LoRA Training on Inference NPUs, Boosting Inference Speed by 3.8x

Inference-dedicated NPU clusters typically only handle forward computation, and fine-tuning tasks still require a separate GPU or CPU environment. VIDRAFT's VKORE software acceleration layer aims to change this: running LoRA training on inference NPUs while boosting single-user inference speed by 3.8x and server throughput by 4.1x.

What Is VKORE

VKORE is a software acceleration layer developed by VIDRAFT that sits between ML workloads and the underlying NPU hardware. It is not tied to any single chip vendor, but rather targets heterogeneous data centers running both GPUs and NPUs.

It has three core characteristics:

• Hardware-agnostic design: Adapts to mixed-compute clusters rather than optimizing for a single vendor's silicon.
• Dual workload support: On the same NPU, it handles both inference acceleration and LoRA training—exactly the capability missing from traditional inference-only NPU deployments.
• Validation model: Benchmarked using Google's open-source Gemma 4 26B model on an eight-card, 32-chip Korean NPU server.

The real problem it aims to solve is architectural: data centers no longer need to maintain a separate GPU cluster year-round for periodic fine-tuning tasks. Inference NPU fleets can absorb these training workloads during off-peak hours.

Architectural differences between NPUs and GPUs. An NPU (Neural Processing Unit) is a specialized accelerator designed for specific machine learning operators (especially matrix multiplication and convolution), typically trading low-precision fixed-point or floating-point data paths for higher energy efficiency and performance-per-watt. Its typical use case is large-scale inference deployment. A GPU, by contrast, is a general-purpose parallel processor with a more flexible programming model (such as CUDA) and a more complete floating-point and memory hierarchy, which is why it has long shouldered training tasks. The most fundamental difference between the two: NPU instruction sets and data paths are often fixed, primarily covering the operators needed for forward inference, and lacking training primitives such as backpropagation and gradient reduction. This is the hardware root cause of the limitation that "inference-only NPUs cannot train." VKORE's approach is precisely to fill in these missing primitives at the software level rather than redesigning the chip.

LoRA (Low-Rank Adaptation). LoRA is a parameter-efficient fine-tuning (PEFT) method. Instead of updating all the weights of a pre-trained model, it attaches a pair of low-rank matrices A and B alongside certain linear layers, decomposing the weight update ΔW into the product B·A, where the rank r is much smaller than the original weight dimensions. During training, only these two small matrices are optimized, and the number of trainable parameters often drops to one-thousandth of the original model. The benefits are threefold: low memory footprint, short training time, and small, pluggable adapter files. It is precisely because the training overhead is so low that LoRA is suitable for scheduling onto idle compute in inference clusters—this is also the premise that makes VKORE's approach economically viable.

What numerical consistency of 0.9999 means. This figure typically refers to the correlation coefficient or element-wise match ratio between the output tensors of two compute backends (VKORE and PyTorch) given the same input. Different rounding orders, accumulation precision, and FMA (fused multiply-add) strategies in floating-point operations lead to slight interval differences in results, so 1.0 is nearly impossible to achieve. 0.9999 represents an extremely high level of consistency, but it is not bit-exact, meaning that downstream applications requiring strict reproducibility still need to verify rather than trust directly.

How It Works

Conceptually, VKORE acts as an abstraction and optimization layer that intercepts the computational graph of transformer model operations and remaps them onto the NPU instruction set—including the gradient computation paths needed for LoRA training.

Making inference chips LoRA-compatible. Standard NPU firmware and driver stacks are typically optimized only for forward computation (matrix multiplication, attention, softmax). VKORE extends the executable operation set to cover the backpropagation primitives needed for LoRA adapter training, and is described as requiring no NPU firmware modifications.

Numerical fidelity. Achieving 0.9999 consistency with PyTorch means VKORE carefully manages floating-point operation ordering and precision handling. The practical significance of this figure is that LoRA adapter weights trained through VKORE should be functionally equivalent to those produced by standard PyTorch GPU training.

Idle fleet utilization. LoRA fine-tuning is itself a parameter-efficient method that updates only a small number of low-rank matrices rather than full weights. Its memory and compute overhead is small enough to be scheduled onto inference NPUs during off-peak hours without interfering with SLA-constrained inference capacity.

Hardware abstraction. Whether the underlying hardware is an NPU or GPU, VKORE exposes a consistent interface, simplifying fleet management in mixed environments.

Benchmarks and Results

The following figures all come from public reports. The test platform was an eight-card, 32-chip Korean NPU server, and the model was Gemma 4 26B.

Metric	Result
Single-user inference speedup (vs. baseline NPU)	3.8×
Server throughput speedup (vs. baseline NPU)	4.1×
LoRA training speed (vs. same-server CPU)	~170×
Numerical consistency with PyTorch (LoRA training)	0.9999

A few points worth engineers' attention:

• The 3.8× and 4.1× figures were measured on the same NPU hardware, with gains coming entirely from VKORE's software optimization rather than a hardware upgrade.
• The 170× CPU comparison targets teams currently using CPU nodes for fine-tuning to save costs—the signal is clear: going the NPU route via VKORE is not just "good enough," it is far faster.
• The 0.9999 consistency figure is especially critical for production-grade ML pipelines. Training reproducibility and cross-compute-backend output consistency are the foundation of model validation and compliance processes.

How to Try It

As of the original article's writing, public channels had not mentioned a VKORE Hugging Face repository, GitHub release, pip package, or API endpoint. Integration details remain undisclosed.

If you are interested in evaluating VKORE, the appropriate next step is to contact VIDRAFT directly through official channels and watch for whether it subsequently releases a public SDK, early access program, or developer preview.

For teams wanting to evaluate this kind of software layer early, the industry-typical path is: first reproduce the benchmarks with your own models on a heterogeneous test cluster, then observe whether its API aligns with mainstream training frameworks (such as PyTorch and Hugging Face PEFT)—the latter determines how much code your existing fine-tuning pipeline needs to change. The greatest risk to watch for is the lock-in risk common to closed-source acceleration layers: once the training path depends on a vendor's proprietary runtime, the portability of model weights and adapters, as well as the migration cost to other hardware in the future, must be clearly assessed before procurement. VIDRAFT has not yet disclosed a code repository, and this risk currently cannot be ruled out through practical verification.

FAQ

Does VKORE require modifying NPU firmware or drivers? Based on public descriptions, VKORE is a software layer above the hardware and does not require firmware changes; its approach is to abstract the hardware interface rather than replace it, consistent with its hardware-agnostic positioning.

Can LoRA adapters trained on NPUs via VKORE be used directly in standard PyTorch or Hugging Face inference pipelines? The 0.9999 numerical consistency strongly suggests yes, but VIDRAFT has not yet published explicit interoperability documentation, and teams should verify on their own before relying on it in production.

Does VKORE only support transformer architectures? The original article only mentions transformer models and the Gemma 4 26B benchmark. Support for other architectures such as CNNs and SSMs is not mentioned in public materials.

One-line Summary

VIDRAFT's VKORE software layer enables inference-only NPUs to run LoRA training while significantly boosting inference performance.

Paragraph Summary

VIDRAFT's VKORE is a software acceleration layer positioned between ML workloads and NPU hardware, aiming to enable inference-only NPUs—previously capable only of forward computation—to support LoRA fine-tuning. It features a hardware-agnostic design targeting heterogeneous data centers mixing GPUs and NPUs, and was benchmarked using Google's open-source Gemma 4 26B model on an eight-card, 32-chip Korean NPU server. Results show 3.8x single-user inference speedup and 4.1x server throughput improvement, with all gains coming from software optimization rather than hardware upgrades; LoRA training speed is roughly 170x that of the same server's CPU, with 0.9999 numerical consistency with PyTorch, meaning trained adapter weights are theoretically functionally equivalent to GPU results. Its core value lies in eliminating the need for data centers to maintain a separate GPU cluster year-round for periodic fine-tuning—off-peak inference NPU fleets can absorb these training workloads. Currently VKORE has no public SDK or code repository; integration requires contacting VIDRAFT directly.