Hibana: A High-Speed Agent Inference Engine for Unified Memory Devices
Hibana is a dedicated inference engine for unified memory devices, co-designing three technologies to boost local agent throughput.
The article introduces Hibana, an inference engine designed for unified memory devices like DGX Spark. While mainstream engines such as vLLM and SGLang target data-center GPUs with separate host memory, local engines fail to fully exploit unified memory's hierarchy—a gap Hibana addresses. It co-designs three technologies: model surgery to shrink memory footprint and data movement; unified memory management to preserve agent state across turns; and adaptive speculation to generate multiple tokens per step. On a single DGX Spark running Qwen and Nemotron models up to 125B parameters, Hibana achieves up to 3.4× more agent tasks per hour and 3.9× higher decoding throughput than vLLM, NVIDIA Flare, llama.cpp, and TensorFold. The code is open-sourced on GitHub, though performance on laptops and phones remains unverified.
Why Local AI Needs a Dedicated Inference Engine
The vision of local AI is to let personal assistants run on users' own devices, with data and computation fully under their control. Open-weight models can already handle practical tasks like programming, but inference engines are holding things back. Mainstream engines like vLLM and SGLang are designed for data-center GPUs and rely on separate host memory; meanwhile, local engines fail to fully exploit the memory hierarchy of unified memory devices. This gap gave rise to Hibana—an inference engine purpose-built for unified memory devices that co-optimizes model representation, memory management, and decoding strategy.
Unified Memory is the key hardware background for understanding this article. On traditional data-center GPUs (such as NVIDIA H100), the GPU has its own dedicated memory (HBM), and the CPU and GPU are connected via PCIe/NVLink. Data must be explicitly moved between host memory and GPU memory, and inference engines (such as vLLM's PagedAttention) schedule precisely around this two-level "GPU memory–host memory" hierarchy. Unified memory SoCs (such as the GB10 in NVIDIA DGX Spark or Apple Silicon's M-series chips) let the CPU and GPU share the same physical memory, eliminating explicit copies—but this also means model weights, KV cache, and agent state all compete for the same capacity and the same bandwidth. This invalidates the memory management assumptions of traditional engines: you can't free up space by "swapping cold data out to host memory" (because it's the same memory), and you must control data movement more carefully to save bandwidth. Hibana's design starts from exactly this hardware constraint.
Co-Design of Three Core Technologies
Hibana's core idea is to co-design three technologies around the shared memory capacity and limited bandwidth of unified memory SoCs, in order to support local multi-agent workloads.
Model Surgery: Changing How Computation Works to Shrink the Memory Footprint
Model Surgery refers to editing the architecture of an already-trained model to fit the hardware. Hibana uses model surgery to change how the model computes, thereby reducing memory usage and data movement. This is not simple quantization or pruning, but rather adapting the model at the architectural level to the characteristics of unified memory devices and reducing pressure on bandwidth.
Unified Memory Management: Preserving Agent State Across Turns
Long-running agents need to maintain state across multiple turns of conversation. Hibana's unified memory management preserves each agent's state across turns, allowing agents to resume instantly instead of recomputing state. This is critical for local multi-agent scenarios that require continuous interaction, avoiding the latency and energy cost of redundant computation.
Adaptive Speculation: Generating Multiple Tokens Per Step
Adaptive Speculation lets the model generate multiple tokens per step, thereby improving decoding throughput. Unlike speculative decoding with a fixed strategy, Hibana's adaptive mechanism dynamically adjusts based on workload and hardware state, maximizing throughput under limited bandwidth.
Speculative Decoding is the technical foundation of adaptive speculation. Its principle is to use a lightweight "draft model" or additional prediction head to quickly generate several candidate tokens, which the target large model then verifies in parallel in a single pass, accepting the correct ones. Because the large model's forward computation during verification is parallel, the overall effect is to obtain multiple output tokens for a single large-model forward pass, breaking through the throughput bottleneck of autoregressive decoding where "only one token can be produced per step." The key is the draft acceptance rate: the higher the acceptance rate, the greater the speedup—and the acceptance rate fluctuates with task type, context, and hardware load. Speculative decoding with fixed length and a fixed draft strategy tends to fail when the load changes, which is exactly why Hibana emphasizes "adaptive"—dynamically adjusting the number and strategy of candidates generated per step to maintain high effective throughput under the limited bandwidth of unified memory.
Performance: Measured on a Single DGX Spark
When serving Qwen and Nemotron models with up to 125B parameters on a single DGX Spark, Hibana completes up to 3.4× more agent tasks per hour and achieves up to 3.9× higher decoding throughput compared with frameworks such as vLLM, NVIDIA Flare, llama.cpp, and TensorFold. These figures come from official descriptions and demonstrate the significant gains delivered by co-design on unified memory devices.
Interestingly, Hibana's code is already open-sourced on GitHub (https://github.com/sharar-project/hibana) for developers to reproduce and further optimize.
DGX Spark is a small desktop-class AI computing device from NVIDIA, powered by the GB10 Grace Blackwell superchip and built on a unified memory architecture in which the CPU and GPU share a large memory capacity (on the order of about 128GB), capable of running models with tens to hundreds of billions of parameters locally. Positioned between a workstation and an edge device, it is a typical embodiment of the "local AI" vision. This is also why comparing Hibana with engines like vLLM and llama.cpp on DGX Spark is representative: the latter are mostly optimized for data-center GPUs or pure-CPU scenarios and struggle to unlock the hardware's potential on unified memory devices. Note that DGX Spark is only one form of unified memory device; laptops, phones, and other devices vary much more widely in bandwidth and capacity, so the speedup ratios mentioned here cannot be simply extrapolated.
Significance for the Local AI Ecosystem
Hibana fills a gap in local inference engines. It does not aim to top benchmarks on data-center GPUs, but focuses on the real bottlenecks of unified memory devices—memory capacity and bandwidth. Through the triple co-design of model surgery, state preservation, and adaptive speculation, it provides viable infrastructure for local multi-agent workloads. As open-weight models grow more capable, such specialized engines are expected to make personal AI assistants truly practical, delivering a near-cloud experience while protecting privacy.
However, Hibana has so far only published test results on DGX Spark, and its performance on other device forms (such as laptops and phones) remains to be verified. Its open-source strategy will accelerate community exploration, but ecosystem building will still take time.
One-line Summary
Hibana builds a dedicated inference engine for unified memory devices, co-designing three technologies to significantly boost local agent throughput.
Paragraph Summary
This article introduces Hibana, an inference engine designed specifically for unified memory devices such as DGX Spark. Existing mainstream engines like vLLM and SGLang target data-center GPUs and rely on separate host memory, while local engines fail to fully exploit the memory hierarchy of unified memory—a gap that gave rise to Hibana. It solves the problem through three co-designed technologies: model surgery reduces the memory footprint and data movement at the architectural level; unified memory management preserves each agent's state across turns, avoiding redundant computation; and adaptive speculation generates multiple tokens per step to improve decoding throughput. When running Qwen and Nemotron models with up to 125B parameters on a single DGX Spark, Hibana achieves up to 3.4× more agent tasks per hour and up to 3.9× higher decoding throughput compared with vLLM, NVIDIA Flare, llama.cpp, and TensorFold. The code is open-sourced on GitHub, but its performance on other device forms such as laptops and phones remains to be verified.
Related articles
Anthropic Model Benchmarking: Fable 5 vs 5.1 vs 5.2 Comparison
Based on Reddit community comparison material, this analyzes benchmark differences among Anthropic models Fable 5, 5.1, and 5.2, and discusses the practical implications of version iterations for developers and model selection advice.
Jensen Huang Steps Down as Nvidia CEO, Asian Governments Scramble to Respond to Chip Supply Shifts
Jensen Huang's departure as Nvidia CEO triggers chain reactions across Asian governments: some confirm chip supply commitments, others monitor local AI project impacts and handle related investigations. This article analyzes supply chain stability, project risks, and compliance uncertainty.
Reddit Player Builds RISK Game Environment and Trains an AI That's Nearly Impossible to Beat
Reddit player builds a RISK training environment and trains an aggressive AI model that's nearly unbeatable. Strong at expansion but weak at defense, the author hits a bottleneck trying human data training, revealing common game AI development challenges.