Hibana
An open-source AI inference engine designed for unified-memory devices, jointly optimizing through model surgery, unified memory management, and adaptive speculative decoding, supporting local multi-agent workloads.
Timeline (last 90 days)
Hibana is an inference engine designed specifically for unified-memory devices, jointly optimizing model representation, memory management and decoding strategy.
Hibana adopts Model Surgery, i.e., editing the architecture of an already-trained model to adapt it to the hardware, thereby reducing memory footprint and data movement, rather than simple quantization or pruning.
Hibana's unified memory management can preserve each agent's state across turns, allowing agents to resume instantly without recomputing state.
Hibana's Adaptive Speculation technology lets the model generate multiple tokens per step and dynamically adjust according to load and hardware state, maximizing throughput under limited bandwidth.
When serving Qwen and Nemotron models with up to 125B parameters on a single DGX Spark, Hibana completes up to 3.4 times more agent tasks per hour and achieves up to 3.9 times higher decoding throughput than vLLM, NVIDIA Flare, llama.cpp and TensorFold.
Hibana's code has been open-sourced on GitHub at https://github.com/sharar-project/hibana.
DGX Spark is only one form of unified-memory device; the bandwidth and capacity of devices such as laptops and phones differ even more, so Hibana's speedup cannot simply be extrapolated to other device forms.
Hibana has currently only released test results on DGX Spark, and its performance on other device forms such as laptops and phones remains to be verified.
All Facts (8)
Hibana has currently only released test results on DGX Spark, and its performance on other device forms such as laptops and phones remains to be verified.
50%UnverifiedHibana adopts Model Surgery, i.e., editing the architecture of an already-trained model to adapt it to the hardware, thereby reducing memory footprint and data movement, rather than simple quantization or pruning.
50%UnverifiedHibana's unified memory management can preserve each agent's state across turns, allowing agents to resume instantly without recomputing state.
50%UnverifiedHibana's Adaptive Speculation technology lets the model generate multiple tokens per step and dynamically adjust according to load and hardware state, maximizing throughput under limited bandwidth.
50%UnverifiedWhen serving Qwen and Nemotron models with up to 125B parameters on a single DGX Spark, Hibana completes up to 3.4 times more agent tasks per hour and achieves up to 3.9 times higher decoding throughput than vLLM, NVIDIA Flare, llama.cpp and TensorFold.
50%UnverifiedHibana is an inference engine designed specifically for unified-memory devices, jointly optimizing model representation, memory management and decoding strategy.
50%UnverifiedDGX Spark is only one form of unified-memory device; the bandwidth and capacity of devices such as laptops and phones differ even more, so Hibana's speedup cannot simply be extrapolated to other device forms.
50%UnverifiedHibana's code has been open-sourced on GitHub at https://github.com/sharar-project/hibana.
50%