SonicBoom v0.0.3: How a Compiler-First AI Runtime Beats llama.cpp

An Inference Runtime Without Python

HardenedLinux is a long-standing free software community, and SonicBoom, which they developed, is an open-source inference runtime under the GPLv3 license. The biggest difference from mainstream solutions: it adopts a compiler-first design, with no Python dependency throughout the build and run process, delivering only a shared library and a C ABI, with Guile Scheme as a first-class language.

This positioning directly addresses a pain point for many developers — although the Python stack has a flourishing ecosystem, it often brings extra overhead in deployment and performance tuning. SonicBoom attempts to prove that bringing compiler thinking into the inference runtime can yield considerable performance gains while maintaining output correctness.

Inference runtime is the software layer responsible for loading trained model weights, executing forward computation, and generating output, typically sitting between model files and hardware drivers. Mainstream solutions such as llama.cpp, vLLM, and ONNX Runtime mostly rely on Python for model conversion, graph construction, and scheduling; Python's interpreted execution and the GIL introduce startup latency and deployment complexity. C ABI (Application Binary Interface) is the binary-level calling convention that allows other languages (C/C++, Rust, Go, etc.) to directly link the shared library without Python bindings. Guile Scheme is the GNU project's Scheme dialect implementation, often used as an embedded scripting engine; here, being a first-class language means users can directly write inference logic and extensions in Scheme, rather than using it merely as a configuration layer.

v0.0.3 Performance: Surpassing llama.cpp in the Decode Phase

On a desktop-class RTX 3050 8GB GPU, using the same GGUF model and the same session, SonicBoom v0.0.3 was directly compared with llama.cpp (-ngl 99). Official results show that SonicBoom achieved better performance than llama.cpp in the decode phase.

More noteworthy is the improvement under multi-turn agent workloads. Since v0.0.2, thanks to prefix reuse and plan caching, time-to-first-token has dropped several-fold. The team emphasizes that output tokens are bit-exact with the golden run results, and the results of prefix reuse and full prefill also remain bit-identical — the performance gains did not come at the expense of correctness.

Time-to-first-token (TTFT) measures the latency from request issuance to the generation of the first output token, primarily determined by the prefill phase, and has the greatest impact on interactive conversation experience; the decode phase generates tokens autoregressively one by one, focusing on throughput and per-token latency. Prefix reuse (prefix caching) means that in multi-turn conversations, the shared system prompt and historical context are computed into KV Cache only once, and subsequent turns reuse it directly, greatly shortening TTFT. Plan cache caches the execution plan generated at compile time, avoiding repeated graph optimization. Bit-exact requires that the floating-point results of two runs be completely identical, a strict standard for verifying that optimizations have not changed numerical semantics.

Boom IR and ggml Kernels: Where Performance Comes From

SonicBoom's performance comes from its compiler architecture. All graph optimization and kernel generation happen on Boom IR, an MLIR dialect; the hot quantized matrix multiplication runs as ggml-derived kernels with attributes.

According to the author, every optimization comes with a toggle and ablation data, so users can clearly see the actual benefit each optimization brings. This transparent approach is uncommon in the inference runtime space and is of high reference value to researchers and performance engineers.

MLIR (Multi-Level Intermediate Representation) is reusable compiler infrastructure under the LLVM project that allows defining custom dialects at multiple levels of abstraction and completing high-level graph optimization and low-level code generation within the same framework. Boom IR is the MLIR dialect defined by SonicBoom, where tensor operators, fusion, and scheduling decisions are expressed and rewritten, thereby moving the graph optimization traditionally done at inference engine runtime forward to compile time. ggml is the lightweight tensor library developed by the author of llama.cpp, providing low-level kernels such as quantized matrix multiplication for CPU/GPU; SonicBoom reuses its quantization kernels but attaches attribute metadata so the compiler can recognize and schedule them. Ablation data refers to the performance difference measured after disabling a given optimization one at a time, used to quantify each optimization's independent contribution.

Honest Gaps and an Invitation for Community Benchmarks

The team does not avoid the current shortcomings: in the Qwen model depth-512 decode scenario, SonicBoom is only 0.98x that of llama.cpp, and validation has so far been completed on only a single GPU. For this reason, they have issued an invitation to the community for benchmark testing.

Each release sets up a benchmark issue. Users with NVIDIA GPUs (any generation, with Ada, Hopper, and Blackwell especially welcome) can generate a paste-ready report with a single command. All contributors whose results are included will be credited in the README. The benchmark issue for v0.0.3 is now open on GitHub, and the team requests that participants submit both results and issues to that issue in a unified manner for centralized management.

Significance for the Open-Source Inference Ecosystem

SonicBoom represents a differentiated path for inference runtimes: replacing the interpreter-style Python stack with compiler infrastructure, providing scripting capability with Scheme, and ensuring embedding flexibility with a C ABI. If community benchmarks can cover more GPU generations and model sizes, the practical scope of this approach will become clearer.

For developers focused on inference performance, lightweight deployment, and compiler technology, participating in benchmark testing is both a contribution and an opportunity to learn early about this runtime's real-world performance.

One-line Summary

SonicBoom, an open-source inference runtime with no Python dependency and Guile Scheme as a first-class language, releases v0.0.3, surpassing llama.cpp in decode performance.

Paragraph Summary

SonicBoom, developed by the HardenedLinux community, is a GPLv3 open-source inference runtime with a compiler-first design, no Python dependency throughout build and run, delivering only a shared library and C ABI, with Guile Scheme as a first-class language. In v0.0.3, compared with llama.cpp on an RTX 3050 8GB using the same GGUF model, it performs better in the decode phase; thanks to prefix reuse and plan caching, time-to-first-token under multi-turn agent workloads drops several-fold versus v0.0.2, and outputs are bit-exact with golden results. Its performance comes from graph optimization and kernel generation based on the MLIR dialect Boom IR, with each optimization accompanied by a toggle and ablation data. The team acknowledges that Qwen depth-512 decode is only 0.98x that of llama.cpp and that only a single GPU has been validated, and is openly soliciting benchmarks across all NVIDIA generations.