SonicBoom
An open-source inference runtime developed by the HardenedLinux community, designed compiler-first. It is built on the MLIR dialect Boom IR, does not depend on Python, provides a C ABI and first-class support for Guile Scheme, and reuses ggml quantization kernels.
Timeline (last 90 days)
SonicBoom is an open-source inference runtime under the GPLv3 license, developed by the HardenedLinux community.
SonicBoom takes a compiler-first design, does not depend on Python at any point in building or running, ships only a shared library and C ABI, and treats Guile Scheme as a first-class language.
On a desktop-class RTX 3050 8GB graphics card, using the same GGUF model and the same session, SonicBoom v0.0.3 achieved better performance in the decoding phase than llama.cpp (-ngl 99).
Since v0.0.2, thanks to prefix reuse and plan caching, SonicBoom's time to first token (TTFT) has dropped several-fold.
SonicBoom's output tokens are bit-for-bit identical to the golden run results, and the results of prefix reuse and full prefill are also bit-for-bit identical.
All of SonicBoom's graph optimizations and kernel generation happen on Boom IR, which is an MLIR dialect.
SonicBoom's hot quantized matrix multiplication runs as an attribute-bearing ggml-derived kernel.
In a Qwen model decoding scenario at depth 512, SonicBoom's performance is only 0.98 times that of llama.cpp.
SonicBoom has so far been validated only on a single GPU.
Each of SonicBoom's optimizations comes with a toggle and ablation data, so users can clearly see the actual benefit each optimization provides.
2 more timeline events
All Facts (12)
SonicBoom is an open-source inference runtime under the GPLv3 license, developed by the HardenedLinux community.
50%UnverifiedSonicBoom takes a compiler-first design, does not depend on Python at any point in building or running, ships only a shared library and C ABI, and treats Guile Scheme as a first-class language.
50%UnverifiedOn a desktop-class RTX 3050 8GB graphics card, using the same GGUF model and the same session, SonicBoom v0.0.3 achieved better performance in the decoding phase than llama.cpp (-ngl 99).
50%UnverifiedSince v0.0.2, thanks to prefix reuse and plan caching, SonicBoom's time to first token (TTFT) has dropped several-fold.
50%UnverifiedSonicBoom's output tokens are bit-for-bit identical to the golden run results, and the results of prefix reuse and full prefill are also bit-for-bit identical.
50%UnverifiedAll of SonicBoom's graph optimizations and kernel generation happen on Boom IR, which is an MLIR dialect.
50%UnverifiedSonicBoom's hot quantized matrix multiplication runs as an attribute-bearing ggml-derived kernel.
50%UnverifiedIn a Qwen model decoding scenario at depth 512, SonicBoom's performance is only 0.98 times that of llama.cpp.
50%UnverifiedSonicBoom has so far been validated only on a single GPU.
50%UnverifiedEach of SonicBoom's optimizations comes with a toggle and ablation data, so users can clearly see the actual benefit each optimization provides.
50%UnverifiedFor each release, SonicBoom opens a benchmarking issue on GitHub, where users with an NVIDIA GPU can generate a pasteable report with a single command, and all contributors whose results are included are credited in the README.
50%UnverifiedThe benchmark issue for SonicBoom v0.0.3 is now open on GitHub.
50%