Lucebox
An open-source local large-model inference acceleration project based on speculative decoding. It supports mixed execution across heterogeneous consumer-grade GPUs (NVIDIA, AMD, Apple Silicon, CPU), is implemented in native C++, and provides an OpenAI-compatible API.
Timeline (last 90 days)
Lucebox is an open-source project that chains together the idle compute of heterogeneous hardware via speculative decoding to increase inference speed by 1.5 to 3 times without sacrificing output quality.
Lucebox splits the draft model and the target model to run on different hardware; in the official example, Llama-3.2-1B runs as the draft model on a second GPU or the CPU, and Llama-3.1-70B runs as the target model on the main card.
Lucebox supports mixing hardware backends, with NVIDIA RTX, AMD ROCm, Apple Silicon and pure CPU able to appear in the same inference pipeline.
Lucebox is implemented in native C++, focusing on low runtime overhead and high concurrency, avoiding the scheduling overhead common in Python inference frameworks.
When deploying Lucebox, one must compile with CMake in Release mode, specify model paths and device numbers at startup via --target-model, --draft-model, --target-device and --draft-device, and the service exposes an OpenAI-compatible /v1/chat/completions endpoint by default.
Lucebox requires models in GGUF format, and the vocabularies of the draft model and target model must be compatible, otherwise the acceptance rate of speculative verification drops sharply and the speedup is out of the question.
The 1.5x–3x speedup given in the documentation is a range under ideal conditions; actual gains depend heavily on how often the draft model guesses the target model's output correctly—the more routine the task and the more predictable the text, the higher the acceptance rate, while creative writing or highly specialized domain Q&A may see discounts.
If there is only a single card, or the model size is itself within a single card's memory, the benefit of using Lucebox may be limited.
All Facts (8)
Lucebox is an open-source project that chains together the idle compute of heterogeneous hardware via speculative decoding to increase inference speed by 1.5 to 3 times without sacrificing output quality.
50%UnverifiedLucebox splits the draft model and the target model to run on different hardware; in the official example, Llama-3.2-1B runs as the draft model on a second GPU or the CPU, and Llama-3.1-70B runs as the target model on the main card.
50%UnverifiedLucebox supports mixing hardware backends, with NVIDIA RTX, AMD ROCm, Apple Silicon and pure CPU able to appear in the same inference pipeline.
50%UnverifiedLucebox is implemented in native C++, focusing on low runtime overhead and high concurrency, avoiding the scheduling overhead common in Python inference frameworks.
50%UnverifiedWhen deploying Lucebox, one must compile with CMake in Release mode, specify model paths and device numbers at startup via --target-model, --draft-model, --target-device and --draft-device, and the service exposes an OpenAI-compatible /v1/chat/completions endpoint by default.
50%UnverifiedLucebox requires models in GGUF format, and the vocabularies of the draft model and target model must be compatible, otherwise the acceptance rate of speculative verification drops sharply and the speedup is out of the question.
50%UnverifiedThe 1.5x–3x speedup given in the documentation is a range under ideal conditions; actual gains depend heavily on how often the draft model guesses the target model's output correctly—the more routine the task and the more predictable the text, the higher the acceptance rate, while creative writing or highly specialized domain Q&A may see discounts.
50%UnverifiedIf there is only a single card, or the model size is itself within a single card's memory, the benefit of using Lucebox may be limited.
50%