Lucebox: Speculative Decoding Acceleration on Heterogeneous Consumer GPUs

Local LLM inference has long been constrained by an implicit assumption: all compute devices should ideally be identical in spec, matched in VRAM, and connected with sufficient interconnect bandwidth. But real-world enthusiast setups are often pieced together—the primary card might be an RTX 4090, the secondary an older card from years ago, plus perhaps an idle integrated GPU. The open-source project Lucebox targets exactly this mismatched scenario: using speculative decoding to chain together the idle compute of heterogeneous hardware, boosting inference speed by 1.5x to 3x without sacrificing output quality.

Core Idea: Small Model Drafts, Large Model Judges

Speculative decoding itself isn't a new concept; its basic premise is simple: use a small, fast draft model to guess several subsequent tokens in a row, then hand them to the large target model for one-shot parallel verification. Correct guesses are accepted directly; from the first divergence point, the rest is rolled back and recomputed. Since verification is parallel, overall latency is significantly lower than token-by-token generation.

Lucebox's differentiation lies in splitting the "draft model" and "target model" across different hardware. In the official example, Llama-3.2-1B runs as the draft on a second GPU or CPU, while Llama-3.1-70B runs as the target model on the primary card. The draft model has low compute demands and fast response, fitting neatly into the idle slot of a low-end accelerator or integrated GPU; the target model monopolizes the primary card's large VRAM and doesn't need to make room for the draft model. This division of labor lets even a mixed machine that "can't handle 70B" achieve usable throughput.

Acceptance Rate and Speedup of Speculative Decoding

The upper bound of speculative decoding speedup can be estimated with a simple model: suppose the draft model guesses γ tokens per round, of which a proportion α (acceptance rate) is accepted by the target model. Then ideally each verification round yields about α·γ+1 tokens (the extra 1 comes from the target model's sampling at the first rejected position). Thus the speedup is roughly bounded by the trade-off between (α·γ+1) and the ratio of draft forward-pass time to target forward-pass time. The higher α and the larger γ, the more pronounced the benefit; but when γ is too large, the draft model's tail guess quality degrades and α drops accordingly, usually requiring experimental tuning. When the draft and target models share the same origin (e.g., both from the Llama family) or the same vocabulary, α tends to be higher—this is the fundamental reason the article emphasizes vocabulary compatibility.

Why Parallel Verification Is Lossless

During verification, the target model performs a single forward pass over the entire candidate token sequence from the draft, obtaining a probability distribution at each position, then compares them one by one using the "accept-reject sampling" rule: if a draft token is accepted under the target distribution, it is adopted; at the first rejected position, that token is resampled from the corrected distribution, and all subsequent draft guesses are discarded. This mechanism guarantees that the final output sequence's distribution is exactly identical to what the target model alone would produce via token-by-token sampling—hence it is lossless acceleration, not approximation or distillation.

Heterogeneous Orchestration: No Need for Identical GPUs or Equal VRAM

The project's most practical feature is that hardware backends can be mixed—NVIDIA RTX, AMD ROCm, Apple Silicon, and pure CPU can all appear in the same inference chain. This is especially friendly for multi-GPU desktop machines: you don't need to buy another card identical to the primary one just to run speculative decoding; old cards, integrated GPUs, and even CPU cores can be enlisted.

From an engineering perspective, the challenge is keeping the host-to-device memory pipeline lightweight enough. Lucebox is implemented in native C++, prioritizing low runtime overhead and high concurrency, avoiding the scheduling overhead common in Python inference frameworks. For developers running local inference as a service, this determines whether it can sustain multiple concurrent requests rather than just a one-off demo.

Engineering Implications of GGUF and Heterogeneous Backends

GGUF is the model file format promoted by the llama.cpp ecosystem, packaging weights, tokenizer vocabulary, and metadata into a single file and supporting various quantization schemes (e.g., Q4_K_M, Q8_0). Quantization significantly reduces VRAM usage but also introduces numerical error, so the draft and target models should ideally use similar quantization strategies; otherwise, the shift between draft and target distributions will lower the acceptance rate. On the heterogeneous backend side, CUDA, ROCm, Metal, and CPU backends each maintain independent VRAM/memory pools and compute streams; cross-device scheduling means inter-layer tensors must be moved between host memory and each device's VRAM. PCIe bandwidth (typically far below NVLink) becomes the bottleneck, so placing the draft model on a device under the same PCIe root complex as the target model, or directly on the CPU, is often more cost-effective than moving across NUMA nodes.

Getting Started: CMake Build, OpenAI-Compatible API

The deployment path is relatively straightforward: clone the repository, build with CMake in Release mode, and at startup specify model paths and device IDs via --target-model, --draft-model, --target-device, and --draft-device. The service by default exposes an OpenAI-compatible /v1/chat/completions endpoint, so existing LangChain pipelines, Ollama, or OpenWebUI frontends can basically be integrated by just changing the address.

The prerequisite to note is that models must be in GGUF format, and the draft and target models' vocabularies must be compatible; otherwise the acceptance rate of speculative verification will drop sharply, and the speedup will be nonexistent. The 1.5x–3x range cited in the documentation is under ideal conditions; actual gains depend heavily on the draft model's "hit rate" on the target model's output—the more routine the task and the more predictable the text, the higher the acceptance rate; creative writing or highly specialized domain Q&A may see discounts.

Who Should Use It, Who Can Wait

Lucebox targets homelab enthusiasts, indie developers, and edge deployment scenarios: people with mixed hardware who want to squeeze every card and aren't willing to pay a premium for a uniform cluster. Its value isn't in reinventing speculative decoding, but in thinning the engineering layer of "heterogeneous hardware orchestration," letting consumer device combinations approach data-center-like latency.

If you only have a single card, or the model size itself fits within a single card's VRAM, gains may be limited; if you're after an out-of-the-box managed experience, this kind of project requiring self-compilation and tuning isn't exactly hassle-free. But for those with a few "useless" old cards lying in the equipment cabinet, Lucebox offers a realistic path to turn idle compute into inference speed.

FAQ

Does Lucebox reduce output quality? No. The verification step of speculative decoding is performed by the target model; accepted tokens are identical to what the target model would generate itself—it's lossless acceleration.

Do I need two discrete GPUs? Not necessarily. The draft model can run on CPU or integrated GPU; as long as that device's inference speed exceeds the target model's token-by-token generation speed, there's room for speedup.

Which model formats are supported? Judging from the startup parameters, it follows the GGUF route, and the draft and target models need compatible vocabularies.

One-line Summary

Lucebox uses speculative decoding to split draft and target models across heterogeneous hardware, achieving 1.5–3x lossless speedup.

Paragraph Summary

Lucebox is an open-source local inference project whose core is using speculative decoding to split the draft model and target model across different hardware: a small model (e.g., Llama-3.2-1B) rapidly guesses tokens on a secondary card, integrated GPU, or CPU, while a large model (e.g., Llama-3.1-70B) verifies them in parallel on the primary card, yielding 1.5–3x speedup without loss of output quality. The project supports mixing NVIDIA, AMD ROCm, Apple Silicon, and pure CPU, and doesn't require identical models or equal VRAM; it's implemented in native C++ to reduce runtime overhead and exposes an OpenAI-compatible endpoint for easy integration into existing local agent pipelines. The prerequisites are GGUF format and compatible draft/target vocabularies, and actual gains depend heavily on the draft model's token acceptance rate—making it better suited for homelab enthusiasts with mixed hardware than for users seeking out-of-the-box simplicity.