X-CoSD Framework Explained: How Cross-Vocabulary Collaborative Speculative Decoding Accelerates LLM Inference

Inference efficiency for large language models (LLMs) has long been a focal point for both industry and academia. The X-CoSD framework leverages an innovative cross-vocabulary collaborative speculative decoding technique to significantly reduce communication overhead while preserving generation quality, offering a practical solution for distributed LLM inference.

The Communication Bottleneck in Collaborative Speculative Decoding

Collaborative Speculative Decoding (CoSD) is a distributed LLM inference paradigm: a small language model (SLM) on the device side rapidly generates candidate tokens, while a large model on the server side verifies them. This architecture effectively balances inference speed and quality, but existing approaches suffer from two key limitations.

To understand the value of CoSD, it's important to first grasp its technical foundation — Speculative Decoding. Speculative decoding has been one of the most closely watched techniques in LLM inference acceleration over the past two years. Its core idea stems from a key observation: during autoregressive generation by a large model, only one token is produced at a time, yet the cost of a GPU verifying multiple tokens in parallel is nearly the same as generating a single token. Therefore, a lightweight draft model can quickly generate multiple candidate token sequences, which the large model then verifies in a single parallel pass, accepting or rejecting these candidates. This technique was formally introduced by Google DeepMind in 2023 and has since been widely adopted in mainstream inference frameworks such as vLLM and TensorRT-LLM. CoSD extends this concept to distributed settings, where the on-device small model serves as the draft model and the cloud-based large model serves as the verifier — a natural fit for device-cloud collaborative deployment architectures.

First, the shared vocabulary assumption rarely holds in practice. Traditional CoSD assumes that the SLM and LLM share an identical vocabulary, but in real-world deployments, models of different scales often employ different tokenization strategies and vocabulary designs, making this precondition difficult to satisfy. Specifically, an LLM's vocabulary is the set of minimal language units the model can recognize and generate. Different model families use significantly different tokenization algorithms (such as BPE, WordPiece, Unigram, etc.) and training corpora. For example, the LLaMA series uses a SentencePiece-based BPE tokenizer with a vocabulary size of 32,000, while the Qwen series has a vocabulary exceeding 150,000, providing finer-grained coverage for Chinese text and code. Even models of different sizes within the same family may adopt different vocabulary versions due to training-stage optimizations. This real-world vocabulary heterogeneity means the traditional speculative decoding assumption that "the draft model and target model must share a vocabulary" simply cannot hold in cross-architecture collaboration.

Second, communication overhead is excessively high. The residual resampling process requires frequent exchange of complete token probability distributions between the device and server, creating a severe performance bottleneck in bandwidth-constrained edge computing scenarios. Residual resampling is the key mechanism in speculative decoding for ensuring unbiased output distribution — when a candidate token generated by the draft model is rejected by the target model, a new sample must be drawn from a "residual distribution." The residual distribution is defined as: for tokens where the target model probability p(x) exceeds the draft model probability q(x), the residual probability is proportional to max(0, p(x) - q(x)). This mechanism ensures that the marginal distribution of the final sample strictly equals the target model's original distribution, meaning speculative decoding is "lossless" and introduces no distribution shift. However, computing and transmitting the complete residual distribution requires exchanging a probability vector over the entire vocabulary, and when vocabulary sizes reach tens or even hundreds of thousands, the communication overhead becomes substantial.

This problem is especially acute in edge computing scenarios. Edge computing refers to performing computation and data processing at the network edge close to data sources, with typical scenarios including smartphones, IoT devices, and in-vehicle terminals. Network bandwidth in these scenarios is typically far lower than that within data centers — 4G upload bandwidth is generally only 5–20 Mbps, and 5G in practical use struggles to consistently reach theoretical peaks. If each residual resampling step requires transmitting a complete probability distribution containing tens of thousands of floating-point numbers (at float16, a distribution over a 100K vocabulary is approximately 200KB), the accumulated communication volume across multiple verification rounds will severely drag down end-to-end latency, potentially negating the speed gains from speculative decoding itself.

X-CoSD's Core Innovation: The Hybrid Resampling Mechanism

At the heart of the X-CoSD framework is the Hybrid Resampling (HR) mechanism, which splits the residual resampling process into two regions handled separately:

• Shared vocabulary region: Tokens present in both the SLM and LLM vocabularies — resampling is completed locally on the device
• LLM-exclusive region: Tokens existing only in the LLM's vocabulary — resampling is handled on the server side

The key advantage of this design is that only the probability distribution over the shared vocabulary region needs to be transmitted between the device and server, dramatically reducing communication data volume. The paper provides rigorous mathematical proofs that this hybrid strategy fully preserves the server LLM's original probability distribution, achieving lossless inference.

The reason hybrid resampling can achieve lossless inference relies on a critical mathematical property: the shared vocabulary region and the LLM-exclusive region form a complete partition of the probability space. For any token x, if x exists in both the SLM and LLM vocabularies, its residual resampling can be completed on the device using the locally available p(x) and q(x). If x exists only in the LLM vocabulary, then q(x) = 0, and the residual probability is simply the normalized p(x), requiring only server-side sampling. Since these two regions are non-overlapping and cover the entire vocabulary, the joint distribution after separate processing is equivalent to performing standard residual resampling over the full vocabulary, thus strictly preserving the target model's output distribution. This set-partition-based proof of distribution preservation provides a solid theoretical foundation for the correctness of hybrid resampling.

X-CoSD-E Enhanced Version: Achieving Even Lower Communication Overhead

Building on the standard X-CoSD, the research team further proposed the enhanced version X-CoSD-E, which adopts a "Server Resampling + Device Verification" (SR-DV) strategy. The workflow is as follows:

1. The server samples a replacement candidate token from the LLM
2. The server sends only the candidate token and its corresponding probability to the device
3. The device performs local verification

Compared to standard X-CoSD, the enhanced version no longer requires transmitting a complete probability distribution. Instead, it directly provides the final sampling result and the minimal information set needed for verification. This design elevates communication efficiency to a new level while maintaining theoretical correctness.

From a communication complexity perspective, the SR-DV strategy reduces the communication volume per verification round from O(|V|) to O(1) (where V is the vocabulary size). In the traditional approach, the server needs to send the entire residual distribution to the device for local sampling, with data volume proportional to vocabulary size. Under the SR-DV strategy, the server completes the sampling process directly and only needs to send back a single candidate token ID along with a small amount of auxiliary information (such as acceptance probability), based on which the device makes the final accept/reject decision. This means that regardless of vocabulary size, the downlink communication volume per verification round is constant. This "trade computation for communication" design philosophy — performing more work on the side with abundant computational resources to reduce network transmission — has universal guiding value in distributed system design.

Experimental Results: Dual Validation of Speed and Quality

Experimental results demonstrate that X-CoSD and its enhanced version perform excellently across multiple benchmarks:

• Inference speed: Token generation speed is significantly improved, with communication overhead substantially reduced
• Generation quality: Output text quality is comparable to directly using the server LLM, validating the framework's lossless property
• Applicable scenarios: The advantages are particularly pronounced in bandwidth-constrained edge computing environments

Technical Significance and Future Prospects

The X-CoSD framework offers multi-faceted technical value:

Breaking vocabulary dependency. Models with different architectures and scales can collaborate more flexibly, no longer constrained by the requirement of sharing the same vocabulary. This means enterprises can deploy the small model best suited for local hardware on the device side (such as models optimized for mobile platforms) while using the most powerful general-purpose large model in the cloud — the two need not come from the same model family, greatly expanding the freedom of model selection.

Reducing deployment costs. The substantial improvement in communication efficiency makes edge inference costs more manageable, clearing a significant barrier for large-scale LLM deployment.

Protecting user privacy. Users can obtain an experience close to cloud-based large models on their local devices while reducing sensitive data uploads and lowering end-to-end latency. In traditional purely cloud-based inference architectures, all user inputs (including chat histories, document contents, code snippets, etc.) must be uploaded to the server, raising serious privacy and data compliance risks — particularly in sensitive domains such as healthcare, finance, and law. Regulations like GDPR and China's Personal Information Protection Law impose strict restrictions on cross-border data transfers and cloud-based processing. Under the X-CoSD framework, user input is first processed by the local SLM to generate candidate tokens, with server interaction occurring only during the verification phase — and the exchanged content is primarily token IDs and probability values rather than raw text. This architecture inherently reduces the exposure surface of sensitive data and, combined with techniques like differential privacy, can enable AI service systems that better meet privacy protection requirements.

As on-device AI applications continue to proliferate, cross-vocabulary collaboration mechanisms are expected to see widespread adoption in scenarios such as intelligent assistants, real-time translation, and code completion, becoming a critical foundational component of distributed AI systems.

Key Takeaways