LatComm: Reusable Context Compression Speeds Up LLM-to-LLM Communication by 3x

Title

LatComm: Reusable Context Compression Speeds Up LLM-to-LLM Communication by 3x

Meta Description

LatComm proposes a reusable context compression method that separates encoding from reading, improving accuracy by 10.2 points across 12 QA datasets, reducing input by 57x and speeding up by 3x on QuALITY, providing an efficient communication interface for multi-LLM collaboration.

Article Content

When multiple large language models (LLMs) collaborate to process a large number of questions, a common bottleneck emerges: context is passed between models as text, and the receiver must reprocess all tokens every time, causing enormous computational waste. A new academic study, LatComm, proposes a latent communication interface that separates encoding from reading, enabling context to be encoded once and reused many times. It significantly improves accuracy across 12 multiple-choice datasets, while reducing input by 57x and speeding up by 3x on the QuALITY dataset.

Problem Background: Redundant Context Processing in Multi-Model Collaboration

In multi-LLM systems, a single context (such as an article or conversation history) is often used to answer multiple questions. If the context is passed as text, the receiving model must re-encode all tokens for each question, even when the context is identical. This redundant computation not only slows down response times but also consumes substantial compute. An ideal solution requires a context representation that simultaneously satisfies three conditions: compact, reusable across questions, and readable by different LLMs. Existing compression methods (such as single-vector summaries) are compact but suffer large information loss; latent communication methods can transmit rich information but typically require re-encoding the context for each question. LatComm attempts to satisfy all three properties at once.

Latent Communication: Refers to a communication method in which models exchange information not through natural language text, but directly in a continuous vector space (latent space). Compared with text communication, latent communication avoids the overhead of tokenization and re-encoding and can transmit richer, higher-dimensional semantic information. However, latent representations are usually tied to a specific model or task, and reusing them across models requires aligning latent spaces. LatComm partially addresses the challenge of cross-model reading by freezing the encoder and receiver and training a cross-attention reading module, but latent space alignment remains an open challenge.

LatComm Core Mechanism: Separating Encoding from Reading

The key design of LatComm is to decouple context encoding from question reading. The system contains a frozen context encoder LLM and a frozen receiver LLM (which may be different models). The encoder compresses the context (without the question) into a fixed-size "packet," i.e., a set of latent vectors. The receiver then reads this packet through a cross-attention mechanism, where the attention gating is computed from the receiver's own prompt, thereby dynamically selecting the information to read for each question. Since the context is encoded only once, all questions share the same packet, and the receiver does not need to reprocess the original context.

Cross-Attention Mechanism: In the Transformer architecture, cross-attention allows one sequence (such as the receiver's prompt) to serve as the query (Query) and another sequence (such as the packet generated by the encoder) to serve as the key (Key) and value (Value), thereby dynamically aggregating information. In LatComm, the receiver uses its own prompt to compute attention weights and performs weighted reading of the latent vectors in the packet, achieving question-aware information extraction. This mechanism does not change the receiver's parameters and adapts only by training the reading module (such as attention layers), thus preserving the model's generality. Cross-attention is commonly used in encoder-decoder models (such as T5) and vision-language models, and is a core component of cross-modal information fusion.

Experimental Results: Accuracy Gains and Efficiency Optimization

On 12 multiple-choice question-answering datasets, LatComm achieves an average accuracy improvement of up to 10.2 percentage points over single-vector baselines. Single-vector baselines typically compress the context into a single vector, creating an obvious information bottleneck; LatComm's fixed-size packet retains more information, and the receiver can extract it on demand. On the QuALITY dataset (long-document reading comprehension), LatComm reduces the receiver's input by 57x and speeds up answering all questions for an article by 3x. This means that in scenarios requiring processing a large number of questions, both latency and throughput improve significantly.

QuALITY Dataset: QuALITY (Question Answering with Long Input Texts, Yes!) is a long-document reading comprehension benchmark containing multiple long articles (averaging about 5,000 words) and multiple-choice questions, requiring models to answer based on the full text. The dataset is designed specifically to evaluate long-context understanding and is commonly used in retrieval-augmented generation and long-document question answering research. LatComm's 57x input reduction and 3x speedup on QuALITY highlight the efficiency advantages of latent communication in long-context scenarios, because traditional methods require re-encoding the long document for each question.

Technical Details and Design Trade-offs

LatComm uses frozen models, does not fine-tune the encoder and receiver, and trains only the cross-attention reading module (inference). This design reduces training costs and preserves the model's generality. The fixed-size packet is a trade-off between information retention and transmission efficiency: the larger the packet, the richer the information but the higher the transmission overhead; the smaller the packet, the opposite. Experiments show that even with a fixed packet size, the receiver can still effectively extract relevant information through question-aware gating. Compared with latent communication methods that require re-encoding for each question, LatComm's reusability is its core advantage.

Industry Impact and Future Directions

LatComm's latent communication interface provides an efficient collaboration paradigm for multi-LLM systems. In scenarios such as retrieval-augmented generation (RAG), multi-agent debate, and long-document question answering, context reuse can substantially reduce inference costs. However, the method is currently evaluated mainly on multiple-choice questions, and its performance on open-ended generation tasks remains to be verified. In addition, latent space alignment between frozen models may limit generalization across model families. Future work could explore dynamic packet sizes, end-to-end fine-tuning, and broader model combinations.

By separating encoding from reading, LatComm makes context compression reusable, offering a new approach to LLM-to-LLM communication. Its code and models have not yet been released, but the paper has already attracted attention.

One-line Summary

By separating encoding from reading, LatComm enables context to be encoded once and reused across questions, improving multi-LLM collaboration efficiency.

Paragraph Summary

LatComm proposes a latent communication interface to solve the bottleneck of redundant context encoding in multi-LLM systems. Its core idea is to decouple context encoding from question reading: a frozen encoder LLM compresses the context into a fixed-size latent packet, and a frozen receiver LLM uses a cross-attention mechanism, gated by its own prompt, to dynamically read the packet and extract question-specific information. The context is encoded only once, and all questions share the same packet. On 12 multiple-choice datasets, average accuracy improves by up to 10.2 percentage points over single-vector baselines; on the QuALITY long-document dataset, the receiver's input is reduced by 57x and answering all questions is 3x faster. The method trains only the cross-attention reading module, reducing training costs, but current evaluation is limited to multiple-choice questions, and open-ended generation and cross-model generalization remain to be verified.