SwaT: Cross-Model-Family KV Cache Transfer Without Prefill

Multi-agent systems have long been a popular direction for LLM applications, but communication between agents has always relied on text. One agent summarizes its findings into words, and another agent reads that text—a process that is not only constrained by the bandwidth of discrete tokens but also introduces substantial redundant computation. A recent study called SwaT (Swarm Telepathy) proposes an interesting solution: let agents directly pass KV caches to each other, and across model families at that.

The Two Hidden Costs of Text Communication

In current mainstream multi-agent frameworks, regardless of how roles are divided, the flow of information between agents almost always follows the path of "generate text → receiver re-prefills." This creates two problems.

First is the bandwidth bottleneck. An agent may have already formed rich internal representations, but when outputting, they are compressed into a limited token sequence, from which the receiver must reconstruct understanding. Information is lost in encoding and decoding, and the capability ceiling is artificially lowered.

Second is wasted compute. The receiver must perform a full prefill on the received text before it can continue reasoning. The same information is computed twice, and much of the intermediate state could have been reused directly.

Prior research has attempted KV cache transfer to bypass this problem, but either requires both sender and receiver to belong to the same model family, or fails to truly work across contexts, leaving the receiver still needing to prefill. This is precisely where SwaT enters.

KV cache (Key-Value Cache) is a core intermediate state of Transformer-based models during autoregressive generation. When a model generates output token by token, the self-attention mechanism at each layer computes a set of Key and Value vectors for each token already generated and caches them, so subsequent token generation can reuse them directly and avoid recomputing historical tokens. The dimensions of these caches are typically [number of layers, number of attention heads, sequence length, head dimension], and they represent a kind of "internal representation snapshot" of the model over its context. In the standard inference pipeline, the prefill stage processes the input prompt and computes KV caches for all input tokens at once, after which the decode stage generates new tokens one by one. In multi-agent scenarios, after the receiver gets the text, it must go through prefill again, essentially discarding the sender's existing KV cache and rebuilding its internal state from scratch—this is the root cause of wasted compute.

What SwaT Does

SwaT stands for Swarm Telepathy. According to the paper, it is the first method to support cross-family KV cache transfer, enabling prefill-free communication in multi-family multi-agent systems.

The key constraint is that both the sender and receiver models remain frozen and are not fine-tuned. Training SwaT itself requires fewer than 50 million tokens and less than 1 GPU-day. This means it is more like a lightweight adaptation layer than a large engineering effort requiring retraining.

In terms of practical results, the paper reports that in agent-to-agent communication scenarios, SwaT combined with recent KV cache compression techniques achieves an average accuracy improvement of 40% over text-to-text communication and is on average 4 times faster in wall-clock time. The experiments cover 12 LLMs, 6 model families, and 17 benchmarks.

Where the Cross-Family Difficulty Lies

The KV cache spaces of different model families are not aligned. Hidden dimensions, layer counts, attention head configurations, and positional encoding methods may all differ, and the semantic structure of the caches is also inconsistent. If one family's KV is directly inserted into another family, the receiver cannot use it directly.

SwaT needs to solve exactly this heterogeneous mapping problem. The paper does not elaborate on the specific mechanism in the abstract, but from the two conditions of "no fine-tuning of sender or receiver models" and "a small amount of training data," it can be inferred that it likely performs some kind of projection or alignment at the cache level rather than modifying the models themselves.

Across different model families, KV caches cannot be directly shared because of differences in architectural hyperparameters. The first is the hidden dimension, i.e., the vector width of each token, which may range from 768 to 8192 across models; the second is the number of layers (depth), which determines the vertical structure of the cache; attention head configuration (number of heads and per-head dimension) affects the shape of the Key/Value tensors; and positional encoding method (such as absolute positional encoding, RoPE, ALiBi, etc.) determines how positional information is organized in the cache. Even if two models happen to have the same dimensions, their semantic spaces may not be aligned—the features represented by values in the same dimension may be completely different across models. Therefore, cross-family transfer must solve the dual problems of shape adaptation and semantic alignment, which is also the technical bottleneck that SwaT's core mechanism needs to overcome.

A Notable Side Capability

The paper also mentions a result beyond expectations: if the sender is a vision-language model (VLM), SwaT can transfer visual capability to a text-only receiver LLM, with results exceeding text-to-text communication—even though SwaT itself was trained only on pure text data.

This result suggests that the information carried in KV caches may be more general than we expected. The intermediate representations formed by visual understanding inside a VLM remain meaningful to a text model after transformation by SwaT. This has direct implications for the design of multimodal multi-agent systems.

VLM (Vision-Language Model) refers to a multimodal large model that processes both image and text inputs, usually composed of a visual encoder (such as ViT) and a language model: the image is first encoded into a sequence of visual tokens, then concatenated with text tokens and fed into the language model for unified inference. Therefore, the KV cache inside a VLM contains not only the text context but also implicit intermediate representations of image content. If SwaT can map this hybrid cache into a format understandable by a pure-text LLM, it means the representations formed by visual information in the VLM have some kind of cross-modal, cross-model general semantic structure. This finding has direct value for multimodal multi-agent systems: a combination of one agent responsible for looking at images and one responsible for reasoning can bypass the inefficient path of "first describe the image in words, then reason" and directly share understanding at the cache level.

Another Possibility for Multi-Agent Communication

The significance of SwaT is not only speed. It points to a shift in the paradigm of agent communication: from "communicating in natural language" to "directly sharing internal states."

Of course, many questions remain on this path. How does the semantic fidelity of cross-family caches decay as model differences increase? Will the information loss caused by cache compression be amplified in complex tasks? Are the performances balanced across different family combinations? These require the full experimental details of the paper.

But at least from the abstract data, a 40% accuracy improvement and 4x speedup are not small numbers. If these results can be reproduced on a broader range of tasks, the communication layer design of multi-agent systems may undergo a substantive adjustment.

One-line Summary

SwaT enables cross-model-family KV cache transfer, allowing prefill-free multi-agent communication with 40% higher accuracy and 4x speed.

Paragraph Summary

SwaT (Swarm Telepathy) is a new study proposing the first method to support cross-model-family KV cache transfer, enabling prefill-free communication in multi-agent systems. In existing frameworks, agents rely on text-based communication, which suffers from bandwidth bottlenecks and wasted compute. While keeping sender and receiver models frozen and without fine-tuning, SwaT uses fewer than 50 million tokens and less than 1 GPU-day of training cost to achieve heterogeneous mapping through cache-level projection or alignment. Experiments cover 12 LLMs, 6 model families, and 17 benchmarks. Combined with KV cache compression, it achieves on average 40% higher accuracy than text communication and 4x faster wall-clock time. The study also finds that SwaT, trained only on pure text, can transfer a VLM's visual capability to a pure-text receiver.