NinaXander: Feasibility and Limits of Stitching Frozen Language Models Across Architectures

Cross-architecture model composition has long been one of the more imaginative frontiers in large model research. If you could directly stitch the first few layers of a Transformer model together with the last few layers of an RNN-style model, could you get the best of both worlds? A new paper on arXiv titled NinaXander offers a cautious and honest answer: technically feasible, but far from the ideal of "free composition."

What Is NinaXander

NinaXander is a family of "composed language models" built around a core idea: using a trained shared-latent adapter to connect layers from frozen language models belonging to different architectural families.

The workflow goes like this: run the first several layers of model A, apply the adapter to transform the resulting intermediate representations, then feed the output into the remaining layers of model B. Both underlying models are frozen — no further fine-tuning occurs; only the adapter in the middle is trained.

One interesting engineering property: once the adapter is trained, it can be applied at different layer positions without retraining, yielding multiple composite models from a single training run. This means one training pass can generate an entire "stitching lineage."

The concept of a shared-latent adapter draws from the "bottleneck adapter" tradition in transfer learning: inserting a small trainable network between two fixed modules so it learns to map one side's representation distribution into a form the other side can accept. Compared to full fine-tuning, this approach requires far fewer parameters and is cheap to train. In the NinaXander context, "shared" means the same adapter is designed to be reused across multiple layer cut points, while "latent" implies the adapter doesn't perform a simple linear projection but instead learns an intermediate implicit representation — a bridge between the activation spaces of two architectures. Notably, whether the adapter can truly learn "architecture-agnostic" representations is one of the central questions the paper interrogates.

Experimental Setup: RWKV Meets Pythia

The study selected two models representing fundamentally different architectural lineages:

• RWKV-4-Raven-7B: A recurrent-mechanism architecture that doesn't rely on large KV caches during inference;
• Tulu-Pythia-6.9b: A standard Transformer architecture.

Referred to simply as RWKV and Pythia in the paper. The intent behind choosing these two is clear — to verify whether frozen models from different architectural families can be recombined post hoc, rather than tackling the easier task of swapping layers within the same architecture.

The composed models were evaluated on multiple-choice questions, and the quality of some generated text was also examined.

RWKV is an architecture that replaces the Transformer's attention mechanism with a linear recurrence formula, combining the constant inference memory footprint of RNNs with the parallel training capability of Transformers. Its core idea is to replace softmax attention with a time-decay-weighted "WKV" operator, eliminating the need to store key-value vectors for historical tokens during inference — memory usage remains constant rather than growing linearly with sequence length. Pythia is a series of standard GPT-NeoX-style Transformer models released by EleutherAI, widely used as research benchmarks due to their fully transparent training process (including complete checkpoint sequences). The two differ fundamentally in attention mechanism, positional encoding, and the mathematical structure of internal hidden states — making them ideal candidates for stress-testing the limits of cross-architecture composition.

Standout Result: 84.4% Reduction in KV Cache

The most striking configuration in the experiments was: Pythia's first 5 layers + RWKV's remaining 27 layers.

This combination reduced the Transformer's key-value cache (KV cache) by 84.4%, with no significant drop in accuracy compared to using RWKV alone.

The significance here is real: KV cache is one of the primary sources of memory overhead and latency in Transformer long-context inference. If a recurrent architecture's back half can "take over" most of the computation while preserving the representational capacity of the Transformer's early layers, you could theoretically compress inference costs substantially with almost no performance loss. This offers a worthwhile engineering direction for hybrid-architecture inference.

The KV cache (Key-Value Cache) is the core memory bottleneck in Transformer autoregressive inference: to avoid recomputing the key and value vectors for historical tokens, inference systems cache them, causing memory usage to grow linearly with sequence length. For long-context scenarios (e.g., 32K or 128K tokens), KV cache typically consumes the vast majority of GPU VRAM and is the primary constraint on batch size. Replacing the back portion of a Transformer with a recurrent architecture like RWKV means those layers no longer generate KV cache — they only maintain a fixed-size recurrent state, fundamentally changing the memory growth curve. The 84.4% reduction corresponds to going from 32 fully Transformer layers to just 5 Transformer + 27 RWKV layers, directly reflecting the structural advantage of recurrent architectures in inference memory efficiency.

Honest Limits: Don't Get Too Excited

The paper also faithfully documents the limitations of composed models — which is part of what makes it more valuable than many works that only report positive results.

No composite model outperforms the parent model Pythia on multiple-choice accuracy. While some combinations match RWKV, none of the stitched versions reach the level of the original Pythia. In other words, composition is a compromise, not a gain.

Cross-domain language modeling performance drops sharply. On WikiText (a Wikipedia corpus outside the training domain), composed models suffer dramatic degradation in language modeling performance. This suggests the adapter has learned in-distribution alignment more than genuine generalization ability.

Correspondence between intermediate representations does not imply a shared semantic space. Researchers only obtained good correspondence between intermediate representations under an "ideal case" — requiring a shared tokenizer, identical depth, and identical hidden layer width. The authors explicitly state that this does not prove the existence of a universal semantic space across different models. This is an important corrective to some of the more optimistic claims about universally interoperable model internals.

The assumption that "model internal representations are universally interoperable" has sparked widespread discussion in the neural network interpretability community in recent years, with some studies claiming that networks from different models — and even different modalities — learn similar geometric structures in their hidden layers (e.g., the "representational alignment" or "universal features" hypothesis). NinaXander's experimental results provide a counterexample: even under the most favorable conditions — shared tokenizer, matched layer depth and width — the alignment learned by the adapter only holds within the training distribution and cannot generalize to out-of-domain corpora like WikiText. This suggests that statistical similarity between intermediate activation vectors is not equivalent to semantic equivalence, and that evaluating cross-architecture representational alignment requires standards more rigorous than single-task accuracy.

Implications for the Field

NinaXander's value lies not in delivering a ready-to-deploy product, but in delineating the feasible domain and boundaries of cross-architecture composition.

On the positive side: the path of using lightweight adapters to bridge heterogeneous models in exchange for inference efficiency is real and demonstrable — the 84.4% KV cache reduction is hard evidence. For engineering teams concerned about inference costs, exploring hybrid "Transformer front-end + recurrent back-end" architectures has practical merit.

On the cautionary side: this work reminds us that "plug-and-play" with frozen models remains an ideal. Representation space alignment is highly dependent on strict preconditions; cross-domain generalization and the ability to surpass parent models are both currently out of reach. The assumption that "models share a universal semantic space" does not hold up under rigorous experimentation.

For researchers, NinaXander reads more like an honest map: it marks which paths are passable, what the costs are, and where the frontiers remain unknown.