Qwengram-0.8B Experiment: Cross-Model N-gram Memory Transfer Reduces Perplexity by 5.05%

A developer used free Kaggle notebook GPUs to run a fascinating experiment: transferring the PLE (n-gram memory) pretrained on the larger Qwen3.8 Flash-Next model to the much smaller Qwen3.5-0.8B, achieving a 5.05% perplexity reduction on a frozen validation set. The entire process required no fine-tuning of the backbone model, and the experiment sheds light on the viability of the "plug-in memory + dynamic gating" approach for small models.

Experimental Design: Freeze the Backbone, Train Only a Lightweight Reader

The core idea is to freeze two "heavy assets": the Qwen3.5-0.8B backbone itself, and the approximately 51-billion-parameter PLE memory module. The only component actually trained is a tiny R=1 reader deployed at layers 3 and 9 of the decoder.

During the late-stage injection phase, the author introduces a token-level linear gate to control the injection intensity of memory signals across different tokens. This gate is "truly dynamic" — its memory strength varies significantly from token to token, rather than collapsing into a learned constant. The entire scheme requires no backbone fine-tuning, meaning external memory capability can be layered on with almost no modification to the original model.

The released "balanced" checkpoint trains the reader on only 15 million tokens, paired with a separately calibrated dynamic gate.

PLE (Pointwise Lexical Embedding / n-gram memory) is a memory structure that pre-encodes statistical co-occurrence information of high-frequency n-grams into a set of retrievable vectors. Unlike KV Cache or RAG, PLE does not rely on runtime retrieval. Instead, it "distills" n-gram-level conditional probabilities into a set of fixed parameters during pretraining, allowing it to be frozen and mounted as a standalone module on other models. Its scale typically far exceeds the backbone — in this experiment, the ~51B-parameter PLE from the larger Qwen3.8 Flash-Next is more than sixty times the size of the 0.8B backbone. The reader acts as a "translation layer": it takes the backbone's current hidden state, retrieves relevant memory vectors from the frozen PLE, and injects the retrieved results into an intermediate decoder layer as a residual. The R value denotes the reader's rank; R=1 means extremely low parameter overhead, while also limiting the information capacity per injection.

Core Result: 5.05% Perplexity Reduction

On the frozen full validation set, the comparison is as follows:

• Qwen3.5-0.8B (original): NLL 2.905585, PPL 18.2759
• Qwengram-0.8B: NLL 2.853786, PPL 17.3534
• Perplexity reduction: 5.05%

The author is careful to clarify a potential misreading: this is a language model validation-level perplexity improvement, not a claim of 5% benchmark accuracy gains. Perplexity measures a model's predictive capability over text and is not equivalent to downstream task accuracy. This kind of restrained framing is uncommon in self-directed experiments and makes the results more credible.

Key Findings That Shaped the Final Design

The ablations performed throughout the experiment are arguably more informative than the final numbers.

Real Pretrained Memory Outperforms Random Baselines

The author ran controlled experiments showing that using real pretrained PLE clearly outperforms both random memory and permuted memory baselines. This demonstrates that it is the genuine statistical information encoded in PLE that matters — not simply the addition of extra parameters. Similarly, under the same memory budget, a learned token placement strategy outperforms random shuffled placement.

Training Volume vs. Task Capability Trade-offs

The reader's loss continues to decrease beyond 5 million tokens. A reader trained on 20 million tokens further reduces overall LM loss, but shows degradation on math tasks. Weighing the trade-off, the author settles on 15 million tokens as the balance point. This phenomenon — where overall metrics improve but specific capabilities regress — is quite common in memory injection research.

Dynamic Gating Recovers Losses from Fixed Injection

Applying memory injection with a fixed strength at later layers hurts LAMBADA performance (a task that tests long-range dependency prediction). Introducing token-level dynamic arbitration largely recovers this loss. The author also found that directing memory injection toward "high-uncertainty positions" recovers some of the advantage, but not as much as the learned dynamic gate.

LAMBADA is a language model benchmark specifically designed to test long-range contextual dependencies: each example presents a narrative passage and asks the model to predict the final word, which can only be inferred by understanding the full context. Fixed-strength memory injection hurts LAMBADA performance because the memory signal applies uniform pressure across all tokens, interfering with the model's ability to perform cross-sentence reasoning via long-range attention. The token-level dynamic gate lets the model autonomously decide "at which positions and with what weight to accept memory assistance," making memory primarily effective at positions with strong short-range n-gram correlations, while automatically yielding at key tokens requiring long-range reasoning. This balances both types of capability. The finding is broadly relevant to any architecture involving external knowledge injection: the spatial distribution of injection intensity matters just as much as the injection volume itself.

R=1 Is the Balanced Architecture

A warm-started R=4 reader produced marginal improvements in overall LM loss, but introduced regression on code and math tasks. The author therefore retains R=1 as the balanced architecture.

Engineering: llama.cpp Inference Path and Quantization Fidelity

Beyond training and evaluation, the author also implemented a complete inference path in llama.cpp and released the model, training code, and modified runtime:

• Model / GGUF: huggingface.co/Ninnix96/Qwengram-0.8B
• Training, controls & evaluation: github.com/Ninnix/qwen-ple-transfer
• Modified llama.cpp runtime: github.com/Ninnix/llama.cpp

A notable architectural choice: the GGUF bundles the Qwen3.5 backbone together with the trained reader and arbitration tensors, while the massive PLE memory exists as an external quantized sidecar rather than being packed into the model GGUF. This separated "backbone + external memory" design is far more deployment-friendly.

In a separate WikiText-2 GGUF runtime test, Q8_0 quantization retained 99.1% of the BF16 reader NLL gain — note this is a distinct measurement from the Kaggle frozen validation set described earlier.

GGUF (GPT-Generated Unified Format) is the mainstream model packaging format in the llama.cpp ecosystem, supporting unified encapsulation of weights, tokenizers, and quantization metadata into a single file for local deployment. Q8_0 is the highest-precision integer quantization scheme, compressing each weight from BF16 (16-bit float) to 8-bit integer — roughly halving the file size with significant CPU inference speedup but typically minimal quantization error. The 99.1% BF16 reader gain retention at Q8_0 indicates that the weight distributions of the reader and gate tensors are quantization-friendly at low bit-widths. Keeping PLE as an external sidecar rather than embedding it in the main GGUF avoids bloated single-file sizes and also allows users to mount or swap memory libraries on demand, preserving flexibility to adapt to domain-specific PLEs in the future.

Significance and Next Steps

The greatest value of this work lies not in the 5.05% number itself, but in what it validates at extremely low cost: a technical approach of using frozen large-model memory to augment small models, training only lightweight readers and dynamic gates. It provides a clear recipe for "reader scaling" and "dynamic memory arbitration."

The author plans to next attempt larger Qwen backbones, particularly the 35B-A3B MoE architecture, but candidly acknowledges this requires far more compute than free Kaggle notebooks provide. The author also openly invites others to reproduce or improve upon the reader, PLE caching, routing, and runtime.

The author discloses being the creator of Qwengram and related repositories, that English is not their native language, that the post was polished with AI assistance, and that coding agents (primarily ChatGPT) were used during development for implementation, debugging, and analysis — while the experimental design, research decisions, and final conclusions were their own. This kind of transparent disclosure is itself a practice worth promoting in the era of AI-assisted research.