Low-Bit N-Gram Embeddings: Saving 75% of Table Memory for Language Models

Language model performance gains often come with a sharp increase in memory overhead, and n-gram embedding tables are a typical example. The Engram n-gram memory module in DeepSeek-V4.1-Flash alone takes up 196 billion parameters, a scale that deters many researchers and developers. A new study proposes storing n-gram embeddings at low precision, greatly compressing memory usage while maintaining performance—offering a viable path to resolving this contradiction.

Why Are N-Gram Embedding Tables So Memory-Hungry?

Many recently open-sourced weight models use n-gram embedding tables to boost performance. These tables store learned representations of short token sequences (such as two or three consecutive tokens), which the model retrieves on demand during processing. The problem is that these tables have an extremely large number of parameters. According to the original data, DeepSeek^Ad-V4.1-Flash allocates 196 billion parameters to its Engram n-gram memory—already approaching or even exceeding the scale of many complete models.

The research team's core question was: can the memory cost of this component be reduced while preserving the performance gains brought by n-gram tables? Their answer is yes—by storing n-gram embeddings at low precision and systematically examining different quantization methods and cross-table bit allocation strategies.

N-gram embedding tables are mechanically similar to the word embedding tables commonly found in Transformers, but differ enormously in scale. A word embedding table's size is roughly the vocabulary size multiplied by the embedding dimension, typically on the order of tens of thousands to hundreds of thousands of rows; an n-gram table, however, must cover the combinatorial space of token sequences—the number of potential bigram combinations can reach the square of the vocabulary size, and trigrams even more. Therefore, even after pruning high-frequency combinations, n-gram tables often have billions or even hundreds of billions of rows, and multiplying each row by the embedding dimension causes the parameter count to balloon rapidly. In addition, these tables are usually stored at full precision (such as FP16 or BF16), further amplifying memory usage. This is precisely the fundamental reason why low-bit quantization can yield significant benefits: the table's parameter count is inherently enormous, and a large number of entries appear infrequently during training, making them relatively tolerant of precision loss.

Four-Bit Quantization: Retaining 99% Performance, Saving 75% Memory

The study trained from scratch on nanochat, Llama, and GPT-2 series models to verify the practical effectiveness of low-bit n-gram tables. The conclusion is quite direct: four-bit tables save about 75% of table memory while retaining over 99% of the validation loss improvement brought by adding n-gram tables.

This means the performance gains that previously required large amounts of memory can now be obtained at a quarter of the storage cost. For resource-constrained training and deployment scenarios, this ratio is enough to change model design trade-offs.

Further reducing the bit count leads to more noticeable performance degradation. However, the study also found that a brief fine-tuning phase can recover a considerable portion of the loss. This leaves room for operation in extremely memory-constrained scenarios—store at lower precision first, then recover performance through a small amount of fine-tuning.

Quantization is the process of mapping model parameters from high-precision floating point (such as FP16) to low-precision representations (such as 4-bit integers), typically implemented with scaling factors and zero points. 4 bits means each parameter occupies only 0.5 bytes; compared with FP16's 2 bytes, the theoretical compression ratio is 4x, corresponding to about 75% memory savings, consistent with the data in this article. Quantization methods broadly fall into two categories: post-training quantization (PTQ) and quantization-aware training (QAT). The former directly converts an already-trained table to low precision—low cost but potentially lossy; the latter simulates quantization error during training—better results but requires additional training. The "brief fine-tuning to recover loss" mentioned in the article is a lightweight post-processing compensation: by continuing to train the low-precision table for a small number of steps, the model adapts to the perturbations introduced by quantization. Cross-table bit allocation refers to distributing the bit budget unevenly across multiple n-gram tables—for example, using 4 bits for more important tables and 2 bits for secondary ones—thereby optimizing overall performance under a total memory constraint.

Reinvesting the Saved Memory: Larger Tables Are Actually Stronger

Low-bit quantization is not just a cost-saving measure; it can also translate into a performance advantage. The study points out that reinvesting the saved memory into larger tables can achieve performance surpassing the original full-precision tables, while overall memory usage remains lower.

This idea is worth noting: with an unchanged memory budget, using the space freed up by low precision to expand table size is equivalent to buying stronger n-gram memory capacity at the same cost. For model architecture design, this offers a new allocation strategy—not mere compression, but reallocation.

Validation on Existing Large Models: Saving 47-55% of Table Memory

The study did not stop at small models trained from scratch. On two existing large models, DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables saved 47% to 55% of table memory, with perplexity increasing by less than 1.2% and downstream task performance essentially maintained.

The significance of this result is that low-bit n-gram embeddings are not just a training-time optimization technique; they can also be directly applied to the compression and adaptation of existing models. Keeping the perplexity increase within 1.2% is an acceptable cost for most practical applications.

Implications for Model Design

This study establishes low-bit precision as a practical method for reducing the memory requirements of n-gram embeddings, and demonstrates a path to improving model performance using the freed-up capacity. Combined with the data showing that four bits retain 99% of the performance gain and that nearly half the memory is saved on large models with only a slight perplexity increase, it is foreseeable that low-bit n-gram tables will become a standard option in memory-sensitive scenarios.

For teams focused on model efficiency, directions worth further exploration include: how bit allocation strategies across different tables change with model scale, the minimum duration of fine-tuning to recover loss, and generalization performance across more model families. The original paper provides systematic experiments in these areas, offering a reproducible baseline for future work.

One-line Summary

Low-bit quantized n-gram embedding tables can save 75% of memory while retaining 99% of performance gains.

Paragraph Summary

This article introduces a study addressing the excessive memory overhead of n-gram embedding tables in large models. N-gram embedding tables store learned representations of short token sequences and have extremely large parameter counts—for example, the Engram module of DeepSeek-V4.1-Flash alone occupies 196 billion parameters. The study proposes storing n-gram embeddings at low precision and systematically examines different quantization methods and cross-table bit allocation strategies. Experiments show that four-bit tables can save about 75% of table memory while retaining over 99% of the validation loss improvement; lower bit counts cause performance degradation, but brief fine-tuning can recover a considerable portion of the loss. Reinvesting the saved memory into larger tables can even surpass the performance of the original full-precision tables. On DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables saved 47% to 55% of table memory, with perplexity increasing by less than 1.2% and downstream task performance essentially maintained. This study provides a practical compression and reallocation solution for memory-sensitive scenarios.