How Do Hierarchical Embeddings and Quantization Trade Off? MTEB Benchmark Reveals the Balance Between Efficiency and Effectiveness

Text embedding models are being optimized for efficiency in two directions: hierarchical representation allows models to output variable-length vectors, while quantization reduces numerical precision. These two techniques appear independent, but what interaction effects arise when they are actually combined? A recent empirical benchmark study provides the answer.

The study is based on 38 tasks in the Massive Text Embedding Benchmark (MTEB), using four Transformer-based embedding models, and systematically evaluates the effects of combining 1D and 2D hierarchical representations with 32-bit, 8-bit, and binary quantization. Each task covers 168 configurations, and reports efficiency along two dimensions: theoretical per-vector storage and end-to-end evaluation time.

Core Finding: int8 Quantization Is Nearly Lossless, While Binary Quantization Has a Significant Cost

The study uses the Friedman test and Nemenyi post-hoc analysis and reaches the following key conclusions:

• int8 quantization is statistically indistinguishable from float32 (for paired classification and reranking tasks), with an overall performance loss of at most 1.5 percentage points. This means that in most scenarios, int8 quantization can serve as a practical substitute for float32.
• Binary quantization is significantly worse across all task categories. Although binary quantization can bring extreme storage compression, the accuracy loss cannot be ignored.

This finding has practical implications for deployment: if the goal is storage efficiency and a small performance loss is acceptable, int8 is the safe choice; binary quantization is better suited to scenarios with extremely low accuracy requirements.

Quantization is the technique of converting model weights or activations from high-precision floating-point numbers (such as 32-bit floating point, float32) to lower-precision representations (such as 8-bit integers, int8, or binary), aiming to reduce storage and computation overhead. int8 quantization maps each value to the integer range -128 to 127, usually using linear scaling, with relatively small accuracy loss; binary quantization uses only 1 bit (-1 or +1), implemented via the sign function, achieving extremely high compression but large information loss. In embedding models, quantization acts on the output vector elements and directly affects the accuracy of similarity computation.

Statistical Detectability of Hierarchical Dimension Reduction

Hierarchical representation allows storage savings by reducing embedding dimensions. The study finds:

• Halving the embedding dimension once results in no statistically detectable performance degradation when the dimension is greater than 16 (m>16).
• Two consecutive halvings can always be statistically detected, meaning the performance loss becomes significant.

This indicates that moderate dimension reduction (one halving) is safe in most cases, but excessive dimension reduction brings observable performance loss.

Hierarchical Representation refers to models outputting variable-length embedding vectors, usually implemented through multi-level dimension reduction: 1D hierarchical representation reduces along a single dimension, while 2D compresses two dimensions simultaneously. Dimension reduction decreases vector dimensions through pooling or projection operations, thereby lowering storage requirements. For example, halving a 768-dimensional vector to 384 dimensions can save 50% storage. However, dimension reduction loses information, so efficiency and accuracy must be balanced. In this study, "m>16" refers to the dimension threshold after reduction; above this threshold, one halving has no statistically significant impact on performance.

Combined Effect: 32x Storage Compression While Retaining 94.4% of Performance

When combining 8x dimension reduction with int8 quantization, per-vector storage is reduced by 32x, while model performance still retains 94.4% of the baseline level. This is a considerable efficiency gain, especially suitable for large-scale retrieval systems.

Evaluation Time: Quantization Does Not Accelerate, While Intermediate-Layer Sampling Is More Effective

The study also measures end-to-end evaluation time, but because it was measured only once, the authors treat it as an indicative result:

• There is no evidence that quantization accelerates evaluation. Quantization mainly reduces storage and has limited impact on computation speed.
• Sampling from intermediate layers can significantly shorten evaluation time. This provides another path for optimizing inference efficiency.

End-to-end evaluation time refers to the complete inference time from input text to output embedding vector, including the model forward computation and possible quantization operations. Quantization mainly reduces storage and memory bandwidth requirements, but if the hardware does not support low-precision acceleration (such as int8 operations on CPU), computation speed may not improve significantly. Intermediate-layer sampling refers to extracting embeddings from intermediate layers of the Transformer (rather than the final layer), which can reduce computation because subsequent layers are skipped, thereby shortening inference time.

Retrieval Tasks Degrade Fastest

The study specifically points out that the performance of retrieval tasks degrades much faster than other categories. Therefore, the authors report results by task category, reminding researchers and engineers to carefully evaluate different task types when applying these efficiency techniques.

Retrieval tasks require the model to find the documents most relevant to a query from a large-scale corpus, and are very sensitive to subtle differences in embeddings. Noise introduced by quantization or dimension reduction can disrupt the relative distances between vectors, leading to lower retrieval ranking quality. In contrast, classification or reranking tasks are more tolerant of local perturbations. Therefore, more careful evaluation is needed when applying efficiency techniques in retrieval scenarios.

Practical Implications

This benchmark study provides a data-driven trade-off guide for efficiency optimization of text embeddings:

1. Prioritize int8 quantization: It is nearly lossless for paired classification and reranking tasks, with overall loss not exceeding 1.5 percentage points.
2. Use binary quantization cautiously: Although storage compression is extreme, performance degradation is significant, and task tolerance must be evaluated.
3. One dimension reduction is safe, two require caution: When the dimension is greater than 16, one halving is statistically undetectable, but two halvings bring observable loss.
4. Combined approaches are highly effective: 8x dimension reduction plus int8 quantization achieves 32x storage compression while retaining 94.4% of performance.
5. Retrieval tasks require special attention: They degrade fastest, and efficiency techniques should be evaluated separately when applied.

The study emphasizes that hierarchical representation and quantization are orthogonal techniques, but their combined effects require empirical evaluation. In real systems, it is recommended to choose appropriate configurations based on task type and accuracy requirements, and to conduct sufficient benchmark testing before deployment.

One-line Summary

An empirical benchmark reveals that int8 quantization is nearly lossless, binary quantization has a significant cost, and combining dimension reduction with quantization can achieve 32x storage compression.

Paragraph Summary

An empirical study based on 38 MTEB tasks and four Transformer embedding models systematically evaluates the effects of combining hierarchical representation (1D/2D) with quantization (32-bit, 8-bit, binary). The study finds that int8 quantization is statistically indistinguishable from float32 on paired classification and reranking tasks, with overall performance loss of at most 1.5 percentage points; binary quantization is significantly worse. Halving the dimension once is statistically undetectable when m>16, while two consecutive halvings are significant. Combining 8x dimension reduction and int8 quantization can achieve 32x reduction in per-vector storage while retaining 94.4% of baseline performance. Quantization does not show evaluation speedup, intermediate-layer sampling can shorten time; retrieval tasks degrade fastest and require separate evaluation.