Peeling the Matryoshka: Extracting Ontological Structure from Retrieval Embeddings

Introduction

In knowledge-intensive NLP tasks, building taxonomies is crucial, but traditional methods face a dilemma: clustering-based approaches require careful tuning for each corpus and produce unstable hierarchies; LLM-based pipelines rely on expensive task-specific training and struggle to transfer across domains. A recent paper proposes a new idea—leveraging the granularity alignment inherent in Matryoshka Representation Learning (MRL) embeddings, a lightweight probe can decode ontological structure from frozen retrieval embeddings. This article refers to the method as PEEL (Peeling Embeddings to Extract Latent ontologies) and interprets its principles, experiments, and significance.

Core Finding: Alignment Between Prefix Length and Taxonomy Granularity

The research team observed an empirical regularity in MRL embeddings: each prefix vector is itself a usable, coarser-grained representation, and there is an alignment between prefix length and the coarse-to-fine granularity of the gold-standard taxonomy. To confirm this alignment is not accidental, they designed two control experiments: shuffled-dimension and reverse-dimension. Results show that the alignment effect exists only in the learned early prefix dimensions, not in arbitrary dimension permutations. This pattern holds across three MRL models with different architectures, indicating the regularity is robust across models.

The significance of this finding is that it reveals a decodable granularity gradient implicitly encoded in pretrained retrieval embeddings. In other words, embedding vectors not only encode semantic similarity but also implicitly preserve hierarchical structure information.

Matryoshka Representation Learning (MRL) is a paradigm for training embedding models: it applies supervision simultaneously to the first several prefixes of the vector (e.g., 64 dimensions, 128 dimensions, 256 dimensions… up to the full dimension) in the loss function, so that each prefix can independently perform retrieval tasks. Its name comes from the Matryoshka doll—a large doll nesting smaller dolls, just as the full vector nests multiple usable short vectors. Unlike conventional embeddings, MRL models do not require all dimensions to contribute equally; instead, they actively compress the most general semantic information into earlier dimensions and push finer-grained discriminative information toward later dimensions. This property gives MRL embeddings a natural "multi-granularity" capability: short prefixes suit coarse-grained matching, long prefixes suit fine-grained discrimination. This is precisely why prefix length can correspond to the hierarchical depth of a taxonomy, providing the physical basis for PEEL to decode ontological structure from frozen embeddings.

PEEL Method: Lightweight Probe Decodes Taxonomy DAG

Based on the above findings, the authors propose PEEL: a minimal probe operating on frozen embeddings. Its core is an order-embedding objective that represents hierarchical structure as vector containment. After training, a taxonomy directed acyclic graph (DAG) is decoded via a minimum spanning arborescence algorithm.

You may not have noticed that PEEL requires only a concept vocabulary, with no document corpus. On two OLLM benchmarks, the taxonomy DAG recovered by PEEL exhibits high structural fidelity and can transfer zero-shot to new domains—without target-edge supervision. For reference, the paper also reports the cross-task performance of an end-to-end OLLM pipeline (jointly discovering concepts and inducing structure from document corpora), but PEEL's lightweight nature makes it more attractive for practical deployment.

Order-embedding is a class of representation learning methods that map partial order relations into vector space. Its core idea is: if concept A is an ancestor of concept B (A is coarser-grained than B), then A's embedding vector is required to be no greater than B's embedding vector in every dimension—i.e., vector containment. Thus the "is-a" relation in hierarchical structure is transformed into per-dimension inequality constraints between vectors, which can be optimized with objectives such as hinge loss. After training, the degree of containment between any two concepts can be directly obtained by per-dimension comparison of vectors, and then a minimum spanning arborescence algorithm—finding a tree structure on a directed graph that covers all nodes with minimum total weight—recovers the complete taxonomy DAG. The advantage of order-embedding is that it does not presuppose a fixed depth or branching factor; the structure is entirely data-driven.

Experimental Validation and Cross-Domain Transfer

The paper evaluates PEEL's structural recovery capability on multiple OLLM benchmarks. Experiments show that PEEL can reconstruct the taxonomy DAG with high fidelity using only the concept vocabulary, and maintains performance on new domains without fine-tuning. This transfer capability benefits from the universality of granularity alignment in MRL embeddings—the coarse-to-fine representations learned by the model are domain-agnostic. To rule out confounding factors, the authors again emphasize the shuffled- and reverse-dimension control experiments, confirming that the decoded structure indeed originates from the inherent granularity information in the embeddings.

In addition, the performance of the end-to-end OLLM pipeline serves as a reference, highlighting PEEL's efficiency advantage: it avoids expensive LLM training and document processing, achieving comparable structural fidelity with only frozen embeddings and a lightweight probe.

OLLM (Open-vocabulary LLM-based taxonomy induction) benchmark is a class of evaluation frameworks for automatically discovering concepts from document corpora and inducing hierarchical structure, typically containing multiple cross-domain corpora and their gold-standard taxonomies. The end-to-end OLLM pipeline requires first extracting candidate concepts from documents, then jointly training an LLM to induce parent-child edges, with high computational and annotation costs. PEEL bypasses document corpora and LLM training, using only the concept vocabulary as input. Therefore, its "zero-shot transfer" means: after training the probe on source-domain concepts, it is directly applied to a new concept vocabulary in the target domain, without target-domain edge supervision or fine-tuning. The paper uses the end-to-end pipeline as a reference precisely to show that the lightweight probe greatly lowers the deployment barrier while achieving comparable structural fidelity.

Significance and Outlook

PEEL offers a new path for automated taxonomy construction. It shows that pretrained retrieval embeddings already encode a decodable granularity gradient, and a minimal probe is sufficient to make it explicit as parent-child (is-a) structure. This reduces dependence on domain-specific training and document corpora, making taxonomy construction more efficient and transferable. Future work may explore how to apply PEEL to more types of embedding models and how to incorporate a small amount of supervision to further improve structural accuracy. This method has potential value for applications requiring hierarchical knowledge organization, such as knowledge management, semantic search, and recommender systems.