Hierarchical Question Construction for Interpretable Text Embeddings: Reducing Manual Review Burden

Text embeddings are a core component of NLP systems, but traditional vector representations are like black boxes that are difficult to interpret. QA-based text embeddings build vectors by answering a series of yes/no questions, with each dimension corresponding to one question, thereby providing fine-grained interpretability. However, this approach faces two major challenges: first, it requires manual review of a large number of questions, and second, the relationship between the number of questions and granularity has not been sufficiently studied. A recent paper proposes a hierarchical question construction method aimed at addressing these issues.

Problem: The Tension Between Interpretability and Manual Cost

QA-based text embeddings represent text as vectors of 1/0 answers, with each question corresponding to one dimension. This representation performs well in both interpretability and task performance, but humans need to inspect a large number of questions to understand the meaning of the embedding. For example, an embedding may contain hundreds of questions, and reviewing them one by one is clearly impractical. In addition, the relationship between the number of questions and appropriate question granularity lacks systematic study, leaving little guidance for designing questions.

Text embedding is a technique that maps discrete text into a continuous vector space, so that semantically similar texts are closer in the vector space. Traditional methods such as Word2Vec and BERT sentence embeddings have no explicit semantic labels for each dimension, and are therefore called "black boxes." QA-based embeddings are different: they predefine a set of yes/no questions, answer each one for the input text, and concatenate the answers (1/0) into a vector, so each dimension directly corresponds to one question, giving the vector dimension-by-dimension semantics. The interpretability of this representation lies in the fact that humans can understand what a dimension is "measuring" by reading the question, but the cost is that the question set must be manually designed, validated, and reviewed. When the number of questions reaches hundreds, checking them one by one is not only time-consuming but also requires reviewers to have domain knowledge, which constitutes the core tension between interpretability and manual cost.

Hierarchical Questions: From General to Specific

The paper proposes a method for constructing hierarchical questions. As the hierarchy deepens, questions become more specific and their number increases. During interpretation, deep questions are replaced with aligned shallow questions to reduce the number of questions that need manual inspection. The question hierarchy is constructed through recursive clustering, contrastive prompting, and question alignment.

Recursive Clustering and Contrastive Prompting

The first step in building the hierarchy is recursive clustering: grouping questions to form clusters at different levels. Contrastive prompting is used to generate more discriminative questions. Question alignment ensures semantic correspondence between deep questions and shallow questions, making upward substitution possible during interpretation.

Recursive clustering is a bottom-up hierarchy construction strategy: first cluster the bottom-level questions (or the text representations corresponding to the questions), grouping semantically similar questions together, then cluster the group representatives in the next round, and repeat this process to form a tree-like hierarchy. Contrastive prompting draws on the idea of contrastive learning by providing both positive and negative examples when generating questions (for example, text pairs that "belong to a certain class" and "do not belong to a certain class"), guiding the language model to generate questions that can distinguish these samples, thereby improving the discriminative power of the questions. Question alignment, after the hierarchy is constructed, establishes a semantic mapping between deep questions and shallow questions, usually by computing the similarity of question embeddings or having a language model judge the coverage relationship, ensuring that deep questions can be reasonably replaced by shallow questions in their ancestor nodes.

Upward Substitution During Interpretation

When the embedding needs to be explained, the system replaces deep questions with shallow questions, thereby reducing the number of questions that must be presented to humans. Shallow questions are more general but have broader coverage, making them suitable for quick understanding.

Upward substitution is the core operation when hierarchical questions are used for interpretation: when the system needs to explain to humans the meaning of a deep question dimension, it no longer directly shows that specific question, but instead moves up the hierarchy tree to find an aligned shallow question and replaces it with a more general question. Because shallow questions are fewer in number and broader in coverage, humans only need to review a small number of general questions to grasp the approximate semantics of the embedding. This substitution is essentially a kind of "semantic abstraction": using coarse-grained but understandable questions to summarize fine-grained questions that are difficult to review one by one. The validity of the substitution depends on the quality of question alignment—if shallow questions cannot sufficiently cover the semantics of deep questions, the explanation will be distorted. Therefore, the alignment step must ensure during construction that deep questions have an inclusion or approximate relationship with their shallow ancestors in terms of semantics.

Experiments: Balancing Performance and Interpretability

Experiments on general tasks show that the method improves interpretability while maintaining task performance. An interesting finding is that when only a limited number of questions are available, shallow questions often achieve higher task performance than deep questions, even when the total number of questions is the same. This means that under resource-constrained conditions, prioritizing shallow questions may be more effective.

Significance and Limitations

This work provides a practical solution for interpretable text embeddings and reduces the cost of manual review. The hierarchical design also reveals the relationship between question granularity and performance, providing a reference for future research. However, the method depends on the quality of question generation, and its generalization ability across different tasks still needs further verification.

Overall, hierarchical question construction finds a balance between interpretability and efficiency and is worth attention.