Label-Wording Augmentation: Effective for Unfamiliar Words, Ineffective for Abbreviations and Code

Introduction: When Label Wording Isn't Under Developer Control

In models based on typed decisions, the system scores options provided by the caller at request time. This means the model's accuracy depends heavily on how labels are worded—and developers cannot control how users will ultimately phrase those labels. A recent study compared two versions of the multilingual cross-encoder Ru-Jev, revealing the actual effects and boundaries of Label-Wording Augmentation.

The study focuses on a multilingual cross-encoder with 307 million parameters trained on thirteen Russian tasks. The newer version introduces an augmentation method that can present labels in multiple surface forms. By fixing the questions and varying only the wording of eighteen routing labels, the researchers evaluated ten label vocabularies generated by three independent processes and checked overlap with the training data. The experimental setup includes five candidate options per question and eighteen candidate options in eight of the settings.

Typed decision refers to a model selecting an output from a set of predefined, typed labels (e.g., routing, intent, department) at inference time rather than freely generating text. Such models are common in intent recognition for dialogue systems, ticket routing, and similar scenarios. The key constraint is that the semantics of the labels are provided by the caller in natural language at request time, and developers cannot predict which synonymous phrasing the user will use. Therefore, the model must be robust to the diversity of label wording—exactly the problem that label-wording augmentation attempts to solve.

Holm correction is a statistical method for controlling the family-wise error rate in multiple comparisons. When testing multiple hypotheses simultaneously (e.g., gains across ten vocabularies and various candidate settings), uncorrected p-values inflate the risk of false positives. The Holm method compares p-values in ascending order against adjusted significance thresholds, being less conservative and more statistically powerful than Bonferroni correction. "Eleven out of fourteen gains pass Holm correction" means these gains remain significant after controlling for overall false positives.

Augmentation Effect: A Clear Lead on Ordinary Words and Loanwords

On held-out vocabularies composed of ordinary words or loanwords (terms derived from English, office slang, department names), the augmented version shows higher accuracy in all settings, with gains of up to 0.24. The study reports that eleven of fourteen such gains pass Holm correction (except for slang with eighteen candidates and descriptive sentences with five candidates), and in the scenario where all labels are jargon, the label-level confidence intervals exclude zero in both settings.

This result indicates that label-wording augmentation provides substantial help for labels expressed in everyday language or foreign loanwords. The model can better handle label phrasings that may not appear in exactly the same form in the training data but are semantically similar.

Cross-encoder is a model architecture that concatenates two inputs (e.g., a question and a candidate label) and feeds them jointly into a Transformer for encoding, outputting a relevance score. Compared with dual encoders (which encode separately and then compute similarity), cross-encoders can capture fine-grained interactions between inputs, typically achieving higher accuracy at greater computational cost. The 307-million-parameter multilingual cross-encoder in this paper is of this type; its scores directly determine the ranking of candidate labels, so the surface wording of labels significantly affects the encoded representation.

Failure Cases: Abbreviations and Opaque Codes

In stark contrast to the ordinary-word scenario, the augmented version yields no usable gains on abbreviations and opaque codes. For abbreviations, the augmented version is not only no better—it actually performs worse on the clean slice. For opaque codes, once labels related to the training options are removed, both versions remain at or below chance level.

This finding reveals the limitations of the augmentation method: when labels are highly compressed or lack semantic transparency, merely changing the surface form cannot help the model establish effective semantic associations. Abbreviations and codes often lack sufficient contextual cues, making it difficult for the model to infer meaning from wording variations.

Risk-Coverage Trade-off: Accuracy Gains Come with More Confident Errors

In most cells where accuracy rises, the risk-coverage area decreases, which is usually regarded as a positive signal. However, when the number of candidates increases to eighteen, confident errors in the jargon and slang scenarios increase. This suggests that while the augmentation method improves overall accuracy, it may in some cases make the model more "confident" about wrong answers.

The study also reports performance on two new datasets unseen by both models: accuracy changed by -8.4 and +5.6 percentage points, respectively. This inconsistency suggests that the generalization ability of the augmentation method remains uncertain.

Risk-coverage area is a metric for evaluating selective prediction: the model can abstain from predicting on some samples, where coverage is the proportion of samples actually predicted and risk is the error rate on those samples. The area under the curve of risk versus coverage is summarized as a single number; a smaller area indicates better performance in being "more accurate when confident and more willing to abstain when uncertain." This metric can reveal problems masked by overall accuracy—for example, a model may improve accuracy while assigning excessively high confidence to certain wrong answers.

Evaluation Hazards and Implications

The paper concludes by identifying four evaluation hazards that arise when comparing the two versions. These hazards remind researchers and practitioners to carefully control variables, check data overlap, and pay attention to the effects of changing the number of candidates when evaluating label-wording augmentation.

For application developers building typed-decision models, the core message of this study is: label-wording augmentation is a targeted technique that is notably effective for unfamiliar vocabulary and everyday language variants, but it cannot be expected to solve problems with abbreviations, codes, or labels lacking semantic transparency. When designing systems, whether to adopt such augmentation strategies should be decided based on the specific type of labels.