Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis

Improving a sentiment classifier under limited resources is far more than deciding how much data to collect or which model to train. A study titled Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis takes document-level entity-targeted sentiment analysis (ELSA) as its setting, primarily using South Slavic news data, and systematically explores data collection ordering, uncertainty-driven prediction re-examination, and model routing strategies. The research finds that merely identifying errors is not enough to yield significant gains—how to selectively escalate predictions to stronger models is the key.

Data Collection Ordering: Minority-Class Priority Is Not a Silver Bullet

The study first compares how different collection orderings affect model performance as annotated data grows. One intuitive strategy is to prioritize minority-class samples to alleviate class imbalance. Experiments show that this minority-class-first approach brings some benefit when the training set is very small, but its advantage does not persist as data volume increases. This means that under limited resources, collection strategies need to be adjusted dynamically and cannot rely solely on minority-class priority.

This finding has direct implications for real-world projects: if the annotation budget is extremely small, prioritizing minority classes may be worth trying; but once the data reaches a certain scale, focus should shift to data diversity and representativeness rather than pursuing class balance alone.

Entity-level sentiment analysis (ELSA) differs from common sentence-level sentiment classification in a key way: a single document may mention multiple entities, and the sentiment polarity for each entity can differ or even be opposite. For example, a news article may praise a minister while criticizing the party they belong to, and the model must separately judge "minister → positive" and "party → negative." ELSA therefore requires joint modeling of entity recognition and sentiment judgment, which raises annotation costs and complicates error sources—the model may misidentify an entity or misjudge the polarity for the same entity. The document-level setting further amplifies the difficulty: sentiment cues are scattered throughout long texts, requiring coreference resolution and cross-sentence reasoning, which is why this study chose a multi-entity, multi-stance corpus such as South Slavic news.

Uncertainty-Driven Prediction Re-examination: The Gap Between the Upper Bound and Reality

The study uses model uncertainty to select which predictions need re-examination, how many to review, and whether alternative encoders or local large language models (LLMs) can correct them. On the SentiCoref dataset, an "oracle" experiment shows that if all errors among the 10% most uncertain predictions could be corrected without harming correct predictions, Macro-F1 could improve by 9.14 points. However, the best action under actual evaluation yields only a 0.90-point gain.

This huge gap shows that error detection alone is not enough—knowing which predictions are likely wrong is easy, but correcting them at low cost is very difficult. Among the samples filtered by uncertainty, only a limited proportion can actually be fixed by existing tools.

Uncertainty estimation in such systems typically comes from the model's output probability distribution, such as the complement of the maximum class probability (1 - max p), prediction entropy, or variance across multiple forward passes (e.g., MC Dropout, deep ensembles). These scores are used to rank each prediction, with top-ranked ones treated as "likely wrong" and prioritized for human review or re-judgment by a stronger model. The limitation is that high uncertainty does not equal correctable error: the model may be "confidently wrong" or "hesitantly right" due to ambiguous entity boundaries, annotation guideline ambiguity, or out-of-distribution expressions. Thus, the gap between the oracle upper bound (assuming perfect correction of selected errors) and actual gains essentially reflects a mismatch between error fixability and error detectability.

Model Routing Strategies: Encoder Averaging vs. Selective LLM Assistance

The study further compares two model escalation approaches: encoder averaging and selective LLM assistance. Encoder averaging helps on AspectBench and NewsMTSC but is harmful on SentiCoref, where selective LLM assistance is more promising. Ultimately, both strategies selected on the development set route 30% of records to an encoder-assisted LLM, yielding a 1.02-point gain on SentiCoref (confidence interval includes zero) and a 1.71-point gain on NewsMTSC, but with significantly different inference costs.

This indicates that model escalation strategies need to be tailored to specific datasets and task characteristics. No single routing strategy wins in all scenarios, and selection on the development set is crucial.

Model routing / escalation refers to a system first processing all inputs with a lightweight model, then handing a portion of samples to a more expensive model based on some signal, thereby balancing average cost and overall quality. Common trigger signals include confidence thresholds, disagreement among predictions, and risk scores from a lightweight classifier. Encoder averaging is the simplest form: averaging the predicted probabilities of multiple encoders (e.g., different pretrained models or different random seeds) to reduce variance through ensembling, with limited cost increase but usually small gains; LLM assistance hands difficult samples along with context to a large language model for re-judgment, offering stronger capability but much higher per-inference cost. Their opposite performance on SentiCoref shows that the benefit of escalation strategies depends heavily on the gap between the base model and the target data: when the base encoder is already near its capability ceiling, ensembling does not help and a model with stronger external knowledge is needed.

Practical Implications: The Trade-offs of Selective Escalation

This study offers several practical lessons for resource-constrained sentiment analysis projects. First, collection strategies should adjust dynamically with data scale, and minority-class priority is only effective at very small data volumes. Second, uncertainty can guide prediction re-examination, but do not overestimate the power of automatic correction—there is often an order-of-magnitude gap between the oracle upper bound and actual gains. Finally, model routing requires empirical selection based on the development set, with full consideration of inference costs.

Overall, the study emphasizes a core point: in entity-level sentiment analysis, finding errors is only the first step. Selectively escalating predictions to the right model and balancing gains against costs is the key to improving performance.

One-line Summary

The study finds that identifying errors is not enough to improve entity-level sentiment analysis; the key lies in selectively escalating predictions and weighing costs.

Paragraph Summary

Taking document-level entity-targeted sentiment analysis (ELSA) as its setting and based on South Slavic news data, this study systematically examines three resource-constrained strategies: data collection ordering, uncertainty-driven prediction re-examination, and model routing. Results show that minority-class-first collection is only effective when the training set is extremely small and its advantage disappears as data grows; uncertainty can flag suspicious predictions, but there is an order-of-magnitude gap between the oracle upper bound (correcting errors in the 10% most uncertain predictions could improve Macro-F1 by 9.14) and the best actual action (only 0.90 points); encoder averaging helps on AspectBench and NewsMTSC but is harmful on SentiCoref, where selective LLM assistance is better. Both strategies selected on the development set route 30% of records to an encoder-assisted LLM, improving SentiCoref and NewsMTSC by 1.02 and 1.71 points respectively, but with significantly different inference costs. The core conclusion: finding errors is only the first step; selectively escalating predictions to the right model and balancing gains against costs is the key to improving performance.