A New Protocol for Evaluating Emotion-Shift Forecasting: Information Matching and Leakage-Safe Auditing

Research on conversational emotion-shift forecasting often reaches conflicting conclusions due to evaluation biases: learned models may receive rich features while transition baselines get only noisy or unavailable current-emotion labels; hyperparameter budgets are inconsistent; and target-turn information may leak into representations. This paper proposes an information-matched, leakage-safe auditing protocol that eliminates these biases through explicit commitment points, executable causal tests, equal-four-configuration validation search, and seed-dialogue-aware inference.

Where Do Evaluation Biases Come From?

In the conversational emotion-shift forecasting task, models must determine whether a speaker's emotion shifts. A common practice is to let learned models receive rich contextual features, while transition baselines (such as simple rules based on the current emotion label) may use only noisy predicted labels, or even lack the current emotion label entirely. This information asymmetry systematically overestimates the advantage of learned models.

Differences in hyperparameter search budgets also introduce bias: learned models may be tuned over many configurations, while baselines use default parameters. More insidious is target-turn information leakage—models encounter information about the turn to be predicted already at the representation stage, inflating evaluation results.

The authors point out that these biases are widespread in the literature but rarely audited systematically. To address this, they design an executable auditing protocol whose core is "information matching": ensuring that both compared parties operate under equivalent conditions in terms of information and tuning resources.

Target-turn information leakage is a common evaluation pitfall in time-series and dialogue forecasting tasks, referring to the situation where, when generating a prediction for turn ( t ), the model's input representation (e.g., a context vector encoded by a pretrained language model) already implicitly contains information from turn ( t ) and beyond. For example, if a bidirectional encoder is used to encode the entire dialogue, or if feature normalization uses full-set statistics that include the target turn, the model will "see" the future. This leakage does not show up in the training loss, yet it leads to inflated evaluation metrics and is difficult to detect through simple data splits. The paper proactively probes for such leakage through executable causal tests (e.g., intervening on target-turn features and observing prediction changes) rather than relying only on static checks.

The Four-Pillar Auditing Protocol

The protocol comprises four key components:

Explicit commitment points: clearly record the information sources accessible to each model before the experiment begins, preventing post-hoc adjustments. Executable causal tests: intervene on features to test whether a model relies on information it should not access. Equal-four-configuration validation search: allocate the same hyperparameter search budget (four configurations) to learned models and baselines, eliminating unfair tuning. Seed-dialogue-aware inference: account for the seed split of dialogues during inference to avoid data leakage.

The authors apply this protocol to six causal model families, four text corpora, and two feature-source sensitivity configurations. All experiments follow the same auditing pipeline and are accompanied by an eight-item leakage checklist for reuse in future research.

Equal-four-configuration validation search is an auditing measure that controls the hyperparameter tuning budget. In machine learning comparisons, learned models are typically tuned over many hyperparameters via grid search or Bayesian optimization, while baselines often use default values or very few configurations; this causes part of the learned model's advantage to stem from "tuning effort" rather than model structure. Equal-four-configuration search requires allocating exactly the same search budget—specifically, four hyperparameter configurations—to every compared model, with selection on the same validation split. This does not fully eliminate bias from differences in search space, but it significantly mitigates the confounding factor of unequal budgets, focusing the comparison more on how information is used rather than on tuning intensity.

Conclusions Under Deployable Conditions

Under deployable conditions—i.e., when the transition baseline uses a current-emotion classifier trained only on training data (rather than gold labels)—learned models outperform the predicted-label baseline on all four main corpora. The improvement on DailyDialog is particularly notable, remaining robust after Holm correction across 72 comparisons (best ROC-AUC gain of +0.167, corrected p=.0144).

This result has practical significance: in real deployments, current emotion labels are often unavailable, and one can only rely on classifier predictions. In this setting, the advantage of learned models is real, not an artifact of information asymmetry.

Reversal Under Gold-Information Matching

When both parties receive gold current-emotion labels, the conclusion changes: the baseline remains competitive on IEMOCAP, and is even stronger on MELD. However, all models retain a substantial advantage on DailyDialog.

This reversal shows that the choice of evaluation conditions directly changes model rankings. The authors emphasize that information conditions must be explicitly stated when reporting results; otherwise, readers cannot judge the source of the advantage. This finding also explains why conclusions across papers in the literature often conflict—they may be operating under different information-matching states.

Holm correction is a statistical correction method for multiple comparisons. When a paper conducts multiple hypothesis tests simultaneously (e.g., 72 model comparisons), uncorrected p-values increase the risk of false positives due to family-wise error rate inflation. Holm correction sorts p-values from smallest to largest and compares them sequentially against adjusted significance thresholds ( \alpha/(m-i+1) ) (where ( m ) is the total number of tests and ( i ) is the rank index), stopping further tests once a test is not significant. It is less conservative and more statistically powerful than Bonferroni correction, and does not require tests to be independent. The paper reports a best gain of +0.167 with a Holm-corrected p=.0144, indicating that the advantage remains robust after controlling for multiple comparisons rather than being a chance significant result.

Beyond Accuracy: Additional Diagnostics

In addition to ROC-AUC, the paper reports precision-recall AUC, calibration metrics, strict utterance-local feature controls, same-speaker self-shift prediction, and future-access dose-response tests. These additional diagnostics verify the robustness of the protocol from different angles.

Strict utterance-local feature controls ensure that models do not gain an advantage through implicit leakage across utterances; same-speaker self-shift prediction tests model performance in single-speaker dialogues; and the future-access dose-response test quantifies the model's sensitivity to future information. The eight-item leakage checklist covers data splitting, feature engineering, hyperparameter search, and inference stages.

Implications for Research Practice

The value of this paper lies not only in specific numbers but in providing a reusable auditing framework. The field of conversational emotion-shift forecasting has long lacked unified evaluation standards; different papers use different baseline configurations and information conditions, making results difficult to compare across studies.

The information-matched auditing protocol requires researchers to specify information boundaries at the experimental design stage and to verify them through causal tests and leakage checklists. Although equal-four-configuration search increases computational cost, it eliminates the common confounding factor of tuning bias. The authors have released code, raw seed predictions, and reporting artifacts (available from the corresponding author upon request), providing a foundation for reproduction and adoption.

For future research, we recommend that when reporting emotion-shift forecasting results, at minimum the deployable-condition and gold-information-condition settings be distinguished, and leakage check results be included. Only then can readers accurately understand the source of model advantages.

One-line Summary

Proposes an information-matched, leakage-safe auditing protocol, revealing how evaluation conditions can reverse model rankings in conversational emotion-shift forecasting.

Paragraph Summary

This paper addresses prevalent evaluation biases in conversational emotion-shift forecasting—information asymmetry between learned models and baselines, differences in hyperparameter budgets, and target-turn information leakage—by proposing an information-matched, leakage-safe auditing protocol. The protocol comprises four components: explicit commitment points, executable causal tests, equal-four-configuration validation search, and seed-dialogue-aware inference, accompanied by an eight-item leakage checklist. The authors apply the protocol across six causal model families and four corpora, finding that under deployable conditions (baselines using predicted labels), learned models consistently dominate, with gains on DailyDialog significant after Holm correction; whereas under gold-information matching, baselines overtake or match on IEMOCAP and MELD. The conclusion emphasizes that evaluation conditions directly determine model rankings, and that information conditions and leakage checks must be explicitly reported.