MoPrO-SD: A Modular Prompt Optimization Framework for Stance Detection

Why Stance Detection Is So Challenging for LLMs

Stance detection requires a model to determine whether a piece of text expresses support, opposition, or a neutral attitude toward a given target (a topic, person, or policy). While this sounds straightforward, the task is full of pitfalls: implicit opinions, ironic expressions, and highly context-dependent phrasing all lead large language models (LLMs) astray.

Compared to traditional fine-tuning, Automatic Prompt Optimization (APO) offers an alternative path that requires no parameter updates. However, most existing structured prompt optimization methods are designed for general use and don't specifically address the semantic characteristics of stance labels or the class-specific ambiguities inherent in stance detection. In short, they don't tackle the core pain points of this particular task.

This is the context in which researchers introduced MoPrO-SD — a prompt optimization framework designed specifically for stance detection.

Automatic Prompt Optimization (APO) refers to algorithmically searching for or rewriting prompt text — without adjusting model weights — to maximize model performance on a target task. Compared to manually crafted prompts, APO systematically explores a much larger prompt space; compared to fine-tuning, it demands far less compute and data. Common APO approaches include gradient-based soft prompt tuning, LLM-driven text-level optimization (e.g., OPRO, ProTeGi), and reinforcement learning-based policy search. The limitation of these general-purpose methods is that they treat the prompt as a monolithic unit, without distinguishing segments that serve different functions or handling label ambiguity in task-specific ways — which is precisely the bottleneck MoPrO-SD aims to break through.

MoPrO-SD's Core Idea: Modular Decomposition and Sequential Refinement

The core approach of MoPrO-SD is to decompose a prompt into a set of functional semantic modules, then use LLM-generated feedback to sequentially refine these modules. This modular design makes the optimization process more targeted — each segment of the prompt that serves a distinct function can be improved individually, rather than treating the prompt as an indivisible whole.

The framework explores three refinement strategies, forming a progressively stronger hierarchy:

• Instruction only (I): Only the instruction itself is rewritten and optimized.
• Instruction + Reasoning (I+R): A reasoning process is introduced alongside instruction optimization.
• Instruction + Per-example Reasoning (I+R+E): Reasoning is further added for each individual sample, at a finer granularity.

A key innovation is MoPrO-SD's use of class-conditioned errors to refine stance labels. Rather than pooling errors from all categories together, it analyzes mistakes separately for each specific stance class (e.g., "support," "oppose"). This directly addresses the fundamental problem in stance detection: different classes carry different kinds of ambiguity.

The concept of class-conditioned errors is worth unpacking further. In a conventional prompt optimization pipeline, the model typically collects all mispredicted samples, aggregates them, and generates a single unified improvement signal (pooled feedback). This "one-size-fits-all" approach ignores the fact that error patterns differ across categories — for instance, the reasons a model misclassifies "oppose" labels (often due to subtle irony) are fundamentally different from why it misclassifies "neutral" labels (commonly caused by vague or two-sided statements). The class-conditioned approach collects error cases for each stance class separately, independently generates diagnostic feedback and revision suggestions for that class, and then integrates the per-class revisions back into the prompt. This fine-grained feedback mechanism makes the optimization signal more precise and prevents error signals from different classes from diluting each other.

VaccineStance: A New Human-Annotated Benchmark

Beyond the methodology itself, the research team has also contributed a new dataset — VaccineStance — a human-annotated benchmark specifically for vaccine stance detection.

Publicly available benchmarks for stance detection are relatively scarce, and vaccine discourse is a prime example of a domain where stance expression is highly polarized and context-sensitive, with implicit opinions and emotionally charged language being especially common. VaccineStance provides a test environment that more closely mirrors real-world public opinion dynamics. Both the dataset and the MoPrO-SD codebase have been open-sourced on GitHub.

Experimental Results: The Advantage of Class-Conditioned Feedback

Experiments were conducted across four datasets and five open-weight LLMs, with results showing that MoPrO-SD outperforms manually written baseline prompts (Base prompts) in many settings.

Several notable findings:

• On the SemEval-2016 dataset, class-conditioned feedback achieved higher average Macro-F1 scores than pooled feedback on four out of five backbone models — validating that treating errors per class is indeed more effective than a pooled approach.
• Both reasoning-based strategies, I+R and I+R+E, performed strongly, but their relative advantage varies across different models and datasets — neither strategy dominates universally in all scenarios.
• Overall, MoPrO-SD remains competitive against recent prompt optimization methods.

These results demonstrate that task-specific customization of prompt optimization — tuned to the semantic characteristics of the task — can yield substantial performance gains. Incorporating reasoning is broadly beneficial, though the optimal granularity needs to be calibrated for specific models and data.

Macro-F1 is a commonly used metric for multi-class classification tasks, particularly suited to imbalanced class distributions. It is computed by first calculating the F1 score (harmonic mean of precision and recall) for each class separately, then taking a simple average across all classes with equal weight — rather than weighting by sample count. In stance detection datasets, the "neutral" class typically has far fewer samples than "support" or "oppose." With weighted averaging, a model could achieve an inflated score simply by performing well on majority classes. Macro-F1 forces the model to perform well across all classes equally, making it a stricter and more meaningful benchmark — and the standard metric used in comparison experiments across the field.

Implications for Practitioners

MoPrO-SD's value extends beyond its performance on stance detection. It demonstrates a transferable methodology: decomposing prompts into functional modules and iteratively refining them with class-conditioned error feedback. For teams constrained by compute resources who cannot frequently fine-tune models, this "optimize the prompt, not the parameters" approach is particularly appealing.

That said, the varying effectiveness of the three refinement strategies across models and datasets serves as a reminder that prompt optimization is not a one-time solution — empirical validation tailored to your specific scenario is still necessary when deploying in practice. With the open release of the code and the VaccineStance dataset, the reproducibility and extensibility of this line of research will only continue to improve.