LLM Prompt Sensitivity: A Case Study on Slovak as a Low-Resource Language

Introduction

Large language models (LLMs) have attracted considerable attention for their performance on multilingual tasks, yet research on low-resource languages remains limited. A case study focused on Slovak migration media discourse takes an in-depth look at LLM sensitivity to prompt variations. The study centers on three tasks: topic relevance classification, geographic relevance classification, and extraction of migration source/destination location terms. The findings reveal key factors in prompt design—most notably, that the instruction portion matters far more than the example portion, and that larger models exhibit greater robustness under prompt variation.

Low-Resource Languages refer to languages that lack resources in terms of digital corpora, annotated data, and computational tools, typically missing large-scale pretraining corpora and mature evaluation benchmarks. Slovak belongs to the West Slavic branch, with roughly 5 million speakers, and is regarded as a medium-to-low resource language in NLP. Compared with high-resource languages such as English, LLMs have an extremely low proportion of pretraining data in low-resource languages, which directly affects the model's language understanding and prompt-response behavior. Studying prompt sensitivity in low-resource languages is essentially examining the generalization boundary of models when training distribution coverage is insufficient, and its conclusions often cannot be simply extrapolated from experimental results in high-resource languages.

Research Methods and Task Design

Using Slovak—a low-resource language—migration media discourse as its backdrop, the research designs several natural language processing tasks. These tasks include judging the relevance of text to migration topics, identifying the relevance of geographic locations in text, and extracting migration source and destination terms from text. By constructing different versions of prompts (e.g., changing instruction wording, adjusting the number or content of examples), the researchers systematically evaluated differences in LLM performance across these tasks.

Prompt Sensitivity refers to the degree to which a model's output quality fluctuates with surface-level changes in the prompt (such as instruction wording, the number and order of examples, and language choice). In controlled experiments, researchers typically fix the model and decoding parameters and systematically vary only one dimension of the prompt to isolate that factor's effect on performance. A highly sensitive model means the same task performs very differently under different prompts, resulting in low reliability at deployment; low sensitivity means the model is more robust to prompt changes. This concept is closely tied to prompt engineering practice and is an important indicator for assessing whether an LLM can be stably applied in real-world scenarios.

Key Findings

Limited Gains from Native-Language Prompts

Prompts written in Slovak (the native language) brought some improvement in task performance compared with prompts in other languages (such as English), but this gain was moderate and highly dependent on the specific task. This suggests that for low-resource languages, directly using native-language prompts is beneficial but not a universal solution.

Robustness Advantage of Large Models

The study compared the performance of models of different sizes and found that large models' robustness to prompt variation was significantly better than that of small models. When faced with changes in instruction wording, example selection, and the like, large models were able to maintain relatively stable performance, whereas small models showed greater fluctuation. This finding underscores the importance of model scale in coping with prompt sensitivity.

Few-shot Examples refer to task demonstration samples provided to the model in the prompt, usually presented as input-output pairs to guide the model in understanding the task format and expected output. The counterpart is zero-shot prompting, which relies solely on instructions to describe the task. The role of examples is to implicitly convey task specifications, but they may also introduce noise, format bias, or inconsistencies with the target task distribution. This study found that the example portion has a weaker effect than the instruction portion, which is in tension with conclusions from some high-resource language studies, suggesting that in low-resource language scenarios, models rely more on explicit instructions than examples to locate the task—a finding with direct practical implications for prompt design.

Instructions Are the Most Critical Component of a Prompt

Analysis shows that the instruction portion of a prompt (whether system instructions or task instructions) is the most critical component, with an impact exceeding that of the example portion. Task-specific performance gains are more pronounced than overall language effects. This means that when designing prompts, priority should be given to optimizing the clarity and accuracy of instructions rather than over-relying on adding or removing examples.

Implications for NLP in Low-Resource Languages

This study offers important insights for LLM applications in low-resource languages. First, although native-language prompts are beneficial, they need to be tuned in line with task characteristics. Second, the robustness advantage of large models makes them better suited for deployment in scenarios where prompts change frequently. Finally, instruction optimization should be the core of prompt engineering, especially in task-oriented applications. These findings help guide future prompt design strategies for low-resource languages.

Conclusion

This case study reveals the patterns of LLM prompt sensitivity in Slovak migration media discourse tasks. The limited gains from native-language prompts, the robustness advantage of large models, and the critical role of instructions together sketch the current state and challenges of prompt engineering for low-resource languages. As LLMs are applied ever more widely in multilingual scenarios, understanding and optimizing the effects of prompts across different languages and tasks will become increasingly important.

One-line Summary

Research reveals LLM prompt sensitivity patterns and design priorities in Slovak migration media discourse tasks.

Paragraph Summary

Using Slovak migration media discourse as its backdrop, this study systematically evaluates LLM sensitivity to prompt variation across three tasks: topic relevance classification, geographic relevance classification, and migration source/destination term extraction. Findings: native-language prompts yield moderate, task-dependent gains over English prompts; large models show significantly greater robustness to instruction wording and example changes; the instruction portion of a prompt matters far more than the example portion, and task-specific gains are more pronounced than overall language effects. These conclusions provide practical guidance for prompt engineering in low-resource languages, emphasizing prioritizing instruction clarity and tuning native-language prompts to task characteristics.