Why Does Few-Shot Prompting Sometimes Hurt? A 12-Model Study Reveals the Truth

The Few-Shot Paradox: Helpful or Harmful?

In real-world LLM applications, few-shot prompting is widely regarded as a standard technique for boosting model performance — simply provide a few examples in the prompt, and the model typically handles the task better. Yet a phenomenon that has long puzzled researchers is that few-shot prompting can sometimes hurt rather than help. The underlying reason has remained elusive.

A newly published arXiv paper (arXiv:2609.15990v1) tackles this problem through a systematic study spanning 12 open-weight models, 2 task types, and 2 architectures, attempting to untangle the mystery through both behavioral evidence and representational analysis. The findings are strikingly counterintuitive: the success or failure of few-shot prompting is the opposite of what the intuitive "distortion" explanation would suggest.

Few-shot prompting is a technique that guides model behavior by providing input-output example pairs in context, without any fine-tuning. Its counterpart, zero-shot prompting, gives the model only a task description with no examples. The theoretical basis for few-shot prompting comes from the concept of "in-context learning" (ICL) introduced in the GPT-3 paper — the idea that models can instantly "learn" task patterns from examples within the prompt window without updating any weights. This capability is considered one of the key emergent properties of large models, but its internal mechanism remains unresolved: whether models perform genuine inductive learning or simply retrieve similar patterns seen during training is still debated. It is precisely because of this ambiguity that failure cases — where examples actually degrade model performance — also lack a convincing mechanistic explanation. This paper aims to fill that gap.

Performance Is Highly Task-Dependent

The research team evaluated 12 models on two Ukrainian-language tasks: news classification and legal case outcome prediction. The results reveal a strong task dependency in few-shot prompting effectiveness.

On the news classification task, the same set of models achieved an average improvement of +24 percentage points — a significant gain. But when switching to the legal text task, the improvement plummeted to just +3.4 percentage points, with two models actually showing performance degradation.

This suggests that asking broadly whether "few-shot prompting works" is not particularly meaningful — the nature of the task is the key variable determining whether examples are beneficial. For well-structured classification tasks with clear patterns, the guiding value of examples is evident. For legal prediction tasks involving longer reasoning chains and more complex context, the marginal benefit of examples is substantially smaller.

The Length Trap: A Blind Spot in Prior Representational Analysis

To understand what is actually happening inside the models, the researchers dove into the models' hidden states. Prior work commonly measured the "shift" in hidden states between zero-shot and few-shot conditions, then tried to use this shift to explain few-shot success or failure.

However, there is an overlooked methodological flaw: few-shot prompts are inherently much longer, and the sheer length difference alone is enough to shift a model's representations. In other words, the "shifts" measured in past work were contaminated by substantial noise from prompt length increases — noise unrelated to the actual content of the examples. This makes analyses based on raw shift magnitudes potentially misleading.

Hidden states are the vector representations produced by each layer of a Transformer architecture after encoding the input token sequence — essentially the model's "internal semantic snapshots" as it processes input. Researchers typically extract hidden states from the last few layers, aggregate them (e.g., via mean pooling) to obtain a representation vector for the entire prompt, and then quantify "drift" by computing cosine or L2 distance between two vectors. This analytical paradigm is widely used in interpretability research, aiming to find interpretable signals inside the model beyond behavioral metrics like accuracy. A fundamental weakness of this approach, however, is that a model's hidden states are extremely sensitive to the overall statistical properties of the input sequence — including length and vocabulary distribution — making it difficult to disentangle "content effects" from "length effects" in any drift measurement that lacks length control.

Random Text Controls: Isolating the True Effect of Content

The paper proposes a clean and elegant fix: replace real examples with length-matched random text, measure the representational drift caused purely by prompt length, and subtract this from the total drift.

The resulting metric is called content delta — it precisely isolates the change in model representations due to "what the examples say" rather than the displacement caused by "how long the examples are." This random-text control design is the methodological centerpiece of the study.

The results completely rewrite the prior picture:

• Raw shift magnitude fails to predict few-shot success (correlation coefficient r = 0.20, essentially no correlation);
• Content delta effectively predicts it (Spearman rho = +0.65, p = 0.043, statistically significant).

A Counterintuitive Finding: More Reconstruction Means Greater Benefit

The most intriguing finding lies in the direction of the effect. The data shows that models which more substantially reconstruct their own representations based on example content actually benefit more from few-shot prompting.

This is the complete opposite of the popular "distortion" hypothesis. Many had previously argued that examples might "contaminate" or "distort" the model's existing representations, thereby causing performance degradation. But this study shows that truly valuable few-shot learning is precisely characterized by the model's willingness and ability to deeply reorganize its representations based on example content — and the more thoroughly it does so, the better the outcome.

To verify whether this finding is causal, the research team conducted example masking experiments on Llama 3.3 70B. When example content was masked out, model accuracy dropped back down — and when examples were restored, accuracy again exceeded the zero-shot baseline. This causal experiment further confirms that the relationship between content delta and performance is not coincidental.

The intuition behind the "distortion hypothesis" has some reasonable grounding: in few-shot settings, if examples are misaligned with the test sample distribution or carry noisy labels, models may over-adapt to surface features of the examples, drifting away from the correct representation of the test input. Early research found that even using randomly mislabeled examples, few-shot performance was sometimes comparable to using correct labels — interpreted as evidence that models don't truly "understand" example content but are instead guided by format or contextual structure. This study offers a different perspective: genuine representational reconstruction (as opposed to mere format adaptation) is the key signal of few-shot success, requiring the model not just to detect the presence of examples but to let their content materially change how it internally organizes the task's semantic space.

Practical Implications for Prompt Engineering

Although this study uses Ukrainian-language tasks as its experimental setting, its methodology and conclusions offer practical guidance for general prompt engineering:

First, when evaluating whether few-shot prompting works, don't just ask whether the prompt has "changed" the model's state — distinguish whether the change comes from content or from length. Length-matched random text controls provide a reusable, clean baseline for future representational analyses.

Second, few-shot failure doesn't necessarily mean "examples interfered with the model" — it may simply mean the model failed to produce sufficient representational reconstruction in response to the example content. This suggests that when few-shot performance is poor, practitioners should try examples that are more informative and better aligned with task patterns, rather than simply concluding that "few-shot prompting is harmful."

Third, prior judgment about task type matters. For complex tasks like legal reasoning, rather than stacking up more examples, it may be worth considering other enhancement strategies.

Overall, this paper uses rigorous controlled experiments to elevate "few-shot degradation" from a vague empirical phenomenon to something quantifiable, interpretable, and causally verifiable.