content delta
A quantitative metric for analyzing the effect of few-shot prompts on language model representations. It isolates the influence of example content on hidden states by subtracting the representational drift caused by length-matched random text from the total representational drift induced by the prompt.
Timeline (last 90 days)
The research proposes replacing real examples with length-matched random text and subtracting the purely length-induced drift from the total drift to obtain a metric called content delta.
Raw drift magnitude cannot predict few-shot success or failure (correlation coefficient r=0.20), whereas content delta can effectively predict it (Spearman correlation rho=+0.65, p=0.043).
Models that more substantially reconstruct their own representations based on example content actually benefit more from few-shot prompting, which is contrary to the popular disruption hypothesis.
All Facts (3)
The research proposes replacing real examples with length-matched random text and subtracting the purely length-induced drift from the total drift to obtain a metric called content delta.
50%UnverifiedRaw drift magnitude cannot predict few-shot success or failure (correlation coefficient r=0.20), whereas content delta can effectively predict it (Spearman correlation rho=+0.65, p=0.043).
50%UnverifiedModels that more substantially reconstruct their own representations based on example content actually benefit more from few-shot prompting, which is contrary to the popular disruption hypothesis.
50%