Poli-SHIFT
A dataset and evaluation framework designed to measure political stance drift in large language models. Covering ten contested political topics across the United States, United Kingdom, and Australia, it quantifies LLM ideological mimicry by controlling variables such as phrasing, presuppositions, and user identity information.
Timeline (last 90 days)
The research team constructed the Poli-SHIFT dataset and evaluation framework, covering 10 contested political topics across three countries: the United States, the United Kingdom, and Australia.
The study systematically tested 7 open-weight LLMs.
The study systematically varied three types of input signals: contested terminology, politically loaded premises, and user information.
In matched comparisons, changing only the wording reversed which side of a contested issue the model supported in 16.9% of cases.
Responses were required in both multiple-choice and open-text formats to verify the robustness of the drift across different output formats.
Testing open-weight models eliminates opaque factors such as server-side system prompts, allowing Poli-SHIFT prompts to be reproduced under identical conditions across multiple models.
Evaluating whether a model is politically neutral cannot rely on a single test with a standardized neutral prompt; robustness evaluation under diverse, real-world interactions with varied biases is required.
All Facts (7)
The research team constructed the Poli-SHIFT dataset and evaluation framework, covering 10 contested political topics across three countries: the United States, the United Kingdom, and Australia.
50%UnverifiedThe study systematically tested 7 open-weight LLMs.
50%UnverifiedThe study systematically varied three types of input signals: contested terminology, politically loaded premises, and user information.
50%UnverifiedIn matched comparisons, changing only the wording reversed which side of a contested issue the model supported in 16.9% of cases.
50%UnverifiedResponses were required in both multiple-choice and open-text formats to verify the robustness of the drift across different output formats.
50%UnverifiedTesting open-weight models eliminates opaque factors such as server-side system prompts, allowing Poli-SHIFT prompts to be reproduced under identical conditions across multiple models.
50%UnverifiedEvaluating whether a model is politically neutral cannot rely on a single test with a standardized neutral prompt; robustness evaluation under diverse, real-world interactions with varied biases is required.
50%