LLM Auditing
A methodology for evaluating LLM systems that observes output shifts by controlling input variables such as user profiles, thereby detecting potential unfairness or quality issues in the model; it differs from the single-score evaluation of traditional benchmarks.
Timeline (last 90 days)
Counterfactual intervention originates from causal inference methodology and, in LLM auditing, takes the form of constructing a pair of inputs that differ only in a certain attribute and comparing the differences in model outputs to determine whether that attribute is a cause of the result.
Moral foundations theory and its measurement tools originate from research on human moral psychology, and their applicability still needs to be tested when transferred to model outputs.
The LLM audit method is borrowed from the audit research tradition in the social sciences, observing whether outputs change systematically by inputting controlled queries that differ only in specific attributes.
Because persona attributes often vary together, audit designs struggle to cleanly isolate the independent effect of a single factor.
All Facts (4)
Counterfactual intervention originates from causal inference methodology and, in LLM auditing, takes the form of constructing a pair of inputs that differ only in a certain attribute and comparing the differences in model outputs to determine whether that attribute is a cause of the result.
50%UnverifiedMoral foundations theory and its measurement tools originate from research on human moral psychology, and their applicability still needs to be tested when transferred to model outputs.
50%UnverifiedThe LLM audit method is borrowed from the audit research tradition in the social sciences, observing whether outputs change systematically by inputting controlled queries that differ only in specific attributes.
50%UnverifiedBecause persona attributes often vary together, audit designs struggle to cleanly isolate the independent effect of a single factor.
50%