AI Audits 900,000 French News Headlines: How Wording and Story Selection Doubly Shape Media Framing

News headlines have never been neutral summaries of events. Through two distinct mechanisms — choosing what to cover and how to phrase it — they quietly shape public understanding of social issues. A new study published on arXiv introduces a two-dimensional analytical framework to audit over 900,000 French news headlines published between 2022 and 2025, breaking down the vague concept of "framing effects" into two measurable dimensions.

Decomposing "Framing" into Two Dimensions

Prior computational communication research often compressed news framing into a single score, obscuring two fundamentally different mechanisms at work. The core innovation of this study is an explicit distinction between two types of framing: salience framing and selection framing.

Salience framing focuses on how things are said, measured through four linguistic devices: loaded vocabulary, blame attribution, threat framing, and rhetorical questions. These language mechanisms determine how sharply or provocatively a given event is described.

Selection framing focuses on what is said, measured by the distribution of coverage patterns at the outlet level and the proportion of high-emotion-density content. In other words, which kinds of stories a media outlet tends to select and how it formats them is itself a framing act. Separating these two dimensions allows researchers to distinguish between media tendencies that were previously conflated.

The concept of "framing" originates from sociologist Erving Goffman's foundational 1974 work and was introduced into news analysis by communication scholar Robert Entman in 1993. Entman defined framing as selecting certain aspects of perceived reality and making them more salient in a communicative text, thereby promoting particular problem definitions, causal interpretations, moral evaluations, or action recommendations. Traditional framing analysis relied heavily on manual coding — costly, limited in scale, and prone to conflating how and what into a single bias score. As computational communication research advanced, researchers began using lexicons, machine learning, and large language models for automated framing detection, but the issue of dimensional separation remained largely unaddressed. This is the theoretical context motivating the study's explicit split between salience and selection framing.

Building a 10,000-Sample Labeled Dataset with LLMs

With nearly a million headlines to analyze, annotation quality is the critical bottleneck. The research team first constructed a supervised dataset of 10,000 French headlines, annotated by three LLM annotators with disagreements resolved via majority voting and supplemented by human arbitration.

To validate label reliability, the team conducted two independent blind human studies, cross-referencing machine labels against human judgments. This "LLM pre-annotation + human validation" pipeline represents a pragmatic approach to large-scale text labeling — approximating human annotation quality while keeping costs manageable. The best-performing classifier was then applied to a deduplicated corpus of 902,111 headlines from 25 French media outlets.

Majority voting is a common strategy for resolving disagreements in ensemble annotation: when multiple annotators — whether human or model — assign different labels to the same sample, the majority opinion becomes the final label. The core assumption is that annotator errors are mutually independent and cancel out in aggregation. However, when multiple LLMs share similar pretraining data or alignment biases, this assumption may not hold — multiple models can systematically make errors in the same direction on the same class of samples, and majority voting cannot correct such shared biases. This is precisely why the researchers additionally introduced independent blind human validation: to check whether machine labels exhibit systematic drift through judgments from people with diverse cognitive backgrounds, rather than relying solely on internal model consistency.

Three Key Findings

Salience and Selection Are Not the Same Thing

The study found a positive correlation between salience deviation and selection deviation, but nearly half of the outlet-level variance cannot be explained by the other dimension. This means a media outlet's performance on "sharp wording" and "story selection tendencies" can be quite inconsistent. Based on this, the study constructs a four-quadrant media typology. Outlets in the "off-diagonal" quadrants — those with moderate language but extreme topic selection, or ordinary topics but inflammatory phrasing — are the most analytically revealing.

Default Thresholds Systematically Overestimate Sensationalism

The second finding carries a methodological warning: a classifier's default decision threshold systematically inflates corpus-level salience estimates. That is, applying a model off-the-shelf without calibration leads to a distorted conclusion that "news is more sensationalist than it actually is." The study proposes a precision-floor recalibration protocol to correct this bias. For any researcher using AI for large-scale content analysis, this is a trap worth watching out for.

A classifier's decision threshold is the cutoff that converts continuous model probabilities into discrete class labels, typically defaulting to 0.5. In settings with class imbalance or asymmetric evaluation objectives, the default threshold is often suboptimal. Precision-floor calibration is a threshold adjustment strategy that prioritizes guaranteeing precision: researchers first set an acceptable minimum precision requirement (e.g., at least 80% of positive predictions must be true positives), then raise the threshold to the lowest value satisfying that requirement, trading some recall for more reliable positive-class predictions. In tasks like detecting news sensationalism, an overly low threshold misclassifies many ambiguous samples as "sensationalist," inflating corpus-level rates. Precision-floor calibration offers a systematic correction method with important methodological implications for research relying on AI for large-scale content analysis.

Unequal Emotional Contexts Around Group Mentions

The most striking finding comes from the group mention analysis. Headlines mentioning Jewish people, the far right, and Muslims showed the highest detected salience rates — differences that cannot be fully explained by the overall composition of event contexts.

The researchers are quite measured in articulating the methodological limits here: the residuals are descriptive, not causal estimates under matched event conditions; and precision rates for each group's lexicon are reported. In other words, the data reveals significant differences in the "emotional context" in which different groups are placed, but stops short of asserting that this reflects deliberate media intent. This careful framing is exactly the kind of rigor that research on sensitive topics demands.

Significance and Open Resources

According to the research team, this is the largest audit of framing effects in French-language news headlines to date. Its value lies not only in the findings themselves, but in its methodological contributions: decomposing framing into two measurable dimensions, flagging the distortion risk of classification thresholds, and maintaining causal caution in sensitive group analysis.

The team has also released the supervised dataset, lexicons, and analysis code, providing a reproducible foundation for follow-up research. For scholars working on media bias, computational communication studies, and the application of LLMs in the social sciences, both the framework and data offer direct reference value. It also reminds us that when AI is used to "audit" the human information environment, the biases of the tools themselves equally warrant auditing.