Stop Removing Stopwords: How an Inherited Preprocessing Default Distorts Legal Text Data Analysis

In empirical legal scholarship, a growing number of researchers treat judicial opinion text as data for analysis. Most of these studies still rely on sparse, interpretable pipelines — TF-IDF features paired with linear classifiers. The reason is straightforward: in this type of research, the text features themselves are often the object of study, not merely a means to prediction. Yet this pipeline inherits a chain of preprocessing defaults from mid-twentieth-century information retrieval, none of which have ever been rigorously validated against classification accuracy. The most deeply entrenched of these defaults is stopword removal.

A newly published arXiv paper mounts a systematic challenge to this taken-for-granted practice, with a blunt central finding: in interpretable legal text analysis, retaining stopwords is the right choice — removing them may distort the jurisprudential and ideological signals the research is meant to capture.

Why the Inherited Default Deserves Scrutiny

Stopword removal is one of the most common operations in text preprocessing. High-frequency function words like "the," "of," and "and" are generally considered to carry no topical information, and so they are stripped out before modeling. This habit originated in information retrieval systems from decades ago — in an era of tight storage and compute constraints, removing these words saved costs and simplified indexing.

The problem is that this rule was never rigorously validated against the downstream goal of classification accuracy when it was introduced, yet it has been carried forward by generation after generation of researchers as an unquestioned starting point. The paper's authors make a sharp observation: a step that quietly changes which features a model can see poses serious measurement validity problems in legal text analysis, where the features themselves are the research object. In other words, you may think you're studying judges' ideological leanings, but preprocessing may have already erased the key signals.

TF-IDF (Term Frequency–Inverse Document Frequency) is the core component of this traditional pipeline. It measures word importance along two dimensions: how often a word appears in a single document (TF), and how rare it is across the entire corpus (IDF). A word that appears frequently in a given opinion but rarely across all opinions receives a high weight — the classic signature of a substantive content word. The reason stopwords are considered safe to remove is precisely that they appear universally across all documents, driving their IDF toward zero and their contribution to final weights to near nothing. But this logic only holds when "we care about topical information." When the research goal shifts to judges' argumentative style, ideological leanings, or rhetorical patterns, distributional differences in function words may themselves be signal — judges of different ideological orientations may exhibit systematic differences in citation, transition, and hedging structures, differences carried precisely by the "ignored" function words.

An Exhaustive Single-Word Ablation Approach

To directly test the impact of preprocessing steps on downstream objectives, the paper proposes an exhaustive single-word ablation method. Rather than relying on expert intuition to compile a stopword list, it removes each of approximately 18,500 candidate words one at a time and directly measures the effect on classification outcomes.

This approach effectively approximates the upper bound of "the best stopword list any expert could construct" — if even this theoretical optimum yields no improvement, then any stopword list actually in use has even less justification. The authors chose stopword removal as their test case precisely because it is the most entrenched default; demonstrating its failure is the most persuasive possible argument.

The study uses two binary classification tasks as test scenarios, chosen to span different ranges of F1 improvement headroom: predicting the ideological direction of opinions (baseline F1 without removal ≈ 0.68), and distinguishing constitutional from non-constitutional law (baseline F1 ≈ 0.92). The data comes from matching Supreme Court Database labels with Caselaw Access Project opinion text, covering 7,668 and 7,001 opinions respectively.

An "ablation study" is a systematic evaluation method common in machine learning: remove one component or input feature at a time, observe the performance change, and quantify that component's actual contribution. The technique originated in evaluating individual modules of neural networks; the paper adapts and extends it to the evaluation of preprocessing steps. Unlike typical ablation experiments that test only a handful of configurations, "exhaustive" ablation means testing every element of the candidate set individually — computationally expensive at vocabulary scales of tens of thousands, but also eliminating the confirmation bias of selective validation, since the researcher cannot cherry-pick a favorable subset to test. The Supreme Court Database is a decades-long judicial database in American political science, containing multi-dimensional coded information on Supreme Court cases since 1791, including the ideological direction (liberal/conservative) of decisions. The Caselaw Access Project, led by Harvard Law School, has digitized and made freely available the full text of millions of American court opinions. Their alignment provides a rare large-scale text-label dataset for empirical research.

Three Counterintuitive Findings

The experiments yield three compounding conclusions, all pointing in the same direction:

First, general-purpose stopword lists underperformed the no-removal baseline in every single test. The off-the-shelf stopword lists commonly used by researchers not only failed to help — they actively hurt classification performance.

Second, even optimized stopword lists were statistically indistinguishable from removing nothing at all. This means stopword removal, in its most favorable possible scenario, still delivers no measurable gain.

Third, a meta-model trained on word-level features cannot predict which words should be removed. This point is especially significant — it means there is no targetable signal for fine-tuning a stopword list, no foothold for human curation. If an algorithm cannot detect the pattern, expert intuition has even less to work with.

Specific Warnings for Legal Text Research

The paper emphasizes that this methodology can generalize to any inherited preprocessing default, not just stopwords. But its warnings are especially sharp for interpretable legal text data analysis: a step that silently changes the features a model can see may distort the jurisprudential and ideological signals on which this type of research depends.

In predictive NLP tasks, where only final accuracy matters, small losses from preprocessing may be tolerable. But in legal empirical research where "text features are the research object," the function words being deleted may themselves carry judges' expressive styles, argumentative logic, and ideological dispositions. Removing them is tantamount to contaminating the evidence before the study begins.

The authors close with a pointed conclusion: "Retaining stopwords is a measurement validity issue." This is no longer a question of efficiency or habit — it is a methodological baseline on which the credibility of research findings depends.

"Measurement validity" is a core concept in social science research methodology, referring to whether what a researcher actually measures truly corresponds to the construct they claim to be studying. In empirical legal research, when scholars claim to be measuring "judicial ideology," the implicit assumption is that their chosen text features reliably reflect judges' actual orientations, rather than being artifacts introduced by the measurement instrument itself. Preprocessing is a potential threat to measurement validity — if the feature extraction process systematically discards a category of information, then the "ideological differences" subsequently discovered may be a byproduct of feature selection rather than a reflection of real judicial behavior. This problem is especially acute in interpretable models: researchers often directly interpret the substantive meaning of high-weight feature words (e.g., "judges of a certain type use a certain word more frequently"), and if preprocessing has already distorted the feature set, such interpretations lose their foundation.

Practical Implications for Researchers

For scholars engaged in text-as-data research, this paper offers two directly actionable takeaways.

First, do not blindly inherit preprocessing defaults of uncertain provenance. Any step that alters the feature set should be validated against specific downstream objectives rather than adopted as an "industry standard."

Second, exhaustive ablation provides a reusable verification paradigm. When you are uncertain whether a given preprocessing operation is beneficial, rather than debating it, simply measure its actual effect on the target metric. With dramatically reduced computational costs today, this rigorous "test each one" approach has become entirely feasible.

This research reminds us that methodological "common sense" does not always hold up under scrutiny — especially when importing machine learning tools into social science and legal research, where every technical step taken for granted may quietly reshape the final conclusions.