LLM Unlocks Criminal Judgments: Flexible Information Extraction for Empirical Legal Research

Criminal judgment documents contain rich descriptions of criminal behavior, and these texts provide valuable material for quantitative legal research. However, transforming these unstructured narratives into analyzable data often faces a core challenge: the required attributes and rules for extraction are difficult to fully determine in advance. To address this challenge, researchers have proposed a flexible information extraction pipeline based on large language models (LLMs) and conducted an empirical evaluation in the field of traffic crime.

When Legal Research Meets Unstructured Text

Traditional empirical legal research relies on manual annotation, which is costly and difficult to scale. The emergence of LLMs has brought new possibilities for automated information extraction, but the complexity of legal texts—especially the free-form descriptions of criminal behavior in judgments—makes it impractical to predefine all extraction rules. The researchers point out that behavior descriptions in criminal judgments require flexible attributes and rules, and these often cannot be fully clarified before annotation begins.

To this end, they designed a set of annotation instructions iterated with legal experts, allowing experts to continuously adjust the rules based on extraction performance. This pipeline includes three key stages: attribute exploration, text span extraction, and value normalization.

Value normalization is a key step in the information extraction pipeline, referring to mapping extracted text spans to a unified, standardized representation. For example, different expressions extracted from judgments such as "blood alcohol content 0.8mg/ml" and "BAC 80mg/100ml" need to be normalized into unified values and units for subsequent statistical analysis and machine learning modeling. Normalization involves not only numerical unit conversion but also date format unification, legal terminology standardization, and entity name disambiguation. In the legal domain, normalization is particularly important because free-text expressions in judgments vary greatly; without normalization, extraction results would be difficult to aggregate and compare. This step usually requires combining domain knowledge bases and rules, and sometimes also uses LLMs for semantic matching.

Dual-Task Testing: DUI and DDB

The study selected two types of traffic crimes for testing: driving under the influence (DUI) and driving during license suspension (DDB). In the DUI task, after revising the annotation instructions, extraction accuracy increased from 52% to 70%, while inter-annotator agreement among human annotators also improved. This improvement indicates that instruction optimization driven by expert feedback can effectively improve LLM extraction performance.

One detail: the consistency among multiple runs of the same model was actually higher than their consistency with gold annotations. The researchers interpret this as: consistent output does not equal correct output; the model may stably make the same mistakes. Divergence among annotators and between models, meanwhile, becomes an effective signal for identifying ambiguities in instructions.

In contrast, in the DDB task, because the instructions defined fixed categories and output formats, annotation consistency was relatively high for most attributes. This shows that when extraction targets are relatively clear, LLM performance is more stable.

Gold annotations refer to high-quality reference standards produced by human experts through a rigorous annotation process (usually including independent annotation by multiple people, discussion of disagreements, and arbitration), and are regarded as the "ground truth" for evaluating model performance. In information extraction research, gold annotations are used to calculate metrics such as model accuracy and recall. However, gold annotations themselves may also contain errors or subjectivity, especially in cases with ambiguous boundaries in legal texts. This study emphasizes that comparing only the consistency of multiple model runs can obscure systematic errors; model outputs must be compared with gold annotations to truly reflect extraction quality. Constructing gold annotations is expensive, so exploring how to use LLMs to assist in generating or refining gold annotations is also a current research hotspot.

Balancing Flexibility and Accuracy

The value of this study lies in the fact that it does not attempt to build an "all-purpose" extraction system, but instead acknowledges the openness of legal texts and addresses uncertainty through a human-machine collaborative iterative process. Legal experts are no longer merely annotators, but designers and optimizers of instructions; LLMs are responsible for processing large-scale texts while exposing ambiguous areas requiring human intervention through divergence detection.

For empirical legal research, this approach provides a scalable path: first define attributes in an exploratory manner, then refine rules based on differences between model outputs and human judgment, and ultimately form reliable structured datasets. The results also remind us that when evaluating LLM extraction quality, we cannot look only at the model's own consistency; external gold standards must be introduced.

Annotation instructions here specifically refer to the definitions and rule descriptions of the extraction task in the prompt given to the LLM, rather than guidelines for human annotators in the traditional sense. In the LLM information extraction pipeline, the quality of instructions directly determines extraction performance. The innovation of this study lies in bringing legal experts into the iterative optimization loop of instructions: experts revise attribute definitions, boundary conditions, and examples in the instructions based on differences between model outputs and expectations, thereby improving extraction accuracy. This "human-in-the-loop" instruction engineering compensates for LLMs' insufficient understanding of subtle legal domain distinctions. The instruction iteration process itself also produces reusable domain knowledge, forming a virtuous cycle.

Practical Implications and Limitations

From the comparison between DUI and DDB, it can be seen that the specificity of instructions directly affects extraction performance. For attributes with clear categories and fixed formats, LLMs can complete the task efficiently; for attributes requiring semantic understanding and boundary judgment, repeated refinement of instructions by experts is more important. The generalizability of this pipeline still needs to be verified in more crime types and language environments, but its "exploration-extraction-normalization-iteration" framework provides a valuable reference for LLM applications in the legal domain.

In the future, with improvements in model capabilities and continuous optimization of annotation instructions, flexible information extraction is expected to become infrastructure for empirical legal research, helping researchers mine more quantifiable insights from massive numbers of judgments.

One-line Summary

This study proposes an LLM-driven flexible information extraction pipeline that improves accuracy in the traffic crime domain through expert-iterated instruction optimization.

Paragraph Summary

To address the challenge of information extraction from unstructured text in criminal judgments, this paper proposes a flexible extraction pipeline based on large language models (LLMs), comprising three stages: attribute exploration, text span extraction, and value normalization, while allowing legal experts to iteratively optimize annotation instructions. Empirical results on two tasks—driving under the influence (DUI) and driving during license suspension (DDB)—show that: in the DUI task, after instruction revision, extraction accuracy increased from 52% to 70%, and human annotator consistency also improved; the DDB task performed stably because its instructions were clear. The study also finds that consistency across multiple model runs is higher than consistency with gold annotations, emphasizing that evaluation must introduce external standards. This framework provides a scalable human-machine collaboration path for empirical legal research.