RAG Framework for Intersection Safety: Using LLMs to Convert Crash Narratives into Countermeasure Recommendations

The Old Problem in Traffic Safety Analysis: Expert Judgment Doesn't Scale

The traditional workflow for improving intersection safety relies heavily on the expertise of traffic safety engineers — identifying crash causes and then matching them to appropriate countermeasures. This expert-dependent model has clear bottlenecks: it's labor-intensive, hard to scale, and highly dependent on the availability of experienced engineers. When there are hundreds or thousands of intersections to evaluate, manual analysis one-by-one becomes practically infeasible.

What's often overlooked is that crash narratives — the descriptive text in accident reports — contain rich information about crash mechanisms, yet this unstructured data is largely untapped in traditional safety analysis. A study published on arXiv (arXiv:2609.15997) targets exactly this gap, proposing a Retrieval-Augmented Generation (RAG) framework that automatically converts crash narratives into intersection-specific countermeasure recommendations.

Framework Design: A Complete Pipeline from Narratives to Countermeasures

The core idea is to first extract key mechanistic attributes from crash narratives, then map them to a library of evidence-based countermeasures. The extracted attributes include: traffic control type, signal indication, driver fault, vehicle movement, and travel direction.

These mechanistic attributes are then linked to two authoritative sources: the Federal Highway Administration's (FHWA) Proven Safety Countermeasures, and the CMF Clearinghouse (Crash Modification Factor database). This means the model's recommendations are not generated from thin air — they are anchored to industry-recognized, evidence-based interventions.

Four Mechanisms Working in Concert

The framework doesn't simply rely on the LLM's free-form generation. Instead, it layers multiple structural constraints:

• Embedding-based retrieval: Uses embeddings to find historically similar intersection cases, providing reference points for the current intersection.
• Association rule mining: Discovers patterns between crash attributes and countermeasures from historical data.
• Statistical guidance: Provides statistical reference for the expected number of relevant countermeasures per intersection, preventing over- or under-recommendation.
• Engineering reasoning guidance: Guides the LLM through a domain-appropriate decision process before selecting countermeasures.

The key value of this design is that it makes the LLM's output interpretable and compliant with engineering standards, rather than a black-box suggestion. This is what fundamentally distinguishes this framework from simply calling an LLM directly.

---

The CMF Clearinghouse is a publicly accessible database maintained by the FHWA that catalogs a large number of empirically validated traffic safety countermeasures along with their corresponding Crash Modification Factors (CMFs). A CMF is a quantitative metric representing the expected change in crash frequency after implementing a safety improvement — for example, a CMF of 0.8 means the measure is expected to reduce crashes by about 20%. Engineers query the CMF Clearinghouse to select measures applicable to specific intersection conditions based on evidence quality and statistical confidence, rather than relying on intuition. Anchoring LLM outputs to this database means every recommendation has traceable empirical support — which is critical for high-stakes decision-making.

Association rule mining is a classic data mining method for discovering co-occurrence patterns in large transactional datasets — essentially finding "if A occurs, B tends to occur as well" — using metrics such as support, confidence, and lift. In this framework, historical crash data is transformed into "crash attributes → countermeasures" transaction sets. The mined rules can reveal patterns like "when an intersection lacks signal control and has left-turn conflicts, 80% of cases adopted protected left-turn phasing." These rules provide structured priors for the LLM's reasoning and also serve as a validation layer — when model suggestions contradict high-confidence historical rules, additional verification logic can be triggered. This is the key design choice that introduces explicit pattern knowledge beyond pure retrieval.

Real-World Performance: F1 Score of 0.82, Recommendation Counts Match Reality

The research team validated the framework on real data from Lake County and Sumter County, Florida, covering 115 intersections and 312 fatal and serious injury crashes, using five-fold cross-validation.

Results show the framework achieved:

• Precision: 0.82
• Recall: 0.85
• F1 Score: 0.82

In terms of recommendation volume, the framework averaged 3.91 countermeasures per intersection, of which 3.14 matched actual recommendations, while the actual average was 3.86 — nearly identical. This data shows the model performs well not only on "are the recommendations correct" but also on the often-overlooked dimension of "are there too many recommendations."

For transportation agencies, the accuracy of recommendation count matters just as much: over-recommending wastes limited improvement budgets, while under-recommending may leave critical hazards unaddressed.

---

Five-fold cross-validation is a standard method in machine learning for evaluating a model's generalization ability: the dataset is randomly split into 5 parts, and the model trains on 4 parts while testing on the remaining 1, cycling through all 5 combinations and averaging the results. Compared to a single train-test split, this approach provides a more robust estimate of real-world performance on unseen data — especially appropriate here given the limited sample size (115 intersections). It's worth noting that in the context of safety countermeasure recommendations, the precision-recall tradeoff carries asymmetric real-world implications: low precision means recommending unnecessary measures and wasting improvement budgets; low recall means missing genuinely needed countermeasures, potentially allowing safety hazards to persist. The fact that both metrics exceed 0.82 in this study indicates the framework achieves a fairly balanced outcome on this tradeoff.

Significance and Limitations: A Scalable Decision Support Tool

This research demonstrates the potential of retrieval-augmented LLMs as a traffic safety decision support tool. Its two main selling points are interpretability and scalability: the former comes from engineering reasoning guidance and evidence-based countermeasure library constraints; the latter addresses the fundamental problem of expert judgment not scaling. For local transportation departments with limited staffing facing large numbers of intersections to evaluate, tools like this can significantly lower the barrier to analysis.

That said, its limitations deserve an honest assessment. The validation is confined to data from two Florida counties — whether geographic differences in road networks, climate, and driving behavior would affect generalization remains to be tested at a broader scale. Additionally, the framework is highly dependent on the quality and completeness of crash narrative text — if original reports are poorly documented, the extracted mechanistic attributes will be correspondingly flawed.

Overall, this work offers a valuable paradigm: putting LLMs inside constrained, evidence-based pipelines rather than letting them make decisions alone. This may also represent a pragmatic path for deploying LLMs in high-stakes, highly regulated domains.