[KongchangAI]
· 2 min read· 1,065 words

Local LLM + Rule-Based Detectors: An Open-Source Toolkit for Clinical Text De-identification

Local LLM + Rule-Based Detectors: An Open-Source Toolkit for Clinical Text De-identification

DeidSharp combines a local LLM with rule/NER detectors to achieve high-recall clinical text de-identification.

DeidSharp is an open-source .NET toolkit that improves recall of Protected Health Information (PHI) in clinical text by running a locally deployed 26B-parameter Mixture-of-Experts LLM layered with regex and clinical NER detectors, using multi-detector voting. Evaluated on PhysioNet nursing notes and semi-synthetic MIMIC-IV-Note discharge summaries against deid, Philter, Stanford/Penn, and others, the recall-first stack achieved 98.9% recall on nursing notes and the precision-first stack 83.2% precision; in discharge summaries both stacks led Stanford/Penn by over 20 percentage points in recall. LLM self-verification discarded 25% of true PHI, requiring external detectors as a backstop. The solution suits local deployment in privacy-sensitive healthcare institutions.

De-identification of clinical text is a prerequisite for medical data sharing and secondary use, but achieving high recall while protecting privacy has always been a challenge. A recent study presents DeidSharp—an open-source .NET toolkit that combines a locally deployed large language model (LLM) with rule-based detectors, achieving performance superior to existing tools across multiple datasets.

rss source: A local language model layered with rule-based and neural detectors for clinical text de-identificat

Why Localized De-identification?

Traditional de-identification tools typically rely on cloud APIs or predefined rules. The former poses data leakage risks, while the latter struggles to cover the diverse PHI (Protected Health Information) expressions found in clinical text. DeidSharp's core approach is to run a 26B-parameter Mixture-of-Experts (MoE) LLM locally, layered with deterministic pattern matching and clinical named entity recognition (NER) models, improving recall through multi-detector voting. When any detector fails, the system "fails closed"—taking a conservative approach to avoid omissions.

Protected Health Information (PHI) is a core concept under the U.S. HIPAA Privacy Rule, referring to any information that can individually identify a person and relates to health status, healthcare services, or payment, including 18 categories of identifiers such as names, addresses, dates (other than year), phone numbers, medical record numbers, and device identifiers. The goal of clinical text de-identification is to detect and remove or replace these PHI so that data can be used for research and secondary purposes without revealing patient identity.

Mixture-of-Experts (MoE) architecture is a sparse-activation neural network design: the model contains multiple "expert" subnetworks and a gating network, activating only a few experts per forward pass. This allows the total parameter count to be large (e.g., 26B) while the actual computation is far lower than a dense model of equivalent size, balancing capability and efficiency for local deployment. DeidSharp chose an MoE LLM precisely for its ability to run larger models on limited hardware.

Multi-Detector Stacking Design

DeidSharp supports pluggable detectors, including regular expressions, clinical NER models, and local LLMs. The research team presets two configurations: recall-first and precision-first. The recall-first stack uses all detectors, while the precision-first stack removes the NER model to reduce false positives. This design allows users to trade off between recall and precision depending on the scenario.

Fail closed is a design principle in security engineering: when a system component fails or is uncertain, it defaults to the most conservative, safest action. In the de-identification context, this means that if a detector errors out or cannot make a determination, the system marks the segment as PHI and masks it rather than letting it through. The opposite is "fail open," where errors default to allowing content through. Medical privacy scenarios typically require fail closed, because the cost of missing PHI far exceeds that of false positives.

Clinical Named Entity Recognition (NER) models are specifically designed to extract entities from clinical text, such as person names, place names, organization names, dates, medications, and diseases. Compared to general NER, clinical NER must handle abbreviations, non-standard expressions, and dense medical terminology. In DeidSharp, the NER model primarily supplements entities such as place names that the LLM may miss, complementing regex and the LLM.

Experimental Design and Comparison Baselines

The study evaluated on PhysioNet nursing notes (889 PHI segments) and semi-synthetic MIMIC-IV-Note discharge summaries (29,023 PHI segments), comparing against tools including deid, Philter, Stanford/Penn de-identifier, OpenAI Privacy Filter, and GLiNER. Tuning and reporting sets were kept separate, using paired patient-level bootstrap confidence intervals to ensure statistical rigor.

Paired patient-level bootstrap confidence intervals are a statistical evaluation method. Bootstrap estimates the sampling distribution of a statistic through resampling with replacement; paired means comparing the predictions of two tools on the same set of patient samples in a paired manner to control for between-patient heterogeneity; patient-level means the resampling unit is the patient rather than the segment, preventing multiple segments from the same patient being split across different samples, which would cause overly narrow confidence intervals. This method is gaining increasing attention in clinical NLP evaluation because it more realistically reflects differences in tool performance on new patient populations.

Key Results: Significant Recall Improvement

In nursing notes, the recall-first stack improved recall by 2.8 percentage points over Stanford/Penn (reaching 98.9%), but precision was 9.3 points lower; the precision-first stack (without NER) achieved 96.5% recall, on par with Stanford/Penn, but with precision 19.6 points higher (83.2%). In discharge summaries, both stacks led Stanford/Penn by more than 20 percentage points in recall (recall-first reaching 95.3%), with nearly all of the gain coming from generated untyped names; on typed labels, all three exceeded 99%. All four preset comparisons held.

Interestingly, on deid's "home turf" corpus, no configuration significantly outperformed deid; Philter found more segments in discharge summaries but had precision 23 points lower.

LLM Is the Workhorse, but Self-Verification Is Risky

The discussion section notes that the LLM is the primary detector, with NER mainly supplementing place names. However, when asked to verify entities it had itself found, the LLM discarded a quarter of true PHI. This suggests that LLM self-verification is unreliable and requires external rules or models as a backstop.

Implications for Clinical NLP Practice

DeidSharp's significance lies in providing a localized, high-recall de-identification solution, particularly suited for healthcare institutions with strict data privacy requirements. Its open-source nature also facilitates community review and extension. However, the drop in precision reminds us that high recall often comes at the cost of more false positives, and deployment requires adjusting configurations based on downstream tasks. In the future, better integrating LLMs with rule-based detectors and reducing false negatives from self-verification remain worthwhile directions for exploration.

One-line Summary

DeidSharp combines a local LLM with rule/NER detectors to achieve high-recall clinical text de-identification.

Paragraph Summary

DeidSharp is an open-source .NET toolkit that improves recall of Protected Health Information (PHI) in clinical text by running a locally deployed 26B-parameter Mixture-of-Experts LLM layered with regex and clinical NER detectors, using multi-detector voting. The study evaluated on PhysioNet nursing notes and semi-synthetic MIMIC-IV-Note discharge summaries, comparing against deid, Philter, Stanford/Penn, and others. Results show the recall-first stack achieved 98.9% recall on nursing notes, and the precision-first stack achieved 83.2% precision; in discharge summaries, both stacks led Stanford/Penn by over 20 percentage points in recall. However, LLM self-verification discarded 25% of true PHI, requiring external detectors as a backstop. The solution is suited for local deployment in healthcare institutions with strict privacy requirements.

#clinical text de-identification#DeidSharp#local LLM#PHI#clinical NER#open-source toolkit#recall-first#privacy
Share:

Related articles