How AI Emergency Triage Works in Practice: An Auditable Approach to Maternal and Newborn Care in India

When Medical AI Meets the Challenge of Scale

At Noora Health in India, nurses respond to over 50,000 medical inquiries per month via WhatsApp, providing always-on support for caregivers. Among all their responsibilities, the most urgent and judgment-intensive task is emergency triage — rapidly identifying which inquiries require immediate in-person medical attention from the massive volume of messages.

Emergency Triage is a core component of emergency medicine, aimed at prioritizing patients based on the urgency of their conditions under limited resources. In developed countries, emergency departments typically use standardized tools like the Manchester Triage System or the Canadian Emergency Department Triage and Acuity Scale (CTAS) to guide this process. However, in community healthcare settings in developing countries like India, triage often occurs outside formal medical institutions — remote consultations through instant messaging tools are a prime example. Noora Health faces a particularly unique challenge: the window for maternal and neonatal emergencies is extremely narrow. Conditions like postpartum hemorrhage and neonatal asphyxia can become life-threatening within hours, making the timeliness and accuracy requirements for remote triage far more demanding than those for general health consultations.

The margin for error in these judgments is razor-thin: a single missed case could mean a mother or newborn loses their best chance at timely treatment, while an over-escalation consumes already scarce clinical resources. To assist nurses with this work, Noora Health had previously built a system based on a large language model (LLM) to determine whether a given message constituted an emergency and provide an explanatory rationale.

However, it was precisely this seemingly advanced system that revealed deep engineering and clinical challenges when deployed at real-world scale.

The "Black Box Problem" of End-to-End LLM Triage

The original system followed a common industry approach: feeding user messages directly to a large language model, which would output an "urgent or not" judgment along with a reasoning chain to enhance explainability. This approach often performed well in demos, but under the real-world load of tens of thousands of queries per month, problems quickly surfaced.

Explainable Doesn't Mean Auditable

The research team highlighted a critical but often overlooked distinction: a model providing a rationale is not the same as a system being auditable. When the system made errors, analyzing them meant reading through the reasoning chain for each individual message — at Noora Health's scale, this was a nearly impossible task.

Reasoning chains, or Chain-of-Thought (CoT) prompting, have been an important paradigm in LLM research since 2022, first proposed by Wei et al. at Google Brain. The core idea is to improve accuracy on complex tasks by prompting the model to show its reasoning step by step. In medical AI, this kind of explainability is especially important because both regulators and clinicians need to understand the basis for AI judgments. However, CoT provides a form of "post-hoc rationalization" — the reasoning steps generated by the model don't necessarily reflect its actual internal computation. This is the fundamental reason the Noora Health team distinguishes between "explainable" and "auditable": the former only requires providing seemingly reasonable justifications, while the latter demands that every decision step can be independently verified and traced.

The Cascading Effects of Prompt Adjustments

Even more challenging was the maintenance cost. In an end-to-end LLM approach, any adjustment to the prompt could trigger unexpected regressions. To ensure changes wouldn't cause previously correct judgments to go wrong, the team had to re-run an entire evaluation pipeline. This was both expensive and operationally unsustainable, severely slowing down the system's iteration speed.

Prompt Engineering is currently the primary way of interacting with large language models, but it fundamentally lacks the mature version control and regression testing mechanisms found in traditional software engineering. In conventional software development, modifying a single line of code can be quickly verified through unit tests. In LLM systems, however, minor prompt changes — even subtle adjustments in wording — can cause unpredictable behavioral shifts in certain edge cases, a phenomenon known as "prompt brittleness." For a system processing 50,000 queries per month, every prompt change requires re-evaluation across a large body of historical cases, with substantial computational and human costs.

The "Gap" in Clinical Decision Knowledge

The most fundamental issue lay in knowledge representation. When clinicians make triage decisions, they actually follow a structured decision tree. But this decision tree had never been formally documented, nor had it been communicated to the model. What the model relied on was merely a flat list of "danger signs" — a clear disconnect from how clinicians actually reason.

Clinical decision trees are widely used decision support tools in evidence-based medicine, breaking down complex diagnostic and treatment workflows into a series of binary or multi-branch decision nodes. For example, a typical maternal emergency decision tree might first assess "Is there vaginal bleeding?" and if yes, proceed to "Is gestational age over 28 weeks?" and "Is bleeding volume greater than menstrual flow?" This hierarchical logical structure is naturally suited to rule-based encoding. However, in actual clinical practice, a vast amount of decision trees exist as "tacit knowledge" in the experience of senior physicians and have never been systematically documented. The field of Knowledge Engineering has long worked to address this problem, and Noora Health's practice essentially accomplished a critical transformation from tacit knowledge to explicit rules.

From Black Box to Two-Stage Pipeline: Deconstructing the Triage Architecture

Facing these challenges, the team made a pivotal architectural decision: decomposing the monolithic triage task into two independent steps.

Step One: LLM Handles Symptom Extraction and Semantic Translation

In the new architecture, the large language model no longer makes direct judgments. Instead, it handles the natural language understanding work it excels at: extracting normalized symptoms and patient context information from user queries. This extraction process is based on a standardized vocabulary written by clinicians, ensuring structured and predictable outputs.

Normalized Symptom Extraction is one of the core tasks in medical natural language processing. Patients often describe symptoms using colloquial, regional, or even metaphorical expressions — for example, saying "my stomach feels like it's being cut with a knife" to describe abdominal pain, or using local dialects to describe specific symptoms. The goal of normalization is to map these diverse expressions to standard medical terminology (such as concepts in SNOMED CT or ICD coding systems). In Noora Health's scenario, this challenge is even more pronounced because consultations occur via WhatsApp, and messages often contain spelling errors, code-mixing (e.g., Hindi mixed with English), and abbreviations. LLMs have a significant advantage in processing such non-standard text, which is why the team chose to have the LLM focus exclusively on this step.

Step Two: Deterministic Rule Engine Executes Triage Decisions

The actual "urgent or not" judgment is handled by a deterministic rule engine. This engine precisely captures the scenarios that indicate emergencies, essentially encoding the decision trees in clinicians' minds as executable rules.

A Deterministic Rule Engine is a classic expert system architecture whose core characteristic is that given the same input, it will always produce the same output. Unlike probability-based LLMs, every rule in a rule engine is transparent, traceable, and independently modifiable. In engineering practice, this means: first, adding a new rule won't affect the behavior of other rules, eliminating regression risk; second, when an output is wrong, the specific rule that triggered it can be precisely identified; third, rule execution is extremely fast with virtually negligible computational cost. In the context of increasingly stringent medical regulations — such as the EU AI Act, which classifies medical AI as a high-risk category — this determinism and traceability is not just an engineering advantage but a compliance necessity.

This decomposition delivers dual benefits: the model handles what it's best at — ambiguous language — while rules handle what they're best at — deterministic logic. Each component plays to its strengths.

A Win-Win for Triage Accuracy and Auditability

The new system's effectiveness is clearly reflected in the metrics:

• Recall improved from 0.565 to 0.810 — the system catches significantly more true emergencies, drastically reducing the risk of missed cases
• F1 score improved from 0.606 to 0.702 — overall accuracy improved substantially

In medical triage, an asymmetric-risk scenario, recall is far more important than precision. Recall measures "of all truly urgent cases, how many did the system successfully identify" — improving from 0.565 to 0.810 means the miss rate dropped from 43.5% to 19%. At a scale of tens of thousands of queries per month, this could mean thousands of patients saved from being incorrectly cleared. The F1 score is the harmonic mean of precision and recall; its improvement indicates that the system didn't simply flag all cases as urgent (which would yield 100% recall but extremely low precision), but rather increased sensitivity while maintaining reasonable specificity.

A notable detail: the research team's analysis found that structured rules were the primary driver of accuracy improvement, while task decomposition delivered the equally valuable benefit of auditability.

How Auditability Is Concretely Achieved

When the system makes an error, clinical experts can now inspect each stage to pinpoint exactly where the problem lies:

• Was the query incorrectly translated?
• Were symptoms incorrectly extracted?
• Was patient context incorrectly inferred?
• Or was a necessary rule missing?

This layered attribution capability transforms error correction from "reading countless reasoning chains" to "pinpointing the specific stage." More importantly, clinical experts can independently add new rules without triggering regressions and without needing to run expensive full-scale evaluations — establishing a rapid correction feedback loop.

Real-World Deployment Data and Operational Results

This system isn't just a paper exercise — it's already running in production at Noora Health. Since deployment, it has completed triage on 152,421 patient queries, of which 28,535 (18.7%) were flagged as emergencies.

The system's over-escalation rate is 17.8%, and while improving recall, there was no increase in missed emergency cases — a particularly critical outcome in healthcare, demonstrating that the system strikes a reasonable balance between "better to over-report" and "never miss a case." Notably, a 17.8% over-escalation rate is quite impressive in the medical triage domain — by comparison, many emergency departments in developed countries have over-triage rates of 30%-50%.

The most compelling evidence of the design's intent: since deployment, clinicians have independently added 48 new rules. This is a direct manifestation of the "faster correction cycle" the team sought — the system is no longer a black box requiring engineer intervention, but a tool that clinical experts can continuously maintain and evolve.

Key Takeaways for AI Deployment in High-Stakes Domains

The lessons from this research extend far beyond a single use case. It reveals a principle that is critically important when deploying AI in high-stakes domains: an end-to-end large model isn't necessarily the optimal solution.

Combining the LLM's language understanding capabilities with the controllability of a deterministic rule engine to form a "neuro-symbolic" hybrid architecture often achieves a better balance among accuracy, maintainability, and trustworthiness. Neuro-symbolic AI is one of the most active directions in current AI research, with the core idea of fusing neural networks' perception and generalization capabilities with symbolic systems' logical reasoning and explainability. Teams at MIT (Josh Tenenbaum), DeepMind, and IBM Research are all actively advancing this paradigm. In practical applications, neuro-symbolic architectures have demonstrated advantages across multiple domains: in drug discovery, LLMs parse literature while rule engines verify molecular structure compliance; in financial compliance, LLMs interpret transaction descriptions while rule engines enforce regulatory provisions. Noora Health's case provides highly compelling evidence for applying this architecture in low-resource healthcare environments.

Especially in fields like healthcare, finance, and law where auditability requirements are extremely high, enabling domain experts to directly inspect and modify system logic may be just as valuable as the intelligence of the model itself.

For all teams exploring AI deployment, Noora Health's practice reminds us: technological success isn't just about how smart the model is — it's about whether the system can be understood, trusted, and continuously improved by people.