Why Health AI Is the Hardest to Ship: Three Key Lessons from a Frontline Engineer

In most jobs, if AI gets something wrong, you just fix the draft. In health, you may not get a second chance.

John is a senior AI and full-stack engineer who spent the past year building Tabibu, a health information assistant. In this article, he shares the question he gets asked most: why is health so much harder than any other domain where he's shipped AI?

The Cost of Errors: A Dosage Mistake Is Not a Typo

In ordinary products, a bad answer costs the user at most a retry. But in health, someone might act on the wrong information. That's why John changed how he asks the question: instead of asking "is this answer good?" he asks "what's the worst plausible wrong answer here?"

For Tabibu, that answer points to dosage. So the system's design principle is unambiguous: never tell anyone to start, stop, or change a medication. This isn't feature-level conservatism — it's a deliberate drawing of the risk boundary.

The lesson applies to any vertical AI developer: first identify the "worst plausible error" in your domain, then design hard limits around it.

The Tone Trap: AI Is Equally Confident Whether It's Right or Wrong

This was the thing that surprised John most as an engineer. The model sounds just as calm and fluent when it knows the answer as when it's purely guessing. Patients usually can't tell the difference, because telling the difference itself requires domain expertise.

In John's observation, the problem isn't whether the model expresses confidence — it's that users lack the basis to judge. His response was to stop treating tone as a credibility signal: Tabibu's answers show their sources, and when guidelines don't cover a question, the system says so explicitly and labels the answer as general information.

This lesson isn't limited to health. Any AI product aimed at non-experts should treat "sources visible" and "say so when uncertain" as baseline design, not optional extras.

The disconnect between model tone and correctness stems from how large language models are trained: they're optimized to generate "plausible-looking" continuations, not to calibrate their own confidence. A model's output probability distribution reflects how common something is in language, not how reliable it is in fact — which is why the model is just as fluent when it's making things up. Academics call this "miscalibration": the model's self-reported confidence doesn't match its actual accuracy. Mitigations include retrieval-augmented generation (RAG), which anchors answers in traceable literature or guidelines, and prompt-level instructions requiring the model to explicitly say "I don't know" when evidence is insufficient. But these methods can only reduce risk, not eliminate it — especially when users themselves lack domain knowledge and can't verify the sources.

The Regulatory Red Line: Strict Rules Have Their Reasons

When software influences medical decisions, regulators in many places treat it as a medical device — and that requires evidence. John is explicit that Tabibu makes no such claim: it's an information tool, it currently has no clinician review, and all limitations are listed on the tabibu.health/trust page.

For developers, the takeaway is to write your product boundaries down and make them testable, rather than hinting vaguely. Spelling out "what we don't do" is often more important than spelling out "what we can do."

The definition of "medical device" varies widely across jurisdictions, but it generally hinges on the software's intended use: if a piece of software is used to diagnose, treat, or prevent disease, or to influence clinical decisions, it may fall under medical device regulation. The U.S. FDA classifies some clinical decision support (CDS) software as medical devices, but may exempt software that merely displays information when a medical professional can independently review its basis; the EU's MDR takes a broader view of software "intended for medical purposes," and most health apps need CE certification. This means the same product can face completely different compliance paths in different markets. For developers, the key isn't to decide "am I a medical device?" but first to define the intended use and keep evidence, then use that to determine whether clinical evaluation, a quality management system (such as ISO 13485), and post-market surveillance are required.

Not Unsolvable: Safety Reminders Reduce Risk, But Never to Zero

According to a study published in Communications Medicine, researchers tested 20 AI models across more than 10 million responses. After adding a brief safety reminder, potentially harmful clinical choices dropped from 16.6% to 10.1%, with declines in 19 of the 20 models. John read the MedicalXpress coverage rather than the original paper, and the study's lead author also noted that reminders are only a safeguard and cannot replace clinical oversight.

ECRI's patient safety report likewise flags AI in diagnostic care as a concern. This matches John's firsthand experience: the question isn't whether AI is good or bad for health, but where it's safe enough — and who's responsible for gatekeeping.

Communications Medicine is an open-access journal from Nature focused on medicine and public health; the model evaluation studies it publishes are typically peer-reviewed, but sample, prompt design, and evaluation criteria can significantly affect how generalizable the conclusions are. The "10 million responses, 20 models" cited here is a large-scale automated evaluation, and its "potentially harmful clinical choices" are determined by scoring rules and may not fully correspond to real clinical outcomes. ECRI (Emergency Care Research Institute) is a nonprofit patient safety organization whose annual Top 10 Health Technology Hazards report is widely cited by hospitals; listing AI as a diagnostic care risk reflects frontline clinical consensus on deployment risk, not a regulatory determination. When reading this kind of research, it's important to distinguish "harm rates in an evaluation setting" from "real-world harm events."

A Question for Developers

John closes with a question: is there any health decision you would never hand over to AI? He plans to discuss the builders of these tools, the business models, the rulemakers, and the failure scenarios in follow-up content.

For teams in or entering health AI, these three lessons all point in the same direction: treat the risk boundary as a first-order product design question, not an after-the-fact patch. Safety reminders help but never reach zero; a confident tone doesn't mean a correct answer; and spelling out your boundaries is more professional than hiding them.

General information, not medical advice.