How Flo Health Built a Safer Open Model with Databricks

From 81 Million MAU to AI Safety: Flo Health's High Bar

Flo Health is one of the world's most popular women's health apps, with 81 million monthly active users. For a product at this scale, the bar for generative AI is far higher than in ordinary scenarios—a single piece of bad advice could carry medical risk, and a single data leak could shake user trust.

According to a Databricks official blog post, Flo Health did not choose to call closed-source LLM APIs directly. Instead, it built a more controllable and safer open-model system on top of Databricks AI Runtime. Behind this choice lies a comprehensive trade-off involving privacy, compliance, and output quality.

Why Open Models Became the Priority

Flo Health's engineering team faced a core tension: women's health data is extremely sensitive, while general-purpose closed-source models often fail to meet medical-grade safety requirements.

• Data sovereignty: Data such as users' cycles, symptoms, and pregnancy plans is not allowed to leave a controlled environment. Open models can be privately deployed, avoiding data egress.
• Customizability: General models have limited understanding of terminology, tone, and boundaries in the women's health domain, so domain knowledge must be injected through fine-tuning.
• Cost control: With 81 million MAU, inference volume is enormous, and token-priced closed-source APIs see costs soar at scale.
• Compliance auditing: The healthcare domain requires model behavior to be traceable, and open weights let the team inspect, intervene, and manage versions.

Databricks AI Runtime provides a complete chain from model fine-tuning and deployment to inference optimization, allowing these goals to be realized on a single platform.

Open Models and Open-Source Base Models: "Open models" here generally refer to large language models whose weights are public, downloadable, and allowed to be privately deployed and fine-tuned, such as the Llama and Mistral series. Unlike closed-source APIs, the weight files of open models can run in their own or a controlled cloud environment, with no need for data to leave the domain. For Flo Health, this means sensitive data such as women's cycles, symptoms, and pregnancy plans always stays within compliance boundaries. At the same time, open weights allow the team to inspect the model's internal behavior, inject domain terminology, adjust output boundaries, and perform auditing and rollback at the version level. It does not mean "free" or "no engineering investment." On the contrary, it requires the team to have a full set of runtime capabilities, including compute scheduling, inference optimization, and evaluation pipelines; otherwise, it is difficult to achieve production-grade reliability and safety.

Safety Guardrails: A Priority Over "Usefulness"

Flo Health made it clear that when building AI features, "safety" and "usefulness" are not并列—safety is a precondition.

On Databricks AI Runtime, the team embedded safety policies into the model lifecycle:

• Input filtering: Identify high-risk queries such as self-harm or emergency medical conditions and route them directly to professional help rather than letting the AI answer.
• Output constraints: Add domain safety examples during fine-tuning to restrict the model from giving diagnostic conclusions.
• Human evaluation loop: Clinical experts participate in annotation and test set construction to continuously calibrate model boundaries.
• Observability: Record anomalous patterns during inference to facilitate rapid rollback or adjustment.

The value of this mechanism is that it does not simply add a filter outside the model, but internalizes safety capabilities into the fine-tuning data and evaluation process.

Model Fine-Tuning and Safety Alignment: Fine-tuning is the process of continuing to train a pre-trained base model on domain data so that the model adapts to a specific task, terminology, and tone. In healthcare scenarios, fine-tuning is not only used to improve answer quality but also carries the responsibility of safety alignment. Flo Health mixes high-risk examples such as self-harm and emergency medical conditions, boundary cases annotated by clinical experts, and instruction data such as "do not give diagnostic conclusions" into the fine-tuning set, so that the model learns at the parameter level to refuse or refer rather than relying only on an external filter to intercept. The advantage of this approach is that safety behavior is more stable and harder to bypass, but it also requires the fine-tuning data itself to undergo strict review to avoid introducing bias or incorrect knowledge. The fine-tuned model still needs continuous evaluation and version management to respond to changes in boundary cases.

Engineering Practice: How Open Models Reach Production Grade

Based on public information, Flo Health's path is not "training from scratch" but domain adaptation based on open-source base models.

Databricks AI Runtime plays several key roles in this:

1. Unified compute scheduling: Fine-tuning, batch inference, and online serving share the same infrastructure, reducing environment migration costs.
2. Inference performance optimization: For latency-sensitive health Q&A scenarios, control response time through techniques such as quantization and batching.
3. Versioning and rollback: Every model update has a complete record, and in case of problems the system can quickly switch to the previous stable version.
4. Evaluation pipeline: Combine automated metrics with human review to avoid "out of control as soon as it goes live."

For teams also handling sensitive data, Flo Health's practice shows that productionizing open models is not simply "download the weights and run them," but requires support from a complete set of runtime capabilities.

Quantization and Batching in Inference Performance Optimization: When a model faces 81 million monthly active users, inference latency and cost directly determine experience and feasibility. Quantization converts model weights from high-precision floating point formats (such as FP16) to lower-precision formats (such as INT8 or INT4), thereby reducing memory usage and improving throughput, but it may cause slight quality loss and requires calibration for health Q&A scenarios. Batching combines multiple user requests into a batch for simultaneous inference, fully utilizing GPU parallelism and reducing the cost per request. Together, these can support large-scale concurrency within acceptable latency. On Databricks AI Runtime, Flo Health uses these techniques to control response time, while version rollback and evaluation pipelines ensure that optimization does not sacrifice safety or answer quality.

Implications for the Industry

Flo Health's case points to a broader trend: in heavily regulated fields such as healthcare and finance, open models plus controlled runtime are becoming a more realistic choice than pure closed-source APIs.

It also reminds practitioners that model capability is only the starting point. What truly determines whether a product can go live is safety evaluation, data governance, and engineering observability. The role Databricks AI Runtime plays in this chain is essentially to transform "research-grade models" into "auditable production systems."

For teams focused on implementing AI applications, it is worth thinking further: in your domain, which safety red lines must be solved inside the model rather than through external patches?

Controlled Runtime and Auditable Production Systems: Controlled runtime refers to a set of infrastructure that uniformly manages model fine-tuning, deployment, inference, monitoring, and version rollback. It turns open models from "research-grade weights" into "auditable production systems." In heavily regulated fields such as healthcare and finance, regulators care not only about whether model outputs are correct, but also about whether every decision can be traced and whether it can be quickly taken offline or rolled back when problems are found. The observability, version records, and evaluation pipelines provided by Databricks AI Runtime are exactly the key to meeting such audit requirements. It integrates safety policies, data governance, and engineering operations into the same platform, reducing the complexity for teams switching among multiple tools. For practitioners, choosing an open model is only the first step; the accompanying runtime capabilities are the watershed that determines whether it can be scaled into production.

One-line Summary

Flo Health built a medical-grade AI safety system with open models on Databricks AI Runtime.

Paragraph Summary

With 81 million monthly active users and highly sensitive women's health data, Flo Health chose to build an open-model system on Databricks AI Runtime instead of calling closed-source APIs directly. Its core considerations include data sovereignty, domain customizability, cost control at scale, and compliance auditing. Safety is treated as a precondition, internalized into the model lifecycle through input filtering, fine-tuned output constraints, a clinical expert evaluation loop, and observability. Engineering relies on unified compute scheduling, inference optimization, version rollback, and evaluation pipelines to bring open models to production grade. The case shows that in heavily regulated industries, open models plus controlled runtime are becoming a more realistic path than pure closed-source APIs. Model capability is only the starting point; safety evaluation, data governance, and engineering observability determine whether it can go live.