Healthcare AI Has a Risk Problem [Guest]

Rethinking How We Evaluate Healthcare AI in the Real World

and

Jul 04, 2025

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction. We run on a “pay what you can” model—so if you believe in the mission, there’s likely a plan that fits (over here).

Every subscription helps me stay independent, avoid clickbait, and focus on depth over noise, and I deeply appreciate everyone who chooses to support our cult.

Help me buy chocolate milk

PS – Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Sarah Gebauer, MD

, is a physician, AI expert, and founder of Validara, where she helps healthcare AI companies validate their products for real-world clinical use. She previously led AI model evaluation work at RAND focused on national security and runs a 500+ member Slack group of physicians interested in AI.

Her work, available at the excellent,

Machine Learning for MDs by Sarah Gebauer, MD

effortlessly bridges the worlds of clinical practice and AI governance with a sharp clarity few can match. Her latest piece cuts through the noise, exposing how outdated risk models leave healthcare dangerously exposed. Rather than fixating on isolated technical flaws and static governance checklists, Sarah proposes a broader, mitigation-based integrity framework that directly addresses how AI systems actually fail in the real world.

As you read the article, I would love to get your answers to the following questions—

Sarah argues that hospitals ask the wrong questions. Why do you think they do this? Is it simple ignorance, or is it a calculated move? Who gains by focusing on a practically useless but legally defensible question?
The framework divides risk into Misuse, Mistakes, and Structural. "Structural Risk" is a huge, abstract category. If you were the Chief Medical Officer of a hospital, how would you translate that category into a concrete go/no-go decision for a new AI tool? At what point does the map stop being useful without a manual for how to navigate the terrain?
Imagine you are an AI vendor being evaluated by a hospital that has adopted this framework. Your model has a known, but manageable, bias issue (a "Mistake by AI"). How do you use the language of this framework to frame the issue in a way that secures the sale? What is the vendor's counter-move to this new mode of scrutiny?
If this risk framework became the industry standard and was rigorously enforced, what are the most likely second-order consequences? Does it kill innovation by making the cost of entry too high for startups? Does it inadvertently create a monopoly for the few tech giants who can afford this level of validation? What happens to VC investment flows under this new standard?
Sarah intentionally excludes AGI-style "misalignment," where the AI is an adversary. But couldn't a more mundane version already be a risk? An AI designed to maximize hospital revenue (e.g., by optimizing billing codes or patient throughput) has different goals than one designed to maximize long-term patient health. Is this a simple "Misuse," or is it a form of systemic misalignment we are not yet prepared to name?
This framework is a powerful tool for a single institution. What would it take to make this kind of thinking the dominant standard for the entire healthcare industry? Which power brokers—regulators, insurance companies, medical associations—would need to be either forced or persuaded to adopt it?

Let’s get into it.

I’ve always been a terrible surfer. The handful of times I’ve tried, I ended up mostly slamming into the ocean while all my kids somehow glided effortlessly over the waves. It’s not for lack of trying; I’ve watched videos and even taken lessons. Despite my complete lack of aptitude, I appreciate the sport for its clear analogy to life’s unpredictability. No matter how much you know what you’re supposed to do, you can’t predict exactly when the next swell will hit or how big it will be. All you can do is learn to recognize the patterns, adjust your stance, and be ready for whatever comes.

Healthcare AI is much the same. Hospitals are starting to implement healthcare AI applications. But when it comes time for vendors to answer basic governance questions like “What training data did you use?” or “Is the model explainable?”, the responses are often unsatisfying and vague.

Why?

Because these questions are the wrong ones. In a complex, evolving system like healthcare AI, focusing solely on the technology itself is like trying to predict every single wave in the ocean. Instead, we need to understand the underlying currents—the risk categories and domains that can fundamentally disrupt clinical practice.

Approaches to Risk Identification

Earlier this year, Google released An Approach to Technical AGI Safety and Security, which describes in 145 pages the way they think about risk and how to mitigate it. They group risks by how they might be mitigated rather than trying to guess every single risk that might occur in the complexity of the real world:

“it is helpful to identify broad groups of pathways to harm that can be addressed through similar mitigation strategies. Since the focus is on identifying similar mitigation strategies, we define areas based on abstract structural features (e.g. which actor, if any, has bad intent), rather than concrete risk domains such as cyber offense or loss of human control. This means they apply to harms from AI in general, rather than being specific to AGI.”

In healthcare, we don’t often use a little devil cat icon (Is it a cat? Or an angry pig?)

Healthcare AI could benefit from a similar shift toward mitigation-based risk grouping. Currently, risk identification in healthcare AI tends to be narrowly focused and static, often linked to the specific technology rather than the broader context of its use. Governance questions often center around technical details, like the training data used or the model’s underlying algorithms. These questions may apply to some, but not situations, and they rarely capture the real-world challenges that arise once the technology is in use.

The Problem with Technology-Specific Risk Thinking

Many of today’s governance frameworks ask questions that feel increasingly irrelevant as AI technology advances. For example, when hospitals evaluate AI tools, they usually include multiple questions about the training data as a way to assess potential bias. But in practice, that focus can be misleading. Consider a dermatology AI model trained predominantly on images of white skin. The problem isn’t just the data itself but the broader domain of subgroup bias. The real question is: How does the model perform across diverse populations?

The root issue is that governance questions are often based on specific technical features rather than broader risk categories. As a result, they fail to adapt as the technology evolves. Instead of focusing on isolated risks, we should be asking: What are the mitigation-based risk categories inherent to any healthcare AI system?

One way to think about mitigations and risk is through the lens of AI biosecurity, which is adapted from a long history of managing biological threats. Instead of predicting every possible failure, AI biosecurity involves identifying potential pathways to harm, as we did in this Nature Communication paper, and building frameworks that can address them regardless of the specific technology involved.

Shifting the Focus to Risk Categories and Domains

I’ve developed a more adaptable approach that I call the Healthcare AI Integrity Framework. Rather than fixating on individual technical flaws, it organizes risks into Domains, which are grouped by the kinds of mitigations that would need to be put into place, and the effects by Impact Dimensions, which are outcome considerations for healthcare interventions.

The Domain categories are Misuse, Mistakes, and Structural Risks. The three Impact Dimensions of Clinical and Safety, Legal and Regulatory, and Reputational and Ethical span the domains. By framing risks in this way, we move from a narrow, technology-specific perspective to one that considers how AI systems integrate into the complex, real-world environment of healthcare.

Domains

Misuse: These risks occur when AI tools are exploited intentionally or through negligence. Financial exploitation, patient data misuse, and user behavioral manipulation fall into this category. These are scenarios where AI is used to maximize profit or control behavior at the expense of patient safety or ethical standards. For example, optimizing documentation to inflate RVUs without clinical necessity is a form of financial exploitation that compromises patient trust.

Mistakes: This category is divided into mistakes by the AI system and mistakes by the user. Mistakes by AI include algorithmic errors, subgroup bias, contextual misunderstandings, and even hallucinations in generative models. Mistakes by users—like clinical overreliance on AI suggestions, data entry errors, and poor workflow integration—can be just as damaging. In practice, these mistakes often manifest as false positives or negatives, mistargeted interventions, or workflow breakdowns that disrupt patient care.

Structural Risks: These are systemic vulnerabilities that emerge from the broader integration of AI into clinical settings. They include multi-agent interactions where different AIs interact in unpredictable ways, systemic workflow failures that create bottlenecks or safety risks, insufficient infrastructure to support meaningful AI deployment, and inadequate governance that leaves gaps in accountability. These risks are less about individual model failures and more about the way AI is embedded into the clinical ecosystem.

Impact Dimensions

Clinical and Safety Risks: These risks address direct impacts on patient outcomes and clinical workflows. For example, algorithmic errors (a Mistake by AI) can lead to missed diagnoses or incorrect treatment recommendations. Human-AI interface errors (a Mistake by the User) can disrupt clinical workflows, leading to delays or miscommunication in patient care. Structural risks, like multi-agent interactions, can compromise clinical decision-making if one AI system's output conflicts with another’s.

Regulatory and Legal Risks: This domain covers compliance with medical regulations, data privacy laws, and standards for clinical validation. Misuse of patient data or financial exploitation by AI tools can result in legal repercussions and breaches of regulatory guidelines. Structural risks like inadequate governance can expose hospitals to fines and legal action if AI-driven decisions violate patient rights or safety standards.

Reputational and Ethical Risks: Trust in healthcare is hard-earned and easily lost. Misuse, like behavioral manipulation or patient data misuse, can rapidly erode patient confidence. Mistakes, especially those that reflect bias or unsafe clinical recommendations, can lead to widespread distrust. Structural failures that result in workflow breakdowns or visible patient harm can damage the reputation of both the technology provider and the healthcare institution.

Note that I did not include misalignment, or the AI system deliberately acting as an “adversary” from Google’s framework. This is a huge area of research but it is out of scope for this discussion. Also note that it might seem like misaligned AI should be the same as AI mistakes, as in “misaligned with goals”. To avoid confusion with the AGI misalignment term that defines it as an AI system as an adversary, I’ve kept “misalignment” to that definition.

The Path Forward

Good surfers don’t try to predict every wave. They study the patterns, anticipate both risks and opportunities, and adapt in real time—shifting their weight, adjusting their stance, and making quick decisions to stay upright. They aren’t trying to control the ocean; they are learning to move with it. That’s exactly what effective governance for healthcare AI should look like. We can’t predict every risk, but we can recognize the domains where failure is most likely, and what impacts that might have, and prepare for those scenarios.

Instead of asking, “How was this model trained?” we should be asking, “What risk domains are inherent to this AI system? How are these risks likely to interact within a clinical setting?” This reframing is more than just a theoretical shift. It can align the questions we’re asking with what we really want to know and make healthcare AI better, safer, and more reliable.

Sarah Gebauer, MD, is a physician, AI expert, and founder of Validara, where she helps healthcare AI companies validate their products for real-world clinical use. She previously led AI model evaluation work at RAND focused on national security and runs a 500+ member Slack group of physicians interested in AI. She’s traveled to over 50 countries with her husband and four kids, and writes at sarahgebauermd.substack.com.

Thank you for being here, and I hope you have a wonderful day.

Dev <3

I provide various consulting and advisory services. If you‘d like to explore how we can work together, reach out to me through any of my socials over here or reply to this email.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.