The Real Risk: Automation Bias in Medical Training (Or: How I Learned to Stop Trusting blindly)

If you’re feeling the same discomfort, you’re not alone. A recent article in BMJ Evidence-Based Medicine titled “Potential risks of GenAI on medical education” lays out the concerns we should all be paying attention to. This paper does not sugarcoat.

The authors identify several critical risks. Let’s walk through them, shall we? Consider this your differential for “Things That Could Go Wrong When We Let Robots Teach Medicine.”

Automation Bias: After extended use and over-reliance, trainees accept incorrect recommendations from AI systems. Generic warnings against this don’t prevent it—the root cause is over-trust. Medical training should include practice in questioning AI outputs, not just using them.

Cognitive Off-Loading and Outsourcing of Reasoning: When students rely on AI to generate answers, summaries, or plans, they bypass the cognitive work that builds expertise. This is especially harmful for novices who need repetition and struggle to develop clinical reasoning (like writing discharge summaries, or clinic notes). Think of it like using a calculator for every math problem in elementary school—sure, you’ll get the right answer, but ask that kid what 7 x 8 is without the calculator and watch the existential crisis unfold.

De-Skilling: The greatest harm falls on learners who are just beginning to build their clinical foundation. If they never practice generating differentials, synthesizing evidence, or drafting clinical reasoning without AI, they won’t develop those skills. And when the AI fails—and it will fail—they won’t have the judgment to recognize it. This is like teaching someone to drive using only cruise control and auto-pilot. Great until you hit a pothole. Or a moose. Or both.

Hallucinated Content and Sources: We’ve discussed this before (looking at you, Episode 54 and the fake “LARK Study”). AI doesn’t know when it’s wrong. It will confidently cite studies that don’t exist, recommend treatments based on outdated guidelines, and present plausible-sounding nonsense as fact. It’s like that one colleague in your med school class who always spoke with confidence, even when they had no idea what they were talking about.

Bias and Inequity: AI models are trained on existing data, which means they inherit existing biases. If we don’t actively audit and address this, we risk amplifying disparities rather than reducing them.

Privacy, Security, and Data Governance: Who owns the data we feed into these systems? Where does it go? How is it used? These questions aren’t theoretical—they’re legal, ethical, and operational concerns that every institution needs to address.

Learning Points and Implications for UME/GME Faculty (Or: How Not to Raise a Generation of Prompt Engineers)

So what do we do with this? Here are some takeaways for faculty.

  1. Teach Verification as a Core Skill: Don’t just teach to use AI—teach them to audit it. Make questioning outputs a routine part of learning. Build exercises where students identify AI errors and explain why they’re wrong.
  2. Embed AI into Curriculum Deliberately: Don’t let AI be the student’s secret study buddy. Bring it into the open. Use it in supervised settings where you can model appropriate use, skepticism, and judgment. Otherwise, you’re just hoping your learners figured it out on their own. And we all know how that goes.
  3. Prioritize Foundational Skills: Ensure that learners master clinical reasoning, differential generation, and evidence synthesis without AI before introducing it as a tool. You have to know the rules before you can break them. Or, as I like to say: you can’t use AI as a crutch if you never learned to walk.
  4. Audit for Bias: Regularly review AI-generated content for accuracy & bias. Make this part of faculty development and quality assurance.
  5. Set Clear Policies: Institutions need guidelines on AI use in assessments, patient care, and communication. Ambiguity breeds misuse. And nobody wants to be the institution that ends up in the news because a resident let ChatGPT write a discharge summary. (Though, to be fair, it probably would’ve been grammatically correct.) At our institution, residents need 6 months of manual ‘note writing’ (I’m not sure that is enough) then need to pass a note review with their faculty mentor before leveraging AI scribes.

The article closes with a warning: AI tools may amplify known human factors problems in health professions education. If we’re not intentional, we risk creating a generation of clinicians who can prompt an AI but can’t reason through a case when the system fails.

Final Thoughts: Staying Human in an Automated World (And Other Impossible Tasks)

Keep in mind disenchantment isn’t the same as despair. It’s my gut telling me to slow down, ask harder questions, and resist the pressure to adopt every new tool just because it’s shiny and makes beeping noises.

AI isn’t going away. These tools will continue to evolve, integrate, and compete for our attention. But we don’t have to accept it uncritically. We can demand better. We can use it thoughtfully. And we can keep the focus where it belongs—on education, patient care, and the humans at the center of it all.

So here’s my ask: Stay skeptical. Stay curious. And keep talking to each other about what’s working, what’s not, and what we’re worried about. That’s how we navigate this. Just a thought