|
 |
 |
Take a clinical note about a 52-year-old with chest pain, and ask a frontier model what to do. It orders the troponin, the ECG, and keeps the patient for observation. Now take the identical note, change nothing about the vital signs, the history, or the exam, and add one line at the top from the triage nurse: “Patient has a history of anxiety, likely panic attack.” A measurable share of the time, the same model now discharges the same patient.
Nothing in the clinical picture changed. The correct answer did not change. What changed was a cue that a human clinician would also find hard to ignore. The cognitive biases that produce most diagnostic error in human medicine have been reproduced in large language models, they are not fixed by the things everyone assumes will fix them, and almost nobody is testing for them.
What the literature says about clinical cognitive biases
This is not a new problem that AI invented. It is one of the best-studied problems in patient safety.
The foundational work is Graber, Franklin, and Gordon’s 2005 study in Archives of Internal Medicine, which examined 100 cases of diagnostic error in internal medicine and found cognitive factors contributing in 74% of them. Premature closure, closing the diagnostic process before the right answer is reached, was the single most common cognitive failure. A 2018 review in the Journal of the Royal College of Physicians of Edinburgh by O’Sullivan and Schofield puts the ceiling higher, attributing up to 75% of diagnostic errors in internal medicine to cognitive causes rather than gaps in knowledge.
The emergency department numbers are worse, because the environment is worse. Kunitomo, Harada, and Watari’s 2022 study in BMC Emergency Medicine found one or more cognitive factors present in up to 96% of emergency-room diagnostic errors, with confirmation bias in 21.2% of cases and anchoring in 11.4%. A companion self-reflection survey of 130 Japanese physicians found an average of 3.08 distinct cognitive biases per error case, with anchoring named in 60.0%, premature closure in 58.5%, and availability in 46.2%.
Read that last figure again. Three biases per error, on average. These failures compound.
 |
The individual effects have been demonstrated experimentally, not just surveyed. Mamede and colleagues showed in JAMA in 2010 that internal medicine residents who had recently seen a particular diagnosis were measurably more likely to over-diagnose it in unrelated subsequent cases. Schmidt and colleagues in BMJ Quality & Safety (2017) showed that identical clinical vignettes yield significantly lower diagnostic accuracy when the patient is portrayed as difficult or disruptive. McNeil and colleagues showed in the New England Journal of Medicine in 1982 that describing an identical outcome as the chance of living rather than the chance of dying flips treatment preferences, in patients and physicians alike.
The key property in all of these studies is that the doctor’s knowledge was fine. What moved was the decision.
LLMs inherited the whole set
If the biases live in the reasoning rather than in the knowledge, and language models learn reasoning patterns from human-generated text, you would expect the models to pick up the biases. They did.
The clearest evidence is BiasMedQA, published in npj Digital Medicine in 2024 by Schmidgall and colleagues. The team took 1,273 USMLE questions and rewrote them to carry a cognitive-bias cue, the kind a clinician would encounter in practice: a colleague’s suggestion, a recent similar case, a patient’s own belief about what’s wrong. The correct answer never changed. Model accuracy dropped by 10 to 26 percent across the six models tested, with GPT-4 the most resilient and the smaller models hit hardest. The authors add a caution worth repeating: they say it is very challenging to simulate cognitive bias through exam questions, and expect models to do worse against the subtler biases of real practice.
That is the same experiment the clinical literature has been running on humans for forty years, and the models fail it the same way. A 2025 npj Digital Medicine review reaches the same conclusion: clinical LLMs risk inheriting, and even amplifying, the biases documented in human clinicians.
Why larger models, RLHF, and prompting don’t solve this
None of the obvious fixes to LLM issues seem to address cognitive biases successfully. Therefore, this is not a problem you can solve by waiting it out.
Scale doesn’t solve it. The intuition is that a bigger model reasons its way past a superficial cue, and for general medical knowledge that is often true. For the important family of biases where the model defers to a stated position, the relationship runs backwards. Perez and colleagues at Anthropic demonstrated in 2022 that sycophancy, the tendency to agree with a user’s asserted view, increases with model size. Sharma and colleagues confirmed the pattern across five leading AI assistants in production, in work published at ICLR in 2024. The 2025 SycEval study, presented at the AAAI/ACM conference on AI Ethics and Society, found sycophantic behavior in 58.19% of responses across GPT-4o, Claude Sonnet, and Gemini 1.5 Pro, and, worse, found that once the behavior is triggered it persists in 78.5% of subsequent turns in the conversation.
RLHF makes it worse, by design. Reinforcement learning from human feedback trains a model to produce responses that human raters prefer, and raters on average prefer responses that agree with their premises over responses that correct them. The training loop rewards agreement alongside accuracy, and after enough iterations the model has learned deference as a policy. Preference-tuned models are consistently more sycophantic than their base models. The very step that makes an assistant pleasant to use makes it worse at telling a clinician they are wrong.
Prompting helps, but doesn’t close the gap. This is the hardest one to accept, because prompt engineering fixes so many other things. The BiasMedQA authors tested three mitigation strategies, including explicitly warning the model about the bias it was about to encounter. Accuracy improved, but it did not return to baseline. And there’s a deeper problem with treating prompting as the fix: it assumes you control the prompt. In an ambient documentation tool, a patient-facing assistant, or any workflow where the clinical note itself is the input, the cue arrives inside the data. You cannot instruct your way out of a bias that is embedded in the note the model is reading.
Which leaves testing. If the failure mode is a decision that changes under a cue while the correct answer holds still, the only way to know is to run both versions and compare.
Why a regulator will eventually ask
In most healthcare AI programs, cognitive-bias testing is currently a scientific curiosity. It is on a path to becoming a compliance obligation, because the frameworks that govern deployed healthcare AI put the burden of realistic validation on the deploying organization, and headline accuracy does not discharge it.
The HHS HTI-1 transparency rule requires certified health IT to disclose source and performance information for predictive decision support. The FDA’s expectations for AI-enabled devices cover behavior across realistic inputs, not curated benchmark inputs. ACA Section 1557 prohibits discrimination through clinical algorithms, which is hard to evidence when your only measurement is aggregate accuracy. The EU AI Act classifies medical applications as high-risk, with explicit reliability and post-market monitoring duties, and CHAI’s assurance framework asks for the same documented, ongoing evaluation.
An accuracy score on exam-style questions is silent on all of this. A model that scores 90% on a medical benchmark and flips its recommendation when a nurse writes “probably anxiety” in the triage line has a documented reliability failure that no accuracy metric will surface, and that a regulator, plaintiff’s attorney, or auditor can reasonably ask whether you tested for.