Prompt Injection Is a Patient Safety Problem, As Well As an Application Security Problem

The Agents Are Already Inside the Workflow

The healthcare sector has moved past the era when AI was a glorified search engine. Large language model (LLM) agents now sit inside active clinical workflows: they summarize intake notes, extract data from referral PDFs, draft replies to patient messages, and connect directly to electronic health records. In the American Medical Association’s 2026 survey, 81% of physicians reported using AI tools professionally, more than double the 2023 figure. By mid-2025, nearly two-thirds of US hospitals running the country’s most widely used EHR platform had deployed ambient AI tools that listen to patient visits and draft clinical notes.

As these agents take on more active roles, they bring a vulnerability with little precedent in traditional software: indirect prompt injection. Put simply, indirect prompt injection is untrusted data acting as instructions. The attacker never types into the chat window. The instructions arrive inside content the system was asked to read. An LLM cannot reliably distinguish between the instructions it was given and the material it is processing. OWASP, the leading application-security standards body, ranks prompt injection as the number-one risk for LLM applications, and the UK’s National Cyber Security Centre explains why: inside the model there is only ever the next token to predict, whatever its source. When the line between instructions and data blurs, text inside a patient record or a referral document can hijack the system that reads it.

When Data Becomes Code, the Harm Can Become Clinical

In traditional software, the boundary between data and code is rigid: a text file is just a text file. In LLM agent systems, that boundary breaks down. Take an agent with fully valid EHR permissions, tasked with summarizing patient histories. If someone plants instructions inside a referral document, the agent may treat those words not as content to summarize but as commands to follow. No firewall is bypassed and no password is stolen. The system does exactly what it was built to do: it processes text.

None of this is speculative. In a study published in Nature Communications, researchers embedded sub-visual prompts (not obvious to human observers) into medical images and manipulated all four of the state-of-the-art vision-language models they tested into producing harmful oncology outputs, including missing cancerous lesions. A study in NEJM AI found that an attacker is not even required: ordinary handwritten labels and pen marks on pathology slides acted as unintentional prompt injections, dragging model accuracy to near zero whenever a label was misleading. And a benchmarking study in The Lancet Digital Health found that models absorbed fabricated medical content most readily when it was inserted into real hospital discharge notes, the exact format clinical agents ingest all day.

Data exfiltration is still a real and expensive problem. Healthcare has the highest breach costs of any industry, and the 2024 Change Healthcare attack ultimately affected an estimated 192.7 million people, so CISOs are right to make it a priority. But alongside that familiar risk sits one that budget lines and incident playbooks don’t yet cover: the silent corruption of clinical decisions.

The following scenarios are entirely hypothetical. None of them are things we have seen, and we describe them because we would like to keep it that way. Each assumes nothing exotic, only what is already true in many hospitals: an agent with valid permissions, reading a document that nobody screened for hidden instructions.

1. Allergy dropped from a summary. A prompt hidden in a patient's history instructs a summarizing agent to leave a penicillin allergy out of the clinician-facing summary. The record itself is untouched, so the clinician reads a summary that looks complete.
2. Dosage nudged in a draft note. A compromised intake form shifts the figures an agent carries into a draft note. The number is plausible for the drug and the patient, so it survives a quick review.
3. Triage priority downgraded. Malicious text in a referral PDF pushes a triage agent to mark a critical patient as routine. Nothing alerts and nothing fails. The patient simply waits.

In each case the output would look completely normal: well-formatted, grammatical correct, delivered on time and still wrong.

Falling Between the Frameworks

The most dangerous property of indirect prompt injection in healthcare is not the technique itself. It is the institutional blind spot it occupies. This problem falls between everyone’s frameworks.

Security teams struggle to own it, because nothing malfunctions. When researchers demonstrated EchoLeak, a zero-click vulnerability in which a single crafted email could silently pull data out of a widely deployed enterprise AI assistant, every component behaved exactly as designed. The flaw was patched, no customers were reported affected, and there was no perimeter failure for a security review to point at.

Medical device rules don’t cover it, because most of these agents aren’t devices. A peer-reviewed taxonomy of 1,016 FDA authorizations of AI-enabled medical devices found no evidence of large language models among them. The regulated pipeline is overwhelmingly imaging and signal analysis. The documentation assistants and summarizers actually spreading through hospitals mostly sit outside device definitions. That is why ECRI’s 2026 list names the misuse of AI chatbots the single greatest health technology hazard, noting they are “not regulated as medical devices nor validated for healthcare purposes.”

And clinical incident reporting misses it, because nothing looks like a mistake. Reporting systems depend on a person recognizing that something went wrong. The NHS’s national service records over 3 million patient safety events a year, and every one of them was noticed by someone. Decades of research on automation bias shows that clinicians tend to over-trust plausible automated output, and that the vast majority of clinical decision support alerts get overridden, in some settings more than 90% of them. A corrupted summary that reads fluently is exactly the kind of failure that incident reporting, which depends on a clinician spotting that something went wrong, was never built to catch.

That gap between security, regulation, and clinical governance is where the threat lives. Closing it is less a technology problem than an ownership problem: someone in clinical risk governance has to hold prompt injection the way medication errors and wrong-site surgery are held.

What Real Mitigation Can Look Like

If clinical risk governance owns the problem, architecture is what it should demand. Treating prompt injection as a bug to patch is a losing game. You cannot write a stricter system prompt or bolt on a text filter and call a safety-critical system secure. When researchers from three frontier AI labs jointly tested twelve published defenses against jailbreaks and prompt injections, adaptive attackers bypassed most of them more than 90% of the time. A filter that catches 95% of attacks sounds impressive; in security engineering, 95% is a failing grade.

What works is architecture: constraining what the system can do, so that a successful injection has nowhere to go. That principle runs through our own work at Upside Lab. When we built the Health Data Nexus, a research platform hosting de-identified medical data for the University of Toronto’s T-CAIREM centre, we assumed some component would eventually misbehave and made the perimeter deterministic: identity-aware access, restrictive egress policies, read-only data mounts, and outright blocking of common exfiltration channels. In Enthusiast, our open-source agentic toolkit, we made the same call at the agent level. Order-processing agents produce ready-to-review drafts, validation agents check outputs before they are published, and the whole design stays model-agnostic, so no single vendor’s guardrails become a dependency.

Translated to clinical workflows, the same principles apply:

Least-privilege tools. An agent that summarizes intake notes should not hold API permissions that can touch medication records. Not as a policy it is expected to follow, but as a permission it simply does not have.

Separate reading from acting. The AI is a recommender, not an executor. Any document can carry injected instructions, including one from a source you trust, so anything that ingests documents at all should be structurally unable to trigger consequential actions. Proposed actions go through independent, deterministic validation first.

Human sign-off that means something. Critical outputs need explicit clinician confirmation, and it has to be designed carefully: the alert-fatigue literature shows that review demanded too often degrades into a reflex click. A few high-stakes confirmations beat a hundred rubber stamps.

None of this eliminates the underlying model behavior, which is that a model cannot reliably tell instructions apart from the data it is reading. The current research consensus is that nothing does yet. But it means a fooled model produces a flawed draft that validation or a clinician catches, instead of a missing allergy or a wrong dose reaching a patient.

The shift to AI-assisted clinical workflows is not going to reverse. Until we stop treating LLM agents as ordinary software and start treating them as capable but easily influenced participants, with the permissions, oversight, and governance that implies, we are leaving a growing part of patient safety to chance. The technology is moving fast. The frameworks now have to move with it.