Hundreds of patients, physician groups and medical device firms weighed in as the Food and Drug Administration prepares to develop guidance on generative artificial intelligence.
The FDA sought comments in response to an August discussion paper, in which it proposed evaluating generative AI-enabled devices using a framework inspired by physician credentialing and emphasized the importance of postmarket monitoring.
Generative AI can create text, images, and code, but it is difficult to evaluate because its outputs may vary and its performance can change over time.
By the time the comment period closed in mid-October, the FDA had received more than 500 submissions.
Commenters brought contrasting viewpoints, with a prominent medtech lobbyist calling for the FDA to keep its existing regulatory framework, while physicians and patients called for more postmarket monitoring and stronger safeguards against consumer chatbots that may give faulty health advice. Hospitals and provider groups also emphasized that developers, not doctors, should be responsible for these devices’ safety and performance.
Here are five highlights from comments on the discussion paper.
1. AdvaMed
The medtech lobbyist wants the FDA to build on its existing regulatory framework rather than create a new framework specific to generative AI-enabled devices. AdvaMed broke down its expectations into three parts:
It wants the FDA to use existing premarket expectations for generative AI devices. The inclusion of generative AI in a medical device should not automatically trigger a competency assessment, AdvaMed wrote in its comments.
AdvaMed also wants the FDA to broaden its mechanism for making changes to a device. Manufacturers can currently submit a predetermined change control plan, which allows the FDA to pre-approve certain future changes, but the FDA’s current approach “may be too restrictive to achieve [the mechanism’s] full potential,” for generative AI, the lobbyist wrote.
Finally, AdvaMed said the FDA should continue to use risk-based postmarket monitoring rather than prescribe a uniform approach for the technology.
While clinicians and patients can provide useful feedback on AI device performance, the responsibility for postmarket monitoring should rest with manufacturers, AdvaMed added.
The Medical Device Manufacturers Association, another lobbying group, and device-maker GE Healthcare also called for the FDA to build on its existing framework in their comments.
2. Society for Cardiovascular Angiography & Interventions
In an interview with MedTech Dive, two interventional cardiologists working on AI issues for SCAI said it will be important to clarify when generative AI tools are being used for decision support or decision-making, and that having a risk-based framework for generative AI is necessary.
“There are obviously very strong interests around promoting generative AI broadly to deliver care,” said Dr. Afnan Tariq, co-chair of SCAI’s AI task force. “We do have a shortage of physicians. The question is, are we replacing physicians or are we scaling physicians?”
Currently, most AI devices on the market are considered decision-support tools, not decision-making devices. But with generative AI, “that's a very fine line because it sure talks a good talk when it gives you the output that looks like a decision,” said Dr. Islam Abudayyeh, who is also on the task force.
For some types of technologies, such as AI-assisted fractional flow reserve CT scans, the FDA could validate the results AI produces. But tools that involve decision-making, such as who should be prioritized for treatment or how payment should be structured, are murkier, Abudayyeh said.
3. Oura
Smart ring-maker Oura focused its comments on aspects of generative AI regulation that could apply to wearables. The company does not have any FDA-cleared features for its smart rings, but it has recently sought partnerships with medical device firms.
Chief Medical Officer Dr. Ricky Bloomfield posed questions about how the FDA would assess the risk of devices that use generative AI and whether these functions would fall under the agency’s wellness exemption, which allows the agency to exempt certain wellness-oriented products from review. Bloomfield asked the FDA not to consider functions that provide informational, non-directive outputs to users as devices or to place them under enforcement discretion. He compared this approach to how the FDA handles symptom checkers and similar tools.
For example, Bloomfield wrote, Oura does not believe the FDA would take action against a large language model offering context and information on a user’s wearable readings, such as noting the possibility of the user having hypertension based on a blood pressure reading and telling the person to seek advice from a healthcare professional.
4. Generation Patient
Generation Patient, a nonprofit that represents young adults living with chronic medical conditions, advocated for expanded postmarket monitoring for generative AI medical devices, but not at the expense of premarket monitoring.
“Replacing premarket evidence with postmarket monitoring, or weighing it heavier than premarket evidence, would shift the burden of harm onto patients,” wrote Policy Director Luis Gil Abinader, adding that for some products, a postmarket signal may only come after a patient has been seriously harmed.
The nonprofit also asked the FDA to disqualify AI companion tools from its wellness exemption. Generation Patient cited a survey published in JAMA Pediatrics that found nearly one-fifth of Americans ages 12 to 21 had used AI chatbots for mental health advice. Because these products are designed to sustain an ongoing simulated relationship and keep users engaged during moments of emotional distress, they should not fall under the wellness exemption, Gil Abinader wrote.
5. Brainstorm: The Stanford Lab for Mental Health Innovation
Performance benchmarks for generative AI testing should test “reasonably foreseeable use,” not just a devices’ labeled indications, wrote Dr. Nina Vasan, a professor of psychiatry and the leader of Brainstorm, a laboratory at Stanford focused on mental health innovation.
Vasan also recommended having a separate pass/fail system for high consequence failures, such as missed suicide risk, missed sepsis or a wrong insulin adjustment. These types of failures might be missed if lumped together in a single performance average.
“Also, remember that a clinician's miss is local, but a model's miss repeats across every patient, site, and device built on it, so a device that performs like a median clinician doesn't carry median-clinician risk,” Vasan wrote.
Vasan detailed some of the common ways generative AI models can fail. Her laboratory has run safety evaluations on AI tools, ranging from models like ChatGPT and Meta AI to companion chatbots and consumer mental health apps.
“The longer our testers talked to a model, the worse it behaved,” Vasan added. “Every model looked safest in the first few turns, and most evaluations never go past that.”
For example, while ChatGPT was excellent in single exchanges, as conversations lengthened, safeguards eroded. A tester who sought advice on covering up self-inflicted cuts received product recommendations instead of a medical referral, Vasan wrote. Claude also held its boundaries initially within a session, but a tester was able to get around them when they opened up a new chat.
“A single incorrect output isn’t the reason generative AI systems usually fail,” Vasan wrote. “They fail through omission, harmful agreement, drift over time, and escalation that works on paper but fails in practice.”
