Medical device industry continues to turn to AI

A growing number of medical devices include artificial intelligence or machine learning features, and the Food and Drug Administration has started tracking ones that have gone through the agency’s approval. The FDA recently updated its list of AI/ML-enabled devices, showing that nearly 700 have now received marketing authorization. 

The latest update added a total of 171 new devices, the majority of which were cleared between August 2022 and July 2023. 

Last year, when the FDA first unveiled the list, MedTech Dive found that the majority of cleared AI/ML devices were used in radiology and went through the agency’s 510(k) clearance pathway. That continued to be true, based on an analysis of the newest additions.

Here are five takeaways from the updated list: 

1. The rate of FDA submissions slowed in 2021 and 2022 but has started to pick up again

The number of AI/ML device submissions the FDA has received has increased every year since 2015. The agency saw the biggest increase between 2019 and 2020, when the number of submissions increased by 39%. The rate of submissions slowed in 2021 and 2022, but is expected to surpass 30% again this year, the agency said in a summary provided with the list.

2. GE HealthCare had the most cleared AI/ML devices

GE HealthCare and Siemens Healthineers led the list with the most cleared AI/ML devices between August 2022 and July 2023, according to an analysis by MedTech Dive. GE HealthCare has had the most authorized AI/ML devices to date. 

Other companies that topped the list include Aidoc, a Tel Aviv-based company that makes triage systems for radiologists, and Brainlab, a Munich-based firm partnering with ZimVie on surgical navigation for spine procedures. 

3. Radiology still accounts for the majority of AI/ML devices

Last year, MedTech Dive found that the majority of AI/ML devices authorized to date have been in the field of radiology. That trend has continued, with radiology accounting for 87% of new devices last year, and 79% of new devices in the first half of 2023. 

Part of the reason radiology might be so represented on the list is because of the early digitization of imaging data compared to other specialties. The 21st Century Cures Act also specifies anything that analyzes a medical image should be regulated as a medical device. The FDA sought to clarify through a final guidance last year that certain types of software should be regulated as devices, such as those that use information from electronic medical records to flag potential cases of sepsis. 

4. None of the devices on the list used generative AI

Despite rising interest in large language models, such as ChatGPT, this type of technology was not used in any of the cleared devices. As of Oct. 19, no device has been authorized that uses generative AI or artificial general intelligence or is powered by large language models, the FDA said. 

The types of models featured in the list vary in complexity. Some were “shallow,” having less than two hidden layers between the algorithm’s input and output, while others used deep learning, the FDA said. Typically, the models used a combination of approaches to achieve a result, such as using one model to generate features and another model to do classification. 

Some examples of the types of devices cleared by the FDA this year include an algorithm developed by Edwards Lifesciences to help clinicians predict the risk of a global hypoperfusion event in patients receiving advanced hemodynamic monitoring, and a radiation planning assistant developed by MD Anderson Cancer Center to help plan cancer treatment.

5. Most devices still went through the 510(k) pathway

The vast majority of AI/ML devices cleared between August 2022 and July 2023 went through the FDA’s 510(k) pathway. Just two devices received de novo clearance during that time period. This is consistent with previous years: 96% of cleared AI/ML devices went through the 510(k) pathway between 1995 and 2022, according to an analysis by MedTech Dive last year. 

Dive Brief:

• The Food and Drug Administration is ramping up production of medtech guidance, adding 18 draft documents to the list of priorities for its 2024 financial year.
• In 2022 and 2023, the FDA’s A-lists featured 10 and four draft documents, respectively. The plan for 2024 is more ambitious, with the administration aiming to draft 18 documents on topics such as artificial intelligence, cybersecurity, real-world evidence and pulse oximeters. 
• Five of the priority drafts were on the FDA’s B-list last year. The B-list for 2024 is relatively short, featuring two final guidance documents and one draft text on 3D printing at the point of care.

Dive Insight:

The list of draft documents touches on many topics the agency has discussed over the past year. Lifecycle management and premarket submissions for devices enabled by AI and machine learning top the list of drafts scheduled for publication in the coming year. The FDA also plans to post drafts on predetermined change control plans, cyber devices and pulse oximeters. 

Three of the drafts cover emergency use authorizations, a process with which the FDA gained experience in the COVID-19 pandemic. The draft documents could provide a framework for future crises by setting out the FDA’s views on enforcing rules during the immediate diagnostic response to an emerging outbreak, the validation of tests and the factors that will inform its policy on unapproved tests. 

Across the five drafts promoted from the B-list, the FDA plans to provide advice on biocompatibility, the evaluation of sex- and gender-specific data in medical device clinical trials, and its conformity assessment accreditation scheme.

The FDA’s final guidance A-list features three documents. One document, on remanufacturing medical devices, is left over from last year. The other two documents cover medical device shortages and advice on predetermined change control plans for device software enabled by AI and ML.

With 21 draft and final documents on the A-list, the agency has published a short B-list of lower-priority texts. The FDA’s B-list includes final guidance on computer software assurance and voluntary malfunction summary reporting, plus a draft document on 3D printing medical devices at the point of care.

Sponsored content

By Nuance

Advancements in technology have improved healthcare for all, especially for medical documentation. While advancements in these technologies have taken awhile, recent developments have propelled the industry forward. The power of conversational AI, ambient AI and now generative AI is enabling new functionality that’s furthering the mission of improved patient care and clinician-patient experiences.

I’ve often thought that advancements in technology are like raising children. When you’re in it day-to-day, child development seems to move slowly and you don’t notice all the changes. But when you look back, you realize how quickly the time went and how fast they advanced through each childhood development stage.

Advancements in medical documentation technology are kind of like that. The technology has advanced significantly and transformed the workflow but it’s been in phases over a long period of time.

Medical documentation began without any technology. Clinicians took pen to paper to document patient care and stored patient charts in closets. Charts were often illegible, they got lost, the quality of the documentation was highly variablen and sharing them with other clinicians outside of the practice was difficult.

Then came transcription, which began as a manual process where medical stenographers converted shorthand into medical documentation, which then evolved into typed notes on typewriters and then on computers. Today, instead of converting written notes into medical documentation, transcriptionists are converting the audio of the patient encounter, which is captured by any number of recording devices (and has evolved over the years), to speech, which relies on Natural Language Processing, a technology pioneered by Nuance.  However, medical documentation created by transcription can take too long to get back and is often incomplete.

Building on the foundation of transcription, is front-end speech. Using a cell phone or a microphone that integrates with a clinical workstation and a computer, a clinician simply dictates patient encounters directly into the electronic medical record, speeding up medical documentation creation significantly. With Nuance’s front-end speech solution, Dragon Medical One, clinicians can pre- and post-chart at any time and perform common tasks such as signing the note, navigating the EHR and placing orders – all with just voice.

The latest technological advancement in medical documentation is AI-driven ambient clinical intelligence (ACI). ACI solutions – like DAX copilot – captures the patient encounter via a mobile application and automatically converts multi-party conversations into specialty specific, structured clinical summaries immediately after a patient encounter. This technology was born three years ago and is used by thousands of physicians in over 30 specialties nationwide. It’s transforming medical documentation and delivering significant results. It’s saving clinicians seven minutes on average per encounter. And that time savings is giving clinicians back time in their day and freeing them from documenting care during or after patient visits. In a recent Nuance survey^[i], 79% of clinicians stated that they have better work-life balance and a 70% reduction in feelings of burnout and fatigue due to DAX Copilot. And because DAX Copilot is enabling clinicians to fully focus on patients and capture their complete story automatically, 85% of patients whose physicians use DAX Copilot say their clinician is more personable and focused compared to their previous visit.

Together, Dragon Medical One and DAX Copilot can be used for all charting needs – with just your voice. Want to review a patient’s history and results, sign a note, or place an order?  Use Dragon Medical One. Want to capture the patient conversation ambiently and document care automatically? Use DAX Copilot.  These solutions surround care teams with an AI-powered ecosystem of support that increases efficiency and documentation quality, while enhancing the quality of care and delivering a better patient-clinician experience. 

This is just the beginning of the benefits that AI-driven technology is bringing to healthcare. The potential for generative AI to further automate and simplify clinicians’ tasks is enormous – particularly when combined with proven conversational and ambient AI in a seamless, workflow-integrated manner. And while it may have seemed like it took a while to get here when in development, looking back, the advancement in medical documentation technology has been fast and transformative.

^[i] Based on Nuance’s survey of over 1,000 clinician survey responses, conducted Feb 2023, US only.

When Rajiv Sethi was first trained as a surgeon, he approached it like a craft. He trained under a surgeon who taught him how to do certain procedures, and he started his own practice.

Sethi, who is executive director of the Center for Neurosciences and Spine at Virginia Mason Franciscan Health in Seattle, now is training a new generation of surgeons interested in using robotic surgery, augmented reality and predictive analytics. Not all surgeons are comfortable with this shift in culture.

“If you think about how a robot now is going to do part or most or all of your surgery, there are a lot of folks that are maybe a little bit older than me that are very, very uncomfortable with that,” Sethi said. “We’re taking the artisan out of it and we’re making the standardizable parts really the future of how physicians are trained.”

Sethi’s comments were part of a broader discussion at AdvaMed’s MedTech Conference on Monday about the use of digital technologies in healthcare and some of the barriers to adoption. These new technologies have implications not just for surgeons, but for nurses, pharmacists, paramedics and other health professionals.

“There’s this fear of losing your capability. Your identity. This is what you’ve done the last 25, 50 years, and now suddenly something else is going to do it for you — that’s the challenge,” said Bakul Patel, Google’s senior director of global digital health strategy and regulatory, and former digital health officer for the Food and Drug Administration.

Christian Eusemann, vice president of research and innovation for Siemens Healthineers, said the company sees “love yet fear from AI,” in that providers love what the technology does, but worry about the future of their jobs. One example is a feature the company implemented that automatically positions patients for CT scans to get the correct amount of radiation. 

“Initially, there was a lot of fear by the technologists, like ‘oh this is the start of AI taking away our job,’” Eusemann said. “But now, because it’s really integrated in a way that they don’t even think about it anymore, they really appreciate it. They love it.” 

No panacea

Although digital technologies promise to improve patient care and lighten clinicians' workloads, they are not a one-size-fits-all solution for the problems hospitals currently face. Developers of medical software must be wary of bias that can be introduced by data, said Robert Cohen, president of digital, robotics and enabling technologies for Stryker. 

“Some people say, ‘Hey, why don't you just get all that data from the hospital and just work with the hospital? Well, the phenotype of those patients at the hospital is not necessarily typical for all levels of income, demographics and things like that. And just inherently by going to that hospital, by getting a couple of thousand patients, you already have inherent bias,” Cohen said. 

Furthermore, getting all of that data in one place — information from surgical equipment, patient records from a hospital, doctor’s office and physical therapist — is currently very difficult. It doesn’t all exist in one place, and people in the U.S. are trying to figure out the rules of data acquisition and data use, meaning each facility can have a different policy, Cohen added. 

Technology also can’t solve all of healthcare’s labor problems, a point that was emphasized by a historic strike of Kaiser Permanente workers that paused just before the conference.  

Shawna Butler, a nurse economist and host of a podcast on nursing innovation, pointed to high vacancy rates and turnover rates at hospitals and more clinicians leaving the profession after just a few years. 

“One of the really scary stats for me that digital technology is not going to help [is] the average tenure in an ICU for nurses. It used to be 10 years … we're down to like two and a half,” Butler said. 

At major medical systems, instruments can’t be sterilized fast enough because there just aren’t enough people — a problem that technology is also unlikely to solve, Sethi said. 

“Is this going to overburden an already burdened workforce, where we don't really have an answer right now to, in my healthcare system, that 450 positions that are open, and there's just no applicants for these jobs?” Sethi said. “And these are people that are going to check you in, these are going to be people that walk you back to your colonoscopy, these are going to be people that recover you after you wake up — these jobs are not being filled. And so this is where I see the importance of empowering the workforce being a very, very key part of how we think about digital in the future.”

GE HealthCare and Mayo Clinic reached an agreement earlier this month to work together on medical technology, building on past work between the two companies that resulted in GE HealthCare including certain features in its products.

The companies laid out four focus areas for the partnership: magnetic resonance imaging, ultrasound technology, AI, and theranostics — a type of cancer treatment. 

Jan Makela, CEO of GE HealthCare’s imaging business, shared more details about the partnership and the future of imaging technology.

This interview has been edited for length and clarity.

What led to your collaboration with Mayo Clinic? 

We've worked with Mayo in collaboration, development and research for many years. And Dr. [Matthew] Callstrom, who's the chair of radiology there, he and I met about a year ago, and we said, how can we sort of take this up a few levels? How can we think about bigger problems, larger goals? 

There are all sorts of products that we sell where there is embedded IP or features from a Mayo collaboration. So we weren’t starting from zero. We talked about the areas that we both feel strongest in and [are] most exciting for growth … and will have the most impact.

Pretty quickly, we signed a master research agreement, and we really broke lots of new ground in terms of how the teams work together, how we'll deal with [intellectual property], how we'll deal with data. A lot of work went into [setting] the right framework that the researchers can just go and not get bogged down with all these intercompany issues we often get. So we feel super, super excited about this. 

What’s the nature of your partnership? Will Mayo Clinic be a research partner, or will they also be involved in the commercialization process? 

Much of this will involve intellectual property. So it depends… but we have a framework to understand where the IP goes. And Mayo themselves can use anything that we’ve worked on. Our objective here is we are the main commercialization partner, but we're going to use all this stuff in Mayo Clinic itself, and then try and apply it more broadly. It’s likely that some of this will also come to light through the Mayo Ventures Organization.

GE HealthCare currently has some AI features for its MRI machines. What are you working on next? 

The thing with MR scans is typically they take longer. One of the big questions is how to make it quicker, so that you don't have to sit there for 45 minutes on the machine. 

There are tens of thousands of MRIs installed around the world, and we've worked a lot on AI to look at how we generate the picture. So you have all these coils and radio waves and magnets, you get all the raw data. How do you get from the raw data to the picture? We’ve worked with AI so you can get a better picture with less data, which means you do [the scan] quicker. 

If you’re a GE customer with a lot of GE MRs, very quickly you can put this AI in your existing machines and suddenly you can do 30% more patients and you get better images and you don't change anything else.

With Mayo, we’re going to keep working on these things around not just the clinical insight, the better picture, but also how to make MR easier to use and more efficient.

Do your hospital customers purchase these AI tools through a subscription or is it embedded in the device? 

Some AI is just embedded software. Some things we sell as a specific sort of upgrade, like this AIR Recon DL. And some of these we sell as more of a subscription. You buy it, then you renew the subscription, and you get updates to the software. So it's quite a range of models. Some customers prefer a subscription. Some customers prefer to buy it and buy upgrades to it. So I think you need a range of solutions.

I’ve heard of theranostics in the context of cancer treatment. How does that work? 

There are two clinical drugs in use today. One is called Pluvicto, one is called Lutathera. One is used for neuroendocrine tumors and one is used for prostate tumors, and they both apply to late-stage patients. 

The big idea is when you’ve got lots of lesions, where are they and how big are they? You have a radiotracer which attaches to protein expressed on the tumor. So you inject the patient with the radioactive tracer, it goes to where all the tumors are, and then [a clinician] can image it on a PET system, so you can measure the tumor burden and characterize the tumors.

 But how do you treat them? Now obviously, there are lots of ways to treat cancer. More of the recent therapies like cell therapy, gene therapies are targeting the cancer. This is sort of similar, because what you do is you take the same molecule which attached to the tumors, and you change the isotope to be one that gives much more radiation. It goes to where the cancer is, it sticks to the cancer and then you get a lot of heavy radiation that goes into the cancer, so it kills it.

So the important thing is you use the same molecule to see where the cancer is, to decide the dose, to decide the treatment regime, and use the same molecule with a different isotope to then kill the cancer.

You can see what you treat. I can see where the radiation dose is going. I can see how the tumors respond. It’s very much precision care.

What’s Mayo Clinic’s role in that?

They are doing more patients in [theranostics] with these drugs than anybody else in the U.S. This is quite a complex procedure. The patient's radioactive. You need to track everything. There's lots of paperwork, lots of tests. They're really leading the field. So we're working with them on how to automate that more, how to make it easier for others to learn and deploy, how to make sure that the tools make sense — some of the software tools we're tracking —  to make it easier for more hospitals to use.

The other thing we're doing with them is really working on what's next. So all of these molecules people are developing, let's get them into trials, let's see how effective they are. 

What about your plans for clinical decision support? 

You have someone who comes with chest pain to the physician, so you get some medical records, then maybe they go for an ECG, and then maybe they go for a stress ultrasound on a treadmill. So now you've got an ECG file, you've got an ultrasound file, and then maybe you decide they've got a partial blockage in their arteries. So then you send them in for CT, see some blockage in the coronary arteries, and you decide they need a stent.

You've already got several data points about this person. Then you have a family history. Do you need to do all those stages to decide to put them into the cath lab for a stent? Maybe with AI, you can take that data, and maybe the AI will learn what are the indicators in that and say, ‘accelerate [the patient] quicker because he almost certainly needs a stent.’ So how do you start to use all the data from different sources to get a better prediction?

Intuitive Surgical in July launched its first digital tool that harnesses artificial intelligence to enable surgeons to study their own procedure data and adapt approaches to achieve better results for patients. Called Case Insights, the technology works with data from Intuitive’s robots and hospitals to find correlations between surgical technique, patient populations and patient outcomes, and develop objective performance indicators.

Intuitive CEO Gary Guthart told investors in July that the company’s AI technologies can also help shorten surgeon training times, help hospitals improve surgical program efficiencies and ultimately reduce costs. The effort builds on several years of work with the robot maker’s clinical research partners.

MedTech Dive spoke with Intuitive’s Tony Jarc, senior director of digital solutions and machine learning, to learn about how the company is incorporating AI into robotic surgery.

This interview has been edited for length and clarity.

MEDTECH DIVE: What are some of the ways that Intuitive is employing artificial intelligence?

JARC: We have AI embedded in some of our instruments. An example [is] our stapler, where we can measure, thousands of times a second, certain things about how much the stapler is clamping down on tissue, [so] that we can do our best to ensure that a good staple fire occurs. And that hopefully would lead to better patient outcomes and trust reliability in an advanced instrument that’s critical for many of our advanced procedures.

We have a platform called Ion that is super exciting and allows for an interventional pulmonologist to be able to navigate to distal parts of the lung for lung biopsy. And we have AI technologies that segment imaging scans to be able to help create a path by which the catheter can get to those distal regions of the lung and then successfully cold biopsy, hopefully earlier on in stages of lung cancer, so that patients have a better chance of recovery.

We have 3D modeling for pre-op imaging for our da Vinci [robot] business. You can take CT scans and segment them into 3D models and use them for pre-op preparation to understand anatomical structures, so that when you go into surgery and start your operation as a surgeon, you may have a mental model of what to expect that is specific to that patient.

And then we continue to build out our analytics capabilities, in terms of some of our consulting services for how hospitals can optimize their [operating rooms] for scheduling. We also have analytics tools that are leveraging AI technology that are finding their way into our My Intuitive channel [an app] to be able to surface objective insights around relevant portions of surgery, which ideally will shorten a learning curve or improve patient outcomes.

How widely in use are these applications today?

For our Ion platform, it's absolutely essential for the behavior of the technology in the platform. It is part of every lung biopsy that there are 3D models created and paths that catheters can follow to do the distal lung biopsies. On the 3D modeling side within da Vinci, it's specific to specific organs and specific structures. We have a few early use cases in that, but it isn't necessarily something that is useful for every single surgery across the many different specialties and procedure types that are performed on da Vinci.

What are some next steps for AI in robotic surgery?

We are investing into expanding our capabilities in terms of being able to run segmentation algorithms on a broader set of anatomical structures from CT scans, so that they can be used in more of those procedures. We're incrementally building those out, [based] on how it fits into the surgeons’ pre-op planning stages and how it might really be validated in solving some of those problems. We see immense potential, which is why we continue driving and releasing and investing in those programs.

As we build advanced instruments with additional sensing capability, like the example I gave in stapler, as we roll out advanced imaging capabilities or abilities to tag tissue with four axis markers that our endoscope can pick up and display to surgeons, they all work together to bring holistic solutions to our surgeon and care team, and hospital customers.

How is AI helping to train surgeons?

Core to some of our more recent work in surgical training is the ability to surface objective feedback from surgery. We can measure what happens during surgery, or we can use [machine learning] models to infer what happens in surgery, in terms of the different procedural steps. We can then quantify the behaviors of surgeons during those particular steps.

Surgeons can use that objective feedback in conjunction with their surgical coaches or the attendings or their peers, to be able to try to drive more personalized loops, whether it be outside [operating room] practice on SimNow [a surgical system simulator], whether it be reviewing additional videos of particular segments of surgery and best practices from other surgeons, etc.

In the emerging field of surgical data science, what role can Intuitive play?

This is a really exciting area, because objective metrics in surgery haven't really existed at scale, and the robotic platform facilitates measurement of those. AI technology facilitates extracting clinically meaningful objective metrics that are interpretable by a surgeon around a particular surgical activity. And we've been investing in not just building the technology, but also engaging the academic community in ushering in this technology.

We've been fortunate enough to partner with many leading academic universities in the U.S. and in Europe and in Asia, to be able to understand, how do we quantify surgery? How do we break it down into its critical steps? What objective metrics matter for which procedure type for assessing skill, objectively quantifying workflow and correlating to patient outcomes? They've been running their research that we've been enabling through the peer review process, and really establishing the foundation of the field.

What are the potential benefits of this technology for patients?

One area that our academic partners have been exploring, and we've been exploring, is the ability to fuse some of these objective metrics we can surface from the sensors on our robotic platform with patient data to make predictions about patient outcomes. You have a detailed readout of what happened during that really important surgical episode that you can pair with characteristics of that patient to make a prediction of how that patient might recover from surgery.

When we talk about some of our analytics products that are launching to our My Intuitive channel, we hope to scale those broadly across our procedure base. That said, we want to do it responsibly. That's one of the reasons why we have invested in collaborations and outfitted some of these academic institutions with the ability to vet the technology so that, as we introduce it, it actually has this ongoing validation element to it.

Google has been keeping busy in the healthcare industry, announcing partnerships with organizations to implement generative artificial intelligence. The search giant announced partnerships this year with health systems like Mayo Clinic to use generative AI, or algorithms that can be used to create new content like text, to improve clinician workflows.

The company is also tackling billing, with financial engagement platform Cedar announcing an agreement with Google Cloud earlier this month to use its generative AI tools to help patients understand their healthcare bills. 

Meanwhile, a Google Research tool trained specifically on medical data, called Med-PaLM 2, can pass medical license tests and may draft medical documentation in the future. 

Healthcare Dive caught up with Amy Waldron, global director for healthcare and life sciences solutions, North America, at Google Cloud, to find out what these developments mean for the future of healthcare. 

What will Google Health’s large language model (LLM) Med-PaLM mean for healthcare? Is Google concerned about stuttering uptake among providers concerning AI and LLM?

WALDRON: Med-PaLM is a large language model, but it's designed to provide high-quality answers to medical questions. Med-PaLM 2 has been trained with a medical corpus, so it's been tuned to the medical industry. To give you a perspective of the volume, there were 540 billion parameters that went into training Med-PaLM. So it's been designed to encode medical knowledge, and to answer questions and summarize insights from a variety of medical texts.   

We're definitely not in a hurry to push something like this out without the confidence of industry experts and ourselves. And yes, we are on the extreme with regards to making sure that we have ethical use of AI. We also are in tight collaboration with our customers that are testing these pilots, because there's a significant responsibility with this type of technology, especially in the clinical environment.

How will generative AI balance accuracy with risks when it comes to healthcare?   

WALDRON: All of us have to make sure this stuff is meaningful, but also safe and used responsibly. No one wants to mess this up.

I think you have to be thoughtful with what information you include with your enterprise search capability, and customers have full control over what information is leveraged. And look at how you're transforming that [search] experience by using conversational AI to be more like a conversation, versus a patient or consumer trying to look at many different sources to find the right answer.

That's an example of very low risk, high reward. The same thing when you look at when clinicians have to search through healthcare payer policies. In many cases, they're looking through tons of PDFs. Now with this technology, that enterprise search can give you the ability to search those PDFs, so when a clinician or someone on their team asks a question, that information can be served to them.

So there are quality controls. There are a lot of use cases in healthcare that make sense to start on where you can get those near-term benefits and not expose people to risk.

Google Cloud introduced its Claims Acceleration Suite in April. What feedback are you receiving from health plans, and how do you plan to grow the platform going forward? Why is prior authorization still such a big challenge for the healthcare industry and how can AI help solve it?  

WALDRON: With Claims Acceleration Suite, that was a great example of how we streamlined the prior-authorization submission, the ingestive information and the decisioning. We use AI capabilities to make that process flow.

People are testing how to use generative AI to help clinicians draft prior authorization letters. It’s that drafting we can do with generative AI that we couldn't do before.

With generative AI, creating communications that resonate with people is really important, and that's something that is going to help with payer-provider communications. But it's also going to be a tremendous asset to the communications that healthcare organizations are having with consumers.

How are health systems such as Mayo Clinic using Google Cloud to explore generative AI’s applications?   

WALDRON: There are four broad categories from a healthcare perspective where we're seeing AI and generative AI leveraged: The first one is administrative and operational efficiency. The second one is clinician and care team support. You also have patient and consumer engagement, and advancing research and development.

That clinician and care team support is where Mayo is definitely doubling down on technology. They're using our Generative AI App Builder to improve the efficiency of clinical workflows. It's a suite that has three major capabilities that healthcare customers are diving into. One is foundation models, but the other two are pretty deep: enterprise search and conversational AI. 

The head of the Food and Drug Administration has warned the U.S. needs to be “nimble in the use and regulation of large language models” to avoid being “swept up quickly by something that we hardly understand.”

FDA Commissioner Robert Califf, speaking at the 2023 Science for Patient Engagement Symposium in May, said the models are “ushering in the revolution that many of us were hoping for,” but cautioned that the sector will need to be regulated appropriately.

Califf identified bias as a specific concern, noting that the models and other algorithms “evolve after they are put into practice” and will likely need continuous adjustment throughout the life cycle to remain accurate.

Large language models (LLMs) such as ChatGPT are a type of artificial intelligence that has advanced quickly in recent years. Califf sees LLMs as having important medtech applications that improve healthcare.

“The great things are really great: imagine a world in which your questions were answered immediately in language appropriate for your literacy and numeracy; also your clinician can actually talk with you rather than spending all their time cutting, pasting and writing clinical notes; I could go on and on,” Califf said.

The commissioner added that he sees “the regulation of large language models as critical to our future.” The speech lacked details of how he envisages the U.S. regulating LLMs, beyond his comment that a “nimble” approach is needed. 

Elsewhere in the speech, Califf, the former head of medical strategy at Alphabet, discussed how digital health technologies “provide opportunities to foster more efficient conduct of clinical investigations by making the operating characteristics of specific digital measurements in their context of use.” He cited the use of digital technologies to remotely record and analyze patient data as an example of their value. 

The FDA chief added that the technologies “can help improve patient access to, and participation in, clinical investigations by potentially reducing the burden of required visits to a research site.” Califf has advanced that idea as head of the FDA by releasing draft guidance on decentralized clinical trials.

Dive Brief:

• The White House has directed the Department of Health and Human Services to create a safety program focused on harmful or unsafe healthcare practices involving artificial intelligence. 
• President Joe Biden outlined the planned program as part of an executive order — released Monday — on “safe, secure, and trustworthy” AI. The program will receive reports of harms or unsafe practices and act to fix them.
• Other sections of the order call for the development of a strategic plan for the responsible use of AI in the health and human services sector, including medical device safety. An HHS AI Task Force will develop the plan. The executive order does not go into detail about medical devices and technology, but the Food and Drug Administration has recently prioritized AI projects, like creating a database for AI- and machine learning-enabled devices.

Dive Insight:

Regulatory scrutiny of AI has increased over the past year. The European Union is developing the AI Act, legislation that it claims will be the first comprehensive AI law, and the FDA is drafting guidance on AI-enabled device software functions. Now, the White House has set out its stance on AI in an executive order. 

The order gives the HHS 365 days to establish an AI safety program. The program should create a “common framework” for approaches to identifying and capturing clinical errors resulting from AI, as well as specifications for a central tracking repository for “incidents that cause harm, including through bias or discrimination, to patients, caregivers or other parties.”

Through the program, the HHS will analyze the captured data and generate evidence to develop and share best practices, recommendations or other informal guidelines to help avoid the harms associated with AI.

The order also requires the HHS to work with the Secretary of Defense and the Secretary of Veterans Affairs to create an AI Task Force within 90 days of the order. Once the task force is in place, it will have 365 days to create a strategic plan. The order envisages the plan including policies and frameworks on the responsible deployment and use of AI.

Specific objectives include monitoring the long-term safety and real-world performance of AI-enabled technology, “including clinically relevant or significant modifications and performance across population groups, with a means to communicate product updates to regulators, developers and users.”

The order also calls for the incorporation of equity principles and security standards into AI development and the incorporation of safety, privacy and security standards in the software-development lifecycle.

Medtech companies will be able to more easily update devices with an artificial intelligence or machine learning component under a new regulatory framework being developed by the Food and Drug Administration.

In March, the agency shared a draft guidance for “predetermined change control plans,” essentially a set of modifications that device developers can include as part of a marketing submission.

Typically, medical algorithms are “locked” and do not change after they are cleared by the FDA. The guidance would allow for some modifications, within a predetermined scope reviewed by regulators.

As the number of AI-enabled medical devices increases, the FDA has been considering how to best regulate these software components. The idea of a “predetermined change control plan,” a way to enable software to “learn” and change within set boundaries, dates back to a discussion paper in 2019.

For example, a manufacturer that makes software used in an intensive care unit to alert clinicians when a patient’s condition is deteriorating may want to make changes to reduce the false alarm rate, while maintaining the software’s efficacy.

Or, a company that makes software used to recommend ideal ventilator settings for a patient might want to change the machine learning model to optimize its performance for a subset of patients with a particular condition.

In its guidance, the FDA said AI- or machine learning-enabled devices can have a “significant positive impact on healthcare.” Its goal is to create a “least burdensome approach to support iterative improvement through modifications to an AI/ML-enabled device, while continuing to provide a reasonable assurance of device safety and effectiveness.”

The agency outlined what elements device-makers should submit as part of these plans, including a description of the planned modifications, how the company plans to validate and implement the changes, and an impact assessment. 

A spokesperson for AdvaMed, the Advanced Medical Technology Association, said the group had no immediate comment on the proposal, but would soon be filing its comments to the FDA.

Medical device companies had until July to comment on the draft.

The World Health Organization on May 16 warned that the “meteoric” rise of artificial intelligence (AI) tools in healthcare threatens the safety of patients, if caution is not exercised.

Because of the excitement that rapidly expanding platforms such as ChatGPT, Bard, Bert and others are generating over the potential to improve patient health, device developers and others are tossing aside the caution that normally would be applied toward new technologies, the organization said in a statement.

“Precipitous adoption of untested systems could lead to errors by health-care workers, cause harm to patients, erode trust in AI and thereby undermine (or delay) the potential long-term benefits and uses of such technologies around the world,” the WHO warned.

Many of the AI systems being introduced in health care use so-called large language model tools (LLMs) to imitate human understanding, processing and communication. The WHO said the “meteoric” rise of the technologies, many still in the experimental stage, warrant a close look at the risks they pose to key values in healthcare and scientific research. Those values include transparency, inclusion, public engagement, expert supervision, and rigorous evaluation.

“It is imperative that the risks be examined carefully when using LLMs to improve access to health information, as a decision-support tool, or even to enhance diagnostic capacity in under-resourced settings to protect people’s health and reduce inequity,” the WHO said.

Those risks include concerns that the data used to train AI may be biased, leading the tools to generate misleading or inaccurate information, including responses that appear authoritative or plausible when they are wrong. Rigorous oversight is needed for the technologies to be used in safe, effective, and ethical ways, the WHO said.

“WHO proposes that these concerns be addressed, and clear evidence of benefit be measured before their widespread use in routine health care and medicine,” the organization said.

The organization’s warning echoes remarks last week from Food and Drug Administration Commissioner Robert Califf, who said in a speech that nimble regulation of large language models is needed to avoid the healthcare system being “swept up quickly by something that we hardly understand.”

Sam Altman, CEO of OpenAI, agreed with the sentiment that AI must be regulated in testimony he gave last week before a Senate subcommittee. “I think if this technology goes wrong, it can go quite wrong. And we want to be vocal about that,” he was quoted as saying. “We want to work with the government to prevent that from happening.”

Recent examples of new AI applications in medtech include Smith and Nephew’s planning and data visualization software for robotic surgery and BD’s new software to look for Methicillin-resistant Staphylococcus aureus (MRSA).

Alison Dennis is a London-based partner with law firm Taylor Wessing, where she co-heads the international life sciences team specializing in regulatory and transactional representation. She works with pharmaceutical and medical device manufacturers to ensure product offerings meet regulatory requirements, and negotiates regularly with regulators. Opinions in this article are the author’s own. 

If you're like me, when you fall ill, you will often resort to at least one of two options: consulting a doctor or checking your symptoms online. The former can be expensive or time-consuming, the latter inaccurate and unreliable. What if there were a middle road, where you could report your symptoms to an AI chatbot that could give you a diagnosis? Would you trust its answer? If so, would you trust it more or less than you would trust a doctor? These tools already exist and are rapidly developing in sophistication, know-how and accuracy. 

This is just one example of the many ways in which AI as a medical device will become ingrained in our lives. One of the key non-technical barriers to adoption of AI medical devices is user trust. As the adage goes, trust is hard to earn but easy to lose. Most people will trust their doctor, yet be aware that healthcare professionals are human and so will inevitably make mistakes. Ironically, it is this humanity which generates trust: These are individuals you can look in the eye and build relations with. 

Not so with AI (or not yet, in any case). Beyond its programming, AI will display no empathy, morality or concern for outcomes. The consequence is that although AI as a medical device has the potential to be far more accurate than a human physician, the public will be less forgiving of any errors that do occur.  

Laying the basis for safety and trust

The foundation blocks for ensuring a high degree of safety, leading to a trusting public, will have to be laid by regulators. Regulation will have to span and classify the spectrum of medical devices, from basic AI-enabled smart wearables with a clinical purpose – such as checking for an irregular heartbeat – to autonomous surgical robots. The detail of the regulation should clearly set thresholds that demarcate increasingly rigorous standards proportionate to the device's risks. 

The healthcare industry will need to educate consumers and physicians on the benefits of AI devices and how to use them optimally to obtain the claimed benefits. 

Manufacturers will need to educate through explanatory notes, FAQs and video demonstrations to give sufficient assurances that their products are acceptably safe, and function as intended. They should also take responsibility for their products, offering customers clear routes to recourse in case of issues. For most, trust will require proof, and seeing is believing. 

Separately, manufacturers will need to demonstrate to regulators the safety, effectiveness and quality of the device before it can be placed on the market. When providing their technical files to the regulator, manufacturers will likely need to build in a technical means for understanding mechanisms, validation and verification, while also noting changes made via AI to the operation of the device.

Will patients and users trust technology once they have received support to understand it? Returning to our diagnostic chatbot example, patients might feel reassured and more comfortable using a device if it includes an explanation of its sources, the degree to which a patient’s symptoms match the diagnosis, and the percentage likelihood that an alternative diagnosis may be correct. 

Don’t be a Magic 8 Ball

To build patient trust, the device might also explain, for example, how the AI was trained on real-world data, whether the data was reviewed by physicians, and which healthcare authorities have approved the device's use. A 'black box AI' chatbot that produces a response without explaining its operations may resemble the digital equivalent of a Magic 8 Ball. 

Manufacturers may wish to go beyond the mandatory requirements and seek additional certification of compliance with international standards or self-certify compliance with good practice principles, such as the ten guiding principles in the Good Machine Learning Practice, agreed between the U.S. Food and Drug Administration, Health Canada and the U.K.'s Medicines and Healthcare products Regulatory Agency. 

One of the commonly cited issues with AI in healthcare is the risk of inbuilt bias, which can be caused when non-representative data sets are used to train AI systems. Any evidence of bias will quickly erode hard-won trust. 

For example, AI medical devices trained to diagnose disease based on data that does not reflect the ethnic or other diversity of the population could alter the effectiveness of the device on the unrepresented or under-represented segments of the population. In response, regulators will require proof that manufacturers have appropriately identified, measured, managed and mitigated risks arising from any bias. 

Supplemental information, such as adherence to stringent cybersecurity standards, maintenance of medical confidentiality and compliance with data processing principles will further reassure patients and users of the manufacturer's commitment to regulatory compliance and high standards, which may further engender trust in the products.

For regulators and manufacturers alike, encouraging trust among healthcare professionals should be an essential target, both in their role as an end-user and as an intermediary with patients. Healthcare professionals will not risk their patients' health, their reputation and possibly their career on untrustworthy technology and so require confidence in the knowledge that they can use and recommend AI medical devices. Gaining the trust of the healthcare community will be a critical stepping stone to achieving a trusting patient population. 

As with all new technologies, it is likely that the younger, more 'plugged-in' generation will initially be the early adopters, leading attitudinal change to the acceptability of AI as a medical device. Ultimately, whether an individual decides to trust AI as a medical device, and the extent of that trust, will be decided by personal preference and access to alternatives. 

It's clear that AI has the technological potential for huge advances, but its success within healthcare will be determined by the ability of the regulatory environment to meet the challenges of a constantly self-developing product, in order to ensure that public trust advances at the same speed.