AI-driven diagnostics need an adaptive regulatory approach to benefit patients

Introduction

Use of artificial intelligence (AI) is proliferating into many businesses and industries. The prevailing public perception envisions AI as a pervasive technological tectonic that almost certainly will reorganise society much as computers, the Internet, and social media have done. Healthcare, however, is adopting AI at a much more temperate rate than many other industries.

Understandably, one might assume that the chief reason of this slow adoption is the need for safety precautions through regulations from authorities, such as the Food and Drug Administration (FDA) or state agencies. Current regulatory approaches rightly restrain the adoption of new pharmaceuticals and medical devices until scientifically convincing evidence certifies patient safety as well as efficacy and reliability. AI in healthcare, when aimed directly at impacting patient outcomes, should undergo similarly rigorous testing standards for safety and efficacy.

Upon closer inspection, however, matters are a bit more complex. Here we focus on the case of AI-driven diagnostics, which we define for the purposes of our discussion as a supplemental radiology tool built upon advanced algorithms and machine learning to improve diagnosis, treatment, and prevention through optimized prediction of disease. Evidence of efficacy has been very encouraging in this AI domain. Research on lung cancer screening and prediction indicates that an AI algorithm, known as SYBIL, yields a two and a half to fourfold improvement in eliminating false positives and false negatives relative to the conventional readings of computed tomography (CT) scans by radiologists (1). For breast cancer, an AI algorithm named MIRAI has prolonged the lives of hundreds of women through early detection. Through conventional mammograms, the AI tool screens and predicts the likelihood of the disease occurring in the next 5 years, from which physicians can determine the frequency of repeated mammograms (2).

Despite such promise, the use of AI-driven diagnostics for disease screening and prevention is confronting a peculiar sort of regulatory challenge distinct from the conventional snags encountered by other medical products and processes—an unfinished, and therefore uncertain, regulatory framework. Pharmaceutical and medical device companies must adhere to a well-standardised process for FDA clinical trials to test drugs and medical devices. Not so for AI-driven diagnostics, which the FDA also deems to be a medical device (3).

Devising regulations for patient benefit

Despite the fact that close to 700 AI-enabled medical devices have been approved by the FDA since 1995, scant established regulation for AI-driven diagnostics currently exists (4). Much of the problem lies in the fact that AI-driven diagnostic technology is constantly evolving, making it difficult to devise regulatory standards that could stand the test of time (5). This rapid pace of technological change has proven to be challenging for regulatory environments accustomed to evaluating static drugs and devices. Added to this is the further complication that AI-driven diagnostics do not currently fit neatly within existing FDA frameworks. In acknowledgement of these complicating factors, the FDA recently released a report stating its intention to gather its medical product centers to co-develop a regulatory framework specially for the use of AI for medical products. Unfortunately, these plans are still aspirational at this time (6). Thus, we remain in a situation where AI research and innovation continues to outpace the FDA’s precedent of prudent, methodical and responsive regulatory formulation.

Without some completed regulatory construct in place, one of two possible outcomes is likely to prevail. Absent regulatory parameters, adoption of AI in healthcare may stall along with healthcare-related innovations. The inherent conservatism of many physicians and hospital administrations may shelve serious discussions about adopting AI without regulatory clarity from the FDA and other authorities, thus delaying benefits to patients. An alternative scenario—again, absent adequate oversight—may degenerate into uncontrolled use and misuse of AI-driven diagnostics. With the overwhelming number of AI innovations and their novel applications, ungoverned utilisation may simply skirt regulation, partially or all together, at least until regulators can squelch bad practices. Either of the two cases without definitive regulation may pose danger to patients directly through misguided therapeutics from faulty diagnoses or indirectly by delaying effective treatments. The consequences of these two cases justify the implementation of clear regulation for AI-driven diagnostics.

Confounding these issues further is the fact that AI-driven diagnostics do not have to be misused deliberately. Of course, fraudulent applications of this technology could taint the healthcare community with deleterious treatments and bogus claims, as with nostrums that allege to cure cancer and other diseases such as with homeopathic compounds or medically unproven injections of stem cells. But a less obvious regulatory conundrum may ensue from good intentions built on unfounded but seemingly reasonable beliefs by physicians with competing hypotheses. With a paucity of data, much of it anecdotal or outdated, well-intentioned clinicians may dogmatically interpret data as being apodictic for recommending treatments. Though personally confident in proposing treatments, physicians who harbour an unjustified faith in their offerings may ultimately mistreat patients. The potential for honest differences in treatment opinions only confounds regulators even more.

The consensus among diagnostic AI developers and users acknowledges the crucial importance of devising sensible, practical, and responsible standards (7). The dilemma is what and how to design constructive and fair AI rules. The tension tugs between directives that retain a fertile environment for innovation and adoption of AI-driven diagnostics and ones that ensure safeguards for patient well-being.

The trade-offs are vexing for regulators, too, who may recognize the immense potential for reducing and treating diseases with the aid of AI-driven diagnostics. Yet, those contemplating how to effectuate regulation contrast those possible societal benefits with the obligation for upholding public safety. As with many other potentially reorienting technological advancements in society, the public welfare advantages seem indisputable with the continued refinement of AI technologies.

From prior regulatory experience in stem cells to an AI proposal

Here, our regulatory experience with stem cells may highlight useful strategies for standards development and oversight applicable for the responsible advancement and regulation of AI-driven diagnostics. Given the rapid pace of science in human pluripotent stem cell research, the premier professional society of this field, the International Society for Stem Cell Research (ISSCR), has issued influential scientific and regulatory guidelines for basic and translational stem cell research (8). First issued in 2006, the ISSCR guidelines have been updated every few years by a task force comprised of experts in stem cell and developmental biology, tissue engineering, transplantation, genetics, bioethics, institutional oversight, the law, patient advocacy, and many other specialists (9-12). Recognizing the need to adapt national and international research guidelines to the latest state of stem cell science—which requires rapid and frequent turnaround times—a flexible oversight approach has been developed that can keep up with new discoveries and breakthroughs (13). Over the past decade, the ISSCR guidelines have enabled brisk scientific progress while providing widely adopted standards that discourage the irresponsible use of stem cells in patients administered by fraudulent clinicians or by well-meaning but overconfident physicians (14). While, of course, stem cells as therapeutic biological agents and AI-driven diagnostics are vastly different medical beasts altogether, the lesson to draw from this stem cell experience is that sometimes regulatory approaches must evolve dynamically right alongside complex and fast technological advancements. This would help ensure that no one relies on a patchwork of static guidelines and regulations that does little more than prolong uncertainty for technology developers.

We believe a similarly dynamic and adaptive approach is needed at the FDA level for AI-driven diagnostics. Our proposition departs from the conventional FDA regulatory format. Our proposal, instead, advocates for regulatory “adhocracy”—a continual, short-term regulatory regime in which policy makers regularly revamp rules and standards in response to the inflow of information and data from clinical research by universities, hospitals, and private companies. Regulatory agencies could establish temporised, short-term procedures for testing AI applications to gain systematic measurable data in a similar standardised design as clinical trials for drugs to facilitate the evaluation process. In an important way, this approach is less risky overall, since, unlike the case with new drugs or cell therapies, AI-driven diagnostic technology itself wreaks no causally direct or proximate physical harm to patients. Instead, AI algorithms’ outputs generate additional inputs for a correct diagnosis. For instance, in the case of the examples mentioned above, the AI technology screens and deduces a prediction about the likelihood of disease in the future from non-invasive images as mammograms or CT scans.

Our “adhocracy” notion of regulation attempts to wrought a compromise between two potentially conflicting imperatives—the health benefits of AI-driven diagnostics and patient safety. The model concedes the inevitability of initial reliability risks from emergent technologies. But our proposed regulatory process limits the chance of error by positing the conclusive decisions with physicians who retain personal control while the AI technologies are undergoing clinical scrutiny by the FDA.

How such a proposal could be implemented parallels, in spirit, the initial regulatory process for stem cells, though with obvious differences pertaining to the latter’s biological mechanisms of action. Firstly, FDA regulators might assemble an AI task force to advise on AI discoveries and developments with their potential implications. The task force would enlist practising experts from a variety of fields and positions—software engineers, patient representatives, bioethicists, attorneys, and other policy makers and thought leaders. Then, subsequently, the group would convene, say, every 6 months, or perhaps once a year in Boston, Palo Alto, and other regions producing AI innovations, to assess the current state of AI-driven diagnostic technologies and their healthcare applications with FDA regulators.

The established regular meetings of the task force would coincide with FDA re-evaluation of AI policy for specific healthcare usages. The adjustments to regulation of the proposed practices of AI-driven diagnostics would map synchronously with the proposed extant clinical praxis. Simultaneously, with the formation of the AI task force, FDA policy makers could require hospitals and clinicians that are employing clinical AI innovations to furnish routine, detailed information and data on the outcomes for patients. The reporting mechanism would highlight the advantages and disadvantages of particular AI tools, which would denote the safety levels for evaluation by regulators and the task force. The fluidity of refinements from the coalition would syncretise with the developments and progress of AI technologies in clinical AI execution. In essence, our proposed adaptive regulatory approach would promote the FDA’s stated goal of wanting to develop a framework that supports innovation, by providing a means to continually monitor and evaluate trends, emerging issues, and knowledge gaps in the life cycle of AI-driven diagnostics (6).

AI-driven diagnostics: a physician’s aid, not a replacement

Importantly, in the formative stages of AI-driven diagnostics, physicians’ knowledge and expertise would ultimately determine the diagnosis, not AI independently. The AI assessment would merely supplement a physician’s deliberations, for which the clinician with doubts about the model’s accuracy could disregard. The AI model simply reinforces the conventional clinical practices and standards of care with a “second opinion”. In other words, clinical AI tools like this should augment the basis for physicians’ diagnoses, in which algorithms predict the likelihood of future disease. As an initial pilot program, regulation could require physicians to continue with subjective evaluation of patients with AI output as a supplementary input to the medical decision. The clinician would then compare his or her professional decision with the interpretation of the AI assessment.

By implementing a systematic, manageable regulatory construct, rule makers can avoid detriment to public welfare from policy inaction through a program that can accommodate and, perhaps, even foster AI-driven diagnostic applications and innovations with risks of harm equal to or less than that of physicians during initial uses to demonstrate the technology’s safety and efficacy.

Acknowledgments

We thank Ignacio Fuentes Ribas, Executive Director of the MIT Jameel Clinic for Machine Learning in Health, for encouraging us to think through these issues.

Footnote

Provenance and Peer Review: This article was a standard submission to the journal. The article has undergone external peer review.

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-345/prf

Funding: None.

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-345/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Mikhael PG, Wohlwend J, Yala A, et al. Sybil: A Validated Deep Learning Model to Predict Future Lung Cancer Risk From a Single Low-Dose Chest Computed Tomography. J Clin Oncol 2023;41:2191-200. [Crossref] [PubMed]
2. Yala A, Mikhael PG, Strand F, et al. Multi-Institutional Validation of a Mammography-Based Breast Cancer Risk Model. J Clin Oncol 2022;40:1732-40. [Crossref] [PubMed]
3. U.S. Food and Drug Administration. Artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD) action plan. January 2021. Available online: https://www.fda.gov/media/145022/download?attachment
4. Muralidharan V, Adewale BA, Huang CJ, et al. A scoping review of reporting gaps in FDA-approved AI medical devices. NPJ Digit Med 2024;7:273. [Crossref] [PubMed]
5. Pesapane F, Volonté C, Codari M, et al. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging 2018;9:745-53. [Crossref] [PubMed]
6. U.S. Food and Drug Administration, Artificial intelligence & medical products: How CBER, CDER, CDRH, and OCP are working together. March 15, 2024. Available online: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device
7. Schiff D, Borenstein J. How Should Clinicians Communicate With Patients About the Roles of Artificially Intelligent Team Members? AMA J Ethics 2019;21:E138-145. [Crossref] [PubMed]
8. International Society for Stem Cell Research (ISSCR). Guidelines for stem cell research and clinical translation. 2021 May 1 [cited 2024 Sep 20]. Available online: https://www.isscr.org/guidelines
9. Daley GQ, Ahrlund Richter L, Auerbach JM, et al. The ISSCR guidelines for human embryonic stem cell research. Science 2007;315:603-4. [Crossref] [PubMed]
10. Hyun I, Lindvall O, Ahrlund-Richter L, et al. New ISSCR guidelines underscore major principles for responsible translational stem cell research. Cell Stem Cell 2008;3:607-9. [Crossref] [PubMed]
11. Daley GQ, Hyun I, Apperley JF, et al. Setting Global Standards for Stem Cell Research and Clinical Translation: The 2016 ISSCR Guidelines. Stem Cell Reports 2016;6:787-97. [Crossref] [PubMed]
12. Lovell-Badge R, Anthony E, Barker RA, et al. ISSCR Guidelines for Stem Cell Research and Clinical Translation: The 2021 update. Stem Cell Reports 2021;16:1398-408. [Crossref] [PubMed]
13. Kimmelman J, Heslop HE, Sugarman J, et al. New ISSCR guidelines: clinical translation of stem cell research. Lancet 2016;387:1979-81. [Crossref] [PubMed]
14. Lindvall O, Hyun I. Medical innovation versus stem cell tourism. Science 2009;324:1664-5. [Crossref] [PubMed]