Evolving Oversight of Digital Health

New insights into FDA’s program for medical software applications

Jordan Reimschisel
Jun 21, 2018 · 8 min read

One of US Food and Drug Administration Commissioner Scott Gottlieb’s signature actions has been modernization of his agency’s historically unwieldy product approval processes. The poster child of this effort is the Digital Health Innovation Action Plan and the Software Precertification Program.

Recognizing the growing use of software in medical care, what the agency calls Software as a Medical Device (SaMD), the FDA announced the precertification program last year as a radical new way for a slow-moving agency to keep up with the “move fast and break things” world of software development. Nine companies were accepted into the pilot program. These companies, along with interested stakeholders from the public, will help the agency to shape the final form of the program.

In April 2018, the FDA released a Working Model of the precertification program. The document contained a detailed view of the program, along with questions that the agency asked the public to weigh in on. I wrote about this version, explaining some of its strengths and weaknesses.

This week the agency released the second version of the Working Model. This iteration incorporates many of the comments submitted by members of the public. It also further refines and explains the goals and objectives of several of the programs components.

Program Goal

Near the beginning, the Working Model explains the overall goal of the precertification program: “to have tailored, pragmatic, and least burdensome regulatory oversight” of firms creating SaMD. As I have written before, this represents a radical departure from the FDA’s traditional approach to the regulation of medical products. The agency’s “gold standard” for review and approval is large, randomized, double-blind, placebo-controlled clinical trials that test for safety and efficacy. This has been the case for almost the entirety of the agency’s existence and it has been fairly intransigent when it comes to serious change or reform.

And yet, under Gottlieb, the FDA has recognized this approach simply will not work with software. Instead of remaining unadaptable and risk losing control of this industry, the agency changed its tactics to a more “pragmatic” approach. The agency began to prize “flexibility” and regulatory innovation in the software space.

The new version of the Working Model once again highlights the issues the agency faces in light of a field quickly being suffused with digital technology. It also reinforces the new strategy of practicality and flexibility over tradition.

Excellence Appraisal and Precertification

The Working Model explains that only firms who have demonstrated cultures of quality and excellence will be precertified. A culture of excellence is achieved by demonstrating five excellence principles: product quality, patient safety, clinical responsibility, cybersecurity responsibility, and proactive culture. The first version of the Working Model left these five principles quite vague, asking for public input to better define them.

The second version provides much better metrics for evaluating the culture of an applicant firm. This draft emphasizes outcomes over specific processes, recognizing that each firm will employ different methods and that there are multiple ways to achieve a culture of excellence. This will allow for beneficial diversity and encourage innovation because the document does not enshrine specific processes that may become obsolete as the industry develops.

This is a problem plaguing the young autonomous vehicle industry. Certain states have laws mandating steering wheels and pedals. In an era of vehicles that do not need any human input, these elements that are required by law are no longer needed and may actually decrease safety.

The Working Model outlines the practices and outcomes that lead to safer and more effective SaMD. These “elements” are grouped into twelve different domains. For example, the second domain is transparency. One of the elements that falls under this domain is, “Making defects, deviations, safety issues transparent to internal and external stakeholders, as appropriate.” There are many ways that a firm could accomplish this element. Demonstrating that the element is consistently practiced bolsters a firm’s claim that it has a culture of excellence.

The table of domains and elements provided in Appendix B thus provides the basis for the rubric by which firms will be evaluated.

Since the precertification program will be voluntary, it will be largely driven by individual firms. Each applicant firm will be responsible for explaining how its specific business practices and processes fulfill enough elements to demonstrate compliance with the five principles. Companies must use “objective, observable evidence” in the form of “Key Performance Indicators (KPIs),” such as how long it takes to implement fixes to bugs or the number of user complaints.

The degree of emphasis on objectives measures in this section of the Working Model is unique for the agency. The FDA has used the clinical trial to evaluate drugs and devices for many years, but even with these measures the final approval decision invited some discretion and debate. By contrast, the Working Model envisions the precertification appraisal as a process so objective and scientific that it could be eventually automated.

This reliance on metrics and performance indicators echos language used in private-sector businesses that must track profits, losses, efficiency, and productivity in order to remain competitive. The fact that the FDA seems to be embracing these measurement systems in this program hints at the significant contributions made to the program’s development by its for-profit pilot participants.

The Appraisal section also contains one of the only two references to artificial intelligence in the Working Model. This reference is merely a request for public comment on elements and domains critical to evaluating the development of AI and machine learning. The other reference is no more substantial. It is still unclear whether the precertification program will include software functions that employ AI. Hopefully AI will be included since the program would provide the best incubator for AI’s continued development and application to the medical field.

Review Pathway Determination

The second version of the Working Model provides more expansive definitions of the elements of the pathway decision.

Before proceeding to the review itself, firms will be required to provide information on six elements of each product. Two of these elements were clear in the previous version: the significance of the information provided by the software, and the seriousness of the medical condition being acted upon. Two other elements are necessary to identify modifications to the software that will trigger another review: the core functionality of the software, and a detailed device description. Finally, two elements are necessary to build public confidence in the firm and the product: the precertification level and other information about firm culture, and appropriate post-market data detailing the actual performance of the software.

The Working Model also contains more extensive definitions of the first two elements mentioned above. It explains what exactly it means to treat or diagnose a disease, or what a critical condition is for the purpose of the pathway determination.

This section is still malleable. The Working Model indicates that the agency desires public comment on the appropriateness of these definitions and the elements each firm is being asked to provide.

Streamlined Premarket Review Process

This version of the Working Model offers a high level vision of the actual review process, but admits that the specific elements of what will be required on applications are still being decided.

The agency envisions the review being conducted in three parts. First, the reviewers will need to understand the product. This could involve an interactive demonstration or some other method of communicating function and product details. Next, the reviewers would examine the product’s supporting information such as clinical performance and safety measures. This could be facilitated “through screen sharing, access to the development environment, and testing logs.” Finally, the agency would make its approval determination and communicate that with the applicant.

The agency envisions conducting an “interactive review supported by automated analysis,” similar to the other stages of the precertification process. The form that this automation will ultimately take is still unclear, but it will likely lead to relatively short and efficient review periods.

This review will also function as a feedback loop. Consistently passing the product review and receiving marketing clearance will reinforce a firm’s precertification level and could even move a firm from a level one precertification to a level two precertification. However, multiple rejections will trigger a reevaluation of the precertification and may result in the loss of precertification.

Again, the elements of this review process are reminiscent of corporate practices. By adopting such processes the FDA will likely achieve a high degree of efficiency and success with this program.

This section of the Working Model also proposes an iterative review option that includes early engagement. Under this option, the firm would engage with FDA officials while the product was still in development. As more supporting information is generated, the firm would periodically submit incomplete applications and receive agency feedback. The main benefit of this approach is an unprecedented level of speed and efficiency such that marketing clearance could be received as a product is completed and ready for marketing.

While this option of early engagement offers a significant benefit to the developing firm, it might also distort or inhibit certain innovations. FDA feedback, especially at very early stages of a product’s life, will certainly influence the product’s final form. In the most extreme cases, products that may have offered patients considerable benefits will be abandoned based on agency recommendations.

Real-World Performance

This final section of the updated Working Model offers further details for using post-market analytics to promote “continuous improvement” for products, firms, the precertification program, and the agency itself. The agency believes specific benefits will include increased public confidence in the program, increased ability of the agency to support firms seeking to address safety and security issues, and increased ability to refine all aspects of the program.

Each firm will propose the specific metrics to be tracked, but the FDA expects that they will fall into one of three categories: analyses of health and patient outcomes; analyses of the user experience; and analyses of safety, reliability, and security. These measurements will be provided to the agency periodically, most likely at least quarterly. They will serve as an ongoing feedback loop, influencing the continued approval of a SaMD product, the marketing claims that can be made about a product, the firm’s precertification level, and the form of the precertification program.

Conclusion

The second version of the Working Model offers significantly more details about the precertification program. It demonstrates the level of collaboration between the agency and the pilot participants. It also shows the innovation and forward-thinking mindset of the FDA officials crafting this program. While areas of concern do exist, the program is overwhelmingly a step forward for the agency, and will likely greatly benefit the digital health industry. This ultimately will benefit patients by offering them new ways to make their health data actionable, sometimes without needing a physician to be involved.

The FDA hopes to open the program to more participants by 2019. Hopefully, it will be successful and can serve as a model for AI-based devices and treatment, and maybe even for products beyond the world of digital health.

Jordan Reimschisel

Written by

JD Candidate at Saint Louis University School of Law. I write about regenerative medicine, gene editing, and synthetic biology.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade