Quick-guide on Software Pre-Cert Program (5) — Real World Performance

During the Excellence Appraisal, all organizations would demonstrate the capability to collect and analyze post-launch Real World Performance (RWP) data, whether by instrumenting their SaMD products to generate needed data, or by leveraging alternative data sources.

Therefore, FDA expects that these organizations will consistently collect and analyze post-launch data related to the safety, effectiveness, and performance of the products they manufacture in order to inform their decision-making related to product or process improvements.

Real world Performance Analytics (RWPA) framework is introduced for achieve above-mentioned goals.

From Pexels byRODNAE Productions

The real-world performance (RWP) component of the Software Pre-Cert Program is designed to use these readily available data analytics to verify ongoing excellence following precertification, identify emerging safety and cybersecurity risks, provide critical feedback to the other components of the Program, and support the appropriate use of postmarket data in clinical evidence generation.

RWP Analytics (RWPA) framework

RWPA are defined as systematic computational analyses of data relevant to the safety, effectiveness, and performance of a SaMD product in real-world settings marketed by a precertified organization.

FDA anticipates that not only data from appropriately instrumented SaMD products may be generated, collected, and analyzed efficiently, but also real-world data from device registries, well-structured data commons, and other electronic health information sources, including patient registries and the National Evaluation System for health Technology (NEST) currently under development.

The FDA considers RWPA to encompass at least three types of analyses, RWHA, UXA, and PPA shown in Fig 1.

Fig 1 illustrates the domains and subdomains of RWPA under Pre-Cert 1.0. FDA anticipates that precertified organizations would identify specific data analytics aligned to each of these domains and subdomains of the proposed RWPA framework based on the intended use, functionality, and risk category of the SaMD product.

Fig 1 domains and Subdomains of RWPA under Pre-Cert 1.0

Real-World Health Analytics (RWHA) are defined as analyses of real-world clinical outputs and outcomes related to the intended use of the SaMD product.

RWHA can inform changes to the intended use of a SaMD product, support expanded functionalities and use in broader target populations, and identify emerging safety issues in postmarket use. While all medical device manufacturers are required to report clinical adverse events related to their products, excellent organizations would leverage the ability of appropriately designed SaMD products to collect and analyze clinical safety data, in order to identify and address issues in a more timely manner. For lower-risk products, RWHA may be collected from sources including user complaints, search analytics, and product-level monitoring of human factors measures, such as use errors. Depending on the claim, FDA anticipates that precertified organizations with higher-risk products may proactively seek out external sources of safety and effectiveness data through activities, such as participation in registries, partnerships with healthcare systems, or utilization of data commons or other structured postmarket data collection efforts.

Table 1 is the example how RWHA is evaluated via CQOE or Exceelance principles.

Table 1 RWHA is evaluated via CQOE or Exceelance principles

User Experience Analytics (UXA) are defined as analyses of user experience outputs related to the real-world use of a SaMD product.

UXA monitoring facilitates timely identification and correction of user issues and enables improvements to the utilization and effectiveness of the software. Depending upon the intended use of the software, UXA may be collected from a variety of sources. Products instrumented to collect UXA may passively monitor measures including use patterns, download rates, and user retention. For metrics related to product satisfaction or user complaints, organizations may use proactive mechanisms of soliciting or incentivizing user feedback. To ensure that feedback is representative of the full range of users, excellent organizations would also actively seek UXA from diverse sources, which may include social media platforms, search analytics, or other third-party online networks.

Table 2 is the example how UXA is evaluated via CQOE or Exceelance principles.

Table 2 UXA is evaluated via CQOE or Exceelance principles.

Product Performance Analytics (PPA) are defined as analyses of outputs and outcomes demonstrating the real-world accuracy, reliability, and security of a SaMD product.

PPA monitoring enables excellent organizations to address software bugs and security vulnerabilities through timely patches and product updates. FDA expects that excellent organizations would instrument SaMD products to track product performance measures related to the reliability and availability of the device. As described for UXA, proactive surveillance may be needed to identify product defects and track time to resolution

Table 3 is the example how PPA is evaluated via CQOE or Exceelance principles.

Table 3 PPA is evaluated via CQOE or Exceelance principles.

RWPA Collection Plan

The RWPA framework for post-launch product monitoring is intended to provide robust evidence supporting the determination of safety and effectiveness of a particular SaMD product from a precertified organization, while retaining sufficient flexibility to accommodate the full range of organizations capable of demonstrating excellence in digital health.

For all products, a RWPA plan should include:

  • Proposed RWP data elements to be collected;
  • Intended frequency of data collection;
  • Intended data structure and format; and
  • Commit and stretch goals for each proposed data element.

For purposes of further optimizing the RWPA framework, FDA intends to review submitted RWPA plans and to work with individual Pilot Participants to refine needed data elements in the 2019 test phase。

RWPA Monitoring

The purpose for which RWPA is being monitored and analyzed influences the type of data to be collected and the duration of data collection. Precertified organizations would use best practices of monitoring and analyzing product-specific data to demonstrate their ongoing commitments to organizational excellence, and to identify and address any safety or security issues in a timely manner.

  • Verification of ongoing excellence.
    Verification that precertified organizations are identifying, tracking, analyzing, and responding to measures related to the safety and effectiveness of SaMD products postmarket enables FDA to have continued confidence in the excellence of the organization. Robust reporting and increased transparency in postmarket data collection on the part of precertified organizations may also enable FDA to explore opportunities to optimize existing postmarket obligations for precertified organizations, including reporting and inspection requirements.
  • Early identification and remediation of safety and security vulnerabilities. Ongoing monitoring of product-specific RWPA enables organizations to respond rapidly to emerging issues, including safety and security vulnerabilities. Regular access to RWPA would increase FDA familiarity with the types of data being collected and any potential signals being monitored by precertified organizations. The Agency believes that this increased familiarity would facilitate collaborative engagement between FDA and precertified organizations when products require modifications or updates. Collaborative engagement aimed at rapidly addressing any postmarket safety or security concerns, in turn, may reduce the need for compliance actions and streamline review of any required product modifications.
  • Addressing premarket uncertainty.
    In certain circumstances, the review team may determine that postmarket data should be utilized as a special control. RWPA used for this purpose will be collected over a defined time period and should be structured in collaboration with the review team to ensure that the evidence generated is appropriate, reliable, and scientifically valid.
  • Generation of adequate clinical evidence to support label expansion. Manufacturers may introduce a SaMD product to market with limited functionality at first, intending to expand its functionalities and associated claims over time. FDA anticipates that continuous collection and analysis of RWPA would support precertified manufacturers in understanding postmarket performance and in improving the product over time. Where postmarket RWPA provide evidence of superior real-world performance as compared to premarket data, FDA would work with the precertified manufacturer to modify claims and labeling to reflect actual performance characteristics. Real-world performance data may identify issues that the manufacturer must address, as appropriate.


Fig 2 is the summarized concept regarding Real World Performance Analytic. RWPA collection plan is based on the RWPA framework. Then RWPA monitoring should demonstrate Verification of ongoing excellence, Early identification & remediation of safety and security, Addressing premarket uncertainty, and Generation of adequate clinical evidence to support label expansion.

The appraisal via CQOE is conducted according to the RWPA framework including RWHA, UXA, and PPA.

Fig 2 Real World Performance Analytics



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store