The use of real-world evidence in ICER’s scoping process and clinical evidence assessments

BACKGROUND: There has been growing interest in using real-world evidence (RWE) for health technology assessment (HTA) in the United States. The Institute for Clinical and Economic Review (ICER) is an independent U.S.-based HTA organization that focuses primarily on pharmaceuticals. RWE is used to inform ICER’s scoping and comparative clinical effectiveness (CCE) assessments, but the extent to which it is used has not been quantified. OBJECTIVE: To systematically evaluate use of RWE in the scoping and CCE assessment sections of the ICER HTA reports on pharmaceuticals. METHODS: We reviewed all ICER reports of pharmaceuticals published between January 2014 and June 2019. We examined the average number of instances and the proportion of RWE use in the scoping documents to inform the population, intervention, comparator, outcome, setting, or timing (PICOTS) elements of the appraisal. We also examined the average number of instances and the proportion of RWE use in the CCE assessments to inform effectiveness, safety, or treatment patterns. Finally, we evaluated use of RWE in clinical guidelines that were cited in the CCE assessments. RESULTS: In ICER scoping documents, the mean (SD) number of instances of RWE use was 3.8 (3.7) per document (55% for outcomes, 20% for population, 14% for comparator, 11% for intervention, and 0% for timing and setting). In ICER CCE assessments, the mean (SD) number of instances per assessment was 0.7 (0.5) per drug (53% for effectiveness, 44% for safety, and 3% for treatment patterns). In clinical guidelines used in ICER reports, the mean (SD) number of instances of RWE use was 1.6 (2.3) per drug per guideline (41% for effectiveness, 30% for safety, and 29% for treatment patterns). CONCLUSIONS: RWE was frequently used in the ICER scoping process, particularly to inform selection of outcomes. RWE was used infrequently in ICER CCE assessments, while more often used to inform effectiveness, safety, and treatment patterns in relevant clinical guidelines. There are opportunities to increase the use of RWE in U.S. HTA processes.

that focuses primarily on pharmaceuticals.
RWE is used to inform ICER's scoping and comparative clinical effectiveness (CCE) assessments, but the extent to which it is used has not been quantified. We also examined the average number of instances and the proportion of RWE use in the CCE assessments to inform effectiveness, safety, or treatment patterns. Finally, we evaluated use of RWE in clinical guidelines that were cited in the CCE assessments.

RESULTS:
In ICER scoping documents, the mean (SD) number of instances of RWE use was 3.8 (3.7) per document (55% for outcomes, 20% for population, 14% for comparator, 11% for intervention, and 0% for timing and setting). In ICER CCE assessments, the mean (SD) number of instances per assessment was 0.7 (0.5) per drug (53% for effectiveness, 44% for safety, and 3% for treatment patterns). In clinical guidelines used in ICER reports, the mean (SD) number of instances of RWE use was 1.6 (2.3) per drug per guideline (41% for effectiveness, 30% for safety, and 29% for treatment patterns).

CONCLUSIONS: RWE was frequently used
in the ICER scoping process, particularly to inform selection of outcomes. RWE was used infrequently in ICER CCE assessments, while more often used to inform effectiveness, safety, and treatment patterns in relevant clinical guidelines.
There are opportunities to increase the use of RWE in U.S. HTA processes.
What is already known about this subject • There has been growing interest in using real-world evidence (RWE) to inform health technology assessment (HTA) in the United States.
• The Institute for Clinical and Economic Review (ICER) is an independent U.S.-based HTA organization that uses RWE to inform its scoping process and comparative clinical evidence (CCE) assessments.
• Existing evidence suggests that RWE is used to varying extents in the HTA context of formulary decision making.

What this study adds
• We provide the first systematic evaluation of the use of RWE in clinical evaluation of pharmaceuticals in the formal HTA process of ICER.
• RWE was commonly used in ICER's scoping process to inform the selection of outcomes but infrequently in ICER CCE assessments.
• Opportunities exist to increase the use of RWE in HTA processes in the United States.
There has been growing interest in using real-world evidence (RWE) for health decision making in the United States. In 2019, the U.S. Food and Drug Administration (FDA) issued guidance for using RWE as part of a regulatory submission for drugs and biologics. 1 According to the FDA guidance, RWE can be used to support postapproval studies or new indications for already approved drugs. 1 RWE may also be used to fill information gaps in the context of health technology assessment (HTA). The Institute for Clinical and Economic Review (ICER) is an independent HTA organization that seeks to assess the clinical and economic value of health care interventions in the U.S. marketplace. Using a 3-step process, ICER investigators first develop a scoping document that outlines the plan for each review. 2 This is followed by an assessment of evidence that includes comparative clinical effectiveness (CCE), long-term cost-effectiveness, and evidence deliberation. 2 Relevant clinical guidelines are also summarized in ICER reports to inform the assessment. 2 The majority of this work focuses on pharmaceuticals.
Despite growing interest in the use of RWE in the United States, the extent to which it is used in evaluations of pharmaceuticals during the HTA process is not clear. The primary aim of our study was to understand the use of RWE in ICER scoping documents, ICER CCE assessments, and in cited clinical guidelines. In a companion publication, we summarize the use of RWE in ICER cost-effectiveness analyses. 3 It is also worth noting that RWE uses might vary across types of diseases and be affected by the amount of time since drug approval. There are a limited number of cases, primarily for rare diseases and oncologic disease, for which RWE has been accepted to support FDA drug approvals. 4 Further, a drug that has been on the market for a longer period of time tends to generate more available RWE to inform its benefits and risks. 5 Thus, the secondary aim of this study was to assess RWE use stratified by disease types and time since drug approvals.

INCLUSION AND EXCLUSION CRITERIA
Our study included scoping documents and CCE assessments found in the final ICER reports published between January 2014 and June 2019. We focused on assessments for pharmaceuticals, excluding ICER reports for behavioral interventions and medical devices. The ICER reports were located in the Materials Library on the ICER website. 6 We also evaluated the original clinical guidelines summarized in the clinical guideline sections of the ICER final reports. Only U.S.-based clinical guidelines were includedthat is, guidelines solely published by U.S. professional societies or jointly published by the professional societies of the United States and other countries. We excluded guidelines that did not include specific sections that described the clinical evidence for the pharmaceuticals evaluated by each ICER report.

DATA
We followed the FDA 2018 definition of RWE: "clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real-world data." 7 Those real-world data are related to patient health status and/or the delivery of health care routinely collected from a variety of sources (e.g., electronic health records, claims, and registries) except randomized controlled trials. 7 We extracted data and captured information such as publication year, disease areas, document title, drugs assessed as interventions, RWE studies cited, reasons for RWE use, and number of instances of RWE use throughout the ICER scoping documents, ICER CCE assessments, and in the drug sections of the clinical guidelines. The reasons for RWE use, were further categorized. For the scoping documents, we adopted the PICOTS framework used by ICER, and assessed whether RWE was used to inform the population, intervention, comparator, outcome, timing, or setting. From the ICER CCE assessments and the drug sections of clinical guidelines, we captured the RWE used to inform effectiveness, safety, or treatment patterns of the drugs.

ANALYSIS
The number of instances of RWE use per document was calculated separately for ICER scoping documents, ICER CCE assessments, and clinical guidelines. We defined RWE use as any instance of RWE used to inform any element of the PICOTS framework in the ICER scoping documents or effectiveness, safety, and treatment patterns of pharmaceuticals in ICER CCE assessments and clinical guidelines. Because there is much variation in the number of drugs evaluated across ICER CCE sections and clinical guidelines, we calculated the number of instances of RWE use per drug included in the report. Additionally, a varying number of clinical guidelines were available across disease areas; therefore, we further calculated the number of instances of RWE use per drug per guideline for each disease.
In the secondary analysis, disease areas covered by the ICER reports were categorized as nonrare and nononcologic diseases, (nononcologic) rare diseases, and oncologic diseases. We examined the distribution of RWE use across these 3 disease categories. Additionally, we stratified instances of RWE use per drug by time on market before the assessments (categorized as preapproval; < 10 years; and ≥ 10 years).

Results
Between January 2014 and June 2019, ICER assessed pharmaceuticals in 33 diseases areas (Supplementary Table 1, available in online article). Some of the disease areas were the subdiseases of others (e.g., genotype 1 chronic hepatitis C [CHC] and CHC), and some overlapped with another (e.g., relapsing-remitting and primary-progressive multiple sclerosis and secondary progressive multiple sclerosis). As the ICER scoping documents for genotype 1 CHC and CHC were not available, 31 ICER scoping documents were included in our study. Thirty-three ICER CCE assessments were included for the analysis. Additionally, we identified 74 U.S.-based clinical guidelines that were summarized in the ICER reports. After excluding the clinical guidelines that did not summarize the evidence for drugs assessed in the ICER reports, 36 reports in 21 disease areas remained (Supplementary Table 2, available in online article). We reviewed the drug sections of these remaining guidelines.
The number of instances of RWE use in ICER scoping documents is presented in Table 1. The number of instances of RWE use ranged from 0 (opioid abuse) to 14 (migraine). The mean (SD) number of instances of RWE use was 3.8 (3.7) per document. In the context of the PICOTS framework, RWE was most frequently used to inform the outcome (55% mean = 2.1 per document, SD = 2.4), followed by the population (20%, mean = 0.7 per document, SD = 0.9), comparator (14%, mean = 0.5, SD = 1.1), and intervention (11%, mean = 0.4, SD = 1.0; Figure 1). RWE was not used to inform the timing or setting. To elucidate rationale  The number of instances of RWE use in the ICER CCE assessments is also presented in Table 1. The mean (SD) number of instances was approximately 0.7 (0.5) per drug per assessment. To inform the effectiveness, safety, and treatment patterns of the drugs, there were 0.3 (53%, SD = 0.7), 0.3 (44%, SD = 0.6), and 0.0 (3%, SD = 0.1) numbers of instances of RWE use per drug per assessment, respectively ( Figure 1). The examples for the categorization in the context of spinal muscular atrophy, cystic fibrosis, and osteoporosis can also be found in Supplementary Table 3. Approximately two thirds of the CCE assessments did not use any RWE. In the other one third, the number of instances per drug ranged from 0.7 (tardive dyskinesia) to 3.0 (hemophilia A).

Number of Instances of RWE Use in ICER Scoping Documents, CCE Sections, and Relevant Clinical Guidelines
The number of instances of RWE use in the drug sections of the clinical guidelines is presented in Table 1. Thirteen of the 21 disease areas included 1 or more instances of RWE. Of these 13, the number of instances of RWE use ranged from 0.1 (non-small cell lung cancer) to 9.3 (hepatitis C) per drug per guideline. Among all guidelines, including guidelines wherein no RWE was used, the mean (SD) number of instances was 1.6 (2.3); 0.6 (41%, SD = 1.2) for effectiveness, 0.5 (30%, SD = 0.7) for safety, and 0.5 (29%, SD = 0.8) for treatment pattern (Figure 1).

CCE = comparative clinical effectiveness; ICER = Institute for Clinical and Economic
Review; RWE = real-world evidence.

FIGURE 1
Mean Previous studies have evaluated RWE use in evidence assessments for U.S. health care payers. Their findings are similar to our findings in the context of ICER CCE assessments. Chambers et al. (2019) reviewed coverage policy documents of 17 U.S. commercial health plans and found that RWE was infrequently cited in the majority of them. 14 Their report also implies that randomized controlled trials are the main sources of the evidence. 14 Hurwitz et al. (2017) evaluated evidence cited in monographs and therapeutic class reviews of pharmacy and therapeutics (P&T) committees from 6 managed care organizations. 15 Their report demonstrates that RWE is not often used in those P&T materials. 15 Malone et al. (2018) interviewed and surveyed P&T committee members from 18 U.S. health plans and health system organizations. 16 They suggest that use of RWE to inform P&T decisions varies across health care organizations and is generally limited. 16

LIMITATIONS
There are several limitations to our analysis. Our study had limited ability to identify the exact reasons for low frequency of RWE use in ICER CCE assessments and especially for oncologic drugs, whether this was due to unavailability of data or ICER's decision not to use it (if RWE were available). However, ICER assessments typically focus on new drugs, 17 and our additional analysis implied that the longer a drug has been on the market, the greater the probability of RWE use-this might be one reasonable explanation. In addition, our analysis only included the clinical guidelines cited in the ICER reports. Newly released guidelines might capture more RWE. However, either might not substantially affect our results, since the majority of the clinical guidelines in our study were released during the past 5 years. Regardless, our study enriches the body of evidence accruing to support the use of RWE in the HTA context.

Conclusions
RWE was often used by ICER to inform the scope of HTAs yet was infrequently used in its CCE assessments. During the ICER scoping process, RWE was mostly used to inform the selection of outcomes within the PICOTS framework. Compared with ICER CCE assessments, relevant clinical guidelines contained a higher number of instances of RWE use. The effectiveness, safety, and treatment patterns of drugs in the guidelines were each informed by RWE. Little RWE was used for the evaluation of oncology drugs, while drugs used to treat rare diseases more often relied on RWE. This report highlights that there are opportunities to increase the

Discussion
We found that RWE was commonly used in ICER scoping documents, with most instances of RWE used to inform the selection of outcomes. Our findings suggest that ICER's scoping process aims to capture the benefits and harms associated with all the meaningful health outcomes over the life span, which might not be evaluated in clinical trials.
Two thirds of the CCE assessments in the ICER reports relied solely on evidence from clinical trials. Although some RWE was used to inform effectiveness and safety endpoints in the other one third of assessments, clinical trials were the primary source of clinical evidence.
Compared with the ICER CCE assessments, RWE was used more frequently in relevant clinical guidelines, with distribution of the number of instances being approximately equal among the 3 reasons. This suggests that the aim of evidence synthesis in clinical guidelines is to comprehensively review and present clinical evidence from all relevant sources that would answer the clinical questions of interest and identify the existing gaps. 8 Those clinical questions come from health care providers, patients, and other potential users or reviewers of the guidelines and, thereby, cover a wide range of topics. 9 Clinical trials are usually conducted in a controlled setting with a limited and homogenous study population. Some relevant clinical questions may not be answered by clinical trials because in real practice patients are often more heterogenous in terms of characteristics and treatment experiences. Therefore, clinical guidelines may rely on RWE to fill the gaps. In contrast to the evidence synthesized in clinical guidelines, the purpose of clinical evidence synthesis in ICER reports is to inform the comparative effectiveness of drugs; other aspects are not the primary interest. 8 In all 3 types of documents, we found that RWE was not used frequently in the setting of oncology. This finding suggests that the evaluation of oncology drugs still relies heavily on clinical trials. It is likely that RWE does not fit into the current framework of the regulatory process for oncology drug development; neither is there an incentive to conduct postmarketing research to assess real-world effectiveness after FDA approval. 10,11 Conversely, RWE has played an important role in rare diseases. In this case, clinical trials are often impractical because of the lifethreatening and progressive nature of the disease and an insufficient number of patients. 12 The FDA has also been relying on RWE for safety surveillance in drug development for rare diseases. 13 use of RWE in HTA processes in the United States, particularly for oncology drugs. This is in line with ICER's recent commitment to augmenting efforts to use RWE.

DISCLOSURES
This study was supported by the Health Tech Fund, University of Washington School of Pharmacy, which was created through unrestricted support from several health care industry companies. Veenstra and Carlson report grant support from the Institute for Clinical and Economic Review outside the submitted work. Carlson reports personal fees from Bayer, Allergan, and Galderma outside the submitted work. Jiao, Lee, and Devine report no support outside the submitted work.