Use of real-world evidence in economic assessments of pharmaceuticals in the United States

BACKGROUND: Despite the increasing interest in expanding the use of real-world evidence (RWE) in economic assessments of pharmaceuticals, decision makers face uncertainty about how RWE should be used. OBJECTIVE: To assess the use of RWE in economic assessments of drugs by the Institute for Clinical and Economic Review (ICER). METHODS: We reviewed cost-effectiveness and budget impact analyses in final evidence reports of pharmaceuticals published by ICER. We calculated the total number of RWE uses and the proportion of model inputs informed by RWE per report. We classified model inputs into 15 categories based on their attributes and then examined what category each RWE informed to classify the reason for RWE use. Finally, we characterized RWE by study design, data source, and sponsor type. RESULTS: We identified 33 reports, all of which used RWE; the mean RWE use per report was 12 (range = 4-26). The average proportion of model inputs informed by RWE per report was 32.7%, but this proportion had a wide range (range = 4.1%-76.9%). RWE was most commonly used for disease progression inputs (28.7%) and health care resource utilization and costs (21.1%), but was rarely used for drug-specific clinical outcomes such as effectiveness (1.5%), adverse drug event rates (0.5%), and discontinuation rates (1.2%). The most frequently used study design was a retrospective cohort (56.6%), and the most frequently used data source was registry data (41.4%). About a third (30.2%) of RWE was industry-sponsored. CONCLUSIONS: RWE has been commonly used to inform pharmaceutical value assessments conducted by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness.


RESULTS:
We identified 33 reports, all of which used RWE; the mean RWE use per report was 12 (range = . The average proportion of model inputs informed by RWE per report was 32.7%, but this proportion had a wide range (range = 4.1%-76.9%). RWE was most commonly used for disease progression inputs (28.7%) and health care resource utilization and costs (21.1%), but was rarely used for drug-specific clinical outcomes such as effectiveness (1.5%), adverse drug event rates (0.5%), and discontinuation rates (1.2%).
The most frequently used study design was a retrospective cohort (56.6%), and the most frequently used data source was registry data (41.4%). About a third (30.2%) of RWE was industry-sponsored.
CONCLUSIONS: RWE has been commonly used to inform pharmaceutical value assessments conducted by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite calls for greater inclusion of RWE in value assessments for real-world drug effectiveness.
What is already known about this subject • A recent FDA guidance enables the use of off-label evidence in communicating health economic information to payers, which is likely to increase the use of real-world evidence (RWE) in economic analyses.
• A majority of U.S. payers recognize the value of RWE in informing health care decision making.
• Recent studies about the use of RWE in health technology assessment in other countries found that RWE is being used in most of the economic evaluations of drugs, but there is a significant variation across countries.

What this study adds
• About a third of model inputs from Institute for Clinical and Economic Review economic assessment models were informed by RWE on average, and this proportion showed an increasing trend over time.
• RWE is commonly used to inform mortality, disease progression rates, and healthcare costs but has a relatively minor role in informing drug effectiveness and safety.
• The most common study design was a retrospective cohort study, and the most common data source was registry data.
Real-world evidence (RWE) is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of data routinely collected from sources regarding patient health status and/or the delivery of health care. 1 There has been a growing interest in incorporating RWE to assess the economic value of drugs because of its ability to reflect real-world outcomes and limited availability of evidence from randomized controlled trials (RCTs) (e.g., long-term effects and economic outcomes). 2 Furthermore, the U.S. Food and Drug Administration (FDA) recently released guidance on how drug companies may communicate health care economic information to payers and formulary committees that enable the use of off-label evidence related to an approved indication, which is likely to increase the use of RWE in economic analyses. [3][4][5][6] A recent study suggested that a majority of U.S. payers recognize the value of RWE in informing health care decision making. They believe that health care decisions should be informed by RWE in addition to RCTs and intend to use more evidence derived from real-world settings once they have a better understanding of the use of RWE. 7 Furthermore, it has been recognized in guidelines for health technology assessment (HTA) that RWE can be used to improve evidence synthesis and integration by supplementing the evidence from RCTs. [8][9][10] Therefore, RWE, while not a replacement for RCTs, can provide the best available evidence for an HTA that fits for purpose when it is difficult to implement a well-controlled clinical trial. 11 Although there were a few studies that examined how RWE has been incorporated in economic evaluation of drugs, primarily in Europe, it has not been systematically assessed in the United States. 12,13 Considering the variations in policies for RWE use across countries, quantitative examination of RWE use in the United States may give payers an objective view of the possible roles of RWE in economic assessments of drugs. 14 This is especially important considering that the use of RWE varies from payer to payer and is generally limited because of the lack of understanding of how to incorporate RWE into the value assessments of drugs. 7 Also, the previous study in European countries was focused on a specific type of disease and thus there is not enough information about the use of RWE across a broad range of diseases. 13 Furthermore, the understanding of the current roles of RWE in the United States can help identify the gap between what has been called for and what has been done in terms of RWE use.
The Institute for Clinical and Economic Review (ICER) is a nonprofit independent research organization that evaluates the clinical and economic values of medical interventions. ICER reports have been increasingly used by U.S. payers in their decision-making processes, such as pharmacy and therapeutics review and pricing negotiations. [15][16][17][18] The goal of this study was to assess the use of RWE in economic assessment of drugs by ICER to understand the recent volumes, trends, reasons, and characteristics of RWE use in the United States.

STUDY DESIGN
We conducted a retrospective analysis of the use of RWE in the long-term cost-effectiveness analysis (CEA) and potential budget impact analysis (BIA) sections of final evidence reports published by ICER. We used the ICER database because ICER is one of the most influential HTA agencies in payer decision making in the United States and makes their evidence reports publicly available. [15][16][17][18] ICER has defined a clear process for performing an HTA that includes topic selection, scoping, clinical evidence review and economic modeling, report generation, public meeting, and finalization of materials. 19 In general, all final evidence reports include CEA and BIA sections, and details about CEA and BIA models, including structure, assumptions, and model inputs, are provided in their methods section. ICER contracts with academic units to conduct the majority, if not all, of the economic models.
We used the FDA definition of RWE as the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from the analysis of real-world data (RWD), and RWD is defined as data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. 1 This definition is consistent with ICER's definition: RWE is clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD, and RWD are data relating to patient health status and/or the delivery of health care collected either prospectively or retrospectively from observations of routine clinical practice. 16

INCLUSION AND EXCLUSION CRITERIA
Final evidence reports that assessed at least 1 pharmaceutical intervention and published between January 2014 and June 2019 were included. We decided to focus on this period because a 5-year period is enough to see the time trend of the use of RWE, the vast majority of ICER reports were published in this period, and 2014 is approximately when ICER started to conduct economic analyses consistently. In case an evidence report has been updated, we only included the most recent version. Reports were identified using the Assessments page on the ICER website (https://icer.org/ explore-our-research/assessments/). all model inputs informed by RWE per report. The trend of this measure was examined over time.
Reasons for RWE use were described using the frequency distribution of model input types for which RWE was used. Model inputs were classified into 15 types based on their attributes (e.g., mortality or disease progression rate, health care costs, and adverse drug events) for ease of categorization.
Study design and data source of RWE were characterized using frequency distribution. We further calculated the frequency distribution for each model input to identify the pattern of study designs and data sources by inputs. In terms of preregistration and study funding, the proportion of RWE preregistered in any study registry (e.g., clinicaltrials.gov) and the proportion of industry-sponsored RWE was calculated. Lastly, we calculated the number of RWE uses by the calendar year of FDA approval of drugs to assess the trend of RWE uses by the post-market periods of drugs.
FDA approval year of each of the drugs assessed in all reports was obtained from Orange Book, 21 Purple Book 22 or package inserts, then the number of RWE uses was counted for each group of drugs approved in the same year. Only RWE that studied specific drugs was included in this analysis. In other words, population-level statistics (e.g., U.S. vital statistics and death certificate data) or nondrug-specific RWE (e.g., transition probabilities, utility, and health care resource utilization among general patients) were not included in this analysis. RWE that studied multiple drugs was counted multiple times for each of the drugs. In addition, we normalized the absolute number of RWE by FDA approval year by accounting for the total number of FDAapproved drugs included in ICER reports in a given year.

ICER REPORTS
Thirty-three reports met the inclusion criteria and were included in the review ( Figure 1 and Supplementary Table 1, available in online article). The reports included a total of 123 pharmaceutical interventions and comparators indicated for 29 diseases, including 5 cancers (e.g., ovarian cancer and prostate cancer), 9 rare diseases (e.g., cystic fibrosis and tardive dyskinesia), and 15 non-rare and noncancer diseases (e.g., atopic dermatitis and osteoporosis). All reports included a CEA, but 2 reports did not include a BIA (Supplementary Table 2, available in online article).

VOLUME OF RWE USE
We identified 407 RWE uses in 33 ICER reports, all of which used at least 1 RWE source. The mean number of RWE uses per

DATA EXTRACTION
We developed a data extraction form (DEF) based on questions adapted from the study by Makady et al. (2018) to abstract information about each report and RWE use in a standardized way (Supplement 1, available in online article). 13 The DEF consisted of 4 parts, each with different types of information abstracted. • In part 1, we collected general information on each ICER report, such as indication and drugs assessed. • In part 2, for each use of RWE identified in the CEA and BIA models, we examined for which model inputs each RWE was used for. To do this, we first identified model inputs and variables on outcomes of a drug or patient characteristics (i.e., clinical, utility, economic inputs).
The description and categorization of model inputs along with sources of their values are available in the model inputs section of the methods section. We searched for, identified, and read the sources of each model input to determine whether it meets the definition of RWE used in our study. Based on the FDA's definition, we considered evidence related to the usage (e.g., health care utilization and costs) and benefits or risks of a medical product (e.g., effectiveness and utilities) as RWE if it was derived from data that meet the FDA's definition of RWD. • In part 3, we abstracted characteristics of each RWE, such as study design, data source, and sponsorships. • In part 4, we determined whether the value of each model input was informed by RWE only, non-RWE only, or both RWE and non-RWE. The included ICER reports were divided among 2 independent reviewers, and the abstraction results were cross-checked (see Supplement 2, available in online article, for the consensus on the data extraction).

OUTCOMES
The extracted data were used to examine (a) the volume of RWE uses and its time trend; (b) reasons for RWE use; and (c) characteristics of RWE including study design, data source, whether or not it was preregistered, funding support for the study, and the number of RWE uses by calendar year of FDA approval of drugs.

ANALYSIS
The volume of RWE use was measured by the absolute number of RWE uses per report. If RWE was used multiple times (e.g., for different model inputs), it was counted as many times as it was used for different inputs. To account for potential bias in the absolute number of RWE uses due to model-specific characteristics, such as the number of drugs and health states included, we measured the relative number of RWE uses per report by calculating the proportion of

STUDY DESIGN, DATA SOURCES, AND OTHER CHARACTERISTICS OF RWE
The most frequently used study design was a retrospective cohort, which accounted for more than 50%  Retrospective studies accounted for more than half of RWE uses for each nondrug-specific clinical input, such as disease progression and mortality rates (64/117, 54.7%), patient characteristics (16/23, 69.6%), and incidence (5/10, 50%; Table 1). Drug-specific clinical inputs, which included effectiveness, drug-specific discontinuation rates,  Among 407 uses of RWE, only 5 (1.2%) studies were preregistered. One was a meta-analysis that was registered in the Prospective Register of Systematic Review system. The other 4 studies were prospective observational studies that were preregistered in ClinicalTrials.gov. Approximately a third of RWE studies (123/407, 30.3%) were sponsored by industry and another 60.7% (247/407) by nonindustry.

Economic inputs
Health care costs 86 Retrospective (68) Administrative claims data (49) Non-health care costs 27 Cross-sectional for survey (16) Patient survey data (15) Treatment patterns/market share 13 Retrospective (9) Administrative claims data (4) Assumptions RWE used to support assumptions 19 Retrospective (8) Prospective (6) Registry data (9) a For each of the model inputs, study designs or data sources that make up at least 30% are included. b Numbers in parentheses indicate the number of RWE uses with the corresponding study design or data source. RWE = real-world evidence.

TABLE 1 Study Designs and Data Sources by Model Inputs
not a noticeable difference in sample sizes, data sources, study designs, or whether or not it was preregistered. These findings may be a result of the lack of guidance on types of RWE used for regulatory decisions (i.e., the FDA does not endorse any one type of data or study design). 29 Our findings suggest the need for approaches to assessing the appropriateness of RWE use that differ based on circumstances (e.g., data availability) or purposes (e.g., outcomes). 30 We found few preregistered RWE studies. Our finding is in line with previous findings that, although much of the ethical and scientific rationale for registering clinical trials also applies to observational studies, only about 17% of registered studies at ClinicalTrials.gov are observational studies. 31 Preregistration can benefit the quality of RWE by improving reliability and transparency. As the need for credibility in RWE increases, there have been several efforts to facilitate the registration of observational studies, such as a certification process for payers and the RWE Transparency Initiative to establish a culture of RWE transparency that was launched by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR). 32,33 The initiative's steering committee, in its recent white paper, outlined an approach designed to facilitate the registration of observational studies, particularly those evaluating effectiveness or safety. Such efforts to enhance the trustworthiness of RWE may further facilitate using drug-specific clinical outcomes estimated based on RWE.
The increasing proportion of model inputs was informed by RWE over time, which is in part explained by the evolution of ICER's value framework. While the essential valuation criteria (i.e., long-term value for money and short-term affordability) did not change, there have been updates in the economic model framework that could have possibly affected the use of RWE. 34 The most notable change was the use of a societal perspective for CEAs. In 2017, ICER announced that the base-case incremental cost-effectiveness ratio will continue to be calculated from the health system perspective, but a scenario analysis from a societal perspective will be performed. Since this requires models to have non-health care sector benefits and costs (e.g., productivity costs) that are hard to be informed by RCTs, this change might have increased the number of RWE uses.
Our findings are for the most part consistent with previous studies. Makady et al. reviewed the use of RWE in European countries and found that RWE is used in the majority of economic assessments, and many of them were used to inform costs or resource use. 13 They also found that registry data and claims data were most frequently used, which was in line with findings from another study by Parody-Rúa et al. (2020), which was a systematic review of economic evaluation studies that used RWD for both that studied drugs approved between 2006 and 2015 and after 2016 made up 20.9% (18/86) and 0% (0/86), respectively (Supplementary Table 4, available in online article). When the number of RWE uses was normalized by the total number of drugs approved in a given year, drugs that were approved by the FDA earlier in the calendar year had a greater number of RWE uses. The numbers of RWE uses per drug were around 1.8, 1.6, and less than 1 for drugs approved before 1990, between 1996 and 2005, and after 2006, respectively.

Discussion
We evaluated the use of RWE in the United States in economic analyses of pharmaceuticals by reviewing ICER Final Evidence Reports between 2014 and 2019. Although all economic models had at least 1 instance of RWE use, there was a wide variation in the extent to which RWE was used to inform model inputs. RWE was commonly used to inform disease progression and health care costs, but it was rarely used for drug effectiveness. The most common study design was a retrospective cohort, and the most common data source was registry data, respectively.
We found RWE has a minor role in informing drugspecific clinical outcomes such as effectiveness, safety, and drug discontinuation in ICER's CEAs; these estimates were primarily derived from RCTs only. A study showed that payers are hesitant to use RWE because of reservations regarding quality of RWE and lack of ability to evaluate RWE. 7 This indicates the lack of awareness and underutilization of quality assessment tools for RWE. 22-26 Another reason for the underutilization is due to the time required to generate RWE. 7 ICER reports are usually published near the time drugs are approved, which does not provide much time for RWE to be generated. Therefore, there are limited resources available for economic assessments that inform initial health care decision making. The lack of RWE for newer drugs often leads to a situation where the RWE for older drugs is extrapolated to a newer drug. ICER often updates the assessments and incorporates recent RWE that was not available originally, which sometimes results in a significant difference in conclusions. In the psoriasis report, for example, discontinuation estimates that were originally informed by extrapolating RWE of older drugs to newer drugs have been updated as RWE for newer drugs became available later. 27 Our finding may suggest the need of reassessment of a drug and adjustment of health care decisions based on updated evidence after the launch. 28 The characteristics of RWE used for drug-specific clinical outcomes did not appear to be different than RWE used for nondrug-specific inputs. To be more specific, there was agencies, since we only focused on ICER reports. There is considerable variation in the absolute number of RWE uses per report, which is due to various factors such as the number of drugs and health states assessed in the economic models and how much RWE is available for the conditions or drugs. Therefore, the numbers should be interpreted with caution. To avoid this bias, the relative number of RWE uses that is less likely to be affected by the aforementioned factors was measured. It is important to note, however, that the absolute and relative numbers of RWE uses alone are not likely to inform whether or not RWE is used sufficiently, since the optimal use of RWE is not simply determined by the number of uses.
Although there was an increasing trend in the relative number of RWE uses over time, the number was exceptionally high in 2015 due to a higher frequency of RWE use in 2 of the 4 ICER publications in 2015: the reports for hepatitis C and high cholesterol. Both reports were class reviews and thus there might have been more RWE available for older drugs. Also, the report for hepatitis C was an updated version, which is likely to include more RWE than the original version. Considering that the number of RWE uses can vary by the types of reports (e.g., class reviews and updated report), future work is needed to adjust for this factor.
Also, some people may find it unclear whether evidence on health care costs should be considered as RWE or the examples of RWD use, because the FDA definition states that RWE is "the clinical evidence." We counted it as RWE if the evidence was derived from the analysis of data of usage or delivery of health care (e.g., claims and billing activities), assuming evidence derived from any RWD is RWE. Our interpretation was also based on the value framework by ICER where RWE is considered as a source to inform nonclinical outcomes. By using a broad definition of RWE, however, we might have overemphasized RWE use.
Finally, we did not take into account the publication year of ICER reports when calculating the number of RWE uses by the FDA approval year of drugs. Since the FDA approval year does not necessarily correspond to how long a drug had been in the market before its ICER report was published, the results should be interpreted with caution. That said, since all ICER reports included in this study were published within the 5-year time window, we expect that there would be no considerable difference in the trend.

Conclusions
RWE has been used for various purposes in the economic assessment of pharmaceutical interventions in the United States by ICER. However, there has been relatively limited use of RWE to inform drug-specific effectiveness, despite cost and clinical inputs. 35 Parody-Rúa et al. also found that only about 5% of the studies that used RWE were published before 2011, suggesting the recent increase in RWE use.
Furthermore, our finding that RWE is rarely used for treatment effectiveness is consistent with existing policies on RWD use. A study that reviewed policies for RWD use in HTAs stated that it is recommended by most agencies using RWE under specific circumstances-for instance, in the absence of RCT evidence. 14 ICER, however, recently announced that it will start to generate decision-grade RWE with partners to improve understanding of the realworld performance of treatments. 36 This is in line with the most recent ICER value assessment framework where it was stated that ICER seeks to use RWE to inform longterm safety or durability of a medication's effect. 10 This movement will be facilitated by assessing the internal and external validity based on the outline presented in the ISPOR RWD Task Force report. 37 ICER also announced that de novo RWE generation will be facilitated to inform clinical and economic outcomes that are not included in RCTs. As such, the increasing use of RWE for drug effectiveness is expected in the United States, along with RWE use for other purposes.
Although most of our findings coincide well with existing studies and policies, there were a few differences from findings in Europe. First, Makady et al. found that 12% of 25 cost-effectiveness models they reviewed did not use any RWE. 13 Our evaluation did not find a comparable frequency of RWE use, likely because Makady et al. only reviewed HTA reports on melanoma drugs while we reviewed all types of drugs. Since melanoma trials generally follow patients for sufficient time to estimate survival probabilities, less RWE is likely needed for disease progression rates compared to other types of diseases. Moreover, Makady et al. found that RWE is most frequently used for extrapolating longterm effectiveness and costs, while we found that RWE is frequently used for mortality and disease progression rates but rarely used for informing drug effectiveness. There are 2 potential reasons for this difference: (1) many of the European melanoma reports used the same registry data to estimate long-term overall survival beyond the trial, and (2) the definition of effectiveness in Makady et al. appears to be broader than ours. We did not classify RWE on mortality or disease progression rates that are not on specific drugs into the effectiveness category to make a distinction between evidence on natural disease progression and drug-specific effectiveness.

LIMITATIONS
This study has some limitations to consider. Our study may not be representative of the RWE use in all U.S. HTA calls for greater inclusion of RWE in value assessments for real-world drug effectiveness. These findings can help identify the gap between what has been called for and what has been done in the use of RWE, which can give directions to the better use of RWE in HTAs.