Oncologists’ Perceptions of Drug Affordability Using NCCN Evidence Blocks: Results from a National Survey

BACKGROUND: The increasing prevalence of cancer coupled with approvals of new drugs and technologies used in therapy have brought increased scrutiny to the cost and value of treatments in oncology. To address the rising concern about oncology drug costs, several organizations have developed value frameworks to help assess the value of oncology regimens. The objective of this study was to assess oncologists’ perceptions, awareness, and knowledge of all oncology value frameworks in the United States and to understand oncologists’ perceptions of affordability in the context of National Comprehensive Cancer Network (NCCN) Evidence Blocks. OBJECTIVES: To (a) assess oncologists’ awareness, knowledge, perceptions, and ratings of the American Society of Clinical Oncology Value Framework (AVF), the Institute for Clinical and Economic Review (ICER) value framework, NCCN Evidence Blocks, and Memorial Sloan Kettering Cancer Center’s DrugAbacus; (b) assess oncologists’ knowledge and perceptions of drug affordability as defined by the NCCN Evidence Blocks methodology; and (c) determine the factors that influence drug affordability ratings. METHODS: Data were collected from an electronic cross-sectional survey of 200 U.S.-based oncologists from a variety of practice settings. Oncologists were asked about their knowledge and perceptions of 4 value frameworks—NCCN Evidence Blocks, AVF, the ICER value framework, and DrugAbacus. Using NCCN Evidence Blocks, oncologists were asked to rate a variety of hypothetical cancer therapies and assign costs (in U.S. dollars) to the 5 levels of affordability. Additional questions that assessed perceived patient out-of-pocket (OOP) costs and comfort level in assessing affordability were also included in the survey. RESULTS: Oncologists were most familiar with NCCN Evidence Blocks (90%), followed by the AVF (84%), ICER value framework (57%), and DrugAbacus (56%). Oncologists rated affordability higher (mean rating 3: moderately expensive) versus the actual NCCN panel affordability rating (mean rating 1: very expensive). The affordability rating was similar across a variety of hypothetical cancer therapies and tumor types (rating: 3). Oncologists estimated the costs for this rating of 3 to range from $4,600 to $6,000 per month, which was inconsistent with actual drug costs. Oncologists estimated the mean monthly OOP costs for patients with insurance to range from $1,260 for a new oral medication to $1,700 for a new infused medication. Only 26% of oncologists were comfortable or very comfortable with rating costs associated with affordability levels. CONCLUSIONS: Surveyed oncologists rated cancer therapies as more affordable (per NCCN Evidence Blocks criteria) than NCCN panel ratings. Costs associated with affordability were not consistent with actual treatment costs; however, most oncologists were not comfortable with rating affordability. Patient OOP costs had the biggest influence on affordability ratings; however, physicians overestimated patient OOP costs significantly. There is an opportunity to improve the value frameworks, especially with regard to affordability assessment.

B y 2020, cancer care costs are projected to reach approximately $158 billion, up from $125 billion in 2010. 1 The increasing prevalence of cancer coupled with the rise in expensive new drugs and technologies used in therapy have brought increased scrutiny to the cost and value of treatments in oncology. To address the rising concern about oncology drug costs, several stakeholders have developed frameworks to help assess the value of oncology regimens. 2 In the United States, these include the American Society for Clinical Oncology Value Framework (AVF), National Comprehensive Cancer Network (NCCN) Evidence Blocks, the Institute for Clinical and Economic Review (ICER) value framework, and Memorial Sloan Kettering Cancer Center's DrugAbacus.
The AVF leverages head-to-head randomized clinical trials to compare 2 cancer treatment regimens. The framework generates a net health benefit score and presents this score along with the direct cost of treatment to form a summary assessment for the treatment. 1,3 DrugAbacus is an online tool that allows users to set the weights for several factors, including • To address the rising concern about oncology drug costs, several stakeholders have developed frameworks to help assess the value of oncology regimens.

What is already known about this subject
• Costs associated with affordability ratings of NCCN Evidence Blocks were not consistent with actual treatment costs and were underestimated by 39%-60% across treatments. • Patient out-of-pocket (OOP) costs had the most influence on affordability ratings; however, physicians overestimated patient OOP costs for oral and infusion treatments. • Affordability ratings did not differ based on oncologist specialty, practice site, or NCCN/National Cancer Institute membership and may be global impressions of what oncologists consider "affordable," rather than being specific to a treatment or tumor type.
frameworks, which included 90 multiple-choice questions or ranking and rating questions that took each respondent about 20 minutes to complete. Physicians who provided informed consent were screened for eligibility. Inclusion criteria were physicians who are board-certified or board-eligible oncologists, in practice for 2-30 years, and see ≥ 5 cancer patients per month. Eligible physicians were permitted to complete the entire survey and submit online. The study protocol and data collection form were reviewed by an independent institutional review board, which granted the study exemption status.

Sample Size Calculation
A total of 200 oncologists were recruited to participate in this study. With this sample size, the estimated precision of descriptive information (e.g., awareness of a framework) would be ±6.86%, assuming a 95% confidence interval and the maximum variance of a binomial distribution (i.e., 0.25).

Study Measures Physician Demographics and Physician Practice
Characteristics. Physicians provided information about their age, sex, ethnicity, region, board certification status, and years in practice. Physicians provided information about their primary practice setting, including years in practice, type of practice site, patient load, state where practice is located, percentage of Medicaid patients seen in practice, and types of cancer treated.

Awareness, Perceptions, and Rankings of Value Frameworks.
Oncologists provided information about their awareness and perceptions of the AVF, the ICER value framework, NCCN Evidence Blocks, and DrugAbacus. Oncologists ranked these 4 value frameworks on the following characteristics: ease of use, comprehensiveness, feasibility for use in the clinical setting, ability of patients to understand the framework, and most favorable overall.
Affordability. Using NCCN Evidence Blocks, oncologists were asked to rate affordability for 4 hypothetical cancer therapies: an anaplastic lymphoma kinase (ALK) inhibitor, 2 immunotherapy agents (1 for breast cancer and 1 for lung cancer), and an oral oncology treatment for leukemia/lymphoma. The NCCN affordability rating scale was used to measure oncologists' perceptions of a drug's affordability as follows: 5: very inexpensive, 4: inexpensive, 3: moderately expensive, 2: expensive, and 1: very expensive. Affordability (as defined by NCCN) refers to the overall cost of an intervention, including drug cost, required supportive care, infusions, toxicity monitoring, management of toxicity, and the probability of care being delivered in an inpatient hospital setting. 5 After rating the drug's affordability, oncologists were asked to assign costs (in U.S. dollars) to the 5 levels of affordability. Additional questions assessing novelty of the drug, to determine the value-based price of an oncology regimen. The ICER value framework assesses clinical benefit, cost-effectiveness, and budget impact of a drug or health intervention. It has a broader audience and is intended for use by payers, policymakers, patients, and physicians. 2 It is not limited to oncology regimens and can be used to assess the value of any drug, device, or procedure. In contrast to these value frameworks, NCCN Evidence Blocks generate 5 scores for a treatment's efficacy, safety, quality and consistency of evidence, and affordability. Evidence Blocks are developed by analyzing the scores of NCCN's expert panel members, which are then plotted onto a 5 × 5 matrix to produce a visual plot of the panel members' responses. 4,5 As cost pressures continue to rise in the United States, NCCN Evidence Blocks may be increasingly used by patients and providers to determine which regimens to prescribe and by payers to determine which regimens to reimburse as part of their health plans. Given that the Evidence Blocks could have considerable impact on the use and coverage of oncology regimens, it is important to gain a better understanding of the validity and key drivers of NCCN Evidence Block ratings. Further, assessment of cost and affordability play an important role in NCCN Evidence Blocks; however, the degree of oncologists' understanding of the cost and affordability of cancer treatments and whether affordability ratings correspond to actual drug costs are unclear. In addition, it is not clear if the NCCN expert panel ratings of affordability differ from other practicing oncologists.
The primary objectives of this study were as follows: To (a) assess oncologists' awareness, knowledge, perceptions, and ratings of the AVF, the ICER value framework, NCCN Evidence Blocks, and DrugAbacus; (b) assess oncologists' knowledge and perceptions of drug affordability as defined by the NCCN Evidence Blocks methodology; and (c) determine the factors that influence drug affordability ratings.

■■ Methods Study Sample
This was an online cross-sectional study conducted in the United States between November 15, 2016, and January 15, 2017. Oncologists were recruited from the Medefield physician panel (http://www.medefield.com/). The Medefield panel is an opt-in panel, in which physicians join with the understanding that they will complete periodic surveys. All physicians in the United States have their credentials validated with the American Medical Association.
Before launching the full study, pretests were conducted among 10 oncologists who provided feedback on the content and structure of the survey. For the full study, email invitations were sent to a random sample of physicians selected from a panel of 1,000 oncologists in the United States. The data for this study were extracted from a larger survey about value perceived patient out-of-pocket (OOP) costs, perceived drug acquisition costs, and comfort level in assessing affordability were also included in the survey.
The NCCN Evidence Block evaluation is administered via survey, during which NCCN provides information to panel members on the efficacy, safety of the drug, and the quality and consistency of evidence. 5 NCCN does not provide panel members with information about the cost of the intervention; panel members are asked to rate the affordability based on their knowledge. 5 For this study, we provided information on efficacy, safety, quality, and consistency, but not on affordability (participants were only told that the drugs were similar in cost to others in the same class of therapy; i.e., similar to other cancer immunotherapies), consistent with NCCN practice.
Factors Influencing Affordability Ratings. After rating the affordability of each of the drug products, oncologists were asked to select and then rank the factors that influenced their affordability rating. Factors included patient OOP costs, drug acquisition cost, duration of treatment, cost to manage drugrelated toxicities, administration costs, site of care, costs of other treatment options, supportive care costs, patient socioeconomic status, efficacy and safety of the drug, potential for hospitalization, or other factors.

Statistical Analysis
Physician demographic and practice site characteristics, knowledge, and opinions were reported descriptively using frequencies and percentages for categorical variables. Continuous variables were reported using means, standard deviations (SDs),  Demographic Characteristics of Oncologists a medians, and interquartile range. Mean values were calculated for affordability ratings and rounded to the nearest integer, consistent with NCCN practice. T-tests and analysis of variance and Kruskal-Wallis tests were used to assess differences in means and medians for continuous variables. Chi-square tests were used to assess differences in familiarity and perceptions of value frameworks by physician demographic/practice characteristics. All analyses were conducted in Stata software package 13 (StataCorp, College Station, TX) or R software package 3.3 (R Foundation for Statistical Computing, Vienna, Austria).

Knowledge, Awareness, and Perceptions of Oncology Value Frameworks
Oncologists were most familiar with NCCN Evidence Blocks (90%), followed by the AVF (84%), ICER value framework (57%), and DrugAbacus (56%; Appendix A, available in online article). More than 3 out of 4 oncologists surveyed (76%) had used value frameworks in the past, with NCCN Evidence Blocks being used most often (61.5%), followed by the AVF (48.4%), the ICER value framework (21%), and DrugAbacus (15%). Among those who were familiar with the value frameworks, almost equal proportions found NCCN Evidence Blocks compared with the AVF to be "useful/very useful" (75% vs. 74%), followed by the ICER value framework (64%) and DrugAbacus (56%; Appendix B, available in online article).
Respondents were asked to rank each framework from 1 to 4 (with 1 being the highest) for its ease of use, comprehensiveness, feasibility for use in the clinical setting, ability of patients to understand, and most favorable overall. NCCN Evidence Blocks was ranked highest on all measures, while the AVF was consistently ranked second, the ICER value framework was consistently ranked third, and DrugAbacus ranked fourth on all measures.

Affordability Ratings for Oncology Treatments
The NCCN Evidence Block affordability rating by oncologists was similar across a variety of hypothetical cancer therapies and tumor types (mean rating: 3, moderately expensive), despite differing monthly drug costs and OOP costs ( Table 2).
These ratings were higher (more affordable) than those given by the NCCN expert panel for drugs in a similar therapy class, such as cancer immunotherapies (CITs) in non-small cell lung cancer (NSCLC) or ALK inhibitors. Affordability ratings did not differ based on oncologist specialty, practice site, or NCCN or National Cancer Institute (NCI) membership (N=200, P > 0.05 for all comparisons).

Factors Influencing Affordability Ratings
The factors influencing an oncologist's affordability rating were consistent across the 4 hypothetical cancer indications, with patient OOP costs being ranked as the most influential factor. Drug acquisition costs and duration of treatment were ranked as the second and third most influential factors for affordability, respectively ( Figure 1).

Perceived Monthly Patient Out-of-Pocket and Drug Acquisition Costs
To further understand the top 2 factors influencing affordability ratings, we assessed oncologists' perceptions of patient OOP costs and drug acquisition costs. Oncologists' estimates for patient OOP costs differed by mode of administration, with estimated mean monthly OOP costs for patients with insurance ranging from $1,260 (SD = $1,127) for a new oral medication to $1,666 (SD = $1,454) for a new infused medication. Estimated OOP costs did not vary significantly by oncologist's specialty, practice site, or comfort level in estimating costs (P > 0.05 for all). Mean OOP costs were also calculated by the percentage of Medicaid patients seen in practice; however, there were no significant differences for infusion or oral product costs (P > 0.05 for both comparisons).
To assess oncologists' perceptions of treatment affordability, participants were asked to attach a dollar value to each affordability level for each drug corresponding to the NCCN Evidence Block methodology (very inexpensive, inexpensive, moderately expensive, expensive, very expensive). Estimated monthly drug acquisition costs associated with affordability ratings by   Figure 2. Oncologists' estimated costs for each affordability rating were similar across indications. Monthly drug costs associated with an affordability rating of 3 were estimated to be $4,600-$6,000; these costs were underestimated by 39%-60% across treatments. In the overall sample, dollar values associated with the 5 affordability categories (very inexpensive, inexpensive, moderately expensive, expensive, very expensive) were considerably lower than the actual costs (based on list price) for the drug.

Comfort Level with Rating Affordability
Only 52 (26%) oncologists were "comfortable/very comfortable" with rating costs associated with affordability levels; 94 (47%) were "somewhat comfortable"; and 54 (27%) were "uncomfortable/very uncomfortable" (Figure 3). Affordability ratings were not significantly different when comparing those who were uncomfortable/very uncomfortable compared with those who were very comfortable/somewhat comfortable/comfortable. These findings were consistent across all drugs evaluated in this study (P = 0.06 for ALK+ inhibitor, P = 0.83 for CIT in NSCLC, P = 0.07 for CIT in breast cancer, and P = 0.29 for oral hematology/oncology drug).

■■ Discussion
The majority of oncologists were familiar with value frameworks and had used them in the past. Of the 4 oncology value frameworks, NCCN Evidence Blocks were perceived most favorably by oncologists, followed by the AVF. Since NCCN Evidence Blocks and the AVF are intended for use by the oncologist and patient as part of the treatment decision-making process, they may be the most directly relevant and hence, may be considered most useful by oncologists. 6 NCCN Evidence Blocks use a simplified metric to assess affordability that ranges from 1 = very expensive to 5 = very inexpensive. However, the scoring criteria are subjective and are based on panel members' knowledge and clinical experience. Additionally, NCCN does not provide panel members with any information pertaining to the cost categories that go into determining the affordability rating-drug cost, supportive care, administration and monitoring costs, and toxicity management costs. Rather, affordability is rated using the panel members' knowledge of overall cost of the treatment regimen. Further, it is not clear if there are experts in health economics on the panel who may provide a deeper understanding of treatment costs. 5 tumor type. Ranges were consistent for each affordability rating irrespective of treatment type, whereas costs associated with each affordability rating were underestimated by 39%-60%. This suggests that oncologists' perceptions of affordability are not affected by the treatment type or cancer type; rather, they are global impressions of what oncologists consider "affordable." Oncology value frameworks can be useful, and their methods to assess efficacy and safety are largely based on known and accepted standards. 2 However, some aspects of their methodology, especially pertaining to affordability assessment, have not been tested or validated.
This study presents the first assessment of oncologists' perceptions of affordability using NCCN Evidence Blocks' approach to affordability and its associated cost ranges. Oncologists in this study overestimated patient OOP costs and underestimated drug acquisition costs; however, only 26% reported feeling comfortable estimating costs associated with affordability. Given that no information currently exists to assess the reliability and validity of NCCN Evidence Blocks' approach to affordability, the results of this study raise a potential issue of whether the affordability ratings accurately reflect costs. Further, the scale used to assess affordability has not been tested or validated. Of the first 10 tumor types to have Evidence Blocks published, none have received a rating of 5, and only 4.79% have received a rating of 1, suggesting that ratings may not span the full spectrum of the scale. 9 As oncologists in this study indicated, most of them are not comfortable with rating the construct of affordability as defined by NCCN. Currently, NCCN Evidence Blocks do not specify the perspective or the context for the construct of affordability. Additionally, the components of the affordability metric are largely undefined. For example, one of the components is drug acquisition cost, but it is not clear from whose perspective (the patient, the payer, or the health care system). Supportive care costs, administration, toxicity, and management costs are all part of the affordability construct, but they are hard to estimate and define.
It is unclear whether oncologists are considering the costs of all of these components when rating affordability and, if so, what sources of information on the costs of these components they are using. Patient OOP costs are not a component of the affordability metric as defined by NCCN; however, oncologists indicated that is the main factor driving their affordability ratings. Future research should assess oncologists' knowledge of costs and cost components as defined by NCCN. Future research should also assess the reliability and validity of the affordability metric of the Evidence Blocks and determine if ratings align with actual costs. As value frameworks evolve, such as with the newly released NCCN Preference Strategy Guidelines, it will be important to refine and validate their metrics.
There are no clearly defined ranges or thresholds for determining the affordability rating, and it is not clear if oncologists are aware of the true costs associated with each of the above categories, given the opaque nature of drug pricing in the United States. In our study, oncologists' ratings of affordability were higher than those of the NCCN expert panel-an indication that oncologists underestimated treatment costs. Affordability ratings did not differ by the respondents' practice characteristics, their specialty, NCCN/NCI membership, or comfort level in rating affordability, suggesting that these ratings are independent of practice characteristics.
The main drivers of the affordability ratings were patient OOP costs and drug acquisition costs. Oncologists estimated the mean monthly OOP costs for patients with insurance, ranging from $1,260 for a new oral medication to $1,666 for a new infused medication. However, actual mean monthly OOP costs for a new oral treatment are estimated to be in the range of $90-$900, whereas those for an infused medication are estimated to be $85-$238. 7,8 Thus, oncologists in the study overestimated patient OOP costs for oral treatments by 140%-1,400%, whereas costs for infusion treatments were overestimated by 700%-1,960%.
Drug acquisition costs were the second biggest factor driving oncologists' affordability ratings. In this study, oncologists' estimated drug acquisition costs did not differ by treatment or

Limitations
This study has some limitations to consider. The study was descriptive in nature without any adjustment for how confounding variables could have influenced affordability ratings. Only members of an Internet panel were eligible to participate; these physicians might differ from the broader oncologist population or from NCCN panel members in their experience and understanding of drug costs and affordability. Since this was an online survey, it is possible that oncologists could have accessed pricing references when ranking affordability. However, given that the average length of time required to complete the survey was about 20 minutes, and there were no large deviations from this time, it is unlikely that this occurred.

■■ Conclusions
Surveyed oncologists rated cancer therapies as more affordable than did NCCN panel ratings. Costs associated with affordability were not consistent with actual treatment costs; however, most oncologists were not comfortable rating affordability. Patient OOP costs had the biggest influence on affordability ratings; however, physicians overestimated patient OOP costs significantly. Given these results, there is an opportunity to both improve oncologists' knowledge of affordability to enable them to have a discussion with their patients as well as improve the validity of the NCCN Evidence Block framework, especially with regard to the affordability assessment.