Using the Delphi method to identify meaningful and feasible outcomes for pharmaceutical value-based contracting

In an effort to demonstrate measurable value of pharmaceuticals in the United States, many payers and drug manufacturers have entered into value-based purchasing contracts that link payment for prescription medications to patient outcomes, creating shared risk between the 2 entities. These agreements have emerged as part of a larger movement within the health care landscape to transition away from volume-based payment models and towards value-based designs that promote high-quality and affordable care. Key to the success of pharmaceutical value-based contracting is agreement on meaningful and measurable outcomes that reflect drug performance. Traditional value-based contracts are developed by pharmaceutical companies and payers and may not reflect values of other important stakeholders, such as patients, providers, and employers (when applicable). One approach to more effectively align the interests of all key stakeholders and to maximize the effect and transparency of value-based pharmaceutical contracts is to use the validated Delphi surveying technique, which can gather information and build stakeholder consensus on key elements before contract development. In this Viewpoints article, we describe our experience conducting Delphi studies in 5 disease contexts to inform pharmaceutical value-based contract development, including insights learned and practical considerations for real-world application. In addition, we outline advantages to using this validated consensus-building tool to solicit vital and underrepresented stakeholder input, foster transparency in the contract development process, and promote shared learning for future value-based initiatives.

is estimated to be much higher. 7 Although the specific terms of these contracts and how they were decided upon are largely unreported, payers and manufacturers typically do not solicit input from other key stakeholders, most notably patients and providers, to inform selection of meaningful outcome measures for contract development. 7 creating shared risk between the 2 entities. [1][2][3][4][5] These agreements can include up-side, down-side, or twosided risk, which refer to the direction that money is exchanged between entities based on how the drug performs in the real world. In the United States, approximately 70 pharmaceutical value-based contracts have been publicly disclosed from 2009 to 2019, 6 while the number of actual contracts In response to pressure exerted on the health care industry to improve quality of care while reducing costs, the United States health care system is experiencing a paradigm shift from volume-to value-based care. [1][2][3][4][5] In the pharmaceutical space, many payers and drug manufacturers have entered into value-based purchasing contracts that link payment for prescription medications to patient outcomes, Using the Delphi method to identify meaningful and feasible outcomes for pharmaceutical value-based contracting

SUMMARY
In an effort to demonstrate measurable value of pharmaceuticals in the United States, many payers and drug manufacturers have entered into value-based purchasing contracts that link payment for prescription medications to patient outcomes, creating shared risk between the 2 entities. These agreements have emerged as part of a larger movement within the health care landscape to transition away from volumebased payment models and towards valuebased designs that promote high-quality and affordable care. Key to the success of pharmaceutical value-based contracting is agreement on meaningful and measurable outcomes that reflect drug performance. Traditional value-based contracts are developed by pharmaceutical companies and payers and may not reflect values of other important stakeholders, such as patients, providers, and employers (when applicable).
One approach to more effectively align the interests of all key stakeholders and to maximize the effect and transparency of value-based pharmaceutical contracts is to use the validated Delphi surveying technique, which can gather information and build stakeholder consensus on key elements before contract development.
In this Viewpoints article, we describe our experience conducting Delphi studies in 5 disease contexts to inform pharmaceutical value-based contract development, including insights learned and practical considerations for real-world application. In addition, we outline advantages to using this validated consensus-building tool to solicit vital and underrepresented stakeholder input, foster transparency in the contract development process, and promote shared learning for future value-based initiatives.
While value-based contracts are often touted as patientcentric, in practice the metrics selected to evaluate drug performance are often surrogate endpoints based on ease of measurement. 6 As such, these contracts do not necessarily incorporate patient-oriented outcomes that are meaningful to patients and providers, 8 as such, they may not truly reflect value. One approach to more effectively align the interests of all key stakeholders and to maximize the effect and transparency of value-based pharmaceutical contracts is to use the validated Delphi surveying technique to gather information and build stakeholder consensus on the value of potential metrics before contract development. [9][10][11] We detail the advantages and rationale for using the Delphi method in this context; describe lessons learned from 5 completed Delphi studies in type 2 diabetes, multiple sclerosis, coronary artery disease, migraine, and atrial fibrillation; and present key considerations for evaluating this approach.

Advantages of the Delphi Method in Seeking Consensus on Disease-Specific Outcome Measures
The Delphi method is a multistage surveying tool that allows a diverse group of individuals across geographic locations and areas of expertise to anonymously contribute their opinions on a given topic. [9][10][11] Following each iteration of a Delphi survey, average group scores are tabulated for each item and provided to participants in subsequent survey rounds. Individuals are then encouraged to consider the group ratings when selecting whether to modify their previous responses. This process continues until a predefined threshold of agreement is reached among the panelists. [9][10][11] The Delphi survey's multiround approach offers a solution to the challenge of identifying which disease-specific metrics are most valued by key stakeholders invested in improved patient outcomes, including patients, providers, payers, pharmaceutical manufacturers, pharmacy benefits managers, employers, and other relevant subject matter experts.
In many down-side (from a manufacturer perspective) value-based contracts, rebates are given when a medication does not provide the expected health benefits observed in clinical trials, which are often based on surrogate endpoints that may not reflect value to those most affected. To illustrate, several publicized value-based arrangements in type 2 diabetes have provided payers with higher rebates if members on the drug in question did not achieve a predetermined hemoglobin A1c (A1c) target, which ultimately lowered the net price paid to manufacturers. [12][13][14] A1c is a commonly evaluated clinical measure in type 2 diabetes, patient Delphi panelists from our survey regarded other treatment outcomes as more important, including preventing heart attacks and preventing kidney disease. 15 As another example, in multiple sclerosis, patient-reported outcomes, including physical disability and functional status as measured by activities of daily living (e.g., dressing, eating, or bathing independently), were viewed as more meaningful in our Delphi survey than surrogate clinical endpoints such as new lesions on imaging studies and claims-based outcomes such as hospitalizations for flares. 16 By gaining a deeper understanding of the perceived importance of disease-specific indicators among a diverse population of patients, practicing providers (e.g., primary care physicians as well as specialists such as neurologists for multiple sclerosis or endocrinologists for diabetes), payers, manufacturers, pharmacy benefits managers, and other relevant subject matter experts through a Delphi survey, a purchasing agreement can be designed in which the evaluated treatment outcomes more closely reflect the collectively desired real-world drug benefit.
The innate flexibility provided by the Delphi method is advantageous to investigators (or those developing valuebased contracts) and participants. As a research instrument, it eliminates conformity bias, group pressure, and geographical impediments and offers ease of administration in a cost-effective manner. [9][10][11] Furthermore, participants benefit from complete anonymity-allowing them to exercise independent thinking and decision making-and the ability to reflect and revise contributions through shared learning and controlled feedback.
When compared with other consensus-building study designs deployed in health care research, we determined that the Delphi method is best suited for achieving our goal of integrating the viewpoints of multiple and diverse individuals into a cohesive ranking of outcome metrics. Other frequently cited, multitiered consensus methods, such as the Nominal group technique (NGT), may be better aligned with research studies focused on generating and exploring ideas to address a problem or open-ended question. 17,18 In contrast with the Delphi method, the NGT does not require a specific numerical value for consensus and requires participants to attend an in-person meeting, for example. 17,18 The Delphi method can be readily structured to facilitate a group's assessment of specific outcome measures through a validated research process requiring a predetermined consensus criterion. Moreover, the process is unencumbered by geographical limitations. [9][10][11]18 Notably, the Delphi survey technique can be applied across a wide range of disease states, and each participant panel can be tailored to include a variety of different stakeholder consensus of ≥ 75% is reached among the panel. 20 Final survey results are used to inform the development of innovative pharmaceutical value-based contracts and submitted for peer-reviewed publication. 15,16,22 In our experience conducting Delphi surveys, top-ranked indicators may align with common clinical outcomes routinely observed or measured in practice, or conversely, stakeholders may rank more unconventional measures highest in importance. 15,16,22 For example, preventing heart attacks and preventing death, 2 commonly used clinical endpoints in coronary artery disease, were expectedly selected as most meaningful outcomes by 25 diverse stakeholders in our Delphi survey. 22 These results informed the development of a value-based contract linking reimbursement for an antiplatelet medication to cardiovascular outcomes in patients recently hospitalized following a heart attack or unstable angina. 23 In the case of multiple sclerosis, however, 100% of 26 stakeholders (patients, caregivers, primary care physicians, neurologists, payers, pharmaceutical company representatives, and pharmacy benefits managers) selected worsening physical disability above all other indicators in its ability to provide information about disease status. 16 Functional impairment, also a patient-reported outcome, ranked second. 16 In this instance, Delphi findings diverged from more commonly used measures in value-based purchasing agreements (e.g., disease relapses, multiple sclerosis-related emergency department visits, and number of hospitalizations 2,24,25 ), highlighting the importance of soliciting multidisciplinary consensus.
While a primary impetus for surveying expert stakeholders is to understand the perceived importance of potential measures of drug benefit, it is also critical to gauge how feasible the measures are to collect. In many cases, value-based contract outcomes may be assessed via administrative claims data, such as hospital admissions, emergency department visits, laboratory or radiology billing, or clinical events.
Some organizations such as integrated delivery and finance systems or accountable care organizations may have access to linked claims and electronic health records data, including test results, expanding the possibilities for contract evaluation. Also, relevant patient-reported outcomes that can be collected by health systems or payers provide additional opportunities for innovative contract development. Including an item within the Delphi survey to gain consensus from nonpatient stakeholders on which outcome measures are perceived as most feasible to collect is informative and recommended.
Of note, in our multiple sclerosis Delphi study, when panelists were asked how feasibly worsening physical disability groups. To this point, when conducting a Delphi study to identify meaningful treatment outcomes for patients with migraines, for example, employer stakeholders might be included on the panel due to the sizable burden migraines impose on work-related outcomes. [19][20][21] Furthermore, consideration should be given to the overall panel size, as well as the sample size of each stakeholder group. Because patient and provider viewpoints are historically underrepresented in value-based agreements, 7 we consistently oversample patients and providers as part of our Delphi strategy. In the Delphi literature, acceptable panel sizes may range from 10-100 individuals in total, depending on the nature of the study. 9,10 A larger panel size might be appropriate for studies that include a larger number of stakeholder groups, for example, to ensure that the number of people per group allows for adequate representation of each stakeholder perspective. While a larger sample size is advantageous for gathering a multitude of diverse viewpoints, it adds to the administrative burden of conducting the study, especially when participants require paper copies of surveys to be distributed and collected through standard mail rather than completing the surveys electronically.

Key Considerations and Lessons Learned for Real-World Application
To date, we have piloted the Delphi technique to inform value-based contract development in 5 disease areas: type 2 diabetes, multiple sclerosis, coronary artery disease, migraine, and atrial fibrillation. 15,16,22 When developing a Delphi survey, we first conduct an extensive literature review to compile all relevant disease-specific outcomes to be considered and ranked. We consult with a subject matter expert who reviews the drafted survey and ensures that the list of chosen indicators is comprehensive and clinically appropriate. The survey is then pilot tested with a committee consisting of specialists, primary care physicians, pharmacists, care managers, nurses, and payers to ensure face validity, outcome suitability, readability, and successful technical deployment.
The multidisciplinary committee also identifies a diverse panel of stakeholders with disease-specific experience, whom we contact individually to inform them of the Delphi survey and to obtain consent for participation. Depending on their indicated preferences, the first-round questionnaire is either emailed or mailed to each panelist. After we collect the round 1 responses, average group scores are tabulated for each item and are embedded within the second-round questionnaire. Rounds continue until application of the Delphi technique exemplifies a stride in the learning health systems movement.
We acknowledge that our use of the Delphi method is nontraditional, in the sense that this survey methodology is not typically used to inform contractual designs between payers and pharmaceutical manufactures. While consensus methods such as the Delphi technique are commonly seen in health services research, there is little documentation around consensus-building studies within pharmacy practice. 17 As such, participants in surveys conducted to inform value-based contracting may not understand the business context in which the survey is intended for use. Consequently, it is possible (and even appropriate) that the outcome of the survey could run counter to the underlying business objectives. In our experience conducting 5 disease-specific Delphi surveys for value-based contracting, we have not yet encountered this. Furthermore, we believe for valuebased contracting to truly reflect value, it is critical to assess which outcomes are most meaningful among diverse stakeholders, independent of potential financial implications. We believe that our use of the Delphi method to identify most important outcomes to drive value-based contracting is thus appropriate and valid.

Conclusions
Building on previous experience and success with the Delphi method, we find that using this validated consensus-building tool to solicit input from relevant stakeholder groups facilitates the identification and prioritization of meaningful and measurable outcomes for inclusion in value-based pharmaceutical contracts. This approach fosters transparency and inclusion of underrepresented but vital viewpoints into business practices, tying drug payments to real-world measures that are most valued by those invested in improved disease-specific patient outcomes.

DISCLOSURES
No outside funding supported this project. and functional impairment could be collected, these 2 measures ranked last among all outcome options. 16 Thus, Delphi results might demonstrate that feasibly collected outcomes may not be viewed as most meaningful, and most meaningful outcomes may not be easily collected. An opportunity for innovation is presented in these instances, where inventive solutions for operationalizing the inclusion of new and meaningful indicators, such as patient-reported outcomes, can enhance the value and impact of a pharmaceutical contract.
Our multiple sclerosis Delphi results directly informed subsequent discussions with our industry partners, and motivated efforts to operationalize the collection of a validated, patient-reported measure of disability progression among our members living with multiple sclerosis. This effectively led to a first-of-its kind value-based agreement linking drug reimbursement for 2 specialty medications used to treat patients with relapsing forms of multiple sclerosis to patient-reported measures of disability progression in a real-world population. 26 Given the broader societal movement towards a more transparent and accountable health care industry, employment of the Delphi process to facilitate collaborative identification and prioritization of value metrics to be incorporated into business practice is a step towards enhanced accountability to all stakeholders. 4 D issemination of D elphi findings through peer-reviewed publication can also inform a wider clinical, research, and industry audience. Furthermore, since the National Academy of Medicine and the Agency for Healthcare Research and Quality are encouraging health systems to become learning health systems, where research informs practice and practice informs research, 27 our