“This Drug May Work, But Is It Worth the Cost?” Can Comparative Effectiveness Research Help Tame Rising Health Care Costs?

BACKGROUND: Comparative effectiveness research (CER) has been proposed in the United States as a way to compare new drugs and technologies with established alternatives and determine not just whether a therapy works, but how well it works compared to other options. OBJECTIVES: To define the current use of CER in the development of new drugs and technologies and explore what is needed for this research approach to reduce or stabilize health care costs in the United States. SUMMARY: In 2010, the Patient-Centered Outcomes Research Institute (PCORI) was established by the Patient Protection and Affordable Care Act (PPACA) to coordinate federally funded CER and recommend research priorities. Hochman and McCormick’s (2010) evaluation of 328 randomized trials, observational studies, and meta-analyses involving medications published between June 2008 and September 2009 in 6 key journals showed that most published research did not fulfill the criteria of CER (defined as comparison to active treatment) and that most study design is driven by FDA requirements rather than the need to develop evidence to facilitate selection of the most effective therapy. Since PPACA provides alternative funding for CER, it could encourage funding more studies to help determine which treatment delivers the best value per unit of investment from clinical, humanistic, and economic perspectives. Manufacturers may avoid CER because it increases product development costs, but a drug proven more effective is more likely to be accepted by formulary committees, increasing the drug’s market share, whereas payers may reject or limit use of a new drug that performs less effectively in comparative studies. CONCLUSIONS: CER may not directly reduce expenditures for drugs and medical technologies. The results may vary widely from case to case; however, despite often significantly higher prices for new drugs, it is important to look beyond product costs to the overall impact on health care costs, including medical cost offsets that may occur through improved health or decreased morbidity. To truly decrease cost and improve quality, cost-effectiveness will have to be integrated into CER with the objective of prioritizing efficient therapies in the real-world health care system. If the methods and output of CER improve, the resulting cost-effectiveness ratios will also be more useful to the payer. CER should ultimately, therefore, be a useful tool to help patients, providers, and decision makers provide the most effective and most cost-effective interventions

she worked closely with internal and external stakeholders in the areas of diabetes, obesity, cardiovascular research, and education.
Prior to entering the pharmaceutical industry in 2001, Dr. Sifford-Wilson spent the majority of her professional career in private medical practice, both in Sumter, South Carolina, and Dover, Delaware. She is board certified in internal medicine and is a Fellow of the American College of Physicians.
Dr. Sifford-Wilson is a native of Baltimore, Maryland, having graduated from Milford Mill Sr. High School and Franklin & Marshall College in Lancaster, PA. She completed her medical school education at the University of Maryland School of Medicine in Baltimore and her internship and residency at the Ohio State University Hospitals in Columbus, Ohio.
Dr. Sifford-Wilson is the recipient of many awards, including the 2004 Pfizer Upjohn Award for Community Service. She served proudly as Delaware President of the National Medical Association (NMA) for 5 years and is currently the NMA Region II Secretary. She is also a board member of Witney's Lights, an organization committed to the eradication of domestic violence. However, her true passion lies in community medical education, health literacy, and youth leadership. This supplement was sponsored by Novo Nordisk Inc. and prepared by the Millcreek Outcomes Group (Salt Lake City, Utah). Diana Brixner reports receiving payment as a manager of Millcreek Outcomes Group to submit this JMCP supplement and also reports being a research investigator on outcomes studies funded by Novo Nordisk to the University of Utah. She also reports a consulting relationship with Novo Nordisk, payment for educational programs from Novartis, and research grants from Abbott, Bristol-Myers Squibb, Novartis, and Novo Nordisk. John Watkins reports no compensation for participation in the roundtable discussion or for writing and revising portions of this supplement and reports no financial or other potential conflicts of interest related to the subject of this supplement. Gary Oderda is a principal in the Millcreek Outcomes Group and reports consulting relationships with Pacira and Novo Nordisk, as a speaker for Jansen, and he has done funded research for King (Pfizer), Novartis, and Takeda. S. Monet Sifford-Wilson is a full-time employee of Novo Nordisk Inc. and shareholder of Novo Nordisk Inc. Joseph Biskupiak reports receiving payment as a manager of Millcreek Outcomes Group for his contributions to this JMCP supplement. Jeffrey Dunn reports receiving compensation from Millcreek Outcomes Group for this project and supplement and for participation on an advisory board for Novo Nordisk Inc. He also reports compensation from Biogen Idec for a Payer Steering Committee and consulting fees from Genentech, UCB, and Teva. Anke-Peggy Holtorf reports receiving payment as a contractor of Millcreek Outcomes Group to participate in a thought leader discussion on Comparative Effectiveness Research in Decision-Making, to survey the use of health economics and outcomes research by U.S. decision makers, and to contribute to this supplement. Kelley J. P. Lindberg reports compensation from Millcreek Outcomes Group for writing and revising this supplement.
Brixner and Oderda conceived the study and design with assistance from Biskupiak and Holtorf. Holtorf collected the data with the assistance of Brixner and Oderda. Lindberg assisted the authors in preparing and revising the manuscripts.

Diana I. Brixner, PhD, RPh
T he rapidly rising costs of health care in the United States, 1 and a corresponding failure to prolong the mean life expectancy, 2 are driving the push to find more efficient and effective ways to provide the best health care for the most value in making decisions in technology reimbursement.
Comparative effectiveness research (CER) is a viable method used to help improve health outcomes and decrease costs in our health care system. CER is a systematic research method that compares new drugs and therapies with established drugs and therapies. This research approach not only determines whether a therapy works, but also how well one therapy compares to another. 3 The goal of CER is to achieve the right therapy for the right patient at the right time. By incorporating CER into early stages of drug development as well as in postmarketing and real-world studies, we can improve health care and de-emphasize inefficient or inferior products and services.
Currently, CER is underutilized. Under the guidance of the U.S. Food and Drug Administration (FDA), most clinical drug trials are designed using a placebo-controlled strategy. This study design fails to compare a new drug product to existing products and may lack important information necessary for formulary decisions. 4,5 CER could be conducted prior to product introduction to understand potential gaps in therapy and help determine where a new agent may fill an unmet need, at reasonable cost. For today's formulary decision makers, the challenge remains of how to incorporate CER into their day-today practice of offering an affordable, balanced drug formulary to their health plan members.
During the October 2010 Academy of Managed Care Pharmacy (AMCP) Educational Conference in St. Louis, Missouri, a payer executive roundtable was convened. This roundtable consisted of 9 representatives from a broad spectrum of health plans as well as government bodies such as Medicaid. The roundtable thought leaders defined CER from their perspectives and described how CER data is currently used in formulary decision making. A comprehensive clinical trial program consisting only of head-to-head comparative trials for liraglutide, a glucagon-like peptide-1 (GLP-1) receptor agonist indicated as an adjunct to diet and exercise to improve glycemic control in adults with type 2 diabetes, 6 was presented to the panel to generate feedback on what was beneficial and what was still lacking in the approach for patients with type 2 diabetes. This example was also used to discuss how CER can be used by formulary decision makers when considering a new drug. The panel then discussed opportunities and hurdles for integrating CER into the health care delivery system. This supplement to the Journal of Managed Care Pharmacy presents 3 articles based on the results of a roundtable discussion and is intended to advance the understanding of CER and its possible future impact on pharmaceutical development and health care in the United States.
T he rising cost of medical care in the United States has triggered an urgent need for a more efficient health care system that achieves greater demonstrated value. While the annual cost of health care was $147 per person in 1960, by 2008 it had escalated to $7,845 per person per year. 1 Ten percent of overall health care spending in the United States in 2008 was for prescription drugs, compared with 31% for hospital expenses and 21% for physician services. The U.S. Department of Health and Human Services (HHS) projects that overall spending for prescription drugs will increase from $234.1 billion in 2008 to $457.8 billion in 2019. 2 The average annual increase in drug spending has been 3%-9% per year since 2005, based on 4 key factors: the increased annual average number of prescriptions per person; drug price inflation; increases in the number of new drug approvals, especially those that address previously unmet medical needs; and the growing market share of expensive specialty drugs. 2 The 2011 report by the Organisation for Economic Co-operation and Development (OECD) shows that the United States, compared with other OECD countries, has the highest spending on health care as a proportion of gross domestic product. 3 From 1970 through 2009, U.S. health care expenditures increased faster than those in "all other high-income OECD countries," with a 5-fold growth rate, even after taking population growth into account.
While there are numerous reason why health care spending in the United States is higher than in other countries, the fact remains that U.S. health costs are some of the highest, and there is an increasing demand to find ways to reduce or stabilize costs while improving health care. As noted by OECD Secretary-General José Ángel Gurría in an August 2009 announcement, "There are opportunities for all countries to improve the performance of their health care system, and making such improvements does not necessarily require higher spending." 4 There is hope that comparative effectiveness research (CER) can be used to help improve the performance of • The United States leads the world in health care spending as a proportion of gross domestic product. CER is being put forth as a possible way to stabilize or reduce health care costs. • Health care reform is a driving force behind the increase in CER, with the U.S. government funding research prioritization, infrastructure, and methodology development.

What is already known about this subject
• The authors (a) describe the progress and development of CER as a health care reform strategy in the United States since 2009; (b) discuss the definition of CER and the types of therapies it can compare, along with possible reasons why a broad interpretation of CER (comparing a drug to a nondrug intervention, for example) may be less relevant to private and public health care payers for whom such broader investments are not considered as part of the budgeting process; and (c) examine the reasons why, from the payer perspective, cost-effectiveness comparison should be part of CER. What Is CER?
The Federal Coordinating Council defined CER as "the conduct and synthesis of systematic research comparing different interventions and strategies to prevent, diagnose, treat and monitor health conditions. The purpose of this research is to inform patients, providers and decision makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances." 12 In other words, CER is any research that helps to identify and monitor the right therapy at the right time for the right patient. The PPACA (2010, Title VI, Subtitle D) also specified what is covered by CER: in addition to medical care, any other strategies or items being used in the treatment, management, and diagnosis of, or prevention of illness or injury in individuals are under the purview of the CER agenda. 13 Hence, CER is intended to encompass any type of intervention while still considering the overall health outcome. Health technology assessment (HTA) is the analysis of evidence coming from randomized controlled trials (RCTs) and, only exceptionally, from real-world studies. CER aims to bring RCTs and real-world evidence together into an integral framework of comparative evidence. Such evidence may extend CER beyond drug therapy to include nondrug clinical and nonclinical interventions (e.g., the decision to invest in accident prevention or the reduction of environmental risk). This interpretation may be less relevant to private and public health care payers for whom such broader investments are not considered as part of the budgeting process, but may be of interest to employers, since interventions to improve workplace safety and encourage employee wellness may directly benefit them.
PCORI has introduced another term to the CER lexicon: patient-centered outcomes research (PCOR). At first glance, this may seem very similar to CER, but it differs in emphasis. The PCORI Methodology Committee defines PCOR as research that "helps people make informed health care decisions and allows their voice to be heard in assessing the value of health care options. This research answers patient-focused questions such as: (a) "Given my personal characteristics, conditions and preferences, what should I expect will happen to me?" (b) "What are my options and what are the benefits and harms of those options?" (c) "What can I do to improve the outcomes that are most important to me?" (d) "How can the health care system improve my chances of achieving the outcomes I prefer?" 14 PCOR emphasizes patient involvement throughout the research process, so that the concerns, perspectives and values of patients are reflected in the methodology and results. This envisions the patient and their health care provider engaging in a collaborative decision-making process as the customers for whom PCOR information is developed. Even though the information generated by PCORI will likely be useful to payers and policy makers, they are not the focus. Patient-centered information and decision support tools will hopefully improve the U.S. health care system, ideally at a lower or at least more stable expenditure. CER compares how various effective medical treatments improve health outcomes, with the objective of eliminating ineffective services or giving preference to more effective services. This argument has been used to justify the role of CER in U.S. health care reform.

CER as a Health Care Reform Strategy in the United States
Recognizing that changes needed to be made, the U.S. government is making extensive funds for CER available, mostly favoring infrastructure and methods development. The American Recovery and Reinvestment Act (ARRA) of 2009 led to the establishment of the Federal Coordinating Council for CER. The council was formed to foster coordination of CER and to recommend priorities for funding. 5 6,7 The new law focuses on 4 main areas: controlling health care costs and identifying funding and savings opportunities; 2 expanding health care coverage for a significantly larger number of U.S. citizens, including access to care for pre-existing conditions at affordable premiums and out-of-pocket costs; 2 improving health care delivery systems; 2 and establishing sustainability over the long term.
To reach these goals, payment reform will be required, and emphasis on health care must shift toward quality, efficiency, wellness and prevention. With the new law, considerable net reductions in federal deficits are expected over the next 10 years, resulting from several new taxes, fees on health-related industries and cuts in government spending on health care programs such as Medicare Advantage. 8 As CER develops there may be enough evidence to consider lifting of current regulations that limit price negotiations by the government.
The PPACA initiated and funded the creation of the Patient-Centered Outcomes Research Institute (PCORI), a public/ private entity, to coordinate CER and recommend priorities. PCORI replaced the council and is charged with identifying priorities, establishing an agenda, and carrying out primary CER and systematic reviews of existing and future studies. 9, 10 The research conducted for PCORI will be peer-reviewed and made available to the medical community and the general public. The Agency for Healthcare Research and Quality (AHRQ), a federal agency within the HHS charged with supporting research that helps people make more informed decisions and improves the quality of health care services, 11 can take proactive steps to disseminate the findings of PCORI to physicians, health care providers, patients, insurance providers, and health care technology vendors. 10 Can CER Be an Effective Tool for Change in the Development and Assessment of New Drugs and Technologies? S8 Supplement to Journal of Managed Care Pharmacy JMCP June 2012 Vol. 18, No. 5 www.amcp.org selected social interventions and combined clinical and social interventions) to prevent obesity, hypertension, diabetes and heart disease in at-risk populations such as the urban poor and American Indians. The first quartile also includes comparing the effectiveness of dissemination and translation techniques to facilitate the use of CER by patients, clinicians, payers, and others. 16 While both of these seem highly relevant to the health plan, nondrug approaches, such as the effectiveness of yoga in depression or the co-location of psychologists and physicians in children aged 0-3 years, may not be directly relevant, since most of them are not covered services; however, payers may benefit indirectly if effective use of these techniques reduces the need for medication or other medical treatment.
Of the priorities, 6 include diabetes as a key research area, and 3 include obesity. Both diabetes and obesity also have been identified by the OECD as priorities for improvement in health care in the United States. 16

How Common Is CER Today?
With the increasing emphasis on CER, a growing number of studies of this type may be expected. Hochman 17 In these studies, either a specific medication or a class of medications was compared with either another medication or a nonpharmacologic therapy (active treatment) or either a placebo or no therapy (an inactive control). 17 Randomized trials, case-control studies, cohort studies, and meta-analyses were included, while systematic reviews and modeling studies were excluded. 17 Classification as comparative effectiveness (CE) was made if the study involved existing (rather than novel) medications or compared active therapies (active-comparator studies), and non-CE studies were defined as involving novel therapies or comparison to an inactive control such as a placebo (inactive-comparator studies). 17 Noninferiority RCT studies were categorized as novel therapy because they are generally done to obtain U.S. Food and Drug Administration (FDA) approval. 17 From a total of 328 studies reviewed, 104 (32%) were classified as CE studies and 224 (68%) as non-CE studies (187 had only an inactive comparator, 81 included non-FDA-approved medications, and 23 were noninferiority trials). 17 Of the 104 CE studies, 45 compared 2 or more medications (43%, 95% confidence interval [CI] = 34%-53%), 11 compared medications with nonpharmacologic interventions (11%, 95% CI = 5%-18%), 32 compared different pharmacologic strategies (31%, 95% CI = 22%-41%), and 16 compared different medication doses, durations or frequencies of treatment, or different medication formulations (15%, 95% CI = 9%-24%). 17 Ninety of the 104 CE studies (87%, 95% CI = 78%-92%) both the outcomes and efficiency of the health care process in the United States. This is important, since most of the future federal funding for CER will likely be directed toward PCOR projects. Kindig et al. (2010) describe CER as a holistic approach of comparing the overall effectiveness and performance of health interventions on target outcomes, with the goal of enabling the stakeholder to make more informed choices. 13 However, it has not yet been defined how far-reaching the actual use of such a CER approach will be and whether CER will focus only on direct health care intervention or will also be used to test preferences for investments in other areas (such as environment, traffic, or education). Using the example of diabetes, CER could be assessed across a broad range of interventions, such as medications, school educational programs, anti-obesity surgery, and preventative behavioral therapy, that all have an impact on diabetes-related health risks. 13 While a study comparing several pharmaceutical interventions seems logical and feasible, a comparison between pharmaceutical interventions and educational school programs, for example, would be extremely complex, time-consuming, and not suitable for helping to make a decision on reimbursement at the time of a new product launch for a health insurance plan. Furthermore, the current budget and incentive structures in health plans do not necessarily support comparison across budget silos. Therefore, for a formulary decision maker working within the limits of the pharmacy budget, any comparison outside of the pharmacy budget might be less relevant than comparisons within the budget.

The Institute of Medicine's Research Priorities
As a part of the ARRA, Congress asked the Institute of Medicine (IOM) to prioritize which research questions should be addressed by CER and funded by ARRA. In its 2009 report, the authoring committee developed a priority list of research topics derived from input from a broad array of stakeholders, including policy makers, academics, researchers, the health care industry, physicians and other health care providers, students, and others in the public and private sectors interested in health policy as well as patients, families, and consumers. 15 For portfolio criteria, the prioritization process examined research areas, populations to be studied, interventions, and proposed methodologies. Condition-level criteria included prevalence, mortality, morbidity, cost, and variability. Priority topic-level criteria included appropriateness of topic for CER, information gaps and duplication, and gaps in translation. 15 From 2,606 nominated topics, 1,268 were voted on in a 5-step voting process, and a final list of 100 top priorities, categorized into 4 priority quartiles, was chosen. 16 These priorities span a broad range of diseases, interventions and investments. The first quartile of priorities includes comparing the effectiveness of various strategies (such as clinical interventions, studies of new drugs against clinically relevant active comparators. The findings of these head-to-head comparisons are used by health technology assessment agencies and provided to government authorities in these respective countries to help make treatment recommendations or to set pricing for public insurance programs. 19,20 However, despite the greater requests for this type of data in other countries, there does not appear to be an increase of CER evidence. In October 2010, Chokshi et al. published another analysis of the design, results, and ultimate impact of past CER studies on practice. They identified 3 areas of special concern: choice of comparison treatments; study time frame; and "external validity"-that is, the extent to which the study's results can be reliably applied to the population as a whole. 20 However, it should be noted that compared with all other medical or clinical technologies (medical devices, procedures, diagnostics, care pathways, etc.), drugs come to the market with the most advanced evidence base. 20 Looking forward, it can be expected that CER data satisfying the need for clinical relevance will be required at the launch of a new product. The considerable investment in CER and the formation of independent institutions for CER will eventually lead to the establishment of quality criteria and guidance, as has been seen with the guidance for registration trials of pharmaceutical products.
Increased requests for CER data in decision making do not guarantee a fast process in achieving such evidence because many different stakeholders are involved and their viewpoints need to be considered to provide broad acceptance and, thus, effectiveness of CER. An early and potentially promising effort to address stakeholder concerns is the Academy of Managed Care Pharmacy/National Pharmaceutical Council/ International Society for Pharmacoeconomics and Outcomes Research (AMCP/NPC/ISPOR) Comparative Effectiveness Research Collaborative Initiative (CER-CI). The goal of CER-CI is to establish a consensus-based set of principles and tools to guide the design and evaluation of nonexperimental studies, including prospective and retrospective observational designs, so that the knowledge gained from these studies can be applied to improve patient health outcomes. 21 Will CER Reduce or Stabilize Health Care Costs? CER compares how effective various medical treatments are at improving health outcomes. But does "effective" include the concept of value for money?
From the payer perspective, cost-effectiveness comparison should be part of CER. By comparing the health outcomes of one diabetes drug with another, it is possible to determine the comparative effectiveness. By comparing the health-related quality of life (HRQoL) of the patients treated with one drug with the HRQoL of patients treated with another, it is possible to examine value from the humanistic perspective. Comparing were funded jointly or exclusively by noncommercial entities. 17 The authors addressed the fact that "commercial entities presumably devote much of their research to the development of novel therapies and to funding inactive-comparator studies aimed at expanding indications for their products." 17 Although analyses are critical for promoting efficient and effective health care, only 1% of the non-CE studies and 2% of the CE studies included cost-effectiveness analysis (which could have been due to editorial preferences of the clinically oriented journals). 17 Overall, Hochman and McCormick found that about two-thirds of research involving medications that was published in high-impact medical journals over 16 months in 2008-2009 does not fulfill the criteria of CER and that much of the study design is driven by FDA requirements rather than the need for evidence allowing the selection of the most effective therapy. 17 Bourgeois et al. (2012) studied the prevalence of clinical trials addressing 15 priority research topics from the IOM list of priorities. Trials conducted in the United States between 2007 and 2010 and registered on ClinicalTrials.gov were included. The authors reported the prevalence of CER studies, the nature of comparators used, funding and how these factors impacted study results. 18 The authors found 1,035 trials that met their selection criteria. An additional 3,384 studies were excluded because they were not conducted in the United States, and 2,124 were excluded as not addressing 1 of the selected topics of interest. Among the different types of interventions studied, drug trials had the largest percentage of studies that met the authors' definition of CER (37.2%), followed by behavior change studies (28.6%), procedural interventions (15.6%) and devices (13.8%). 18 These results align with those reported by Hochman and McCormick, 17 suggesting that their observations are generalizable beyond those 6 journals. In both cases, the majority of studies did not qualify as CER, and the prevalence of CER studies was even lower for nonpharmacologic interventions. Many of the studies found on ClinicalTrials.gov were placebo controlled, and some did not have a control group at all. 18 The low prevalence of CER studies found by Bourgeois et al. is noteworthy, given that the topics were drawn from the IOM priority list. Furthermore, a large number of studies (3,384) were excluded because they were not conducted in the United States, and without further explanation of the nature of these studies, their data may be considered CER and may be used as such in U.S. formulary decisions. Clearly, there is a great deal of work yet to be done in developing the body of CER evidence that is needed to facilitate informed patient-centered treatment decisions.
To obtain FDA approval, it is possible to demonstrate efficacy of a new product in studies that compare the new product with a placebo. Other countries (government agencies in Canada, the United Kingdom, Japan, and Australia) require or decreased morbidity. In addition, the cost-effectiveness of a technology strongly depends on the efficiency of using it and may differ between individual sites of care (hospitals, practices, care teams, etc.). For example, the cost of a magnetic resonance imaging (MRI) machine used across several hospitals versus the cost of 1 machine purchased for each hospital would significantly decrease the cost per scan based on efficiency of use, not the cost of the machine itself. If cost became a part of CER, the results of such research could contribute to streamlining health care expenditures. To truly decrease cost and improve quality, cost-effectiveness would have to be integrated into CER with the objective of prioritizing efficient therapies in the real-world health care system. 25,26 A final consideration for CER and cost is that if the methods and output of CER improve, the resulting cost-effectiveness ratios will also be more meaningful to the payer.
CER should ultimately, therefore, be a useful tool to help patients, providers, and decision makers provide the optimal and most cost-effective interventions.
the overall cost consequences of one treatment with another, it is possible to determine the economic perspective of effectiveness. Hence, theoretically, CER could help to answer the question of which treatment delivers the best value per unit of investment from clinical, humanistic, and economic perspectives.
The current framework of CER in the United States makes very limited use of cost data, and the PPACA and the Centers for Medicare & Medicaid Services (CMS) specifically prohibit use of dollars per quality-adjusted life year as a threshold to determine which treatments are cost-effective or recommended. For payers, however, formulary decisions are business decisions as well as clinical. Payers need to offer a comprehensive and reasonable formulary for their membership within the constraints of a given budget. Therefore, cost is an important part of formulary decision making. Payers need to be aware of budget changes when a new drug is integrated into the formulary and whether the expected improved clinical outcome justifies such a change.
In the face of increasing numbers of people becoming eligible for Medicare Part D coverage, options for cost savings will have to be identified and utilized. While it is currently prohibited under the noninterference provision (Medicare Prescription Drug, Improvement, and Modernization Act [MMA] of 2003) for Medicare to negotiate on prices of prescription drugs, overall drug cost could be reduced by starting to negotiate the price. 22 In 2007, rules such as the safe-harbor guidelines established by CMS and MMA's requirement that Part D plans cover at least 2 drugs per class were put into effect. 23 In addition, CMS required at least 1 drug in each subclass and gave special protections to 6 classes of drugs, requiring that "all or substantially all drugs" in those classes be included in the formularies. Therefore, Part D drug-price negotiations over anticonvulsants, antidepressants, antineoplastics, antipsychotics, antiretrovirals and immunosuppressants were effectively eliminated. Part D plans regularly exclude some drugs as part of the normal commercial formulary process with other classes of drugs. 22 One study found that these 6 protected classes accounted for 16.8% to 33.2% of Part D drug costs. 24 Reversing the rule would decrease prices in these classes by 9% to 11%, for a projected Part D savings of $511 million per year. 22,24 Government regulations can thus create challenges for health plans in managing budgets in the absence of CER.

■■ Conclusions
The impact of CER on reduced drug expenditures may be limited. In fact, CER may lead to higher costs in some cases because it may support a preference for solutions that are clinically more effective and more expensive.
When assessing the impact of cost, it is important to look beyond drug costs to the overall impact on treatment costs, including cost offsets that may occur through improved health

DISCLOSuRES
This supplement was sponsored by Novo Nordisk Inc. and prepared by the Millcreek Outcomes Group (Salt Lake City, Utah). Diana Brixner reports receiving payment as a manager of Millcreek Outcomes Group to submit this JMCP supplement and also reports being a research investigator on outcomes studies funded by Novo Nordisk to the University of Utah. She reports receipt of payment for educational programs from Novartis and research grants from Abbott, Bristol-Myers Squibb, and Novartis. She is a member of the International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Vision 2020 task force regarding the appropriate communication of outcomes study results to stakeholders and serves on the Academy of Managed Care Pharmacy (AMCP) eDossier Advisory Committee.
John Watkins reports no compensation for participation in the roundtable discussion or for writing and revising portions of this supplement and reports no financial or other potential conflicts of interest related to the subject of this supplement. He is a member of the AMCP Format Executive Committee including oversight of the AMCP Format for Formulary Submissions, and he is a member of the Research Committee of the Foundation for Managed Care Pharmacy (FMCP). He is a member of the ISPOR HTA Council, North America.  OBJECTIVES: To examine how CER can help strengthen a new drug's entry into the market and integration into formularies, and how ongoing CER might be valuable as a drug is implemented in the real world.
SUMMARY: A roundtable of 9 representatives from health plans, including formulary decision makers, evaluated how CER in phase 3 development of a new drug can add to the drug's strength of evidence, helping decision makers understand how and where to integrate that drug into a formulary. The round table participants viewed, as a case study, the development of liraglutide, a glucagon-like peptide-1 (GLP-1) receptor agonist for adults with type 2 diabetes that was approved by the FDA in January 2010. With this drug, CER was incorporated into an extensive type 2 diabetes clinical development program, comparing how the drug worked in comparison with other established therapies. Although there are many antidiabetic drugs available for use, patients with type 2 diabetes often need additional agents. The FDA approved liraglutide with the conclusion that benefits of the drug outweighed potential risks but noted the association with pancreatitis in humans and animal data that showed rare medullary thyroid cancer associated with liraglutide. Roundtable participants agreed that while prelaunch CER can be valuable, ongoing real-world research is also important for confirming expected results, identifying additional uses and indications and managing risks. The participants also suggested opportunities for additional CER studies and made recommendations for manufacturers. CONCLUSIONS: Roundtable thought leaders agreed that well-planned trial designs incorporating CER result in high-quality evidence that may provide sufficient data to support adoption of a new therapy onto the formulary. When more real-world data become available and confirm the phase 3 clinical trial results, decision makers may be able to use the results to change the drug's position and either lessen or extend its use.
T reatment of type 2 diabetes mellitus includes several classes of drugs with multiple agents in each class that are generally safe and effective, yet the proportion of patients at goal is suboptimal. There still remains an unmet need for drugs to treat type 2 diabetes with better overall riskbenefit profiles, especially in light of the weight gain and hypoglycemia frequently seen with some commonly used agents.
As and much of the study design is driven by U.S. Food and Drug Administration (FDA) requirements rather than the need for evidence allowing the selection of the most effective therapy. 3 At the October 2010 Academy of Managed Care Pharmacy (AMCP) meeting, a payer executive roundtable was convened, comprising 9 representatives from a broad spectrum of health plan types, including Medicaid, those providing Medicare • For FDA registration in the United States, it is common to demonstrate efficacy of a new product by comparing the new product with a placebo rather than using head-to-head comparisons with clinically relevant alternatives. Without a requirement for clinical trials using active comparators, many manufacturers opt to avoid the extra expense and risk of unfavorable results in these types of studies. • Comparative effectiveness research (CER) may help reduce barriers to a new drug's adoption and integration into formularies. The safety and efficacy of liraglutide were investigated in an extensive clinical development program in adults with type 2 diabetes and showed that the benefits from the drug outweighed the risks.

What is already known about this subject
• It was a consensus of the 9 roundtable participants that wellplanned trial designs that result in a broad range of relevant data, such as that presented for liraglutide, can provide a good basis for the decision maker. • Even if a drug comes to market with a solid set of CER data, roundtable participants felt that additional post-launch comparative data, such as studies showing how the drug might benefit subpopulations or studies pursuing alternative combination therapies or other comparators, might even strengthen the case for the drug. • The roundtable participants recommended that manufacturers should translate CER and clinical data into tools that will enable decision makers to create effective benefit designs and make good formulary decisions; that manufacturers be proactive in communicating with all stakeholders to keep them informed of ongoing data (especially with safety issues); and that manufacturers should pursue opportunities to validate data from randomized controlled trials in the real world. drug, especially an innovative drug, may provide the compelling data necessary to persuade decision makers. For FDA registration in the United States, it is common to demonstrate efficacy of a new product by comparing the new product with a placebo using a pivotal phase 3 trial. 5,6 Headto-head comparisons with clinically relevant alternatives are not done in most cases. Without a requirement for CER-based studies, manufacturers may opt to avoid the extra expense and risk of these types of studies. However, some manufacturers are beginning to see value in such studies, especially if the data shows formulary decision makers that the new drug presents a clear advantage in at least some populations.
Safety and Efficacy of Liraglutide. The safety and efficacy of liraglutide were investigated in one of the most extensive clinical development programs ever conducted in type 2 diabetes. [7][8][9][10][11][12][13] The phase 3 clinical trials for the drug included 4,445 patients and comprised 5 randomized, placebo-controlled, double-blind trials plus 1 open-label trial (liraglutide vs. exenatide). 14 There was 1 monotherapy study of liraglutide versus the sulfonylurea glimepiride of 52-week duration and 5 combination-therapy studies of 26-week duration. 7-12 All 6 trials were randomized, multicenter, parallel-group design, and all trials were doubleblinded except the metformin plus glimepiride add-on trial, which included open-label insulin glargine and the metformin and/or glimepiride add-on open-label trial versus exenatide. 11 In addition, the monotherapy trial and the combination therapy trials with metformin or glimepiride add-ons were double-dummy design. The program compared once-daily liraglutide injections with 3 widely used diabetes therapiesglimepiride, rosiglitazone, and insulin glargine-and 1 study also directly compared liraglutide with exenatide. [7][8][9][10][11][12] In addition, one 26-week, randomized, parallel-group, open-label head-to-head trial with sitagliptin was performed. 13 The studies were designed to investigate the efficacy and safety of liraglutide at each step in the treatment continuum, from monotherapy to combination with 2 oral antidiabetic drugs in adults with type 2 diabetes. All but 1 of the studies had an active comparator. A substantial effect on A1c reduction, combined with significant weight reduction and low risk for hypoglycemia, was consistently shown.
Zinman et al. (2012) conducted a meta-analysis of the Liraglutide Effect and Action in Diabetes (LEAD) trials using a composite endpoint (A1c < 7% with no weight gain and no hypoglycemia) to compare liraglutide to other agents from standard classes of antidiabetic therapy. 15 The composite endpoint at 26 weeks was attained in 40% of the liraglutide 1.8 milligram (mg) group, 32% of the liraglutide 1.2 mg group and 6%-25% of comparators (6% rosiglitazone, 8% glimepiride, 8% placebo, 11% sitagliptin, 15% glargine, and 25% exenatide). This is a clinically relevant endpoint, given that many antidiabetic therapies increase body weight and risk of Part D drug formularies, and drug formularies as defined by contracts with health plans or employers. This roundtable of thought leaders was invited to examine the case study of liraglutide [rDNA origin] injection (Victoza), a glucagon-like peptide-1 (GLP-1) receptor agonist approved by the FDA on January 25, 2010, with the indication as an adjunct to diet and exercise to improve glycemic control in adults with type 2 diabetes. 4 The label lists the following as "important limitations of use:" • "Not recommended as first-line therapy for patients inadequately controlled on diet and exercise. • Has not been studied sufficiently in patients with a history of pancreatitis. Use caution. • Not for treatment of type 1 diabetes mellitus or diabetic ketoacidosis. • Has not been studied in combination with prandial insulin." 4 Subsequent to the conduct of the executive roundtable, the warnings and precautions section of the label was updated in May 2011 regarding renal impairment, that "has been reported postmarketing, usually in association with nausea, vomiting, diarrhea, or dehydration which may sometimes require hemodialysis. Use caution when initiating or escalating doses of Victoza in patients with renal impairment." 4 After viewing the evidence from 6 clinical trials, roundtable participants offered their perspectives on the strength of the CER data reported in this case and how it would affect their decisions about the drug. In the phase 3 clinical trial program, the randomized portions of all phase 3a trials (LEAD 1-5) were double-blind (except the insulin glargine arm of LEAD-5), placebo-controlled (except LEAD-3), active-comparator (except LEAD-4), parallel-group studies. In addition, participants offered ideas for how continuing real-world CER might be valuable for future decisions about a drug post-launch.

Incorporating CER into Clinical Trials: Beyond the Norm
Launching a new drug into a therapy area that already contains existing effective treatments can be challenging. Without comparative effectiveness data that demonstrates how the new drug works versus those existing treatments, formulary decision makers may not see any compelling reasons to help them decide where or even whether to integrate the new drug into their formularies. The drug development process involves 4 general phases of human trials. Phase 1 trials administer the drug to a small number of healthy volunteers to study activity and safety. In phase 2 the focus is on effectiveness, and a few hundred volunteers with the disease of interest are treated. Phase 3 trials generally involve several thousand patients with the disease of interest, are the primary trials used for registration, and they evaluate both efficacy and safety. Phase 4 trials are done after the FDA has approved the drug for marketing and look at long-term efficacy and safety in real-world patients. Incorporating CER into the phase 3 development of a new

Need to Confirm RCT Findings in the Real World
According to roundtable participants, when making a decision regarding whether and how to add a drug to the formulary, efficacy, safety, and cost are the most important considerations. If a new product comes to the market with strong differentiating data, including head-to-head trials, those data will improve access to the formulary. However, the concern remains that results in clinical trials do not translate into meaningful differences in the real world. It is critical that post-market data be made available as soon as possible after launch.
Decision makers want to be certain that what happened in the RCTs is also happening in their plans' memberships. Roundtable participants suggested that the manufacturer should therefore use opportunities to validate RCT results in real-world studies. The advantages and disadvantages of clinical trial data versus real-world data have been contrasted by a few recent publications on a general basis or, specifically, for diabetes. 20 As noted by Schneeweiss et al. (2011), the role of efficacy data from clinical trials and comparative effectiveness data from observational studies changes over a drug's life cycle, with RCT and placebo trials more important in the early phases, and CER becoming more important late in phase 3 and particularly in phase 4 studies. 21 As Schneeweiss et al. point out, there are a number of methodological challenges in using post-marketing observational data for CER. These include bias such as confounding at the physician and patient level, sparse data during early marketing, and disease and data issues. Disease issues include the increased likelihood that first-in-class medications will be used in patients with more severe disease, and the long lag-time between product launch and the development of the disease in patients for a drug that is designed to prevent that disease. A number of data issues also are important and include the lag time for data to be available, lack of necessary granularity of information, and lack of data on suitable outcome measures. 21 A major initial challenge is selective prescribing (or channeling) when a new drug enters the market. Patients who are well controlled and tolerating existing agents are much less likely to be switched to a new agent than patients who are not. Therefore, these early patients are not likely to be representative of patients who will end up taking the drug later. 10 For example, when exenatide came to market, Segal et al. (2007) showed that early users (those who began taking exenatide in its first 3 months) had higher A1c levels and were more likely to have taken insulin and other oral antidiabetic drugs than patients who began exenatide 6 to 7 months after introduction. 22 Sparse data available close to launch causes several problems, including a small sample size and difficulty in controlling for bias due to small numbers.
hypoglycemia. The composite endpoint also aligns nicely with CER and is a reasonable way to compare antidiabetic therapies, yielding some indication of risk-benefit.
Because liraglutide caused dose-dependent and treatmentduration-dependent thyroid C-cell tumors at clinically relevant exposures in both genders of rats and mice, the liraglutide label has a boxed warning. It is not known whether liraglutide causes these tumors in humans, but liraglutide is contraindicated in patients with a personal or family history of medullary thyroid cancer and in patients with multiple endocrine neoplasia syndrome type 2. 4 No cases of medullary thyroid cancer were observed in patients treated with liraglutide in the 6 phase 3 clinical trials of 26-to 52-weeks duration.
Liraglutide also has a Risk Evaluation and Mitigation Strategies (REMS) program including an FDA safety warning on June 13, 2011, regarding the risk of thyroid cancer and pancreatitis with the recommendation to "observe patients carefully for signs and symptoms of pancreatitis (including persistent severe abdominal pain, sometimes radiating to the back, and which may or may not be accompanied by vomiting)" after use of liraglutide and after dose increases. 16 Additional interpretation of CER data for antidiabetic drugs in general and the GLP-1 agonists in particular, including liraglutide, were presented in the Agency for Healthcare Research and Quality (AHRQ) comparative effectiveness review on antidiabetic agents, released in March 2011. 17 This CER report included 5 studies with liraglutide as a comparator. For A1c reduction, 3 randomized controlled trials (RCTs) found "conflicting results" in comparison of sulfonylureas with liraglutide; 1 small RCT found no difference in A1c reduction, and the 2 larger RCTs favored liraglutide, but 1 of the 2 larger RCTs "underdosed the sulfonylurea [glibenclamide] arm." 17 The 3 RCTs were not combined in a meta-analysis due to dosing differences within and between studies. The small phase 2 RCT that found no difference in A1c reduction compared liraglutide (0.045 mg, 0.225 mg, 0.45 mg, 0.60 mg, or 0.75 mg) with glimepiride (1-4 mg); the changes in A1c were comparable with glimepiride at the 2 highest dosage levels of liraglutide. 18 The effect on glycemic control with liraglutide was superior to glimepiride in the other 2 RCTs. 9, 19 The larger RCT that "underdosed" the sulfonylurea was conducted in Japan and compared glibenclamide 2.5 mg per day with liraglutide at 0.9 mg per day. 19 Although 3 studies showed weight loss with liraglutide and weight gain with sulfonylurea, both relative to baseline body weight, the AHRQ CER found a 2-fold higher rate of adverse gastrointestinal events in liraglutide patients (50% vs. 26% for the sulfonylurea glimepiride) and rates of nausea, vomiting, and diarrhea approximately 2-3 times higher for liraglutide versus glimepiride, consistent with the known adverse effects profile of GLP-1 receptor agonists. toward the optimal therapy. A higher degree of personalization created by using an algorithm that predicts improved chances for response to therapy would improve health outcomes and, therefore, the value of the therapy. If differences are found in behaviors among the patients responding or not responding to therapy, such findings could be used to target behavioral programs and support patients at risk for inferior outcomes.

Opportunities for Additional Comparative Data
Even if a drug comes to market with a solid set of CER data, some roundtable participants felt that additional comparative data might even strengthen the case for the drug. For example, additional studies might show whether a new drug might benefit subpopulations, such as patients with previously inadequately controlled diabetes, or additional studies could pursue alternative combination therapies or other comparators. For example, participants suggested that studying the use of liraglutide plus insulin presented an interesting opportunity and relevance for use of the product outside of available clinical trial data. Rosenstock et al. (2011) recently presented data in a poster abstract that described evaluation of the addition of insulin detemir to patients on metformin and liraglutide 1.8 mg daily who had not achieved the A1c goal of < 7%. An additional 0.5% decrease in A1c was achieved, and 43% of the patients not at goal reached goal. Hypoglycemia rates were low and no major hypoglycemia events occurred. 25 Such data validated across large health plan populations would be useful.
The expectations of formulary decision makers frequently differ from those of regulators regarding the endpoints used in clinical studies and observational studies. Formulary decision makers note that A1c is still only a surrogate marker for adults with type 2 diabetes, and significant A1c reduction in a clinical study does not necessarily equate to satisfactory long-term control of diabetes and related negative cardiovascular outcomes, despite strong evidence that A1c reduction is associated with microvascular risk reduction.
The concept of disease modification might open a new, very attractive set of therapeutic opportunities. Studies on disease progression, disease reversion, beta-cell function and insulin sensitivity would be of major interest for all stakeholders in the field of type 2 diabetes, including patients with pre-diabetes, providers, policymakers, and payers.
Another interesting opportunity recognized by the thought leaders addressed whether the accrual of outcomes (such as cardiovascular improvement in type 2 diabetes) could be shown to outweigh the increased drug costs. This is particularly an issue since the direct drug cost of GLP-1 receptor agonists such as liraglutide is much higher than the oral agents used to treat type 2 diabetes. Such data would definitely support price and reimbursement decisions. The LEADER (Liraglutide Effect Schneeweiss et al. propose several solutions to these problems. 21 Although most pre-marketing clinical trials use placebo controls, more are being done with active comparators as well, as was the case with liraglutide and the LEAD trial program. Simulations using simulation software may also be of value. Some health plans, such as Kaiser Permanente, use software such as Archimedes in making coverage decisions when data are limited. 23 The roundtable participants came to the conclusion that observational studies using large health care databases to assess treatment effectiveness in patients encountered in day-to-day clinical practice can complement RCTs that are conducted prospectively in multiple sites with large patient numbers. Use of outcomes from observational studies, which include larger and more diverse patient populations with common comorbidities and longer follow-up periods, can expand upon RCT data. Finally, identification of clinically important differences between therapeutic options and data on long-term drug effectiveness and safety can result from well-designed observational studies. 14 Although real-world data are considered to be important, initial formulary decisions for new products have to be made on the basis of the clinical trial results before additional real-world data become available. Likewise, economic data are important, but the majority of such data come from observations of realworld resource utilization and thus are not available at the time formulary decisions need to be made. For example, the incidence and importance of a drug's safety concerns, such as the boxed liraglutide label warning of thyroid cancer and the REMS program and recommendation for clinicians to monitor liraglutide initiation and dose increase for symptoms of pancreatitis, can only be judged after longer use in larger patient populations and will also be matters for real-world observation. Approaches to better understanding of patient subpopulations are usually observed with interest by decision makers. The elderly are an important subgroup in diabetes, and data on liraglutide in this subpopulation are available. Data from a pooled analysis of 6 randomized, placebo-controlled multinational trials include data on 797 patients aged 65 years or older, or about 20% of the patient population. Between 57% and 67% of patients on liraglutide 1.2 mg and 1.8 mg per day reached their A1c goals with no difference in response between those less than 65 and older patients. There also was no difference in the amount of weight lost or in the nature and frequency of adverse events between the age groups. 24 If an algorithm for identifying patients with a high chance of good response were developed by this approach and could be used to triage patients based on their potential responses to the existing therapies, the approach to future therapy could be changed fundamentally. Using such an algorithm could drive patients and Action in Diabetes: Evaluation of cardiovascular outcome Results, NCT01179048) trial will address this with the primary composite outcomes of cardiovascular death, nonfatal myocardial infarction, and nonfatal stroke. 26 With any drug, information on long-term use is desired. Whereas an RCT may provide such data, the results are often not available for 5 to 10 years after the launch of a drug. Evaluation of long-term outcomes using real-world data can often provide results much sooner. Not only do such data show long-term outcome trends, but they can be used to resolve safety issues such as the boxed warning for liraglutide and the additional label warning regarding renal impairment added 14 months after initial FDA approval. The decision makers want the option of proactively managing risks instead of having to react to them. Therefore, ongoing information concerning the safety of a drug is essential. However, it is clear that potential risks can be confirmed or countered only after many patients have been treated with the drug.
Recently Ahmann (2011) has summarized the data on new classes of antidiabetic drugs (incretin-based agents including dipeptidyl peptidase-4 [DPP-4] inhibitors and GLP-1 receptor agonists) and application of CER to drugs used to treat type 2 diabetes mellitus. Ahmann determined that ongoing CER can help guide individualized, patient-centered treatment of patients with type 2 diabetes mellitus and potentially reduce trial-and-error therapy. He also said that CER can help identify which treatments would be preferred for patient subgroups and in patients at high risk of developing specific adverse events. 27 In the area of type 2 diabetes, for example, patients with renal impairment, cardiovascular disease and increased age may have been excluded from the clinical trial program, but they are prevalent patient types. Data documenting the positive impact of a therapy on health and cost outcomes in patients with multiple morbidities can support the drug's placement in the formulary and provide useful information to those patient groups and their health care providers. As Ahmann suggests, the best way to make CER useful in diabetes is to further encourage head-to-head clinical and observational studies to generate high-quality evidence that can be incorporated into CER evaluations. 27

■■ Conclusions
Liraglutide came to the market with unusually robust data demonstrating clinical comparative efficacy and safety. The thought leaders involved in the roundtable discussion agreed that well-planned trial designs that result in a broad range of relevant data, such as that presented for liraglutide, can provide a good basis for the decision maker. There are frequently challenges due to the lag time between trial design, completion and market availability. New competitors into the marketplace and Authors changes in treatment guidelines sometimes make it difficult to obtain relevant CER data.
Based on the roundtable discussion, the thought leaders had several recommendations for any manufacturer implementing CER. First, manufacturers should translate clinical and CER data into tools that will enable decision makers to create effective benefit designs and make good formulary decisions. Second, manufacturers should be proactive in communication with all stakeholders, including decision makers, keeping them informed of further developments, information, or new evidence gathered, especially with safety issues. Manufacturers should pursue opportunities to validate data from the RCT program in the real world. Manufacturers should pursue methods to better identify patients who will best respond to their products. And finally, the roundtable participants recommend that manufacturers perform CER along the lines envisioned by the federal government.

DISCLOSuRES
This JMCP supplement was sponsored by Novo Nordisk Inc. and prepared by the Millcreek Outcomes Group (Salt Lake City, Utah). Gary Oderda has served as a consultant for Pacira and Novo Nordisk and as a speaker for Jansen; he has done funded research for King (Pfizer), Novartis, and Takeda. S. Monet Sifford-Wilson is a full-time employee of Novo Nordisk Inc. and shareholder of Novo Nordisk Inc. SUMMARY: While decision makers from different types of organizations, such as pharmacy benefit management (PBM) companies and MCOs, may have varying definitions and expectations of CER, most thought leaders from a roundtable of stakeholders, including formulary decision makers, see value in CER's ability to enhance their formulary decision making. Formulary decision makers may be able to use CER to better inform their coverage decisions in areas such as benefit design, contracting, conditional reimbursement, pay for performance, and other alternative pricing arrangements. Real-world CER will require improvement in the health information technology infrastructure to better capture value-related information. The federal government is viewed as a key driver and funding source behind CER, especially for infrastructure and methods development, while industry will adapt the clinical development and create increasing CER evidence. CER then needs to be applied to determining value (or costefficacy).
CONCLUSIONS: It is expected that CER will continue to grow as a valuable component of formulary decision making. Future integration of CER into formulary decision making will require federal government and academic leadership, improvements in the health information technology infrastructure, ongoing funding, and improved and more consistent methodologies. T o increase the understanding of what comparative effectiveness research (CER) means to formulary decision makers today and how the results of CER are expected to be used in the assessment of new technologies in reimbursement and benefit design, a moderated roundtable in formulary decision making for some executives from payers was convened in a session held coincident with the October 2010 Academy of Managed Care Pharmacy (AMCP) Educational Conference. The roundtable consisted of 9 representatives from a broad spectrum of health plan types as well as government bodies. This roundtable of thought leaders was invited to discuss the topic from their respective payer perspectives.
"Is the drug better?" and "Which drug produces the best outcome?" are key questions for the drug formulary decision maker that relate to efficacy and safety. Both questions require a direct comparison of the alternatives. The motivation of using CER to inform decisions may be driven by the need for utilization management: "How do we get the most out of the • Liraglutide has a direct drug cost of approximately $300-$450 per month of therapy compared with less than $10 per month for first-line drug therapy with generic metformin and $5 to $20 per month with the 3 sulfonylureas available in the United States as second-line therapeutic alternatives. • Currently, there is significant variability in the use of health economics outcomes research (HEOR) data for making coverage decisions. While the 9 roundtable participants indicated that HEOR information provided to Pharmacy & Therapeutics (P&T) committees is reviewed, when it comes to making a formulary inclusion decision, safety and efficacy from the randomized controlled trials (RCTs) and cost information from the manufacturer carry the most weight. • Comparative effectiveness research (CER) studies that compare a new drug to a variety of existing therapies, including nondrug therapies, would be of more interest to MCOs that have a broader view of patient care than to pharmacy benefit management (PBM) companies that are primarily concerned with drug-based therapies.

What is already known about this subject
• This roundtable of 9 MCO decision makers opined that the results of CER studies would be helpful in making formulary decisions at product launch and may be useful in the creation of coveragewith-evidence-development (CED) agreements in which coverage is provided under the condition that further evidence on the treatment's efficacy and safety is gathered over time. • In the absence of adequate CER data, modeling outcomes can better inform formulary decisions, as long as the model enables the decision maker to predict the impact of the drug in a plan's population in a transparent manner. However, CER data is still viewed as superior for the decision-making process, especially when it is available to be integrated into the model. • The 9 roundtable participants expected that formulary decisions will be more patient-centered in the future as compared to today where patients are not part of the decision-making chain. Currently, information on indirect cost, patient quality of life and patient opinion (or patient-group opinion) is not yet considered important for making formulary decisions. • The 9 roundtable participants opined that health-related quality of life (HRQoL) might be considered in formulary decisions, but the methods and data presented in HRQoL studies were generally considered insufficiently valid and reliable. • According to the roundtable participants, reassessments of medicines for inclusion, exclusion, or change in position in the drug formulary are currently rarely performed. Real-world CER can only effectively add value if reassessments become a standard part of the formulary decision process throughout the life cycle of pharmaceutical products. • The effective use of CER for understanding the performance of products in the real world will require significant improvements in health information technology and will require an evaluation of more effective ways to bridge health care sectors.  2 The IOM definition of CER covers a very broad spectrum of research. However, there are many different players with many different perspectives (e.g., social, individual, and health-plan perspectives). It would be extremely difficult to solve all the interests with CER, especially because some of them may be contradictory. Even for a simple definition such as "the right medication for the right patient at the right time," there will be contradictory interpretations and conclusions. Some organizations or key opinion leaders only want to look at results of randomized controlled trials (RCTs)-"comparative efficacy"-while the IOM priorities apply to a much broader range of studies. For manufacturers, it is important to know and to get a clear commitment as to what kind of evidence is considered. The roundtable participants mostly agreed that there are advantages and disadvantages to all study types and that, depending on product maturity, different data sets might be relevant. Although there is a preference for prospective trial designs, retrospective data analysis was mentioned repeatedly as a useful way to assess the performance or issues in the membership population of a health plan.

What this article adds
Still, many of the 100 CER priorities defined by the IOM do not seem to generate information relevant for payer decision makers or, more specifically, formulary decision makers.

Current Use of CER in Formulary Decision Making
Consensus among the thought leaders at the roundtable suggests that available health economics and outcomes research (HEOR) data are always reviewed before making a formulary decision. HEOR assesses the clinical, economic and humanistic outcomes associated with health care interventions and can include observational, retrospective-data and decision models along with other related resources and tools. At the time of initial formulary decisions, there is usually a limited amount of real-world data available to the decision maker, and a majority of HEOR information will be supplied by the manufacturer. Once the drug is on the formulary, real-world data may be assessed within the membership population if the IT systems and analytical capabilities support such internal analysis. Several recent publications have drawn attention to the need for a more inclusive understanding of evidence when assessing the impact of technologies on health outcomes. [3][4][5] RCTs serve as the gold standard for evidence of safety and efficacy but may be limited in generalizability due to the population studied. Observational studies assessing treatment effectiveness in actual practice can complement the findings of RCTs and expand the generalizability of the findings to a broader population. resources in our budget?". If a new drug has to be assessed, the question will be "How does this drug directly compare with what is on formulary today?". Some organizations allow a broader comparison of available data. In systems in which members tend to remain for a long time, the view tends to be more comprehensive and less focused on the pharmacy silo. Specifically, the thought leader representing Medicaid felt that, in such an environment, CER helps to define value in the general health care context.
There are many proposed definitions of CER. Although all follow a common theme (comparison of how effective various medical treatments are at improving health outcomes), the perspective of the group or individual proposing the definition is important to consider. Two of these definitions, one from a pharmacy benefits management (PBM) company perspective and the other from a managed care organization (MCO) perspective, are discussed here.
PBMs, by the nature of their business, take a drug-focused perspective. Because they are responsible, for the most part, only for the drug budget for a health plan or employer, information about nondrug therapies is of little relevance to their decision making. CER is of interest to PBMs as long as it compares drugs or drug classes. This includes questions such as how a decision for 1 drug class will impact utilization of another drug class for the same or better outcome. The panelist with a PBM perspective thought that, within this framework, PBMs are interested in total drug-related outcomes.
MCOs have a somewhat different perspective. Because MCOs provide total health care coverage, their definition of CER tends to be broader, with a willingness to recognize that a higher drug cost may decrease overall costs to a health plan.
Decision makers will pick information from CER studies as it is relevant to them. In the short term, there will be no consensus-based way of using CER. Although cost-effectiveness has been excluded from the current definition of CER, cost will have to be brought into the equation for CER to be truly useful. CER must translate incremental cost to "value," or whether a drug is more cost-effective. If CER shows that a drug leads to better health-related outcomes, the decision maker must be enabled to value this improvement in comparison with the treatment cost for each therapeutic choice. Without this information, decision makers will default back to comparing net price per package of the therapies in question. A high price differential may intuitively override the incremental outcomes revealed in CER. For example, liraglutide has a direct drug cost of approximately $300-$450 per month of therapy compared with less than $10 per month for first-line antidiabetic drug therapy with generic metformin and $5 to $20 per month with the 3 sulfonylureas available in the United States as second-line therapeutic alternatives. 1 That cost information alone would lead payers to favor the established and cheaper alternatives. But if the CER incorporates cost-effectiveness research in addition to the health outcomes data, payers will be able to make a more informed real-world experience data are missing and will need to be collected over the next months and years when patients are using the medication.
Another challenge for integrating CER into decision making is the fact that post-launch data need to come from payers, and those data are currently not available in a single integrated manner. Furthermore, decision makers may not always agree with a CER study's design, inclusion/exclusion criteria, outcome measures (such as reduction in events vs. improvement in disease markers, primary vs. secondary outcomes, and so on), or other elements, thereby decreasing the applicability of that comparative data for decision making. A possible solution may be to include payers in a CER study's initial design phase, so that applicable data for decision making is more likely to be gathered and presented. Increasingly, as new drugs come to market with more comparative data for efficacy and safety, it is important to determine how the results translate into the membership population and their current standard therapies. In addition, cost must be factored in and, depending on the perspective and cost items included, the value (or incremental cost benefit) may not be as clear.

Current Use of CER for Coverage Decisions
Decision makers still make indirect comparisons on an ongoing basis. CER has the potential to make these comparisons more standardized and easier to execute. Also, in cases where CER data are not available at a product's launch, a conditional agreement based on a target for (comparative) effectiveness might help limit the risk to the decision maker and the patients.
While one-third of formulary decision makers in the national Internet survey stated that they are not using health economic or outcomes data for conditional reimbursement, another one-third claimed to use outcomes targets for conditional reimbursement agreements often, and the remaining one-third stated they rarely do so. 6 According to thought leaders from Medicaid, CER data are used for a variety of situations and for all new drugs in Medicaid. The primary question for the Medicaid organizations is not whether the drug will be reimbursed, but which is the best place in the formulary for the new drug. If the manufacturer participates in a federal rebate program, not covering the drug is not an option in the Medicaid environment, although controls such as prior authorization (PA) and step-therapy can be used to limit access. This is particularly true for newly approved drugs. Limited access allows the payer organization to gain some initial experience with the drug in a "protected" situation within the Medicaid patient population and to ask the manufacturer to bring additional data or analysis before the final place in the Medicaid program is determined after another review.
CER data may be useful in defining risk groups and limiting access to high-risk groups if safety problems arise. An example would be if a drug is more effective at controlling a specific disease (such as stroke), but it increases adverse events (such as serious bleeding). It may be appropriate to limit initial use to While all HEOR information provided to the P&T committee is reviewed irrespective of source (PBM, MCO, manufacturer, or literature), when it comes to making a formulary decision, safety and efficacy from the RCTs and cost information from the manufacturer carry the most weight. The results of a 2010 Internet survey, reported as a poster abstract, for 72 U.S. formulary decision makers (22 health maintence organizations [HMOs], 12 integrated health care systems, 26 PBMs, 8 preferred provider organizations, and 4 Veterans Affairs [VA] health systems) found that 73% of respondents said that they review HEOR data often or sometimes, while 14% indicated that they never review HEOR data in formulary decision making, and 13% responded they rarely review HEOR data. 6 In addition, only 32% of those reviewing HEOR data indicated that they applied quality standards to such data. This reflects significant variability among organizations in the use of such data for making formulary coverage decisions. Panelists noted that for HEOR data to increase influence on decision making, there would need to be a clear definition of HEOR data requirements (54%), more HEOR expertise on the P&T committee (47%), and a high need for in-house data analysis (42%). 6 The roundtable participants saw the key role of comparative HEOR data in deciding on the best place of a product in therapy in which "best place" could also mean exclusion of the product from the formulary. CER studies can be used to demonstrate the real-world performance of products; other applications of CER studies may be used to limit the target patient population, limit the financial risk and better understand patient behavior. The more information that can be assessed before making a decision, the easier it is to define the level of patient access to a product. Traditionally, clinical development of pharmaceutical products is planned with registration in mind, and data are often missing that would be useful for the decision maker. In other words, managed care would like to see data beyond those required to obtain drug approval. The acceptance of such data will grow as the data increase the ability of managed care decision makers to prioritize the use of therapies in the formularies, to reduce uncertainty, and to achieve maximum health outcomes in the reality of MCO membership mix, considering factors such as comorbidities, poly-medication, patient behavior, or adherence.
The greatest difficulty with CER is that it is usually not available when the drug comes to the market. In addition, there are limited incentives for providing CER data at launch. This is partly because a lack of U.S. Food and Drug Administration (FDA) guidance on what constitutes "competent and reliable scientific evidence" makes companies reluctant to make superiority claims even with supportive data. In some cases, it may also be because companies run the risk that CER data may clearly delineate the appropriate patient population that will benefit from a new medication, potentially reducing the market size for the manufacturer's new drug and thereby reducing revenues. Even if the clinical trials include CER, delivering data from RCTs include active comparators, the comparative well-defined endpoints and targets at the outset and that the time frame needs to be far shorter than in this first example for multiple sclerosis therapy. Often, however, it is difficult to define clear, measurable and nonambiguous outcomes targets that can be detected in a useful time frame, without putting additional burden on health care providers. An alternative to CED could be outcomes-based contracting. In an outcomesbased contract, the manufacturer is held responsible for the promised health outcome such as reduced adverse events, therapeutic response (e.g., A1c reduction), or less weight gain. The structure of such agreements could, for example, foresee a delayed payment after reaching the target outcome or a credit note by the manufacturer if the target outcome is not achieved by the individual patient or in a patient population. However, outcomes-based contracting requires an electronic infrastructure, in which conditional outcomes-related payments can be administered through automated functions and feedback routines. Currently, the complexity of outcomes and data tracking make the implementation of outcomes-based contracting impossible for most plans.
From the manufacturer's perspective, a conditional reimbursement agreement may help to overcome decision makers' fears regarding the uncertainty surrounding the new product, but it may also set a precedent for rebates. Once the manufacturer has agreed to a rebate on a conditional-access basis, it may be difficult to withdraw this rebate later.

Usefulness of Models to Estimate CER and Cost
At the time of product launch, there is a strong desire for more economic information and more information about the impact of a new treatment versus existing alternatives. However, this information cannot be delivered without the use of the product in the real-world health care setting. Modeling economic and the population less at risk for the adverse event. On the other hand, it would be very difficult to take drugs off the formulary if they do not perform as expected or intended. Nonetheless, there are published examples of drugs that have been taken off the formulary-usually because of new or additional safety concerns, or because a generic alternative became available that facilitated placing the brand drug in a nonpreferred position. 7 The more alternatives that are available in any indication, the easier it is to discontinue reimbursement for a specific drug, especially if there is no CER proving that the more expensive drugs work better or are cheaper when considering total health outcomes and total cost of health care.
Towse and Garrison (2010) attribute a number of deviceand drug-reimbursement agreements with the Centers for Medicare & Medicaid Services (CMS) to the coverage-withevidence-development (CED) concept. 8 Under CED agreements, reimbursement coverage is provided by CMS under the condition that the patient is enrolled in a registry in which further evidence on the treatment's efficacy and safety is gathered over time. 9 An early international example of a CED payment scheme was the outcomes-based agreement between the United Kingdom and manufacturers of interferon that treatment for multiple sclerosis would be covered with the understanding that data would be collected and evaluated later to make a final coverage decision. Initial evaluations of the 10-year data were published in 2009 and showed poor results for drug therapy. This dramatic finding, however, did not trigger any price reduction. 10 Instead, the paper reports, "The scientific advisory group considered that it was premature at this stage to reach any decision about re-pricing the drugs without further follow-up and analyses." 10 Such experiences have taught decision makers and manufacturers that conditional reimbursement agreements need Implementing CER: What Will It Take?

Key Factors to Consider When Building a Model for the Use of Formulary Decision Makers a factors Concerning Model Development
• Author/creator of the model needs to be identified • Methods of quality control and validation • Transparency of the model • Transparency of the factors that were included and excluded • Any budget impact model must be available to the decision makers to populate with data from their own member populations • There is no gold standard for how to make assumptions, but assumptions are an essential component of models. This uncertainty makes adapting the model conclusions to the needs of multiple stakeholders even more complicated (for example, for PBMs helping their various customers to make decisions) factors Concerning the Population in the Model • A change of the model population for demographic-or disease-related factors must be possible to make the results relevant to the local health plan • The more a plan covers a nonstandard membership, the more important it is that a model is all-inclusive so as to be a useful indicator of what might happen • High transparency and flexibility of the model, to allow combination with the plan's electronic data systems, will increase acceptability among decision makers in such plans • Models enabled to analyze subpopulations can be used by the plan to estimate the impact on specific patient segments (for example, single mothers, elderly people, disabled people) a These factors were defined by the 9 participants in the Health Plan Executive Roundtable Discussion on Comparative Effectiveness Research conducted on October 12, 2010, and  clinical outcomes based on data from the experimental study designs is an option used to simulate the impact of the new product, thus enabling decision makers to estimate what overall outcomes may be expected.
Still, debate among thought leaders continues over whether modeling clinical or economic outcomes can better inform formulary decisions. About half of the thought leaders in the roundtable discussion would consider data resulting from validated models, while the others remained skeptical. The CORE Diabetes Model, which was used to project and compare lifetime clinical and economic endpoints for liraglutide or rosiglitazone therapy added to glimepiride in the treatment of type 2 diabetes, was one model that the panelists mentioned as believable. 11 The properties of a model may determine whether it is perceived as useful and relevant. The roundtable participants suggested factors considered important when evaluating a model (see Table 1). The essential point is that a model must enable the decision maker to predict the impact of the technology in a plan's population in a transparent manner. However, the thought leaders felt that a model will likely not eliminate the need for CER, unless there would be a very high trust that the model reflects reality.

Future Role of CER in Formulary Decisions
Seven of the 9 thought leaders of the roundtable discussion thought that in 5 to 10 years, CER will always be used when making decisions about adding new drugs to a formulary. Among U.S. formulary decision makers from a broad range of payer organizations in the national Internet survey, the majority (81.9%) expected an increasing influence of HEOR data in the future, while only 2.8% of the 72 survey participants expected no increase in use (and 15.2% expected only limited use). 6 However, it seems that the organizations are not yet fully prepared to use HEOR or CER information. There are concerns about the quality of the evidence because unlike an RCT with accepted methodologies, HEOR and CER methodologies are still evolving. Participants in the national survey and the roundtable discussion found it very important to obtain clear definitions of the role and use of CER and HEOR data in their organizations. In addition, both groups of decision makers agreed that a higher level of in-house competence would be needed in their health plans, MCOs, PBMs and hospital management to be able to analyze their own data and to interpret the results.

Which Outcomes Data Will Be Important?
When asked what type of information is important for making formulary decisions with a focus on diabetes, the 9 thought leaders at the roundtable ranked the following criteria highest: the risk for adverse events, medication adherence and persistence, and health care utilization. (Again, adherence and persistence cannot be shown until the medication has Implementing CER: What Will It Take? been available for an extended amount of time.) Medication adherence and cost information were deemed more important by the thought leaders than HRQoL data, and with regard to economic information, cost-effectiveness was rated above budget impact and direct medical cost data. None of the panelists believed that societal cost (for example, caregiver time or patient productivity) would play any major role in formulary decision making.
While medication adherence may not be the most important factor for decision makers when it comes to making formulary decisions, it is nevertheless an important factor in health plans or PBMs to improve utilization of all formulary medications once coverage criteria have been defined. For example, 1 major PBM has its own therapeutic research center, the major goal of which is to increase medication adherence. The thought leaders also suggested that patient copayment levels could help to achieve goals (e.g., via lower payment with consistent adherence or weight loss). While decreasing their premiums, patients can be more responsible for their own health. Adherence is something that PBMs could measure with their data, and this activity could make a difference in patient outcomes. The value of looking at adherence data is somewhat tempered by the fact that adherence in most drug classes is far from ideal. Realworld comparative compliance or adherence data may be used to determine continuation of coverage or preferred formulary status upon subsequent class reviews.
Still, safety and efficacy data from active-comparative RCTs (RCT-CER: comparative efficacy research) and observational CER studies (actual practice comparative effectiveness) are priorities and will continue to drive decisions. In addition, real-world CER and HEOR offer a broader body of evidence. However, since evaluating HEOR data takes additional time and expertise than evaluating RCTs, it will not be used for all products or therapeutic classes. A higher degree of standardization for the generation of all outcomes data, as well as for the use of these data, will support a broader adoption of such data in standard assessment procedures.
An additional dimension in determining the value of product could be patient-reported outcomes including, for example, HRQoL, patient preferences or patient satisfaction. Despite public discussions on increasing consumerism in health care, patient satisfaction was not deemed important for better decision making by any of the thought leaders. Patient satisfaction is often measured on a general level through surveys by payer organizations but is not relevant to understanding the effectiveness of a specific treatment. Patient satisfaction may be relevant to consumers selecting a health plan but is not currently influencing formulary decision makers on drug selection. In the thought leaders' opinion, HRQoL would be a more relevant measure, but because methods or standards are not sufficiently defined and robust data are difficult to compare, quality of life is not usually a key consideration. Improvement might mean including more than 1 endpoint or even a standardized list of outcomes (e.g., cost, short-and long-term clinical outcome, out-of-pocket cost, adherence or persistence). The authors felt that these programs need to control medication use and lower overall medical costs without hindering patient outcomes over the long term, as opposed to the short-term benefits displayed by many.
Several studies have attempted to analyze the impact of benefit design on patient behavior. For example, an analysis of retrospective claims in 2001-2002 performed by Brixner et al. (2007) found that a change of benefit design involving a pharmacy benefit design change (BDC) was associated with medication adherence for 4 of 5 drug classes studied. 14 BDCdefined as a copayment increase in the second or third tier of $5 or greater or as a change from a flat copayment to a percentage coinsurance (intended to raise the incentive to use generic medications and formulary brands by increasing the copayment differential between generics and tier 1, between formulary brands and tier 2, and between nonformulary brands and tier 3 by increasing the tier 2 copayment or tier 3 copayment or both)-compared to a group with no BDC was associated with a higher proportion of patients who discontinued drug therapy (67% vs. 54% for allergic rhinitis), (66% vs. 50% for asthma), (61% versus 36% for osteoarthritis) and hypertension (39% vs. 18% for hypertension; P < 0.05 for all). BDC was not associated with discontinuation of diabetes drug therapy, and BDC was not associated with reduced adherence measured by the medication possession ratio (MPR). The BDC groups' year-to-year pharmacy costs per patient were lower in 2 of the 5 drug classes, but there was no effect of BDC on total overall health care costs in any of the 5 drug classes. 14 The roundtable thought leaders commented that, ideally, a reassessment of formulary decisions should be performed, but reassessment is rarely done. When a new drug is launched, an initial decision on preliminary inclusion or exclusion from the formulary is usually made relatively quickly using the best information available at the time. Once sufficient realworld data become available, the thought leaders believe a reevaluation should take place to confirm or revise the decision. According to the thought leaders, this approach would result in a transformation of the contracting process, acknowledging the lack of sufficient relevant data at launch and encouraging manufacturers to develop additional data during the postmarketing period. The decision will be delayed by 6 months or longer if needed, and meanwhile, limited market access will be granted.
A similar approach was suggested recently as a future model for CMS. 15 Medicare would cover and reimburse an innovative new technology or new drug therapy, as it does today, for a maximum of 3 years. During that time, manufacturers and clinicians would have to carry out research to compare the performance of the new treatment with the prior standard of care in the real-world setting. If evidence showed that the treatment did not offer clinical advantages, CMS would cut

The Role of Patient-Centered Outcomes Research Institute (PCORI) Data
PCORI's patient-oriented objectives will give more weight to the patient perspective of CER. However, according to the payers represented in this roundtable, it remains unclear whether research from PCORI will be useful and timely for better formulary decision making, since the PCORI's research priorities do not necessarily match payer priorities. In addition, as stated by decision makers, certain patient priorities (such as the humanistic outcomes like quality of life, productivity, and caregiver burden) and societal costs are not their concern and will not influence their decision making.
Cost remains a crucial factor for decision makers as long as they have to balance the use of limited resources. However, a current provision in the Patient Protection and Affordable Care Act of 2010 (PPACA) reflects the political nature of this subject, and thus, PCORI may not have the responsibility or even the authority to measure value as determined by cost per outcome. 12 In the equation used to calculate value to an organization, PCORI may help to better define the desired clinical and patient benefits. However, PCORI is not expected to define the cost of these benefits or whether the benefits are affordable.

CER Data in Formulary Design, Benefit Design, and Re-Evaluations
Today, decision makers do not consider further assessment a priority after the primary decisions on inclusion, exclusion, and position of a drug in the formulary have been made. With a few exceptions, the impact of drug-formulary or benefit design decisions on health outcomes is generally not measured. 6,13 As 1 thought leader noted, "We rarely go back and see what the impact of our decision was." This emphasizes the issue described earlier about the lack of real-world CER data at a product's launch. If there is a desire for more real-world CER data, then decision makers must use the data and make it a priority to follow up on class reviews.
Benefit design may also impact the development and utility of real-world data and retrospective analysis and re-evaluation. This observation is confirmed by a systematic review of publications that analyzed the impact of benefit designs on health and economic outcomes. 13 Seventy-seven studies were included in the review by McAdam-Marx et al. (2008) and were analyzed for the use of 11 types of outcome endpoints. Sixty-eight percent of studies incorporated an economic endpoint; of these, 68% reported only economic data and did not address clinical or humanistic outcomes. Overall, clinical or humanistic endpoints were evaluated by 43% of studies; of these, an additional economic endpoint was reported by 52% of studies. 13 The authors concluded that "the efforts of these researchers to assess the overall quality of drug-management programs have fallen short. To ensure that drug-management tools have a desired effect on outcomes and medical costs, measures used to evaluate drug-management programs must be improved." 13 HEOR data are already used in contracting, and up to twothirds of respondents believe their use is expected to increase in the future. 6 Outcomes-based contracting will most likely be used to cover gray areas where there is insufficient information available at the time of decision. For example, to make it more cost-effective, a new drug may have to offer larger rebates until CER is available. Contracting affects the value equation for a drug. Additional use of real-world CER in contracting will require improvement in the health information technology (HIT) infrastructure-specifically, improved electronic medical records and data integration across sectors. This alone is an enormous challenge. It is not enough for each physician's office to maintain a patient's medical records, but those records must also incorporate patient data from other sites of care, wherever that patient is seen. Every physician's office and hospital must share patient data and communicate with each other. In addition, financial incentives across sectors need to be aligned. A PBM and a MCO have very different financial incentives, for example.
Improved HIT may help to better capture value-related information. When this information is available, outcomesbased contracting will be more operationally manageable. In addition, the same outcomes targets and criteria could become the basis for new disease-management programs.
Outcomes-based contracting or outcomes-based disease management will work better when the long-term-oriented payer systems are more standardized. On the contrary, in organizations with a multitude of different customers with different needs and patient populations, as in the case of PBM companies, it will be more difficult to determine the targets for each customer and to then differentiate among various contract types.
There is an additional dimension of outcomes-based contracting: To capture the full potential value of a therapy, decisions will increasingly have to bridge health care sectors (e.g., pharmaceuticals vs. physical therapy vs. surgery). The current budget structures and incentives limit the usefulness of cross-sector comparison. In addition, many plans and PBMs currently don't have the necessary integration of medical and pharmacy data to make this type of integrated contracting feasible and would need better HIT.
For practical reasons, not all data can be generated locally for each plan or system. Decision makers will, however, be interested in seeing how applicable the data are to their own populations and how to improve outcomes in their own populations. In addition to using drugs in the right patients (highchance responders), alternatives are needed to improve the chances for patients at high risk for nonresponse. Thus, targeting and outcomes-improvement programs could be additional aspects of CER.
CER can support contracting on 2 levels. Using CER with the supply side would mean that the payment depends on concrete and measurable outcomes. The contracts will mostly rely on health outcomes (clinical parameters). According to the 9 formulary decision maker thought leaders in our roundtable, the price to the level it pays for the equivalent, conventional treatment. But, if the evidence showed that the new intervention was superior to the traditional method, Medicare would continue to pay the higher rate. 15 Where will CER lead formulary decision makers? The majority of thought leaders (7 of 9) in the roundtable discussion agreed that benefit design will increasingly consider CER data over the next 5 to 10 years. CER will probably not tip the scales for or against coverage, but it is expected to help in finding the best place in therapy and on the formulary (e.g., copayment tier, PA, or step therapy).
With all the investments in CER in the United States, it can be expected that in a few years more CER data will have been published and be available. Comparative research is also available from other regions worldwide and may add to the evidence base. While this information will be useful, it will not necessarily change the process of how decisions are made. Even if health care reform stagnates in the United States, some changes, such as the advances already made in CER that provides clearer information for decision makers, cannot be ignored. The more that CER data become available, the higher the expectations of decision makers will be. Thus, the main change will be in the intensity of how the data are used.
However, the use of CER data will not only be accelerated by payer demand or government policies. Each additional head-to-head comparative study submitted with a new drug in a therapeutic area or class will raise the expectations for future development of subsequent drugs. Because of pressure from early CER adopter manufacturers, other manufacturers will follow, despite the fact there are limited incentives for providing CER data at launch from the registration and marketing perspective.
Eventually, with more experience and standardization, and as both RCT-CER and real-world CER become more abundant, CER will become more important and a fully integrated aspect of formulary management. While, initially, there might be more pressure on new drugs with perceived high-cost consequences (including drug costs, costs of adverse events, costs of monitoring and utilization) for the health plan, CER should be evaluated for all drugs on the formulary.
In addition, CER may offer the opportunity for earlier market access under the condition of continuing data collection in the real world: coverage-with-evidence development (CED). Under such arrangements, the drug is reimbursed, but the manufacturer has to build a registry to collect data documenting the course of the disease in patients treated with the new product. As CER evolves and becomes a more important component of decision making, the impact of benefit design on outcomes will become more important in CER design.

CER's Effects on Health Information Technology, Contracting, and Bridging Health Care Sectors
According to the (unpublished) national Internet survey of formulary decision makers, one-third of the respondents felt A very small percentage of ARRA funding for CER has been earmarked for understanding the genetic or genomic basis of diseases or response to treatment ($4.8 million or 2.5%). 16 At this point in time, the research projects funded by the National Institutes of Health (NIH) are not necessarily matching up against the priorities set by the IOM. The highest 4 NIH expenditure areas for evidence development and synthesis are psychiatric disorders; oncology and hematology; cardiovascular and peripheral vascular disease; and alcoholism, drug dependency, and overdose. The IOM's top 4 research priority areas are functional limitations and disabilities, cardiovascular and peripheral vascular disease, psychiatric disorders, and neurological disorders. 16 The debate continues as to how well either of these priority lists align with those of health plans and formulary decision makers.
According to the 9 roundtable participants, government is playing a leading role in defining the cornerstones for CER. Many other players are still puzzling over how CER applies to them and are waiting for some government or academic leadership. An immediate and extreme change-legislation requiring companies to conduct clinical trials against a current standard of care, for example-is unlikely in the short term because of the combined mass and inertia of both the industry and government. Marketplace demand will probably drive CER's growth, at least in the beginning. Pharmaceutical companies may be a secondary driver of CER. As a result of CER becoming a general expectation, new products in development will have to fill the gap in CER by integrating comparative trials into their study programs. Comparative clinical studies have been supplied with an increasing number of drug applications in recent years. 20 With the increasing request for comparative data, some drug companies proactively take on a leadership role in designing active comparative trials (e.g., with liraglutide). [21][22][23][24][25][26] Based on such studies, formulary decision makers will have more comparative efficacy data on hand for newly developed drugs to support their decisions. At the same time, the bar of expectations will be raised and active comparator trials will be expected if the study designs and data produced are judged to be valid.
In addition to funds coming from the government, some funding for CER will come from pharmaceutical manufacturers who will voluntarily invest in active comparator trials as a competitive business decision. Increasing experience with manufacturer-driven CER will begin to shape how CER is integrated into drug development. Moreover, there are voices requesting CER to be mandatory for FDA approval. [27][28][29] Health plans do not have the resources to fund CER. They will require it, but the thought leaders at the roundtable discussion felt that government rather than any other stakeholder will have to be the power behind the impetus for change, potentially through CER mandates. Making the right investment of government funds, defining the appropriate health targets, and deciding on the acceptable value limits in terms of cost and humanistic outcomes such as HRQoL can only shift the weight of a decision in the United States. Quality of life is not expected to become a key decision criterion to the payer, and payers are not expected to pay an extra margin based on quality-oflife outcomes. Similarly, patient time or productivity is not expected to be highly valued.
Using CER for contracting with the user side would mean that the amount of copayment contributed by the patient will depend on achieving goals (such as consistent adherence or losing weight). The purpose of these types of contracts is to motivate the patient to be responsible for his or her own health. More re-evaluation will need to be undertaken and made a priority. Value-based benefit designs have been in use for some time, but more data are needed to evaluate the clinical and financial benefit to the payer organization.
In disease areas like type 2 diabetes, hypertension, and obesity, increased patient participation in the cost may appear attractive to the payer organization. The perceived benefit of cost-sharing programs directed toward the patient could be that the patients become more motivated to adhere to the therapy, and thereby, better outcomes are achieved. The expected benefit of contracting with drug manufacturers based on clinical outcomes would be that the financial risk for the payer is limited to patients who respond to therapy.

Who Will Initiate and Fund CER?
Several sources of funding are available to support the establishment of CER. Additionally, future funding has been promised by the president of the United States and the PPACA. [16][17][18][19] Sources of funding include: • In 2010, $1.1 billion in stimulus funding for CER was made available through the ARRA. A considerable share of the original funds was used for infrastructure and methods development. 16 • The Agency for Healthcare and Research Quality (AHRQ) fiscal year 2011 budget request included $286 million to broaden patient-centered health research. 17 • The PPACA provides sustained federal funding for CER through 2019. 19 Forty-six percent of the $1.1 billion funding has been allocated to synthesizing existing evidence through systematic reviews or developing new evidence. A further 41% will be allocated for CER to develop the data, methods and workforce that can, in turn, increase the capacity for future research. The majority of spending in both of these categories will be dedicated to using the knowledge acquired from the delivery of care or observational research. 16 Moderate percentages of the funding were dedicated to: • Comparing models for the integration and coordination of care ($38.6 million or 15.1%) • Studying the delivery of care in nontraditional settings or by nonphysician providers ($34.3 million or 13.5%) • Evaluating economic incentives for patients and providers to choose effective treatments ($8.0 million or 3.2%) 16 appropriate outcomes would be the government's responsibility. From the perspective of the pharmacy formulary decision makers, the real question is whether the tremendous investments in CER through stimulus funds and expected tax contributions, along with the accompanying changes in the investment priorities of health care-related organizations, will really lead to studies that deliver results with a higher impact on health care efficiency. A review of recent CER trials suggested that poor quality research continues to emerge and that there are methodology issues that must be addressed to improve the degree of usability of CER. 30 This includes many aspects of the studies such as the selection of less-than-optimal comparators, inadequate dosages of comparators, the use of surrogate study endpoints that may or may not have been validated as clear indicators of specific clinical outcomes (e.g., using biomarkers such as low-density lipoprotein [LDL] cholesterol levels for coronary artery disease), and accelerated time frames. 20 The PCORI is expected to assist in defining criteria for CER trial design in the future.

■■ Conclusions
As noted by the roundtable participants, both RCT-CER and real-world CER studies are highly desirable for formulary decision making, and both are used when available. However, such research is not yet routinely available, and it does not yet apply to all formulary decision-making organizations and is not yet used routinely. It is expected that the role of real-world CER will increase as more results are published and accessible, and as the guidance and training on how to use real-world CER research and results emerge. The government is viewed as a driver behind CER, while the pharmaceutical industry will shape it by adapting the clinical development process and creating increased evidence from both pre-and post-approval CER. To make it sustainable, however, decision makers will have to require CER evidence. For formulary decision makers, CER is useful only if it has application. Ideally, health plans will develop processes to effectively conduct, analyze, and use CER data to understand the results in their own populations, enabling formulary managers to create effective benefit designs or make better formulary decisions. CER must then be applied to determining value (or cost-effectiveness).
The evidence generated in RCTs should be validated in real life to show that the expected advantages are also seen in realworld populations of various compositions. Decision makers need to improve retrospective analysis of data within their own health plans to ensure validity of their decisions.
The roundtable panel recommended that manufacturers maintain close communication with decision makers when planning and gathering further evidence in real-life studies and that such communication remain proactive and transparent.