Can We Develop Sustainable and Sharable Cost-Effectiveness Models for Value Assessment in the U.S. Health Care System?
Publication: Journal of Managed Care & Specialty Pharmacy
Volume 25, Number 5
Abstract
There is currently a movement to make economic models more transparent, with some arguing for completely open-source models. However, increasing model transparency raises several logistical challenges, including ownership and funding. This article outlines recent experience and approaches to some of the logistical hurdles that must be overcome in pursuit of more transparent economic models. The Institute for Clinical and Economic Review (ICER) has recently completed a pilot transparency initiative that made drafts of executable economic models available to involved drug manufacturers during the review process. By directly viewing and interacting with models’ structures, inputs, key assumptions, and results, stakeholders were better equipped to provide valuable feedback as part of the health technology assessments. This experience, along with feedback received from the modeling collaborators and relevant manufacturers during this pilot, have helped determine ICER’s approach to sharing economic models associated with ongoing and future evidence reviews. This process has been expanded to all assessments going forward, making it the standard approach to model transparency.
DISCLOSURES: No outside funding supported the writing of this article. Both authors are employees of the Institute for Clinical and Economic Review (ICER), which is an an independent organization that evaluates the evidence on the value of health care interventions. ICER receives grant funding from the California Healthcare Foundation, Laura and John Arnold Foundation, and New England States Consortium Systems Organization. ICER’s annual policy summit is supported by dues from Aetna, AHIP, Allergan, Alnylam, Anthem, AstraZeneca, Biogen, Blue Shield of California, Cambia Health Services, CVS Caremark, Editas, Express Scripts, Genentech, GlaxoSmithKline, Harvard Pilgrim Health Care, Health Care Service Corporation, Health Partners, Johnson & Johnson, Kaiser Permanente, LEO, Mallinckrodt, Merck, National Pharmaceutical Council, Novartis, Premera, Prime Therapeutics, Regeneron, Sanofi, Spark Therapeutics, and United Healthcare.
The accompanying article “Toward a Relevant and Credible Cost-Effectiveness Analysis for Value Assessment in the Decentralized U.S. Health Care System” by Jansen et al. points out the importance of cost-effectiveness analyses (CEAs) in the dispersal of high-value care and argues that there is a need for transparency and adaptability of these in a multipayer health care system as in the United States. However, initiatives for model transparency must address several logistical challenges in open-source models for economic analysis, including ownership, funding, and sources of bias, that may not be fully addressed in the accompanying commentary. There is currently lively debate in the field regarding the feasible extent and ultimate sustainability of open-source models for economic analyses.1-4
Because the Institute for Clinical and Economic Review (ICER) is currently one of the most prolific producers of CEAs in the United States, its analyses have faced increasing scrutiny and criticism by some organizations over the transparency of ICER’s work. Over time, ICER has attempted to address concerns regarding model transparency by increasing stakeholder engagement throughout model development, presentation of preliminary model structure and assumptions to manufacturers and patient groups, model sharing with manufacturers, and publication of model details in its reports and in peer-reviewed publications. Accordingly, ICER’s health economics staff has experience in addressing some of the logistical hurdles that must be overcome in pursuit of a more transparent, and eventually open-source, CEA environment. This article outlines some of our experience and approaches to addressing these hurdles.
Practical Requirements and Logistics
As Jansen et al. point out, health care in the United States is provided through a fragmented system with multiple payers, which provides different incentives and barriers to the use of CEAs, and the models on which they rely, than are found in many other countries’ more unitary health care systems. First, there is no centralized health technology assessment (HTA) authority in the United States, and coverage decisions are made by multiple payers using numerous insurance benefit plans covering varied populations. Second, clinicians face increasing incentives to consider the cost and value of the care provided, but there is lingering skepticism of CEAs and their use for coverage or payment decisions in the United States.5-7 Finally, the U.S. health care system includes several private sector agents who may have their own, separate concerns over intellectual property, which can make the sharing of data and models more difficult or impossible in the current environment.
All of these factors can make it more difficult to create an environment in which CEA models can be shared in a sustainable manner that clearly creates value for all of the stakeholders involved. While we applaud efforts to increase model sharing and transparency, such as the open-source rheumatoid arthritis model developed by the Innovation and Value Initiative (IVI),8 we are also aware from our own experience in such efforts that there are several important logistical and practical issues that must be addressed to create a sustainable process for increased sharing of CEA models. This article discusses these issues and potential solutions based on our experience in sharing the CEA models used in several recent HTA projects.
In March 2018, ICER announced a pilot transparency project, a new program to make drafts of executable economic models available to involved drug manufacturers during our HTA review process. ICER has always been committed to providing the level of detail necessary for independent replication of our economic analyses, and we have been informed that external stakeholders have done so in cases. However, by directly viewing and interacting with a model’s structure, estimates, key assumptions, and outputs, the hope is that manufacturers (and other stakeholders) may be better equipped to provide valuable feedback as part of the public comment period that follows the release of an ICER draft evidence report. This enhanced feedback can then be used to further refine the economic models and analyses.
Benefits of Model Transparency
As previously stated, ICER has always shared information about economic model structure and inputs with manufacturers and other stakeholders during our review process. One purpose of this information sharing is to help interested parties better understand and critique these models during their development and use in the HTA process. The ability to change inputs or assumptions and observe the effect on results can provide opportunities for manufacturers to gain even greater insights into draft models. Ultimately, patient groups and other qualified stakeholders may also be interested in attaining access to economic models for this purpose. Our goal throughout this process is to obtain enhanced feedback that can improve the accuracy and relevance of our final model versions and HTA reports.
However, another important objective of this information sharing is to enable third parties to develop new models or modify existing models based on insights gained from the ICER models. This allows other analysts to attempt to duplicate the results using a parallel, independent modeling process. To the extent that this occurs, this independent verification can provide additional assurances about the validity of a given model. In cases where important inconsistencies are found, analysts can work, either in parallel or collaboratively, to try to understand what variations in model structure, inputs, or assumptions may be driving differences in results.
Description of Transparency Pilot Project
In ICER’s model-sharing process, manufacturers of the interventions and comparators of interest for a given review are asked if they would be interested in gaining access to the draft model during the HTA process. If so, manufacturers complete a contract with and pay a nominal fee to ICER’s academic collaborators to cover the time needed to produce sharable executable files and all supporting documentation. Throughout this process, we have remained focused on balancing increased model transparency with the intellectual property rights of our academic collaborators and their associated institutions.
Model Preparation
Economic models may be developed using different software platforms, structures, formats, and levels of documentation. Decisions about each of these may affect the ease with which a model may be shared, as well as the accessibility of the model to external users or reviewers, and may increase the time and resources (and funding) required to conduct economic analyses. Additional sources of funding may be required to prepare models for broader sharing, especially if not originally developed with that in mind.
Additional decisions involve what levels of user access and modification will be allowed and whether multiple levels will be provided. Different stakeholders will have varying levels of experience with and expertise in health economics and outcomes research and so will desire different user interfaces, levels of detail, and ability to modify model structure or parameters. Parts of models may need to be “locked down” to ensure the validity of modeled results. The facility with which these varying needs can be accommodated is affected by many of the decisions above, such as which software platform is used.
Confidentiality and Intellectual Property Issues
One concern throughout this model-sharing process has been to balance increased model transparency with the intellectual property rights of our academic collaborators and their associated institutions. To protect the intellectual property of the researchers and their institutions, models are shared under nondisclosure/confidentiality or licensing agreements. Under these agreements, manufacturers are unable to alter the shared model for their own purposes or to retain copies of the shared model. In addition, their access is for a limited time frame and confined to the team of individuals responsible for the model review. The expectation is that this process will inform the written public comments on the draft report.
A related issue is the potential need to include data in models that may be confidential for commercial or academic (i.e., publication) purposes. ICER’s HTA reviews are often conducted around the time of a product’s launch, when some product-specific data that may serve as key model inputs may not yet be published or may be considered proprietary by manufacturers. In such cases, any model sharing must include a way to mask or redact specific data in the model or to create separate versions of the model to share with different stakeholders.
Beyond redacting model inputs, there is also the issue of how to present model results. For example, results may need to be presented as a range rather than a point estimate to lessen the possibility that redacted data could be back-calculated or reverse engineered using specific model outputs. These concerns, of course, increase the effort and time required to prepare such models for wider access. In these cases, our model-sharing agreements include provisions requiring that users not attempt to back-calculate confidential inputs that have been redacted.
Access/Support Infrastructure
An additional concern is how to ensure that the correct infrastructure is in place to provide secure access to the model and to support users of the model. To date, ICER and its collaborators have provided access to models via cloud-based file hosting services and a web-based modeling platform (heRo3, Policy Analysis, Brookline, MA). Use of these services allows us to ensure that access is secure and confined to the teams of individuals responsible for the model review. In addition, access can be specified for a limited time frame, with no saving or retention of materials. (Note that these restrictions can be eased to allow for more open-source, public access with no time limit, if desired.)
Simply providing access to an economic model may be of limited use, especially to those who have not been involved in or are less familiar with the modeling process. To ensure that economic analyses are truly relevant and credible often requires additional technical support to users and reviewers. Such support can range from a brief initial orientation document, to more extensive user guides, follow-up email/phone contacts to answer queries, or even in-person workshops to orient or provide support. One important consideration is what level and duration of support to provide, since increased levels of intensity require increased levels of funding, which could restrict access or sustainability of such initiatives.
Sustainability
If access to models is provided beyond the initial HTA review, analysts must consider whether provision will be made for continued updating and support so that models remain relevant and credible over time. This also requires consideration of funding and processes to ensure that the update and support infrastructure required are sustainable over time.
Transparency Pilot: Results/Lessons Learned
The transparency program was first piloted during ICER’s reviews of therapies for migraine prevention and of the drug elagolix for pain associated with endometriosis. Academic collaborators from the University of Illinois at Chicago College of Pharmacy and the University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences, respectively, developed the economic models for these reviews. (Note that the manufacturer declined to review the elagolix model, so it was not shared.) The second transparency effort involved an ICER-developed economic model that assessed medications for addiction treatment in patients with opioid use disorder, which was shared with relevant manufacturers via the heRo3 platform. Through our experience and the feedback received from manufacturers during this pilot program, we have determined several areas that have worked well and others where issues remain.
In terms of what has worked, ICER and its modeling collaborators provided access to models for several manufacturers and their consultants during the draft review phase of our assessments. In the process, basic technical support processes were not onerous or overly resource intensive. Contracting and licensing documents have been developed, and secure platforms and accessible procedures have been used for model sharing, with the ability to lock down specific inputs or to set auto-expiry dates on shared models. We will continue to develop and revise these processes as we gain more experience. In both cases where models were shared, we received positive feedback on the model-sharing experience, with the highlight being that this process aided manufacturers in providing more detailed and structured feedback on our economic analysis.
However, we encountered some obstacles in rolling out this program. Given the timeline of ICER’s reviews, the schedule for contracting or licensing access to models has been challenging for some manufacturers, as has been the time allotted for review of the models once accessed. This has been a contributing factor in some manufacturers choosing not to participate in the opportunity to access a model (e.g., none chose to do so for endometriosis).
Another issue that has been raised is the need to clearly define the level of support and documentation that should be expected (e.g., the type and number of interactions), as well as how to provide updates when a model changes during the review process. As previously discussed, processes for dealing with confidential and redacted data had to be developed, and there may be additional concerns regarding this as models are provided to broader audiences or made more open source. Finally, there are concerns about the potential for financial barriers to access, especially for nonmanufacturer stakeholders, and how to ensure access for such stakeholders in a sustainable manner.
Next Steps
Our experience, along with feedback received from our modeling collaborators and the relevant manufacturers during the transparency pilot, have helped determine ICER’s approach to sharing economic models associated with ongoing and future evidence reviews. We are also considering the steps that will be needed to expand model access beyond manufacturers to other stakeholders and qualified researchers, including the need to tailor the extent and types of technical support to be provided. Over time, we hope to continue to increase the overall level of model access, for example, by publicly releasing the models that are internally developed at ICER within a few months of publication. Finally, we will continue to explore funding mechanisms to ensure the broadest possible access to our models for various stakeholders in a sustainable manner.
Despite the obstacles encountered to date, we have expanded this process to all assessments going forward, making this our standard approach to model transparency. We believe that the ICER and IVI transparency initiatives underscore our field’s collective commitment to fairness, transparency, and scientific rigor and hope that they will contribute to the relevance and credibility of health economic analyses.
REFERENCES
1.
Cohen JT, Neumann PJ, Wong JB. A call for open-source cost-effectiveness analysis. Ann Intern Med. 2017;167(6):432-33.
2.
Sampson CJ. A call for open-source cost-effectiveness analysis [letter]. Ann Intern Med. 2018;168(7):528.
3.
McQueen RB, Padula WV, Campbell JD. A call for open-source cost-effectiveness analysis [letter]. Ann Intern Med. 2018;168(7):528-29.
4.
Cohen JT, Neumann PJ, Wong JB. A call for open-source cost-effectiveness analysis [letter]. Ann Intern Med. 2018;168(7):529.
5.
Bryan S, Sofaer S, Siegelberg T, Gold M. Has the time come for cost-effectiveness analysis in U.S. health care? Health Econ Policy Law. 2009;4 (Pt 4):425-43.
6.
Gusmano MK. Objections to the use of cost-effectiveness analysis in the US: reflecting on ‘Has the time come for cost-effectiveness analysis in U.S. health care?’. Health Econ Policy Law. 2015;10(4):419-24.
7.
Neumann PJ. Why don’t Americans use cost-effectiveness analysis? Am J Manag Care. 2004;10(5):308-12.
8.
Innovation and Value Initiative. IVI-RA Value Model: part of the open source value platform. Available at: https://www.thevalueinitiative.org/ivi-ra-value-model/. Accessed March 28, 2019.
Information & Authors
Information
Published In
Journal of Managed Care & Specialty Pharmacy
Volume 25 • Number 5 • May 2019
Pages: 521 - 524
PubMed: 31039064
Copyright
© 2019, Academy of Managed Care Pharmacy. All rights reserved.
History
Published online: 30 April 2019
Published in print: May 2019
Authors
Metrics & Citations
Metrics
Citations
Export citation
Select the citation format you wish to export for this article or chapter.