Industry's Perception of Presenting Pharmacoeconomic Models to Managed Care Organizations

BACKGROUND: Previous research has shown that pharmacoeconomic (PE) data are considered important but may not be optimally utilized by decision makers. No research has compared the effectiveness of different types of PE models. OBJECTIVES: The purpose of this study was to examine the perceived value and understanding of PE models among decision makers in managed care organizations. The perspective of this study was from research scientists working in the pharmaceutical industry who present PE models to managed care clients. The study objectives were to (1) examine what types of models are best received by decision makers, (2) investigate the barriers to using PE models, and (3) recommend methods for improving PE models. METHODS: A telephone survey of 39 items was conducted with 20 PE research scientists from various U.S. pharmaceutical and biotechnology companies. Topics addressed included factors contributing to how well PE models are received, barriers to using PE models, and recommendations for improving PE models. RESULTS: Models have an impact on health policy decision making. Nineteen of 20 respondents had at least one experience where a PE model played a role in optimizing the formulary positioning of a product. No single model format (e.g., decision analytic tools, spreadsheet analyses, Markov models, multivariate regression models) was regarded as the most effective model type. Although 7 of 20 respondents said simple spreadsheet models were most effective, well-designed, scientifically sound regression models were also reported to be very effective. CONCLUSIONS: The respondents commonly used models to share PE information, which was said to play a role in making health policy decisions by decision makers in managed care. There was no consensus regarding the type of model that was most effective. Study participants indicated that a variety of model designs are effective, ranging from simple spreadsheet models to multivariate regression models. Recommendations for improving PE models include (1) producing scientifically sound models, (2) customizing models where possible, (3) making models transparent, (4) making models user friendly, and (5) involving a nonbiased third party for model development.

here has been tremendous growth in the number of pharmacoeconomic (PE) studies conducted in the past 10 to 15 years. 1 This growth has occurred in the United States as well as in other developed countries worldwide. 2,3 With this growth, guidelines have been promoted to provide some structure when creating studies and evaluating literature for formulary analysis. 4 Pharmacoeconomics provides a toolkit to integrate the assessment of clinical, economic, and patient-centered (humanistic) outcomes. These techniques have also been useful to emphasize the importance of overall health care costs, rather than limiting the focus to strictly pharmaceutical expenditures in a "silo" budget environment. 5 Although the basic economic techniques are not new, they have recently enjoyed widespread use in evaluating products for formulary inclusion, positioning of products within different formulary tiers, and selection of products in treatment guidelines. 6 Previous research with pharmaceutical industry representatives suggest that 90% of respondents provide PE information to their customers. 7 Very limited research has been conducted evaluating the use of PE data in the interface between product marketing representatives and health care decision makers. Since the pharmaceutical industry plays a large role in supporting and disseminating the results of PE research, its perspectives on the effectiveness of these tools provides important information for those developing or sharing these types of data.
One issue that has been raised as a concern is the use of PE models for illustrating analysis results. 6 In this context, models may be broadly defined to include decision analytic tools, spreadsheet analyses, Markov models, and multivariate regression models. These methodologies may not be well understood by many clinicians or health plan decision makers. However, there may be a growing appreciation of PE models. For example, a recent editorial in Pharmacotherapy praised a multivariate regression analysis as a "model pharmacoeconomic evaluation." 8 It is useful to determine the perceptions from industry professionals who share these models with health care systems and identify mechanisms to improve the usefulness of PE models.
The purpose of this study was to examine the value and understanding of several types of PE models that are currently being marketed to health care decision makers. The study objectives included the following: (1) to examine what types of models are best received by decision makers, (2) to investigate the barriers to using PE models, and (3) to recommend methods for improving PE models.

Industry's Perception of Presenting Pharmacoeconomic
Models to Managed Care Organizations

■■ Methods
Approval was obtained from the University of Arizona' s Human Subjects Protection Program Committee for the interview discussion questions and recruitment invitations prior to initiation of the study. Individuals within health outcomes departments of pharmaceutical and biotechnology companies across the United Staes were contacted. These individuals were known to utilize PE information with customers, and these people were queried to obtain names of professionals within their organizations who would be best qualified to discuss their experiences in presenting PE models. Potential subjects were emailed and telephoned between March and May 2002 to ask if they would participate in a 30-minute interview. Twenty-three industry representatives (e.g., directors, managers, or medical liaisons) were invited to participate. When potential participants agreed to be interviewed, this served as consent to participate in the study.
Subjects did not receive compensation for their participation. All interviews were completed by June 2002. A survey instrument was created from a review of the literature on the use of PE models. Using this information and expertise in the areas of PE and human behavior (i.e., anthropology), the 4 researchers from the University of Arizona developed a draft questionnaire. The authors had previous experience developing questionnaires regarding the use of PE and the role of PE models in decision making among managed care executives. 6,7,9 The questionnaire content was reviewed for face validity by an independent health outcomes researcher at the University of Arizona. A formal pilot test of the questionnaire was not conducted.
The final telephone survey consisted of 39 items, focusing on demographic characteristics of participants; a description of the types of PE models presented; participants' perceptions of their clients' knowledge of statistics (i.e., tools, methods, terminology); and techniques for effective demonstration and communication of PE models Appendix A). The interview questions were primarily open ended, with the purpose of gaining as much information from subjects as possible. This was a qualitative study by design.
Because of the open-ended nature of the questions, participants did not have an opportunity to agree or disagree with all issues. Study participants were asked to discuss issues related to all PE models as well as specific types of models, including decision-analytic models, spreadsheet analyses, Markov models, and multivariate regression models.
A series of items was included in order to assess the level of experience among participants. Information regarding their company position, years of experience presenting PE models, and types of PE training was obtained. The frequency of presentations, types of clients served, and geographic areas in which models were presented were also gathered to illustrate sample characteristics.
Several items were developed to assess the types of PE models being presented by the pharmaceutical industry and how well these models were received by decision makers. In this context, decision makers were primarily described as those working in various health plans and organizations across the United States who have substantial influence over the medications that are included on drug formularies. The format by which the models are commonly presented (e.g., laptop presentations, slide presentations, article reprints of model) was recorded as were differences in reception by the end user associated with these various presentation designs. Factors contributing to how well the models are received (e.g., audience, user training, model format, presentation format, therapeutic area, level of model complexity) were assessed in order to identify potential barriers to using PE models in decision making. The ability and extent by which clients are able to "customize" the PE models developed by the pharmaceutical company were also discussed with survey participants. Reasons for and against leaving electronic copies of the PE models with clients, commonly asked questions, and participants' perceptions of client understanding of the PE models were recorded.
Several items focused on the participants' perceptions of their clients' knowledge of statistics. The purpose of these ques- tions was to identify statistical tools or concepts that were presented in PE models but poorly understood by decision makers in managed care. Participant ability to identify his or her audiences' level of understanding with respect to statistics and successfully explain complex concepts was assessed. The usefulness of "pop-up" windows (e.g., explanation of concepts and/or definitions embedded in the model) was discussed in the interviews as well. This information was used to develop recommendations for improving the understanding of PE models presented to decision makers.

Demographic Characteristics of Sample
Finally, a series of items addressed the effectiveness of PE models and potential opportunities for improving their usefulness. Participants' most common problems in presenting PE models were gathered, including the most common misinterpretations of the data presented. Because of their extensive experience in presenting PE models, participants were also asked to make recommendations for how models could be changed to improve their effectiveness and enhance their usefulness to health care decision makers.
The primary objectives of this study were to identify barriers to using PE models and to make recommendations to improve the utility of these models. The analysis was, therefore, primarily descriptive in nature. Study investigators summarized data gathered from the interviews and outlined the key issues described by study participants. This method was used in order to organize a set of recommendations for improving the usefulness of PE models.

Sample Characteristics
Twenty (87%) of the 23 pharmaceutical representatives invited to participate agreed to be interviewed; 3 of those representatives contacted were unavailable to participate (Table 1). Participants were spread across the United States, representing 10 pharmaceutical and biotechnology companies. The mean interview length was approximately 32 minutes, with a range of 20 to 57 minutes. Participants had mean years of experience presenting models of 4.6 years, with a range of 1 to 10 years of experience. Respondents (15 of 20, 75%) indicated that on-thejob training was the most common method for obtaining PE education or training. All respondents had given presentations involving PE models within the United States, and 4 respondents had presented PE models internationally. On average, PE models were presented between 1 and 2 times each month by survey participants, with managed care organizations (MCOs) being the most common clients of these presenters.

Factors Contributing to How Well Models Are Received
Audience. Most participants preferred small group presentations (18 of 20, 90%) consisting of both pharmacists and physicians (14 of 20, 70%). The combination of these 2 groups was thought to be effective in bringing together multiple viewpoints in the decision-making process and balancing the opinions of each. An overall theme was that physicians and pharmacists view model information differently (i.e., with a different focus). Eleven participants (55%) reported that physicians were more interested in clinical outcomes and the overall model results (or take home messages) than the cost components of the model (e.g., cost-effectiveness, budget impact). Thirteen participants (65%) indicated that pharmacists were more educated in cost issues and tended to focus more on the economic impact demonstrated by PE models as opposed to concentrating on the clinical outcomes.
Respondents noted that because many pharmacists continue to operate in a silo-based environment (i.e., required to focus on the budget constraints of one department rather than the total health care budget impact across multiple departments), it is helpful to have physicians in attendance to represent the broader issues (e.g., impact on overall health care costs). Participants viewed physicians as being less receptive to technical cost issues presented in the models. Therefore, presenting to both groups simultaneously was perceived to be advantageous.
In an effort to bring these disciplines together, one participant reported success in assembling decision makers from multiple health plans for presentations. This was found to be an extremely effective means of sharing model information, according to one participant. While presenting to pharmacists and physicians together was preferred by the majority of participants, one respondent had a preference for speaking with pharmacists and physicians separately until they were ready to make a decision.
Two participants commented that younger professionals were more aware of some of the newer tools such as PE models for decision making, perhaps due to more recent training. It was also noted that the best audience to whom to present models is composed of those individuals responsible for decisions affecting total health care costs. However, respondents agreed that PE model effectiveness would be enhanced with supplemental training for decision makers across health care systems.
In addition to audience characteristics, respondents identified several other factors that determine whether a model is well received by decision makers. The most frequently mentioned factors included (1) ease of understanding (i.e., model simplicity and transparency) (19 of 20, 95%); (2) the ability to customize the model to individual practice settings (12 of 20, 60%); (3) presenter credibility and training (9 of 20, 45%); (4) model format and content (6 of 20, 30%); and (5) availability of reprints (i.e., model results have been published) (3 of 20, 15%). The contribution of each of these factors is detailed below.
Ease of understanding. Simplicity (14 of 20, 70%) and transparency (9 of 20, 45%) were the most frequently mentioned factors in determining how well a PE model is received. Transparency describes the ability of the end user to "see through" the design of the model and easily understand how the model reaches its conclusions. Being able to clearly see what is happening "behind the scenes" of the model (e.g., description of the calculations and methods, identification of all of the assumptions and the data elements that were included) was thought to be of great importance to decision makers.
Keeping the model simple, while maintaining scientific quality was a challenge described by participants. One respondent stated that it was more valid and effective to develop a complex model that could be explained well than to oversimplify the disease represented by the model. An overall theme that resonated from these interviews was that even when PE models are well received, they are not necessarily fully understood by clients.
Ability to customize the model. Nearly all models presented by study participants (19 of 20, 95%) included the ability to modify variables to tailor for specific customers. The ability to customize models was thought to be important for the purpose of generalizability across practice settings. Although some variables were thought to be inappropriate to alter, several participants (5 of 20, 25%) noted that their models do allow all variables to be modified, regardless of whether the change makes sense. Increased credibility of the model was said to justify these modifications and add to the robustness of the model. Two participants said that for those models that cannot be customized, the use of extensive sensitivity analyses (e.g., tornado diagrams) was an important means by which to compensate for the lack of customization. Although all participants agreed that the ability to customize PE models is helpful, seven participants noted difficulty in actually obtaining customized information from clients. In many cases, participants noted that the required data were unavailable or inaccessible to the clients that they typically interact with.
Presenter credibility and training. Presenter credibility was also considered to be an important factor in delivering a well-received PE model. The presenter must understand all aspects of the model and be able to effectively answer questions in order to gain the attention and respect of the audience. Understanding what is wanted and needed by decision makers was also described as a key factor in the presentation of a wellreceived model.
Model content and format. Nearly all participants (19 of 20, 95%) said that referencing cost information and citing all sources of information improves the credibility of the model. Three participants noted that the face validity of the model is also an important factor. Similarly, sensitivity analysis was mentioned by 2 participants as important in determining how well a model was received.
There was no consensus regarding which model format was most effective. Seven participants (35%) reported that decision makers prefer spreadsheet and budget impact models most often and regression models least often. However, those respondents with extensive experience presenting regression models (6 of 20, 30%) felt that they were very well received by decision makers. Cost-minimization models, cost-of-illness models, and cost-offset models (3 of 20, 15%) were also included in this category of "best received" formats. Two participants (10%) specif-ically mentioned that Markov models were not well received because of the lack of understanding associated with this modeling technique.
There were also different opinions regarding the type of model interface that is best received by decision makers. Most participants (18 of 20, 90%) commented that "pop-up" windows that detail source information, definitions, or background information were helpful to both the presenter and the end user. Five participants (25%) indicated that models with a visually pleasing front end (referred to by one respondent as a "glitzy interface") improved the usefulness of the model because of its user-friendly presentation. Others stated that, in their experience, a "barebones" spreadsheet was best and that a "fancy front" was not well received because of skepticism caused by its apparent "black box." Participants reported that results of PE models were shared with decision makers via laptop and slide presentations as well as published articles of the model results. The format by which these presentations were delivered was said to be dependent on the individual product area and the audience to which the model was presented.
Availability of reprint. All participants agreed that having a publication of results derived from a PE model available for distribution at the time that the model was presented increases the credibility of the model itself. Four respondents (20%) noted that models were better received compared to publications because of their interactive nature. However, 16 of 20 participants (80%) indicated that the combination of these formats was most effective for demonstrating the validity of the model while allowing decision makers to actively apply the model in their own practice setting.
Therapeutic area. There was no consensus regarding which therapeutic areas were best suited for model presentation. Nine participants (45%) noted that models work well for chronic diseases because of their ongoing cost structure and the ability of an intervention to have an impact over time. Individual respondents stated that they thought models were helpful in therapeutic areas with well-accepted end points; very expensive products, where head-to-head comparisons were lacking; and in instances where products are differentiated only by cost. One participant mentioned that for models to be effective, there need to be competitive products available in the market, especially when trying to create awareness or treatment demand for conditions that may be under-treated. For example, practitioners may still view obesity as a "lifestyle" issue rather than a disease that should be treated with medication. In such cases, PE models may only be effective if competing products were being compared.
Barriers to using pharmacoeconomic models. Participants identified several barriers to the effective use of models. Model complexity was stated by 8 respondents (40%) as a barrier to full acceptance of the model. Several participants (7 of 20, 35%) identified skepticism surrounding model assumptions as a barrier. Five participants (25%) perceived that their customers felt an industry bias existed because of industry funding and/or the development of models by pharmaceutical companies. Lack of model transparency was also mentioned as a barrier by 4 participants (20%).
Leaving the model with the decision maker. The majority of participants (16 of 20, 80%) stated that electronic copies of the models were not provided to their customers. Reasons for not leaving the model behind included company policy, legal implications, FDA restrictions, competition, and the proprietary nature of the model. Another reason models were not provided was described as a fear that decision makers may misuse the model (via misunderstanding the assumptions or model structure) and make inaccurate conclusions based on their own manipulation of the model. Respondents agreed that decision makers seldom have time to "play" with the model, even when they were left behind. One participant added that this practice (i.e., leaving the model behind) may diminish the role of the presenter of PE information. Pending publication of model results was also described as a reason why PE models may not be distributed to the end users.
Few participants (4 of 20, 20%) offered to leave the company' s model with the customer audience. If a customer requested that a model be provided, they may be asked to sign confidentiality agreements to account for issues associated with competition (e.g., keeping the model out of the competitor' s hands) and avoiding lawsuits by competitors who may feel that their own product was inadequately represented. Even those participants who reported leaving models with their customers to facilitate model utility maintained that the models were unlikely to be used by decision makers in the absence of the presenter (i.e., pharmaceutical company representative).
Statistical knowledge of decision makers. Most participants (18 of 20, 90%) agreed that there was wide variation in the statistical knowledge of clients who attend their presentations. Nine participants (45%) noted that, overall, clients did not have a strong understanding of statistical methods. It was agreed that statistical issues (e.g., interpretation of P-values, confidence intervals, and odds ratios) are rarely the focus of the model or the presentation; however, "pop-up" windows were thought to be helpful in some situations where basic statistical concepts could be illustrated. For those presenting regression models, a strong background and explanation of statistics was considered more important.
Use of Academy of Managed Care Pharmacy (AMCP) formulary guidelines. Our study was conducted in the fall of 2002. At that time, the AMCP Format for Formulary Submissions had been introduced into the marketplace. 10 When asked about the impact of the recent formulary submission guidelines recommended by AMCP, 11 participants (55%) felt it was too soon to say what kind of impact they will have on managed care organizations or on the pharmaceutical industry. Twelve participants (60%) had no or few requests to follow the AMCP Format for Formulary Submissions to date. These respondents felt the number of requests was growing, particu-larly in state Medicaid programs and in the northwest region of the United States.
None of the participants discussed the unique benefit of the recommendation in the AMCP Format to build a model specific to a health plan. Five participants (25%) mentioned that the guidelines may be helpful in an ideal world but were contrary to the environment in which the pharmaceutical industry operated. Specifically, AMCP Format expectations for the availability of "quality" PE data at time of launch (or soon after) were said to be unrealistic. Respondents agreed that, although the AMCP Format guidelines have raised awareness of PE models, decision makers were often not equipped to evaluate them, thereby negating their effectiveness in certain settings.
Five respondents (25%) noted that some customers have requested the AMCP Format for Formulary Submissions for aspects other than PE models (i.e., clinical efficacy and safety information). One participant noted that the guidelines were useful for presenting and standardizing information. Two respondents (10%) commented on disadvantages of the guidelines, including the fact that the respondents felt the guidelines did not define what constituted a "good" model. Another participant thought that the guidelines were onerous and questioned their clinical or economic relevance. As the adoption of the AMCP dossier model (AMCP Format for Formulary Submissions) expands, the perceptions of its usefulness in practice should be reevaluated.
From an international perspective, concerns were expressed by respondents that the AMCP Format for Formulary Submissions does not apply to other countries because of the unique managed care environment in the United States. However, one respondent suggested that pharmacoeconomics might be more important in other countries because of limited government health care budgets and a strong interest in balancing health outcomes and cost issues.
Effectiveness of pharmacoeconomic models. Nearly all participants (19 of 20, 95%) agreed that PE models have had an impact on decision making for their clients. However, 14 respondents (70%) mentioned that models were just one piece of an entire package necessary for decision making by their customers. These participants mentioned that PE models had contributed either to formulary adoptions, priority or preferred formulary status, shifts in tier placement, maintaining formulary listing, or reimbursement without formulary status. There was no consensus regarding the type of model (e.g., spreadsheet, decision analysis, regression models) having the greatest impact on decision making. Instead, a combination of multiple factors discussed above was thought to contribute to the acceptability and effectiveness of the PE models presented. One participant mentioned that although PE models had minimal impact on HIV/AIDS products (i.e., products were routinely added to the formulary regardless of PE information), they had been helpful for estimating budget needs for health plans.
Recommendations for improvements.
Participants were asked what changes could be made to improve the effectiveness of the PE models they presented. The most common response (7 of 20, 35%) was to customize the model (i.e., the ability to adapt the model to include data from specific organizations). Further, participants said that models could be improved if clients would routinely provide data for model customization.
Outlining the data elements needed from clients in advance of the presentation was recommended. This approach was thought to encourage clients to participate in the customization of PE models in an effort to make information more applicable to their organization. Six respondents (30%) recommended simplifying the models, making them more user friendly (e.g., easy to manipulate and interpret). Another theme mentioned by 5 respondents (25%) was to make models more transparent to decision makers. Gathering additional data during model development in order to decrease the number of assumptions necessary was also recommended. One participant mentioned that a societal perspective, in which all possible costs and benefits were considered regardless of who the payer is, should be avoided. A more focused payer perspective was thought to be most relevant to the decision makers with whom study participants interact. One participant noted that having head-to-head comparative models, including overall health care costs, would improve their effectiveness. Having the model developed by an academician with whom the decision maker was familiar was also recommended to address issues of industry bias. Survey participants also stated that having a publication to distribute at the time of model presentation would improve the model' s effectiveness. It was also thought to be of value to understand what the customer wants to see in the model prior to developing it.

■■ Discussion
Studies focused on PE research have been conducted by a number of stakeholders. Researchers in academia, industry, managed care organizations, hospitals, and government have each contributed to this literature. In 2000, Hill and colleagues reported on problems associated with interpreting PE analyses. 11 These problems were revealed through a comprehensive review of submissions to the Australian Pharmaceutical Benefits Scheme Department of Health and Aged Care (DHAC) between 1994 and 1997. 11 These authors concluded that the resources required to fully evaluate PE analyses are beyond the capacity of many organizations and peer-reviewed journals. Looking specifically at PE models, Hill and colleagues found several problems associated with technical aspects of the models (e.g., discounting costs but not benefits, failing to appropriately relate costs and outcomes, and uncertainties arising from extrapolating short-term benefits). 11 In addition, unsubstantiated assumptions and cost estimates were criticized by the DHAC evaluators. Hill and colleagues also stated that the models were not transparent in their calculation of cost-effectiveness.
The current study took on a different perspective compared to the study by Hill and colleagues; however, the primary goal of identifying problems associated with PE models was the same. In our study, individuals who presented PE models to clients were asked to self-evaluate the usefulness and effectiveness of that information tool and to offer recommendations for how PE models can be improved. This is an important step in developing future PE models, especially since it is apparent that the use of economic analyses in managed care decision making across the United States is likely to expand. The responsibility to understand and appropriately evaluate these models will fall on health plan decision makers. Developers of these models (e.g., the pharmaceutical industry), however, must also assume responsibility for providing objective and accurate analyses. Several studies have been conducted evaluating how decision makers are actually using PE data. Previous research reported that it was difficult to locate examples where PE data constituted the primary end point by which drug policy decisions were made (e.g., adding a product to formulary). 12,13 In contrast to previous literature, our study suggests that PE models are perceived by pharmaceutical manufacturers to be useful in influencing drug policy decisions. 1,3,[5][6][7]9,[12][13][14] This appears to be a growing trend that can be useful to both the industry and health care organizations. Guidelines, such as those recommended by AMCP, are further encouraging the use of modeling techniques for the evaluation of pharmaceuticals and may receive increased attention over the next few years.
Numerous barriers have been suggested as to why PE studies have not played a larger role in drug policy decisions to date. 1,6,12,14 Limitations of the usefulness of PE information found in this study were similar to those cited in previous literature. 6,7,9 The most important barriers were skepticism of a "black box" model design, credibility of model assumptions, and perceived or actual biases in the model results. Since government funding for these types of studies is uncommon, the pharmaceutical industry has funded the majority of this research. 5 Participants in this study continued to perceive that leaders in managed care organizations feel uncomfortable with and untrusting of the potential bias that this funding source may introduce. This study supports the idea that model credibility is enhanced with scientific soundness of the model, transparency of model specifications and resource unit costs, the ability to customize the model, and involvement of nonbiased third-party researchers in the development of models. In addition, publication of the model in peer-reviewed journals may enhance credibility.
Issues such as relevance and generalizability of PE studies may be limitations for some health systems. The timeliness of some studies may be a concern as well since much of the PE data collected are not available until after a product has been introduced into the market place. Inadequate understanding of PE methods may also be an important barrier for some decision makers. Participants in this study suggested that educating the end user may be an important step in enhancing the effectiveness of PE information, especially with respect to model-build-ing. To strengthen the appropriate use of models, many health plans may benefit from enhanced training of staff in developing and evaluating PE models. Significant education and training is needed across a broad range of health care professionals. Although most decision makers have received training to evaluate clinical trials, few are familiar with modeling techniques commonly used in PE research. Decision analysis will likely become more commonly used by health plans given the expansion of software systems that can easily be used on personal computers. Training opportunities will speed the adoption of these evaluation tools within health plans. With the increasing use of databases within managed care organizations, it is also anticipated that multivariate (regression) models will become more important and more prevalent in the future.
More recent studies have demonstrated that although PE data are considered important, this information remains secondary to safety and efficacy data when making drug benefit decisions. 1,6 Despite this, studies suggest that health plans are desiring additional PE data and that this information is beginning to play a larger role in health policy decision making. 6,9 The PE model may not be the most important piece of data; however, models have demonstrated utility in combining important data that is useful to decision makers. All participants in our study believed that PE models were effective in promoting informed decision making. The majority of respondents had at least one experience where a PE model played a pivotal role in optimizing the formulary positioning of a product.
By customizing PE models and pursuing collaborations with academia, perhaps some of the perceived industry bias can be avoided and other barriers to using PE information can be overcome. As the role of formulary submission guidelines continue to expand, the usefulness and necessity of quality PE models will likely grow in parallel. Based on the findings of this study, the researchers compiled a list of recommendations for improving the usefulness of PE models (Table 2).
Prior to this study, there was a gap in the literature with respect to the effectiveness and application of specific PE modeling techniques. In our study, there was no single model format (e.g., spreadsheet model, decision analysis, regression analysis) that was regarded as the most effective model type. Although many respondents said simple spreadsheet models are most effective, complex regression models were reported to be very successful with well-trained presenters. Well-designed, scientifically sound regression models were also reported to be very effective by several respondents. It is difficult to compare these results to existing literature, as previous studies have not examined the impact of various model designs.

■■ Limitations
The results of our study were derived from a small convenience sample of pharmaceutical company representatives. While the sample was developed to reflect a wide range of companies across the United States, the results are not necessarily general-izable to all people involved in the dissemination of PE research to managed care organizations (MCOs) and other recipients. In addition, there may be a social desirability bias since some respondents may have answered questions in a manner they believed would be preferred or anticipated by the interviewers. Furthermore, health policy decision makers (e.g., from MCOs) may have had different perceptions than the participants in this study. Because of the qualitative study design and open-ended nature of the interview questions, participants did not have an opportunity to agree or disagree with all issues. Comments made by individuals, however, can still be insightful and generate ideas for future research.

■■ Conclusion
Research scientists from the pharmaceutical industry who participated in this investigation suggested that PE models are useful in influencing drug policy decisions. Nineteen of the 20 respondents could provide examples where a PE model contributed to a health policy or medication policy decision (e.g., drug coverage, formulary position). No single model type surfaced as the single most effective means by which to communicate PE findings. Additional user training will be an important component for optimizing the usefulness of PE models. This study provides several recommendations for enhancing the effectiveness of PE models such as ensuring that model assumptions and calculations are transparent, creating and encouraging model customization by working directly with managed care decision makers, and incorporating head-to-head comparisons wherever possible. Great opportunities exist for further research in the area of effectiveness and utility of PE models in decision support for drug formulary content, placement, and coverage. The uptake of health plans employing the AMCP Format for Formulary Submissions process 15 may provide the impetus for increased collaboration among academia, the drug industry, and MCOs in maximizing the usefulness of PE models in making formulary coverage and placement decisions.