Ability and Use of Comparative Effectiveness Research by P&T Committee Members and Support Staff: A 1-Year Follow-up

BACKGROUND: In recent years, comparative effectiveness tools and methods have evolved to assist health care decision makers in identifying optimal therapies. In-person training programs on comparative effectiveness research may be helpful in understanding and applying this information. OBJECTIVE: To provide a follow-up assessment of the use of comparative effectiveness research (CER) in the pharmacy and therapeutics (P&T) committee decision-making process, using information collected from participants 1 year after attending a live continuing education program, in which participants were taught about CER designs and how to access available CER resources through the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program. METHODS: A retrospective, cross-sectional questionnaire was developed and mailed to 2 groups of individuals: CER workshop attendees and interested nonattendees (expressing an interest in attending a workshop but did not attend for unknown reasons). The questionnaire asked respondents to indicate personal and organizational use of CER in the decision-making process. Participants were asked to indicate whether their knowledge, ability, and use of CER studies increased since participating in the program. Data were analyzed using nonparametric tests to compare the responses of attendees and nonattendees, as well as overall reliability of the instrument. RESULTS: A total of 164 respondents completed the questionnaire (63 attendees and 101 nonattendees; overall response rate = 44%). The majority of respondents were pharmacists (n = 157, 95.7%) and were affiliated with a hospital (n = 106, 64.6%). Proportions of attendees and nonattendees differed significantly in the use of EHC research reviews/reports (45% and 28%, respectively; P = 0.02) and EHC executive summaries of research reviews/reports (48% and 29%, respectively; P = 0.01). At 1-year follow-up, the majority of attendees reported an increase (“somewhat” or “very much”) in knowledge of CER (91.5%), ability to use CER (83.0%), and use of CER studies (58.7%). CONCLUSIONS: Health professionals attending a continuing education CER program reported higher use of EHC CER materials compared with nonattendees. Additionally, attendees reported increased use of CER in clinical decision making. A continuing education program such as this may provide an effective avenue for introducing CER methods and resources to the P&T committee and clinical decision-making processes.

P harmacy and therapeutics (P&T) committees are composed of health care professionals who evaluate medications for use in the formularies of their health systems. Members of P&T committees may include physicians, pharmacists, nurses, other health care professionals, and occasionally consumer representatives. P&T committees evaluate published and unpublished research, including comparative effectiveness research (CER), to determine if medications should be included in the formulary of their facilities or health plans. Additionally, P&T committees may develop medication-use policies to address (a) how medications are used; (b) any monitoring guidelines for the medications; and (c) methods for ensuring safe prescribing, administration, and monitoring of medications. When reviewing formulary changes-either adding new medications or removing previously approved medications-P&T committees may rely on clinical experts to provide insight • Comparative effectiveness research (CER) is a research method that provides real-world evidence for improving quality of care and achieving better patient outcomes. • Clinical trials, clinical guidelines, and pharmacoeconomic studies have been shown to influence decision making among pharmacy and therapeutics (P&T) committees. • The Agency for Healthcare Research and Quality developed the Effective Health Care (EHC) Program to provide tools and resources to help improve health care decision making. These tools include published and unpublished scientific evidence on CER.

What is already known about this subject
• A high percentage of respondents indicated that they were using CER research, such as clinical practice guidelines and primary literature involving systematic reviews, in P&T decision-making processes. • After 1 year, respondents who attended a CER workshop indicated that they were more likely to use the materials provided by the EHC Program and that their ability and use of CER increased after the workshop. • Reported use of CER studies by all respondents and their organizations have not increased.

What this study adds
ability regarding the use of CER in decision making after participating in a tailored education program and found that the program increased attendees' knowledge and self-perceived ability to evaluation CER materials. 11 The goal of this program evaluation was to assess the longerterm impact of a CER training program by comparing the use of CER materials in P&T committee decision-making processes by attendees and nonattendees. The data for this evaluation were collected as part of a 1-year follow-up to a live continuing education program, 11 where P&T committee members and support staff learned about CER methods and how to use EHC Program resources in making drug-use policy decisions.

■■ Methods Design
This prospective cohort study was conducted to evaluate the effectiveness of a 4-hour, live, educational CER program designed specifically for P&T committees and their support staff. Support staff members were individuals who were not members of P&T committees but who supported the committees' activities (e.g., a clinical expert). This cross-sectional design compared the self-reported use of CER evidence in clinical decision making by attendees and nonattendees 1 year after attending the CER training program. The focus of this article is on the 1-year follow-up; more detailed information about the program design is described elsewhere. 11

Participants
Participants were recruited from 3 collaborating health care organizations: (1) the Academy of Managed Care Pharmacy (AMCP); (2) the American Society of Health-System Pharmacists (ASHP); and (3) the Indian Health Service (IHS). To participate in the CER program, participants met these eligibility criteria: (a) involvement in the P&T process at a managed care, health system, or institutional care organization and (b) willingness to attend a 4-hour, live, educational CER session. The CER programs were held in conjunction with these professional meetings: AMCP's annual and fall educational meetings; ASHP's midyear clinical and summer meetings; and IHS's annual spring P&T committee meeting. Attendees included individuals who participated in the live program; nonattendees were eligible and invited to attend but did not ultimately participate.
Approximately 1 year after completing the program, attendees and nonattendees were mailed a paper-based questionnaire packet. Attendees and nonattendees were invited to complete the questionnaire, which assessed the extent that they personally used CER reviews and guides and the extent that these resources were used by their organizations over the past year. The packet contained the following items: (a) a personalized letter from the principal investigator printed on official letterhead; (b) the questionnaire; (c) a self-addressed, stamped return envelope; and (d) a small token of appreciation ($1 bill). After the first mailing, attendees and nonattendees received a reminder and knowledge regarding medications and their use within a health system. Committee members and clinical experts review evidence-based research reports, including clinical trials, postmarketing reports, and other medical literature, to inform safe and cost-effective decision making. According to previous reports, published studies containing stronger evidence (e.g., randomized controlled trials and meta-analyses) are commonly used in making formulary decisions. 1 One published report found that clinical trials, clinical guidelines, and pharmacoeconomic studies were the most relevant factors that influenced decision making among P&T committees. 2 Committees may also use internal data on prescribing patterns and outcome information (e.g., drug utilization reviews and evaluations) when making formulary decisions, in order to gain an understanding of how a medication is being used within their organizations. 1 The rationale for using CER is to support better decision making regarding health care interventions 3 and provide additional information to decision makers, health care providers, and patients by comparing different methods used to diagnose, treat, prevent, or monitor health conditions. CER provides real-world evidence for improving quality of care and achieving better patient outcomes. 4 To accomplish these tasks, it is important that CER is clear, concise, and unbiased so that clinicians and policymakers can generate informed decisions. [5][6] In addition to P&T committees, many other health care groups may use CER to help make better health care decisions. The Agency for Healthcare Research and Quality (AHRQ) developed the Effective Health Care (EHC) Program to provide tools and resources to assist health care providers, practitioners, policymakers, and consumers in improved decision making. 7 Additionally, AHRQ supports research on CER by synthesizing published and unpublished scientific evidence, generating new evidence and tools for clinicians, and compiling the results into documents for different stakeholders. While these resources are available at no cost through the EHC Program website (http://www.effectivehealthcare.ahrq.gov/index.cfm) and elsewhere, it is unclear whether health care providers use these tools when making formulary decisions.
Previous research has found that some hospital P&T committees use pharmacoeconomic data in their formulary decision-making processes. 8 However, committee members may be reluctant to use these types of research data, given their potential limited understanding of the respective methodology and terminology. 8 Additionally, relevant information in the published literature may be lacking or its usefulness may be limited. 2,9 To date, few studies have reported on the use of CER and/or EHC Program materials by P&T committees. Villa et al. (2013) reported that two thirds of focus group participants knew about the AHRQ EHC Program CER reviews, 10 yet only one quarter of the participants actually use the reviews within their organizations. 8 Another study by Tang et al. (2014) assessed P&T committee members' knowledge, attitudes, and postcard in the mail. Nonrespondents received a second (reminder) packet approximately 2 weeks after the initial packet was mailed. A reminder postcard was sent approximately 2 weeks after the reminder packet. This study is considered to be a program evaluation, so informed consent was not required. The University of Arizona Human Subjects Protection program reviewed and approved the project.

Questionnaire
The questionnaire was specifically designed by the research team to assess the extent of the impact that attending the CER program had on adoption and use of CER reviews and guides within health care organizations. The items of interest included use of the AHRQ CER materials and use of other CER materials over the past year by the participants and their organizations ( Table 1); perceptions of the usefulness of the CER training for attendees of the workshop ( Figure 1); perceptions of the usefulness of the EHC materials by nonattendees ( Figure 2); actions taken to routinely receive information about CER reports ( Table 2); and self-reported changes in behaviors by attendees ( Table 3).
The questionnaire was composed of 44 items for participants (attendees) and 37 items for nonparticipants (nonattendees). Thirty-two items used a dichotomous (yes/no) format. One item used a 5-point rating scale ("not at all useful," "not very useful," "somewhat useful," "very useful," and "I haven't used the materials"). Four open-ended questions allowed participants to provide further information on how EHC and other CER materials were used in making drug use policy decisions (if applicable). The final 7 items were designed for attendees of the live  program and assessed the CER workshop, using a 4-point rating scale ("not at all," "very little," "somewhat," and "very much").

Data Analysis
Initially, a database was created in Excel for data entry and analysis purposes. Chi-square and Fisher's exact tests were used to compare differences in dichotomous responses between (a) attendees and nonattendees and (b) the organizations represented by the attendees and nonattendees. The Wilcoxon rank sum test was conducted for the ordinal scale measuring usefulness. An a priori P value of 0.05 was used to assess significance. These nonparametric tests were conducted using STATA SE, version 12.1 (StataCorp, College Station, TX). Cronbach's alpha reliability measure was calculated to determine internal reliability of the questionnaire.

■■ Results Participants
A total of 373 paper-based questionnaires were mailed to attendees and nonattendees; 161 questionnaires were mailed to all CER training program attendees; and 212 questionnaires were mailed to individuals who were eligible for inclusion but did not attend 1 of the training programs. Of these, 167 (45%) questionnaires were completed and returned-64 were from attendees (40%), and 103 were from nonattendees (49%). Three questionnaires (1 attendee and 2 nonattendees) were excluded from data analysis because of incomplete data. The majority of respondents were affiliated with hospitals (n = 106, 64.6%), and almost all respondents were pharmacists (n = 157, 95.7%).
Additionally, almost half of the respondents were employed at a private, not-for-profit organization (n = 79, 48.2%), and one third (n = 49, 29.9%) of the respondents were located in the Midwest and were aged 35-45 years (n = 48, 29.3%). In general, the demographic characteristics were similar between the attendee respondents and the nonattendee respondents (P > 0.05). However, there was a significant difference between attendee and nonattendee respondents regarding their roles in their P&T committees. Attendees were more likely to be the chair of their P&T committees compared with nonattendees (P = 0.02). Also, nonattendees were more likely to be affiliated with hospitals, whereas attendees were more likely to be affiliated with managed care or pharmacy benefit management plans (P = 0.04). See Table 4 and Table 5 for demographic information. The Cronbach's alpha reliability for the questionnaire was 0.85.

Use of CER Materials
Respondents were asked to report their personal use or use by their organizations of CER materials when making drug-use policy decisions over the past year. CER materials included those specifically designed and posted by the EHC Program, as well as other non-EHC materials. There were significant differences between attendees and nonattendees regarding the use of specific EHC Program materials (Table 1). Twentyeight (45%) of the 63 attendees reported using EHC research reviews/reports compared with 28 of the 101 nonattendees (28%, P = 0.02). Additionally, a significantly higher proportion of attendees used the EHC executive summaries of research reviews/reports (n = 30, 47.6%) compared with nonattendees (n = 29, 28.7%, P = 0.01). One attendee respondent indicated that "clinical pharmacists can use them to provide reliable  evidence when making recommendations for medication therapy." Another attendee respondent indicated the use of EHC materials, especially "cumulative and comprehensive reports, which helped to make final protocol decisions when the P&T committee has been on the fence regarding criteria and/or coverage for many drugs." There were no significant differences in the proportion of attendees and nonattendees regarding their use of other CER materials. The highest reported use of other CER materials by individuals included clinical practice guidelines (73% of attendees and 68.3% of nonattendees, P = 0.52) and primary literature involving systematic reviews with or without meta-analyses (71.4% of attendees and 60.4% of nonattendees, P = 0.15). One attendee respondent stated that his/her organization used CER materials "as part of our medication reconciliation and for our process on the transition of care from acute care to home." Another attendee respondent indicated that EHC materials are used to "review information for certain formulary decisions to see if our organization's recommendations are sound." When asked to rate the usefulness of the EHC materials, 89% of attendees (n = 46) and 93% of nonattendees (n = 38) felt that they were either very useful or somewhat useful (Figure 1 and Figure 2). Overall, there were no significant differences between attendees and nonattendees on the usefulness of the EHC materials (P = 0.90). However, there was a significant difference in the number of attendees who reported searching for EHC materials online (n = 40, 63.5%) compared with nonattendees (n = 28, 27.7%, P < 0.001; Table 3).

Attendee Self-Reported Behavior 1-Year After Workshop
In addition to the questions previously mentioned, attendees were asked to report on the impact of the CER workshop after 1 year (Table 3). Attendees reported that the workshop "very much" increased their knowledge of CER (n = 22, 37.3%). However, the workshop did not appear to have overwhelmingly increased the use of CER by their organizations (n = 17, 29.3%, reported not at all). Finally, nearly 75% of attendees responded that the workshop did not have an impact on the formulary   Demographic Characteristics of Respondents care providers, students are taught to review evidence-based medicine resources, including practice guidelines and systematic reviews, when treating patients. 12,13 Thus, it is feasible to assume that respondents are most comfortable with finding and using guidelines and systematic reviews because they have used it frequently in the past. This finding is encouraging given that the Institute of Medicine recommends that health care professionals utilize CER when making policy decisions. The results of this study appear to support that they are doing so. However, about 48% of attendee respondents indicated that the CER workshop had "slightly" or "very little" impact on increasing their use or the use by their organizations of CER.
Previous research has shown that continuing education programs are effective in enhancing knowledge, yet they have little effect on changing practice activities. 14 Additionally, any acquired knowledge or skills resulting from attending a continuing education program do not necessarily alter practice behaviors. 15 However, educational research has provided evidence to support the notion that repeated presentation of information may provide an avenue for learning and behavior change. 16 Because the majority of respondents were clinical experts within P&T committees, and not members, committee members may not be aware of the studies included in the drug monograph. According to focus group participants, a potential reason for respondents or their organizations not using CER studies may be because of the difficulty of finding relevant information. 10 The authors noted that a little more than half of focus group members were aware of reviews published by AHRQ, but only one quarter of participants had personally used them. 6 Based on these results, feedback was provided to AHRQ on the difficulty of finding EHC Program materials and other CER studies. The agency has since updated its website to make the EHC Program resources easier to find within the website, as well as linking EHC Program studies within MEDLINE.
Additionally, a 2009 report by Academy Health found that CER studies represented a small proportion of current health research, based on searches from ClinicalTrials.gov and Health Services Research Projects in Progress (HSRProj). 5 Less than 5% of new studies added to these databases involve CER. Therefore, the difficulty in finding CER materials and the low volume of CER reported may be a factor influencing the use of CER as part of P&T committee reviews. When practitioners do locate CER materials, the reviews may not include all of the medications or devices of interest. 10 In fact, previous research shows that EHC CER reviews are most useful for therapeutic class reviews. 9 However, a study of hospital pharmacists reported that only 56.2% of P&T committees complete a review of therapeutic categories (i.e., class reviews), while 83.2% of hospitals have their process (n = 44, 74.6%). However, 1 respondent stated that "the physician and I on the P&T committee frequently research EHC to solidify policy criteria and coverage decisions."

■■ Discussion
Respondents who attended the live program on CER were significantly more likely to report using EHC research reviews/ reports and research review/report executive summaries 1 year later compared with nonattendees. It is feasible to assume that these differences are attributable to exposure to the EHC Program during the workshop. Attendees received information on the EHC Program that included lists and explanations of the depth and breadth of materials available and how to access this information. Attendees agreed that the CER workshop increased their knowledge, ability to use, use, and ability to find CER research reports, with over 70% responding that the workshop increased these behaviors "very much" or "somewhat." These results are consistent with those from Tang et al. who evaluated an earlier phase of the project, 11 which found that attendees of the same CER workshop reported significant improvements in their perceived knowledge of and ability to use CER methods immediately after the workshop.
CER materials developed by sources other than the EHC Program were used by over half of all respondents, including attendees and nonattendees. Respondents reported that they mostly used clinical practice guidelines, while systematic reviews with or without meta-analyses were second most commonly used. This result is not surprising given that pharmacists and physicians are familiar with practice guidelines and systematic reviews as a result of their health professional education. In the training and education of health

Characteristics of Respondents' Organizations and P&T Committee Activities
P&T committees review minimal duplication of therapeutically equivalent products. 17 Thus, EHC CER materials may be a good start for P&T committee reviews but may not contain all of the required information for the committee; for example, EHC CER reviews do not incorporate new drugs or technologies compared with the current standard of treatment.

Limitations
This program evaluation was based on a cross-sectional questionnaire mailed to attendees and nonattendees 1 year after training. Actual practice and use of CER materials may differ from that which has been reported because of recall and response bias. This study reflects those individuals who expressed an interest in learning more about CER. Furthermore, nonattendees were drawn from a convenience sample of health care professionals who expressed an interest but did not attend the program. Therefore, some selection bias may exist within this sample population. Individuals who expressed an interest in the program may have been more likely to be aware of or seek EHC program resources or other CER materials. Based on the purpose of this research, it was decided to focus on this group in order to determine the impact of the CER workshop among individuals who may have been more likely to seek CER information and research (attendees and interested nonattendees). Additionally, there may be some selection bias among professionals who were able to attend professional conferences. Professionals who attend these conferences may be more interested in learning about new medications or learning about research on these medications.

■■ Conclusions
At the 1-year follow up, CER workshop attendees were using materials from the EHC Program significantly more than nonattendees. These materials included research reviews/reports and executive summaries of research reviews/reports. Other types of CER research, such as clinical practice guidelines and primary literature involving systematic reviews, were being used by a high percentage of attendees and nonattendees. At follow-up, attendees indicated that their knowledge, ability, and use of CER increased after attending the workshop. However, reported use of CER studies by the attendees and their organizations did not increase. Future research is warranted to investigate the reasons behind this apparent lack of use of CER materials. Additionally, further study is needed to identify tools, instruments, and resources to assist P&T committee members in making better use of these studies for clinical decision making.