Quality Assessment of Medical Apps that Target Medication-Related Problems

BACKGROUND: The advent of smartphones has enabled a plethora of medical apps for disease management. As of 2012, there are 40,000 health care-related mobile apps available in the market. Since most of these medical apps do not go through any stringent quality assessment, there is a risk of consumers being misinformed or misled by unreliable information. In this regard, apps that target medication-related problems (MRPs) are not an exception. There is little information on what constitutes quality in apps that target MRPs and how good the existing apps are. OBJECTIVE: To develop a quality assessment tool for evaluating apps that target MRPs and assess the quality of such apps available in the major mobile app stores (iTunes and Google Play). METHODS: The top 100 free and paid apps in the medical categories of iTunes and Google Play stores (total of 400 apps) were screened for inclusion in the final analysis. English language apps that targeted MRPs were downloaded on test devices to evaluate their quality. Apps intended for clinicians, patients, or both were eligible for evaluation. The quality assessment tool consisted of 4 sections (appropriateness, reliability, usability, privacy), which determined the overall quality of the apps. Apps that fulfilled the inclusion criteria were classified based on the presence of any 1 or more of the 5 features considered important for apps targeting MRPs (monitoring, interaction checker, dose calculator, medication information, medication record). Descriptive statistics and Mann-Whitney tests were used for analysis. RESULTS: Final analysis was based on 59 apps that fulfilled the study inclusion criteria. Apps with interaction checker (66.9%) and monitoring features (54.8%) had the highest and lowest overall qualities. Paid apps generally scored higher for usability than free apps (P = 0.006) but lower for privacy (P = 0.003). Half of the interaction checker apps were unable to detect interactions with herbal medications. Blood pressure and heart rate monitoring apps had the highest overall quality scores (67.7%), while apps that monitored visual, hearing, and temperature changes scored the lowest (35.5%). CONCLUSIONS: A quality assessment tool for evaluating medical apps targeting MRPs has been developed. Clinicians can use this tool to guide their assessments of medical apps that are appropriate for use in the health care setting. Although potentially useful apps were identified, many apps were found to have deficiencies in quality, among which was poor reliability scores for most of the apps. Continued assessments of the quality of apps targeting MRPs are recommended to ensure their usefulness for clinicians and patients.

S martphones and rapid development of mobile health care apps, together with advanced features such as Bluetooth, accelerometers, and 3G/4G networks, are currently supporting patient care activities in many health systems. [1][2][3] The use of such mobile technologies in health care provision has paved the way for a new domain known as mobile health, or mHealth. 1,2 Eighty-one percent of U.S. physicians own a smartphone and are incorporating their use in practice. [4][5][6] Moreover, 56% of the U.S. adult population was reported to own smartphone in 2013. 7 Mobile subscriptions within the Asia-Pacific region account for more than half of the global subscription rate with 3.5 billion subscriptions. 8 In Singapore, 75% of smartphone users utilize mobile apps, ranking Singapore fifth among 43 countries in terms of app usage. 9,10 As of 2012, there are approximately 40,000 health care apps available. 4 Thirty percent of these apps specifically target clinicians and the remaining 70% target the general consumer. 4,11 Up to 63.5% of clinicians use apps in clinical practice and 1 in 5 people with smartphones download an app with a healthrelated purpose. 12,13 Little is known about the quality of mobile apps used by clinicians and patients in health care settings. Recently, medical app reviews have discussed the use of apps for • Medication-related problems (MRPs) are events or circumstances involving medication therapy that can potentially or actually interfere with desired health outcomes, which, in turn, can lead to morbidity and mortality, as well as increased health care costs. • As of 2012, there are approximately 40,000 health care apps available, 30% of which specifically target clinicians with little focus on app quality or ability to address MRPs.

What is already known about this subject
• A quality assessment tool has been developed to help clinicians and medical app reviewers assess the quality of medical apps that target MRPs. • This study showed that apps with medication interaction checkers and monitoring features had the highest and lowest overall qualities, respectively; reliability of apps was generally poor; and paid apps had better usability but poorer privacy scores. • A recommended list of ranked apps is provided as a guide for clinicians to select or recommend an appropriate app in their clinical practices.

What this study adds
apps that target MRPs. In the context of patient care, such apps can serve as potentially useful tools for MRP prevention interventions. As such, the objective of this study was to develop a quality assessment tool to evaluate health care apps that target MRPs and provide a non-exhaustive recommended list of apps for clinicians in practice settings. The study focused on the 2 main operating systems, Android and Apple, since they account for over 90% of the market share. 22

■ Methods Creation of Quality Assessment Tool
Articles from PubMed and the iMedicalApps website were analyzed to generate the evaluation criteria for the quality assessment tool. 23 The PubMed search included a list of medication-related and mobile technology-related MeSH terms medication information. 14,15 However, there is little focus on how these apps can address medication-related problems (MRPs). MRPs are events or circumstances involving medication therapy that can potentially or actually interfere with desired health outcomes. 16,17 The Pharmaceutical Care Network Europe (PCNE) classifies MRPs as problems with treatment effectiveness, adverse drug events, increased treatment costs, and others. MRPs are associated with increased morbidity and mortality, as well as increased health care costs. 18-20 For example, it is estimated that MRPs cost the U.S. health system over USD$170 billion annually, with hospital admissions accounting for 69% of total costs. 21 MRPs can be caused by prescribing or medicationuse errors and can be prevented with proper measures in place. 17 While there are numerous health-related apps in the different app stores, it is hard to find scientific studies evaluating  Multi-feature apps n = 6 Monitoring n = 27 a Apple: n = 18 (9 free) Android: n = 9 (5 free)

Medication Interaction
Checker n = 6 a Apple: n = 2 (2 free) Android: n = 4 (4 free) Total medication apps screened across both platforms and versions, n = 400 Inclusion Criteria: English language; contains features (monitoring, medication interaction checker, dose calculator, medication information, medication record) targeting MRPs that can be evaluated by the categories within the quality evaluation tool Exclusion Criteria: non-English language; apps designed specifically for students, such as medical dictionary and simulation apps; ovulation calculator apps; pregnancy testing apps; preview version of apps that require in-app purchases; apps that require subscriptions or those that are exclusive to patients from a particular medical institution  Figure 1). The full texts of the 70 articles were reviewed to extract potentially relevant quality assessment criteria for apps targeting MRPs. The criteria extracted from the articles were classified into 2 major categories of quality as "appropriateness of the app to serve its intended function" and "general quality." The first category was based on the features of the apps that were discussed in the reviewed articles. The features of the apps were broadly categorized into monitoring, medication interaction checker, dose calculator, medication information, and medication records. Each app in the reviewed articles had 1 or more of these features. Criteria for evaluation of the quality of such features were listed under each section. Depending on the number of criteria for each feature, the scores for the quality of each feature varied. For example, the maximum score for the monitoring feature was 24, while for the medication record feature, it was 8. The general quality category included sections on the reliability, usability, and privacy of apps. The corresponding criteria were listed under each of these sections (Appendix A, available in online article). The quality assessment tool was developed in such a way that all apps were rated for their general quality out of a total of 38 points, whereas each app was assessed for the presence of features listed under the appropriateness category and corresponding criteria were used to evaluate the app (the maximum possible score in this section varied with the evaluated feature).

Selection of Apps and Process of Quality Assessment
The top paid and free medical app categories on iTunes and Google Play were searched to identify the first 100 apps in each category as ranked by the app stores themselves. The top 100 apps in each category were screened based on their title and description. Apps were included for assessment if they were available in English and contained 1 or more of the 5 features targeting MRPs ( Figure 2). All eligible apps were downloaded to test devices for final evaluation. Android apps were evaluated on a Samsung Galaxy S3 smartphone (version 4.1.2) and Apple apps were evaluated on an iPhone 4s (version 7.0.3).
Upon download to the test devices, an app was first classified based on the presence of features for the purposes of monitoring treatment effectiveness, checking medication interactions, calculating doses, providing medication information, and keeping medication records. Once the app was identified to have any 1 or more of these features, the corresponding assessment criteria under the section of "appropriateness of feature to serve its intended function" were employed to evaluate its quality. Accordingly, the maximum possible score for an app having any of the features could vary from 46 for the medication record feature to 62 for the monitoring feature. The maximum possible scores for apps with a dose calculator, medication interaction checker, and medication information features were 49, 56, and 60, respectively. App comparisons were made at the level of each feature. To facilitate comparison among apps with different features, percentages of the raw scores of the apps on the assessment tool were calculated. The overall quality of the apps was then calculated as a percentage based on the composite scores for each section (appropriateness, reliability, usability, and privacy).

Statistical Analysis
Apps were grouped according to which MRPs they could target based on their descriptions in the app stores. Descriptive statistics for each feature were computed based on the apps' feature scores on the quality assessment tool and reported as medians and interquartile ranges (IQRs). A Mann-Whitney U test was used to evaluate the differences in scores between free and paid apps and Apple and Android apps. All tests were 2-tailed and

Median General Quality Scores of Apple and Android Apps: Free and Paid Versions
P-values below 0.05 were considered statistically significant. Statistical analyses were conducted using Statistical Package for the Social Sciences (SPSS) version 21 (IBM Corp, Armonk, NY).

■■ Results
A total of 400 apps were screened for inclusion in the final evaluation based on review of their app store description pages. Only 59 (14.8%) apps fulfilled the inclusion criteria, of which 5 were available on both platforms. There were 27 apps with monitoring features, 17 apps with medication information features, 14 dose calculators, 8 apps with medication record features, and 6 medication interaction checkers. Six apps contained multiple features and were evaluated according to their respective feature sections ( Figure 2).

Assessment of Feature Scores
Monitoring. Paid apps with monitoring features had higher overall quality scores than free apps across both platforms, but the score difference for Android apps was statistically significant (65.3% vs. 53.2%, respectively; P = 0.032; Table 2). The Cancer.Net Mobile app ranked highest (79.0%), followed by Blood Pressure Diary Pro, HeartWise Blood Pressure Tracker, and iBP Blood Pressure (67.7% each; Table 3). Other than the Cancer.Net Mobile app, the top-scoring apps based on their overall quality scores were those that monitored blood pressure and heart rate (66.1%-67.7%), while apps that monitored visual, hearing, and temperature changes scored the lowest (30.6%-38.7%). The appropriateness and usability scores for the top 4 apps were similar, but Cancer.Net Mobile performed better for reliability and privacy (Appendix B, available in online article). A majority of the apps did not allow users to set target goals for physiological parameters such as blood glucose, body temperature, and heart rate (n = 20) nor provide a management plan when an abnormal measurement was recorded (n = 23), thus affecting their appropriateness scores. Only 3 apps (Blood Pressure Diary, Blood Pressure Diary Pro, and iBP Blood Pressure) allowed syncing of data from the blood pressure monitoring device to the app via Bluetooth.

Medication Interaction Checker.
All the evaluated apps in this category were free apps ( Table 2). The top app based on overall quality was Medscape (76.8%), while the Drugs.com Medication Guide scored the lowest (57.1%). The Apple version of Epocrates had a higher score than its Android counterpart due to better usability (score: 17/22 vs. 14/22, respectively; Appendix B). A majority of the apps (n = 4) were unable to provide users with a list of all possible interactions associated with a single medication, and half of the apps (n = 3) were also unable to detect interactions with herbal medications. Dose Calculator. In general, free apps scored higher for overall quality compared to paid apps ( Table 2). The highest-scoring apps based on overall quality were Medscape (75.5%), MedCalc (67.3%), and Epocrates (Apple version: 61.2%; Table 3).

Median Overall Quality and Appropriateness Scores of Apple and Android Apps: Free and Paid Versions
physical features (e.g., shape, color, imprint; n = 14). Furthermore, most apps did not provide supporting references for the content provided (n = 13).
Medication Record. Paid apps in this category were ranked higher in terms of overall quality. The top-ranked apps were Med Helper Pro Pill Reminder and Pillboxie (67.4% each), both of which were paid apps ( Table 3). The appropriateness and usability scores were similar for both apps, but Pillboxie had a higher reliability score of 6/12 in contrast to Med Helper Pro Pill Reminder, which had a higher privacy score of 4/4 (Appendix B). More than half of the apps were unable to provide medication reminders or keep track of users' medication schedules (n = 5). Most apps were also unable to alert users if an allergy-causing medication was among their active list of medications (n = 7).

■■ Discussion
According to the recently released U.S. Food and Drug Administration (FDA) guidelines, regulatory oversight is applied to mobile apps that satisfy the definition of a medical device. Depending on the level of risk the use of a mobile medical app poses to a patient's safety, the FDA may require app developers to apply for premarket notification (510k) or premarket approval. 24,25 However, a majority of the healthrelated apps in the app stores do not fall under the FDA's regulatory oversight. This can be a concern for the increasing number of consumers who download these medical apps. The quality assessment tool developed in this study is to serve as a guide for clinicians to assess the quality of the medical apps in the market before using them in their practice settings or recommending them to patients. Although non-exhaustive, this study identified mobile apps that can be potentially useful for both patients and clinicians in addressing MRPs. The study also identified important deficiencies of currently available apps targeting MRPs, which can serve as an input for future improvements.
Paid apps with the monitoring feature are of higher overall quality than their free versions, in contrast to the other evaluated app features. We postulate that the higher price of these apps may be linked to technical complexity of this feature (e.g., Bluetooth syncing with external devices, data storage, and trending capabilities) and the greater effort put in by developers. Clinicians may choose to use or recommend to patients paid versions of monitoring apps but can also consider the free versions of apps with other features.
Free apps scored higher for privacy than paid apps in this study. Our results were consistent with a recent survey that showed that free apps were more likely than paid apps to include privacy policies. 26 A reason could be that most paid apps were usually developed by individuals rather than organizations, hence they are less likely to take user's privacy into account Epocrates performed better for overall quality on Apple compared to Android platforms (61.2% vs. 57.1%) due to better usability (score: 17/22 vs. 14/22, respectively). Similarly, the Android version of Calculate by QxMD also fared better than its Apple counterpart due to slightly better usability (score: 16/22 vs. 15/22, respectively; Appendix B). Most apps in this category were unable to calculate dose requirements for antibiotics (n = 10) and did not provide any administration instructions after calculation of doses (n = 13).
Medication Information. Free apps in this category had higher overall quality and appropriateness scores compared to paid apps ( Table 2). The top-ranking apps were Medscape and Epocrates (Table 3). The Apple version of Epocrates scored better for usability than the Android version (score: 17/22 vs. 14/22, respectively), thus affecting its overall quality scores and ranking (Appendix B). Although apps that were among the top 3 ranks had similar appropriateness scores, most of them did not allow for identification of medications based on their

Top 3 Apps Within Each Feature Category Based on Overall
Quality Scores ferent mobile platforms are likely to achieve similar scores for appropriateness, reliability, usability, and privacy, suggesting that the quality of apps across other platforms is likely to be similar. It is likely possible to apply this assessment tool across other platforms with minor adjustments. Finally, the apps evaluated in this study represent a snapshot of the current medical apps that were available during the period of the study. Apps may be removed or updated with time, so the list of recommended apps may change in the future. There is a need to frequently assess the quality of apps to ensure that they remain within certain standards. It is also important to note that user preferences and experiences with the apps will determine whether they are going to continue using recommended apps. Despite the inclusion of criteria to evaluate issues of usability in the quality assessment tool, actual user experiences may have significant influence on the adoption of an app. While this study did not assess if the use of such apps could really reduce or prevent MRPs in clinical practices, future studies can possibly be conducted to determine if there is indeed a role for using medical apps that target MRPs in the health care setting.

■■ Conclusions
A quality assessment tool has been developed to assess the quality of medical apps that target MRPs. This study showed that apps with medication interaction checkers and monitoring features had the highest and lowest overall qualities, respectively. Reliability of apps was generally poor. Paid apps had better usability but poorer privacy scores. Apart from Android apps with monitoring features, the free/paid status of an app generally did not reflect its overall quality. We hope that this tool can assist clinician users and medical app reviewers assessing the quality of apps from a more objective approach. Potential developers of health care apps can also use this tool as a guide to create apps of a better quality standard for appropriate use in the health care setting. The recommended list of ranked apps provided can be utilized as a guide for clinicians to select or recommend an appropriate app in their clinical practices. during app development. 27 However, a clearly stated privacy policy on how user information is handled is important to safeguard the privacy of users, which is critical for handling confidential information in certain apps, such as medical record apps.
In general, the reliability scores of apps assessed in this study were low regardless of platform or free/paid versions. In order for apps to be current and reliable, they should be validated regularly either by quality assessment tools, such as the one in this study, or by third-party certifications. For example, Cancer.Net Mobile was the only app that was certified by the Health On the Net Foundation (http://www.hon. ch/HONcode/), thus scoring the highest (10/12, 83.3%) for its reliability. Such evaluations can help filter out useful and reliable apps from app stores.
It was interesting to note that the overall quality of the Apple and Android versions of some apps were different. For example, the Android version of Epocrates contained an additional intravenous-drip rate calculator that was not available on its Apple version. Similarly, iBP Blood Pressure on the Apple platform was able to sync with an external monitoring device for automatic transfer of readings, but not on Android. Therefore, clinicians should be aware that there might be slight differences in the functionalities of the apps when used on different platforms.
A list of recommended apps within the top 3 ranks for each feature category based on their overall quality scores is provided in Table 3. The recommended apps in this list are consistent with those provided by reviewers from the iMedicalApps website (e.g., Medscape and MedCalc). 28 However, the difference is that our study classifies the apps into feature categories that target MRPs. In addition, this study assesses the quality of apps with monitoring and medication record features, which, from our knowledge, are not evaluated by iMedicalApps. Furthermore, this study assesses the quality of the apps objectively through a scoring system, which can complement the recommendations provided by the iMedicalApps reviewers based on their experiences with the apps.

Limitations
Owing to the ever-changing nature of information, some of the questions in the quality assessment tool may be time-sensitive. For example, content evaluation of apps with the medication information feature was based on the recent black box warning for carbamazepine to conduct genetic screening in Asian patients who use this medication due to the possibility of Stevens-Johnson syndrome and toxic epidermal necrolysis. 29 Clinicians who intend to use this tool for evaluation of their apps will need to update the content based on their area of specialization.
The apps evaluated were only on the Apple and Android platforms, so the external validity of using this assessment tool across other platforms, such as Windows and Blackberry, is uncertain. This study's results show that the same apps on dif-

DISCLOSURES
No outside funding supported this study. The authors have no conflicts of interests directly related to this study.
Study concept and design were contributed by Loy and Yap. Loy collected the data and took the lead in data interpretation, along with Ali and Yap. The manuscript was primarily written by Loy, along with Yap, and revised primarily by Ali, along with Yap.

ACKNOWLEDGMENTS
The paid apps that were used for evaluation in this study were sponsored by the final year project fund from the Department of Pharmacy, National University of Singapore.