- Research article
- Open Access
The evolution of mobile apps for asthma: an updated systematic assessment of content and tools
BMC Medicine volume 13, Article number: 58 (2015)
Interest in mobile apps that support long-term conditions such as asthma is matched by recognition of the importance of the quality and safety of apps intended for patient use. We assessed how changes over a 2-year period affected the clinical suitability of apps providing self-management information and tools for people with asthma by updating a review first performed in 2011.
Systematic content assessment of all apps for iOS and Android examining the comprehensiveness of asthma information, consistency with the evidence base for asthma self-management and adherence to best practice principles for trustworthy content, comparing the quality of apps available in 2011 to those released since.
Between 2011 and 2013, numbers of asthma apps more than doubled from 93 to 191, despite withdrawal of 25% (n = 23/93) of existing apps. Newer apps were no more likely than those available in 2011 to include comprehensive information, such as the use of action plans, or offer guidance consistent with evidence; 13% (n = 19/147) of all apps, and 39% (n = 9/23) of those intended to manage acute asthma, recommended self-care procedures unsupported by evidence. Despite increases in the numbers of apps targeting specific skills, such as acute asthma management (n = 12 to 23) and inhaler technique (from n = 2 to 12), the proportion consistent with guidelines (17%, n = 4/23) and inhaler instructions (25%, n = 3/12), respectively, was low, and most apps provided only either basic information about asthma (50%, n = 75/147) or simple diary functions (24%, n = 36/147).
In addition to persisting questions about clinical quality and safety, dynamic aspects of app turnover and feature evolution affect the suitability of asthma apps for use in routine care. The findings underline the need for coordinated quality assurance processes that can adapt to changing clinical and information governance-related risks, ensure compliance with the evidence base and reflect local variations in clinical practice. It is unclear if substantial clinical benefits can be realized from a landscape dominated by low quality generic information apps and tools that do not adhere to accepted medical practice.
Medical apps for patients are software programs that run on personal mobile devices and offer functions such as disease tracking, access to clinical and peer support, health information, and reminders . Survey data indicate that approximately three fifths of US and UK adults own an app-capable smartphone [2,3] and around half of adults report knowing about and using apps . In 2012, a fifth reported having downloaded a health or fitness app, increasing amongst people with recent changes in health status . Industry estimates suggest that, by 2018, over one billion individuals worldwide will use a health or fitness app  drawn from a pool of consumer health apps estimated to number in excess of 15,000 . In 2014, over a third of US clinicians reported having recommended an app to a patient in the past year , and a survey of US adults suggested that nine of ten would use one issued ‘on prescription’ by a healthcare professional . Among policymakers and clinicians, interest in apps reflects the potential to improve patient support and, potentially, health outcomes by equipping individuals with tools for self-management and remote monitoring [9-11]. However, there is also increasing recognition of the importance of ensuring that medical apps are effective [12,13], safe [14,15], meet patient needs [16,17], and comply with existing evidence-based clinical practice [18-20].
In 2011, we mapped  the types of apps available for patients with asthma, chosen as a prevalent long-term condition with potential social and economic [22,23] impacts and which requires a range of self-management skills in everyday life . We explored the degree to which apps conformed to evidence-based recommendations derived from international asthma guidelines. We characterized the range of possible functions supported by apps for asthma, including information provision, peak flow and symptom diaries, and medication trackers. While highlighting the potential to support tailored education and assist clinical decision-making using monitoring data, the review uncovered a range of potential issues. Few information apps addressed key educational topics recommended in international guidance. Many advocated strategies for acute asthma management and trigger avoidance that were unsupported by current evidence. Assessment and diagnostic tools were largely unvalidated, some had errors of design, and the small number of apps offering therapeutic interventions had no basis in evidence.
Rapid evolution is characteristic of mobile technologies. New hardware and software capabilities create opportunities for health; for example, remote monitoring enabled by always-on network connections and wearable sensors . Modern development methods mean that apps commonly undergo cycles of refinement in response to testing and user feedback . Changes in mobile devices and software have been accompanied by a broadening discussion about quality and safety that has involved clinicians [27,28], policy groups , and, more recently, regulators [30,31]. Compared to 2011, there is greater understanding of potential risks and more resources targeted towards medical app developers, aiming to improve the quality of medical apps [32-35]. As a result, there is an opportunity to explore the impact that these changes may have had on apps intended for patient use.
We aimed to explore changes in the content, function, and clinical quality of apps since 2011. We used a systematic approach, based on a systematic literature review, to identify, classify, and review apps for asthma. By updating our earlier review, we sought to differentiate persistent quality issues from those that are new or have proven transient.
All apps from the two most widely-used  mobile platforms (Android and iOS) incorporated in the original review  were automatically eligible for inclusion. We chose not to include apps running on other platforms, such as Blackberry, whose market share is currently declining . To identify new apps, we performed a stepwise process of search, screening, and review. We searched the public app stores of the two most popular smartphone operating systems using the terms ‘asthma’, ‘inhaler’, ‘peak flow’, and related word variants, e.g., ‘asthmatic’. After removing apps already included from the previous study, results were screened to eliminate obviously irrelevant content. The remaining apps were downloaded and reviewed to reach a final decision for inclusion. Screening and review were performed by three reviewers (KH, JO, and AG), working independently. Final decisions were reached by consensus.
Free and paid-for smartphone and tablet apps were eligible for inclusion if they contained asthma-specific content or tools intended for use by patients or the public (see list below). Apps targeted exclusively towards clinicians were not included. Apps were excluded if they could not be downloaded from UK app stores or, after at least two attempts on different test devices, would not start. Apps from any country were eligible provided they supported English.
Smartphone or tablet app
Free or paid
Running on Android or iOS platforms
Content or tools addressing one or more aspects of asthma diagnosis, management, or support as either the sole function or in a way that means asthma-related elements can be isolated from the rest of the content
Presenting content in any format
Targeted at patients of any age
Not available through an approved device marketplace
Explicitly disclaimed use for a health-related purpose
Could not be downloaded because of country restrictions that prevented access in the UK
Could not be used because of technical problems, after two attempts
Apps were assessed using a systematic approach to characterize functions and content. Apps containing information about asthma were evaluated using a set of criteria derived from US , UK , and international  asthma guidelines. Information content was evaluated for coverage of eight domains identified as priorities for self-management education (basic facts about asthma, basic principles of treatment, trigger avoidance, how to use treatment, self-monitoring skills, the role of action plans, managing exacerbations, and personalized care), using a set of operational criteria derived from guidelines. These criteria were those used in our earlier review except for one criterion in the self-monitoring domain which was modified in response to peer feedback to acknowledge symptom-based methods for self-monitoring (criteria and change detailed in Additional file 1: Table AF1). Coverage of each educational domain was rated as either complete (all criteria satisfied), partial (some criteria satisfied), or absent. Content was also evaluated for consistency with evidence-based recommendations concerning eight aspects of asthma management derived from guidelines (Additional file 1: Table AF2). For those apps with content addressing a particular topic, the advice given was rated as being either consistent or inconsistent with underlying evidence. Consistency was judged both on the basis of factual accuracy and appropriate qualification of advice to reflect either uncertainty or limitations in the applicability of evidence, such as recommendations relevant only to individuals with more severe asthma, in which case we expected any information to be appropriately qualified. Information content was evaluated regardless of the delivery medium. However, apps focusing only on complementary and alternative medicine, using the definition of the NIH National Center for Complementary and Alternative Medicine , were excluded from this analysis because there was no basis to expect that they should comply with guideline recommendations. For diaries and other self-management tools, the nature of inputs and outputs and methods for summarizing and sharing data were characterized. For all apps with calculator and questionnaire functions, we attempted to locate validation information in the form of an authoritative citation. Where validation information was found, the behaviour of the app in handling inputs and producing appropriate outputs was assessed. All apps were assessed for compliance with principles for ethical disclosure of authorship and data protection practices, using criteria previously adapted from those assessing online health information devised by the Health on the Net Foundation  (criteria in Additional file 1: Table AF3). We also recorded any software issues that were encountered during testing, using a schema devised during the first review to classify problems such as crashes or errors in the user interface (detailed in Additional file 1: Table AF4).
Apps included in the original review had previously been assessed using these criteria. To ensure scoring consistency, we conducted a process of calibration in which a random selection of 10 apps was re-assessed and the outcomes compared to previous judgments. Apps that had been updated between 2011 and 2013 were also reassessed. Apps were reviewed by an author of the original review (KH) and one of two additional reviewers (JO and AG), working independently. After reconciliation, the results of the assessment process were recorded, along with basic characteristics of included apps, in a structured data extraction form (template in Additional file 1: Table AF5) for analysis.
Post hoc evaluation
In addition to the planned analyses described above, we performed a post hoc evaluation of inhaler technique guidance in response to the growing number of apps that include this function, using criteria derived from manufacturer guidance to assess key steps, such as inhaler preparation, necessary for effective drug delivery into the lungs (detailed in Additional file 1: Table AF6).
Apps available on multiple platforms were grouped for subsequent analysis. When an app was available as multiple versions on the same platform, for example ‘demo’ and ‘full’ editions, the most fully featured version was used for analysis (irrespective of free/paid status). The decision to define the unit of analysis as the platform-independent app reflected learning from the original review concerning the possible risk of bias resulting from double counting some apps. In that review, we treated each app version separately because we anticipated different quality issues arising from the need to tailor each app according to the platform and version. However, the final results suggested that this was not, in general, the case. While in 2011 only a small number of apps were affected (n = 13), and sensitivity analysis undertaken during that analysis demonstrated minimal impact on results, by 2013 the number of duplicates was substantially larger (n = 77) with consequently greater scope for bias.
Descriptive statistics were used to summarize the types, functions, and quality characteristics of apps available in 2011, those released since 2011, and overall. When reporting comparisons between 2011 and 2013, data are always written in chronological order, e.g., for “(X vs. Y)”, X are data from 2011 and Y reflect data for apps released since 2011. We hypothesized that newer apps might have different quality characteristics arising from greater experience amongst developers and discourse in both medical and computer science literature emphasizing the importance of high-quality medical apps. To explore differences in the quality of information apps, we compared the proportion of educational domains completely, partially, or not satisfied, and the proportion of guidance consistent with the evidence base in those available in 2011 and those released since. We also compared the proportion of apps satisfying principles for ethical disclosures of information and the proportion of apps with specific software issues. Apart from these planned comparisons, we performed a small number of additional tests to compare differences in the proportion of paid-for apps, apps addressing complementary and alternative medicine, and the incorporation of features such as support details, in-app help, and social sharing features. These are marked as post hoc comparisons in the results.
For all comparisons, we used a two-tailed Fisher’s exact test , with the Freeman-Halton extension  to handle two by three cases for information comprehensiveness and (post hoc) content orientation. To adjust for multiple comparisons, we used a Holm-Bonferroni  sequential adjustment to control the group-wise error rate at 5%, instead of specifying a fixed significance level for comparisons. All analyses were performed using R (Version 3.1.2).
Searches performed in June, July, and August 2013 identified 764 apps (Figure 1, n = 538 Android, 226 Apple). After excluding unsuitable apps (n = 643), 191 apps were selected for evaluation. These included 70 apps identified in the 2011 review that remained available. Details of apps excluded at the final review stage are provided in Additional file 2: Table AF7.
The number of apps for asthma more than doubled between 2011 and 2013 (from 93 to 191), dominated by growth in apps available for Android (from 41 to 103, 151% increase, vs. iOS 52 to 88, 69% increase, charted over time in Figure 2). Forty-one apps were available on both Android and iOS platforms and a further 36 had multiple versions on a single platform. After removing 24 platform duplicates and 20 demonstration versions where a more fully-featured version was available, analysis was performed on 147 unique apps. Over a quarter (28%, n = 22/78 unique apps) included in the original review had been withdrawn, and a similar proportion (24%, n = 19/78) updated, between 2011 and 2013. Withdrawals are included in reported 2011 statistics but not in overall 2013 totals. At the conclusion of analysis in June 2014, a further 20 apps had been withdrawn. These are retained for analysis, but annotated as withdrawn in data tables provided in Additional file 2.
Table 1 summarizes the basic characteristics of included apps. Compared to apps available in 2011, apps released since were significantly more likely to be free (33%, n = 26/78 vs. 64%, n = 59/92, P <0.001, post hoc) but also more likely to carry advertising (18%, n = 14/78 vs. 34%, n = 31/92, P = 0.024, post hoc), although the difference was not significant after sequential adjustment. Between 2011 and 2013 the median cost of paid-for apps decreased from £1.27 GBP to £0.89 (£1.12 overall), reflecting a reduction in the proportion of apps charging more than £1.00 (37%, n = 29/78 vs. 22%, n = 20/92). The proportion of apps offering in-app upgrades to unlock addition functions remained unchanged at 9% (n = 13/147) overall, (8%, n = 7/78 vs. 7%, n = 6/92, P = 0.576, post hoc). While the US remained the main source of apps for asthma, accounting for almost half of apps with a known country of origin available in 2013 (48%, n = 52/108), the number of countries represented increased from 13 to 21. Although the overall proportion of apps indicating that the product was targeted at a particular group remained low (16%, n = 24/147), apps released since 2011 were significantly more likely to provide information targeting parents/caregivers (4%, n = 3/78 vs. 16%, n = 15/92, P = 0.011, post hoc). There was also a non-significant increase in the proportion of apps targeting children (3%, n = 2/78 vs. 8%, n = 7/92, P = 0.182, post hoc).
Information provision remained the function most commonly offered by nearly three fifths of apps for asthma (56%, n = 83/147; Table 2), followed by self-management tools in almost half (48%, n = 70/147). The number of apps combining self-management tools, such as diaries, with any form of information content grew from one (1%) in 2011 to nine (6%) in 2013. As in 2011, small numbers of apps available in 2013 provided functions for assessing asthma symptoms using questionnaires (n = 11) or sensors (n = 2) or assisting with aspects of therapy (n = 6). Although there was an increase in the proportion of apps released since 2011 incorporating mixed content addressing both conventional and alternative medical traditions, the change in relative frequencies was not significant (P = 0.268, post hoc; Table 2) and the overall proportion of apps presenting content oriented towards conventional medical practice changed minimally (69% n = 101/147 vs. 73% n = 57/78 in 2011).
The number of apps offering information about asthma increased from 37 in 2011 to 83 by 2013, with 60 new apps and 14 withdrawals. Most information apps available in 2013 incorporated general information about asthma (89%, n = 74/83), although a growing proportion of apps (23%, n = 18/78 vs. 34%, n = 31/92, P = 0.174, post hoc) also incorporated specific instructions about the management of acute asthma, inhaler technique, and other aspects of self-management (discussed further in Therapeutic Instructions and Tools, below.) Apps used a range of media to communicate information including text (64%, n = 53/83), video and animation (10%, n = 8/83), or by combining different media types (27%, n = 22/83). A significantly greater proportion of apps released since 2011 used a medium other than text to communicate information (11%, n = 4/37 vs. 42%, n = 25/60, P = 0.001, post hoc). A new development was the use of gameplay (n = 3) to provide interactive education in three products targeted at children and teenagers. Full details of information-containing apps are provided in Additional file 2: Table AF8.
Although increasing in number, the proportion of products presenting information based exclusively around conventional medical practices was unchanged (51%, n = 19/37 vs. 50%, n = 30/60). Just over a fifth of all apps available in 2013 (23%, n = 19/83) combined alternative and conventional information and the remainder (28%, n = 23/83) presented information based on alternative medical principles alone. Consequently, there were 59 apps (vs. 23 in 2011) incorporating conventional medical information for which it was possible to assess coverage of self-management education topics and consistency with guideline recommendations.
Over two thirds (83%, n = 49/59) of all apps available in 2013 provided at least some details about basic aspects of asthma pathophysiology (Table 3). However, only just over half provided information about basic principles of medical management (53%, n = 31/59), and fewer addressed key self-management skills, including allergen and trigger avoidance (44%, n = 26/59), self-monitoring skills (39%, n = 23/59), or how to use treatment appropriately (39%, n = 23/59). Less than a third of apps addressed how to recognize the signs of deterioration (31%, n = 18/59), the importance of personalized treatment and goal setting (31%, n = 18/59), and the role of an action plan (22%, n = 13/59). There was no significant difference in the proportion of apps released since 2011 addressing domains other than basic facts (27%, n = 6/22 vs. 20%, n = 9/45, P = 0.542), nor in the proportion of apps addressing each domain considered individually (P values shown in Table 3). While the overall number of products addressing three or more educational domains, at least partially, grew from 14 to 31 between 2011 and 2013, new apps were not more likely to address multiple domains (64%, n = 14/22 vs. 51% n = 23/45, P = 0.435). In the same period, the number of products addressing all eight domains at least partially grew from two to five (with one withdrawal). As in 2011, however, no product provided complete coverage of all eight domains by satisfying all criteria.
Thirty-six apps (vs. 14 in 2011) were evaluated for consistency of recommendations against nine evidence-based topics (Table 4) because they contained content addressing at least one topic. Smoking cessation and passive smoking avoidance were the topics most commonly addressed by apps (by n = 19 and 21, respectively) and, as in 2011, were correctly recommended as beneficial by all apps. All apps addressing immunotherapy (n = 5) and weight reduction (n = 7) correctly advocated their use in specific patient populations. However, apps were also likely to recommend actions for which there is unclear evidence of benefit, such as mold (n = 12/12) and cockroach (n = 7/7) avoidance measures, removal of pets (n = 7/8), and avoidance of air pollution (n = 15/17). Most apps covering seasonal influenza vaccination recommended it (n = 8/10) without acknowledging the uncertain impact on asthma . Compared to 2011, there were no significant differences in the proportion of apps released since that presented advice consistent with the evidence base, either when considering each statement individually (exact P values computed from proportions shown in Table 4 were all 1.0) or when comparing the proportion of apps containing no statements contradicting evidence (36%, n = 5/14 vs. 43%, n = 12/28, P = 0.747). A new finding in 2012 concerned a small number apps (n = 6) containing advice emphasizing the possible harmful effects of conventional medicine and, in two cases, promoting a commercial product, for example a homeopathic remedy. Previously, apps containing mixed content had tended to emphasize the right to choose between medical approaches rather than denying accepted medical practice.
Diaries and trackers
Thirty six apps available in 2013 (24%, vs. 27% in 2011) were diaries for recording asthma symptoms (n = 27), peak flow measurements (n = 24), and medication (n = 24, detailed in Additional file 2: Table AF9). Six apps allowed a user to document an action plan. Six incorporated inhaler trackers, including two where this was the sole function, designed to alert a user when a metered dose inhaler or other medication was running low. One app connected to an electronic inhaler to automatically log doses. Two thirds of diaries (67%, n = 24/36) offered a mechanism to summarize either recorded symptoms (n = 4), peak flow (n = 11), or both (n = 9) in chart form. Over half (55% n = 11/20) of apps charting peak flow used predicted (n = 6), personal best (n = 7), or manually entered (n = 3) thresholds to classify charted data to aid interpretation, for example, as ‘green’, ‘yellow’, and ‘red’ zones. Nine apps provided alerts based on changing input values. Seven provided specific advice, either by excerpting relevant parts of a user supplied-action plan (n = 3/6 apps with action plan support) or displaying standardized text (n = 4). Sixteen allowed reminders to be set to take peak flow measurements (n = 9), medication (n = 3), appointments (n = 2), and vaccination (n = 1). Compared to 2011, there were increases in the proportion of apps synchronizing and storing data online (10%, n = 2/21 vs. 44%, n = 8/18, P = 0.025, post hoc) but not those offering a mechanism for sharing data with clinicians (62%, n = 13/21 vs. 61%, n = 11/18, P = 1.000, post hoc), for example, email or file export.
Other self-management tools
Nineteen apps (compared to 6 in 2011, Table 2) provided status information about pollen (n = 10), air pollution (n = 7), or both (n = 2; Additional file 2: Table AF10). Most (n = 12) provided coverage of locations in the US. Only one product, which relied on used-supplied data, offered global coverage. Seven apps (vs. 4 in 2011) offered databases of chemical additives to allow patients to identify potential triggers in food and cosmetic ingredients. One app combined an electronic version of an action plan with links to information about asthma. Another product connected to a forum where users could view topics about asthma management and post their own questions. A third app offered UK patients the ability to purchase asthma medication online.
Diagnostic and assessment tools
Fifteen apps (vs. 6 in 2011) contained diagnostic or assessment functions. Ten products offered questionnaires for asthma diagnosis (n = 4) or assessment (n = 6; Additional file 2: Table AF11). Two apps used the validated Asthma Control Test under license and one app reused a questionnaire developed by the University of Maryland. The remaining apps either did not provide a citation or validation information or the information provided was insufficient to verify the basis for assessment. Nine products provided predicted peak flow calculators as either the main function (n = 3) or integrated into a diary (n = 6). No calculator app provided a clear citation, but the underlying formula was identified for two apps. Issues first identified in 2011 resulting in incorrect calculation output for two apps where the formula was deduced were unresolved. Three products used the device microphone to attempt to quantify lung volume (n = 1) and analyse breath sounds (n = 2), but were either unvalidated or experimental products that had not yet received regulatory approval.
Therapeutic instructions and tools
Forty-one apps (28% of n = 147) contained specific instructions for the use of inhalers (8%, n = 12/147), first aid for acute exacerbations (16%, n = 23/147), or other approaches to the management of asthma such as the use of herbal remedies (11%, n = 16/147, breakdown in Additional file 2: Table AF12). After pressurized metered dose inhalers (pMDI), either alone (100%, n = 12/12) or with spacer devices (92%, n = 11/12), the devices most commonly addressed by inhaler technique guidance were Accuhalers (58%, n = 7/12) and Turbohalers (50%, n = 6/12), although a range of additional devices were also covered (Additional file 2: Table AF12). Apps used text (33%, n = 4/12), video (25%, n = 3/12), animation (8%, n = 1/12), or a combination (33%, n = 4/12) to provide guidance. Post hoc evaluation (criteria in Additional file 2: Table AF6) examined the proportion of apps providing complete and correct information about inhaler preparation (67%, n = 8/12), positioning in the mouth (67%, n = 8/12), and the sequence of inhalation and actuation (50%, n = 6/12). While a quarter (25%, n = 3/12) addressed all three domains, the majority of apps omitted or misstated at least one step necessary for effective delivery of drugs to the lungs [45,46] for at least one of the inhalers discussed. The most common error (in 25%, n = 4/12) was the failure to explain how inspiration should start before, and continue after, actuation of a pMDI. A variable proportion of apps addressed other aspects of inhaler use such as holding the breath after inhalation (83%, n = 10/12), timing of a second dose (58%, n = 7/12), and mouth rinsing to reduce the risk of candidiasis with inhaled steroids (50%, n = 6/12). While five apps (42%) provided information about identifying empty Turbohalers or Accuhalers, no app addressed dose-tracking using pMDI. The two apps (17%) that covered inhaler care both provided incorrect information about spacer cleaning that risked static build-up interfering with subsequent doses.
Of the 23 apps incorporating advice about the management of acute asthma, approximately half (52%, n = 12/23) provided a clear description of the signs of acute asthma and gave practical advice such as sitting a patient up (48%, n = 11/23) and offering reassurance (52%, n = 12/23). Three fifths (61%, n = 14/23) provided appropriate instructions about when to seek further help. While almost two thirds (65%, n = 15/23) recommended the use of a rescue inhaler, less than a third (30%, n = 7/23) provided specific dose and timing instructions and only four apps (17%) provided instructions appropriate for children in addition to adults. Overall, four apps (17%) addressed all five components (Additional file 2: Table AF12). The proportion of apps released since 2011 that made treatment recommendations based on alternative medical practices did not change significantly (76%, n = 13/17 vs. 56%, n = 14/25, P = 0.207, post hoc), and almost three fifths of apps available in 2013 (58%, n = 19/33), and 39% (n = 9/23) of those intended to manage acute asthma, recommended procedures unsupported by evidence, for example, the use of a hot water bottle to warm the chest, cider vinegar administered orally, or fasting for the duration of an attack.
A small number of products (n = 6 vs. 2 in 2011) were tools intended to have a therapeutic effect. Three apps provided training in aspects of Buteyko breathing, which may have a future role in asthma management . The remaining three were based on alternative medical principles using hypnosis, mantra, and motivational messages.
Disclosures and software issues
We also assessed software issues encountered during the test process (using criteria detailed in Additional file 1: Table AF5). Compared to 2011, there was no change in the proportion of new apps lacking appropriate data entry validation to prevent, for example, textual data being entered in numeric data fields (33%, n = 9/27 vs. 32%, n = 8/25, P = 1.000). The proportion of apps with problems with the user interface, such as mislabelled data fields (32%, n = 25/78 vs. 37%, n = 34/92, P = 0.522), or core functionality, such as the correct operation of a calculator (17%, n = 13/78 vs. 10%, n = 9/92, P = 0.252), was also unchanged. However, there was a significant increase in apps released since 2011 experiencing network problems when trying to access online content or services (4%, n = 3/78 vs. 22%, n = 20/92, P <0.001). Crashes affected about one in ten apps during testing, unchanged from 2011 (9%, n = 13/147 overall).
Our updated findings present a mixed picture for clinicians interested in integrating apps into routine care. While choice has increased, newer apps for asthma were no more likely than those available in 2011 to satisfy evidence-based recommendations for information content or the design of self-management tools. Our data extend findings of low overall compliance with evidence in recent assessments of medical apps for disease-management [20,48] and health-promotion [49,50] by suggesting that issues of quality are sustained and are not isolated to first-generation apps. However, as in 2011, our results question the scope for benefit offered by current apps for asthma. Firstly, rather than focusing on known gaps in self-management where there is evidence that intervention can improve outcomes, such as the use of action plans [22,51], most apps offer either very basic background information about asthma or simple peak flow diaries. Only a fifth of information apps available in 2013 addressed action plans and the proportion of diaries offering a mechanism for recording a plan was smaller. Secondly, although it is recognized that information provision alone has a limited impact on health behaviour , the proportion of apps combining information and tools remains small. While apps using multimedia methods, such as video and gameplay, presented welcome new opportunities for tailored support that focuses on specific needs, such as improving inhaler technique , or on specific groups, such as those with limited literacy , our assessment highlights the risk of clinically-relevant impacts resulting from inadequate content. For example, three quarters of inhaler training apps left out at least one step considered necessary for effective delivery of drug into the lungs. Thirdly, while apps incorporate a range of features, such as reminders, that could plausibly play a role in interventions designed to improve asthma self-management, apps themselves do not currently provide effective scaffolding to support individuals in changing, and sustaining, behaviour. For example, although many apps offered the ability to send data by email, and the proportion of apps using a mechanism to share data online almost doubled between 2011 and 2013 to 52%, no app offered a built-in mechanism for communication between a professional and patient, potentially limiting the scope for feedback that could assist self-management. Nor was it clear that the online services linked to currently available apps could support the kind of summary and flagging activities necessary to manage cohorts rather than single patients. Evidence is needed that app-supported remote monitoring can generate reliable data, prompt consistent and sustained clinical improvements, and meet the needs of patients across the spectrum of asthma severity . Although more apps allowed information to be shared using a social network like Facebook or Twitter, this was never in the context of a structured activity relating to asthma self-management and no apps took advantage of the possibilities for peer support. Similar to the findings of others [19,56], only one asthma app appeared to be linked to an active research project , highlighting a broader disconnect between guidance on the design of interventions that incorporate evidence-based components and evaluation  and what occurs in practice.
Dynamic aspects of the asthma app marketplace, including rapid growth and turnover, have potential consequences for both clinical services and individuals. Over a quarter of apps included in the original review, including those from reputable content providers, had been withdrawn. While patients may choose apps to suit their changing needs, the costs associated with app selection, staff training, setting up support mechanisms, and appropriate governance create incentives for planned care to be based around small numbers of trusted apps. Unexpected withdrawal of an app may be an inconvenience for individual users, but can threaten the function of an established clinical service. App updates are also potentially disruptive, requiring testing and training. As a result, formal arrangements with app developers, such as service-level contracts, should be considered where apps are being used as part of planned care. Systematic identification of factors influencing app lifespan would be helpful and future work should address this.
A new finding concerned apps that repackaged existing information either to create content focused on promoting specific commercial products, or to serve as a more general vehicle for hosting adverts operating on a pay-per-click basis. A small number specifically emphasized negative effects associated with conventional medical practice, with the intention of promoting commercial alternatives. Some apps were developed using online tools that specifically target developers with no or limited coding experience. Marketing materials for these tools emphasized their ability to produce apps quickly and in number. If the goal of these apps is simply to generate advertising revenue or an initial purchase, content need only to appear superficially plausible. This kind of low-quality information, coupled with high volume distribution and commercial motive, shares similarities with spam or junk email and websites . Indeed, some apps for asthma reused material originally produced by online content farms . If sustained, the junk medical app would be an unwelcome development, making it harder for users to locate higher-quality and unbiased content. Unlike the web, where search engines tend to prioritize higher-quality content in results , conventional app stores offer limited mechanisms for filtering out lower quality and junk apps.
Apps offering unvalidated diagnostic, assessment, and therapeutic functions with no basis in evidence are a continuing issue that raises questions about patient safety and clinical liability . While some apps are likely to fall under the scope of medical app regulation recently proposed in both the US and Europe [30,31], aspects of how this will work in practice are yet to be agreed, not least because regulators must balance competing demands of risk, costs, and the possible impact on innovation . As a result, rapid action appears unlikely, even where standards already exist. For example, while physical sensors, such as peak flow meters, are classified as moderate risk (class 2) medical devices, neither of the two asthma apps that use a device microphone to analyse breath sounds appear to have undergone regulatory approval, and both remain available in November 2014, over 1 year since guidance first came out. Regulation is also constrained by the scope of current legislation which applies only to apps intended to diagnose, treat, or prevent disease. While not ruling out future changes, regulators have stated that information apps do not satisfy this definition, even if they contain procedural guidance such as first aid , and that they intend to apply discretion when enforcing rules for apps that probably qualify as medical devices but pose a low overall risk such as diaries, pollen alerts, and inhaler trackers . While ensuring that potentially useful apps are not burdened by costly and time-consuming approvals, some apps will fall outside established reporting mechanisms that would allow a more complete picture of risk to emerge over time. Further work is necessary to understand the consequences of known defects in tools such as predicted peak flow calculators and identify unanticipated risks, for example, those arising from novel services like online pharmacies that do not involve a usual clinician.
Our findings have relevance for emerging medical app accreditation programs [32,64,65], which aim to reduce the burden of assessment placed on professionals and patients, provide recognizable badges of app quality, and collate certified apps to make them easily accessible. Rapid app turnover and feature evolution suggest that accreditation processes will need to be revised regularly and programs will need to allocate appropriate resources to reassess their growing cohort of existing certified apps as they are updated. Programs must also consider the possibility of best practices that vary by location and target group. For example, some US inhaler technique guidance apps advised actuating a metered dose inhaler in front of an open mouth, a method not taught in UK self-management education. The growth in connected apps, and those based around advertiser models, highlights the importance of assessment processes which examine privacy and security-related risks as personal and medical data start to be shared online. While we did not explore the governance arrangements of cloud-based apps, privacy and security are topical concerns for patients [55,66]. Additionally, recent findings highlighting gaps in privacy disclosures [67,68] and security arrangements [69,70] in popular health apps suggest that further scrutiny is required. High rates of app turnover emphasize the importance of due-diligence that focuses on the commitment of developers to maintaining their products. The range of functions seen within a single long-term condition in this study highlights the potential complexity of defining comprehensive assessment standards. Similar to the content-independent criteria applied in this study, simplified assessments which focus on high-risk functions and proxy markers for quality, such as the involvement of clinicians in design, have been proposed [71,72], but their predictive validity is unknown. While most accreditation programs appear to be developing their own criteria, a collaborative, open approach to standard development could share the workload, for example, by involving professional bodies with specific content expertise, enable transparent input by patients, professionals, and developers, and permit evaluation. Further work should explore the extent to which developers can be engaged proactively to increase awareness of accreditation requirements and the potential value of patient and professional input into health app design, which, in turn, may increase the likelihood that apps meet accreditation requirements without potentially costly cycles of rejection and resubmission. Finally, because accreditation programs are voluntary and have yet to prove their ability to control quality issues at scale, most health apps available in public app stores will not be badged for quality. Patient education could address this by, for example, explaining the phenomenon of junk medical apps and characterizing the risks as similar to those presented by the range of information available on the general internet.
The study used a systematic method to exhaustively sample apps from the two most popular app stores and included steps to minimize bias during the assessment process. By repeating an assessment first performed in 2011, we were able to explore aspects of the dynamics of app turnover and feature development. However, the study had a number of limitations. Apps in languages other than English and those available on platforms other than Android and iOS were not included. It is possible that there are systematic differences in app quality by language or platform that this study could not detect. Because the study involved only two time points, there should be substantial caution in inferring that any changes observed are indicative of trends. Our data reflect issues present in apps available at particular points in time, and while representative categories are likely to persist, the specifics will change, necessitating updated assessment. Finally, because of limitations in the data provided by app stores, we were unable to precisely quantify the scale of app use by patients nor the ways in which apps are used. The potential for a mismatch between available features and those desired by patients was highlighted by a recent review of apps for patients with Parkinson’s disease . Future work could explore this issue in relation to asthma, examine whether user reviews and ratings have any predictive validity for app quality, and continue to develop the currently limited evidence-base specifically addressing the design of apps for asthma self-management .
In addition to established findings about variable clinical quality and possible safety risks, dynamic aspects of the asthma app marketplace, such as rapid growth and turnover, affect the suitability of apps for use in clinical settings. Efforts to assure the quality of medical apps need to be responsive to the changing features and risks posed by apps, consider governance as well as clinical factors, and recognize the differing needs of patients using apps on their own initiative and those supported by clinical care. Accreditation programs may need to be supplemented by proactive intervention by regulators to address potentially unsafe apps and advice to help patients navigate the increasing numbers of poor quality or ‘junk’ apps for asthma in general app stores.
Klasnja P, Pratt W. Healthcare in the pocket: mapping the space of mobile-phone health interventions. J Biomed Inform. 2011;45:184–98.
Ofcom. The communications market 2014 (August). 2014. http://stakeholders.ofcom.org.uk/market-data-research/market-data/communications-market-reports/cmr14/.
Pew Research Internet Project. Mobile technology fact sheet. http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/.
Fox S, Duggan M. Mobile health 2012. Washington DC: Pew Research Center; 2012.
research2guidance. Mobile health market report 2013–2017. http://www.research2guidance.com/shop/index.php/mhealth-report-2.
Aitken M, Gauntlett C. Patient apps for improved healthcare – from novelty to mainstream. Parsippany, NJ: IMS Institute for Healthcare Informatics; 2013.
Manhattan Research. 2014 Environment. http://manhattanresearch.com/Products-and-Services/Physician/Taking-the-Pulse-U-S.
Digitas Health. mBook 2013 edition – marketing mobile health. 2013. http://m.2013.digitashealth.com/Mbook2013_web.pdf.
Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? JAMA. 2013;310:2395–6.
European Commission. Green paper on mobile health ("mHealth"). Brussels: EC; 2014.
England NHS. Five year forward view. London: HM Government; 2014.
Rosser BA, Eccleston C. Smartphone applications for pain management. J Telemed Telecare. 2011;17:308–12.
McCartney M. How do we know whether medical apps work? BMJ. 2013;346:f1811.
Cortez NG, Cohen IG, Kesselheim AS. FDA regulation of mobile health technologies. N Engl J Med. 2014;371:372–9.
Wolf JA, Moreau JF, Akilov O, Patton T, English 3rd JC, Ho J, et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol. 2013;149:422–6.
Huckvale K, Car J. Implementation of mobile health tools. JAMA. 2014;311:1447–8.
Atienza AA, Patrick K. Mobile health: the killer app for cyberinfrastructure and consumer health. Am J Prev Med. 2011;40:S151–3.
Buijink AWG, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med. 2013;18:90–2.
De La Vega R, Miro J. mHealth: a strategic field without a solid scientific soul. A systematic review of pain-related apps. PLoS One. 2014;9:e101312.
Bender JL, Yue RY, To MJ, Deacken L, Jadad AR. A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer. J Med Internet Res. 2013;15:e287.
Huckvale K, Car M, Morrison C, Car J. Apps for asthma self-management: a systematic assessment of content and tools. BMC Med. 2012;10:144.
Partridge MR, Negro RWD, Olivieri D. Understanding patients with asthma and COPD: insights from a European study. Prim Care Respir J. 2011;20:315–23.
Masoli M, Fabian D, Holt S, Beasley R, Global Initiative for Asthma (GINA) Program. The global burden of asthma: executive summary of the GINA dissemination committee report. Allergy. 2004;59:469–78.
British Thoracic Society, Scottish Intercollegiate Guidelines Network. British guideline on the management of asthma – a national clinical guideline. May 2008, revised January 2012. Edinburgh: SIGN; 2012.
Patel S, Park H, Bonato P, Chan L, Rodgers M. A review of wearable sensors and systems with application in rehabilitation. J Neuroeng Rehabil. 2012;9:21.
Heredia A, Garcia-Guzman J, Amescua-Seco A, Velasco-Diego M. Agile practices adapted to mass-market application development. J Softw (Malden). 2014;26:818–28.
Charani E, Castro-Sanchez E, Moore L, Holmes A. Do smartphone applications in healthcare require a governance and legal framework? It depends on the application! BMC Med. 2014;12:29.
Barton A. The regulation of mobile health applications. BMC Med. 2012;10:46.
Alliance ECD. Green paper on mHealth apps – input from the European Chronic Disease Alliance (ECDA). Brussels: ECDA; 2014.
Food US, Administration D. Mobile medical applications – guidance for industry and food and drug administration staff. US Department of Health and Human Services: Rockville, MD; 2013.
Medicines and Healthcare Regulatory Agency. Guidance on medical device stand-alone software (including apps). http://www.mhra.gov.uk/Howweregulate/Devices/Software/index.htm.
Ferrero-Alvarez-Rementeria J, Santana-Lopez V, Escobar-Ubreva A, Vazquez-Vazquez M. Quality and safety strategy for mobile health applications: a certification programme. Eur J ePractice. 2013;21:14–26.
Marvel FA, Chase J, Madruga M. Ideas to iPhones: A 10-step framework for creating mobile medical applications with case report from Madruga and Marvel’s Medical Black Book app. J Mob Technol Med. 2014;3:35–41.
Information Commissioner's Office. Privacy in mobile apps – guidance for app developers. https://ico.org.uk/media/for-organisations/documents/1596/privacy-in-mobile-apps-dp-guidance.pdf.
Macharia E. Five tips for healthcare app development. http://www.software.ac.uk/blog/2013-10-08-five-tips-healthcare-app-development.
IDC. Smartphone OS market share, Q3 2014. http://www.idc.com/prodserv/smartphone-os-market-share.jsp.
National Asthma Education and Prevention Program. Expert panel report 3: guidelines for the diagnosis and management of asthma. Full report 2007. Bethseda, MD: National Institutes of Health; 2007.
Global Initiative for Asthma. Global strategy for asthma management and prevention. Updated 2012. http://www.ginasthma.org.
National Centre for Complementary and Alternative Medicine (NCCAM). Complementary, alternative, or integrative health: what’s in a name? http://nccam.nih.gov/health/whatiscam.
Health on the Net Foundation. Operational definition of the HONcode principles. Guidelines v2.0. http://www.hon.ch/HONcode/Guidelines/guidelines.html#Review.
Fisher RA. On the interpretation of X2 from contingency tables, and the calculation of P. J R Stat Soc. 1922;85:87–94.
Freeman GH, Halton JH. Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika. 1951;38:141–9.
Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat. 1979;6:65–70.
Cates CJ, Rowe BH. Vaccines for preventing influenza in people with asthma. Cochrane Database Syst Rev. 2013;2, CD000364.
van der Palen J, Klein JJ, Kerkhoff AHM, van Herwaarden CLA, Seydel ER. Evaluation of the long-term effectiveness of three instruction modes for inhaling medicines. Patient Educ Couns. 1997;32:S87–95.
Melani AS, Bonavia M, Cilenti V, Cinti C, Lodi M, Martucci P, et al. Inhaler mishandling remains common in real life and is associated with reduced disease control. Respir Med. 2011;105:930–8.
Freitas DA, Holloway EA, Bruno SS, Chaves GS, Fregonezi GA, Mendonca KP. Breathing exercises for adults with asthma. Cochrane Database Syst Rev. 2013;10, CD001277.
Chomutare T, Fernandez-Luque L, Årsand E, Hartvigsen G. Features of mobile diabetes applications: review of the literature and analysis of current applications compared against evidence-based guidelines. J Med Internet Res. 2011;13:e65.
Abroms LC, Lee Westmaas J, Bontemps-Jones J, Ramani R, Mellerson J. A content analysis of popular smartphone apps for smoking cessation. Am J Prev Med. 2013;45:732–6.
Breton E, Fuemmeler B, Abroms L. Weight loss – there is an app for that! But does it adhere to evidence-informed practices? Behav Med Pract Policy Res. 2011;1:523–9.
Gibson PG, Powell H. Written action plans for asthma: an evidence-based review of the key components. Thorax. 2004;59:94–9.
Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. JAMA. 2002;288:2469–75.
Newcomb PA, McGrath KW, Covington JK, Lazarus SC, Janson SL. Barriers to patient-clinician collaboration in asthma management: the patient experience. J Asthma. 2010;47:192–7.
Paasche-Orlow MK, Riekert KA, Bilderback A, Chanmugam A, Hill P, Rand CS, et al. Tailored education may reduce health literacy disparities in asthma self-management. Am J Respir Crit Care Med. 2005;172:980–6.
Hall JL, McGraw D. For telehealth to succeed, privacy and security risks must be identified and addressed. Health Aff (Millwood). 2014;33:216–21.
Mosnaim GS, Powell LH, Rathkopf M. A review of published studies using interactive internet tools or mobile devices to improve asthma knowledge or health outcomes. Pediatr Allergy Immunol Pulmonol. 2012;25:55–63.
Burnay E, Cruz-Correia R, Jacinto T, Sousa AS, Fonseca J. Challenges of a mobile application for asthma and allergic rhinitis patient enablement-interface and synchronization. Telemed J E Health. 2013;19:13–8.
Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46:81–95.
Markines B, Cattuto C, Menczer F. Social spam detection. 5th International Workshop on Adversarial Information Retrieval on the Web. 20 April 2009; Madrid: ACM; 2009. p. 41–8.
McCreadie R, Macdonald C, Ounis I, Giles J, Jabr F. An examination of content farms in web search using crowdsourcing. 21st ACM International Conference on Information and Knowledge Management. 29 October – 2 November 2012. Maui, Hawaii: ACM; 2012. p. 2551–4.
Colledge A, Car J, Donnelly A, Majeed A. Health information for patients: time to look beyond patient information leaflets. J R Soc Med. 2008;101:447–53.
Yang YT, Silverman RD. Mobile health applications: the patchwork of legal and liability issues suggests strategies to improve oversight. Health Aff (Millwood). 2014;33:222–7.
Food US, Administration D. Appendix A – examples of mobile apps that are not medical devices. Mobile Medical Applications – Guidance for Industry and Food and Drug Administration Staff. Rockville, MD: US Department of Health and Human Services; 2013.
Agencia de Calidad Sanitaria de Andalucia. [Distintivo app saludable]. http://www.calidadappsalud.com/.
NHS Health Apps Library. Review process. http://apps.nhs.uk/review-process.
King J. “How come I’m allowing strangers to go through my phone?” – smartphones and privacy expectations. Symposium On Usable Privacy and Security (SOUPS) 2012. July 11th–13th, 2012. Washington, DC; 2012.
Office of the Privacy Commissioner of Canada. Results of the 2014 Global Privacy Enforcement Network sweep. https://www.priv.gc.ca/media/nr-c/2014/bg_140910_e.asp.
Sunyaev A, Dehling T, Taylor PL, Mandl KD. Availability and quality of mobile health app privacy policies. J Am Med Inform Assoc. 2014.
Privacy Rights Clearinghouse. Fact Sheet 39: Mobile health and fitness apps: what are the privacy risks? https://www.privacyrights.org/mobile-health-and-fitness-apps-what-are-privacy-risks.
Dolan B. Happtique suspends mobile health app certification program. http://mobihealthnews.com/28165/happtique-suspends-mobile-health-app-certification-program/.
Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res. 2013;15:e89.
Albrecht UV, von Jan U, Jungnickel T, Pramann O. App-synopsis – standard reporting for medical apps. Stud Health Technol Inform. 2013;192:1154.
Divakaruni A, Armstrong M, Gruber-Baldini A, Shulman L. Discordance between medication apps and needs of patients with Parkinson's disease. Neurology. 2014;82:Supplement P3.067.
Marcano Belisario JS, Huckvale K, Greenfield G, Car J, Gunn LH. Smartphone and tablet self management apps for asthma. Cochrane Database Syst Rev. 2013;11, CD010013.
This work received no specific funding. However, KH’s doctoral studies are funded by the National Institute for Health Research (NIHR) under the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) programme for North West London. They played no role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: KH’s doctoral studies of which this work forms a part are funded by the NIHR CLAHRC for North West London; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
KH designed the study and data collection tools, performed and supervised searches, screening, and data extraction, cleaned and analyzed the data, and wrote the first draft of the paper; he is guarantor. JO and AG performed screening and data extraction and revised the draft paper. CM reviewed the analysis and revised the draft paper. JC contributed to the original study design and revised the draft paper. All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors read and approved the final manuscript.
Tables AF1–6. Contains: Table AF1. Operational criteria for assessing the comprehensiveness of asthma self-management educational materials. Criteria covering eight key educational domains, derived from UK British Thoracic Society/Scottish Intercollegiate Guideline Network (BTS/SIGN), US Expert Panel Report 3 (EPR-3), and Global Initiative for Asthma (GINA) guidelines. Table AF2. Management strategies used to assess consistency of asthma information with evidence-base. Strategies and evidence-base derived from UK BTS/SIGN, US EPR-3, and GINA guidelines. Table AF3. Ethical disclosure principles for smartphone apps. Adapted from the Health on The Net Foundation principles for health information on the internet. Table AF4. Categories of software issue considered during assessment. Categories used to group any software errors, user interface problems, or other issue encountered during testing. Table AF5. Data extraction template. Template into which details of apps were recorded for subsequent analysis. Table AF6. Operational criteria for assessing inhaler technique education. Criteria covering eight domains, derived from product manufacturer guidance and UK BTS/SIGN, US EPR-3, and GINA guidelines.
Tables AF7–12. Contains: Table AF7. Characteristics of excluded apps. Table detailing apps excluded from analysis and the reasons for exclusion. Table AF8. Characteristics of information apps. Table of apps containing written or multimedia information about asthma. Detailed results of evaluation for consistency and comprehensiveness with evidence-based guidelines and adherence to best-practices for content development disclosures. Table AF9. Characteristics of diary apps. Table of apps containing asthma diary and related tracking functions. Details of diary function types, action plan support, adherence to best-practices for content development disclosures, and any issues identified during testing. Table AF10. Characteristics of apps with other self-management functions. Table of apps containing self-management functions other than diaries and trackers such as air pollution information. Details of function types, adherence to best-practices for content development disclosures, and any issues identified during testing. Table AF11. Characteristics of apps with assessment functions. Table of apps containing calculator and questionnaire functions related to asthma diagnosis or management. Details of function types, validation information (if identified), and adherence to best-practices for content development disclosures and any issues identified during testing. Table AF12. Characteristics of apps with therapeutic instructions and functions. Table of apps with an intended therapeutic purpose. Details of function types, validation information (if identified), adherence to best-practices for content development disclosures, and any issues identified during testing.
About this article
Cite this article
Huckvale, K., Morrison, C., Ouyang, J. et al. The evolution of mobile apps for asthma: an updated systematic assessment of content and tools. BMC Med 13, 58 (2015). https://doi.org/10.1186/s12916-015-0303-x
- Cross-sectional study
- Mobile health
- Systematic assessment