Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions

  • Sarah J. Iribarren ,

    sjiribar@uw.edu

    Affiliation University of Washington, Department of Biobehavioral Nursing and Health Informatics, School of Nursing, Seattle, Washington, United States of America

    ORCID http://orcid.org/0000-0003-2980-0717

  • Kenrick Cato,

    Affiliations Columbia University, School of Nursing, New York, New York, United States of America, Office of Nursing Research, EBP and Innovation, New York-Presbyterian Hospital, New York, New York, United States of America

  • Louise Falzon,

    Affiliation Center for Behavioral Cardiovascular Health, Department of Medicine, Columbia University Medical Center, New York-Presbyterian Hospital, New York, New York, United States of America

  • Patricia W. Stone

    Affiliations Columbia University, School of Nursing, New York, New York, United States of America, Columbia University, School of Nursing, Center for Health Policy, New York, New York, United States of America

What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions

  • Sarah J. Iribarren, 
  • Kenrick Cato, 
  • Louise Falzon, 
  • Patricia W. Stone
PLOS
x

Abstract

Background

Mobile health (mHealth) is often reputed to be cost-effective or cost-saving. Despite optimism, the strength of the evidence supporting this assertion has been limited. In this systematic review the body of evidence related to economic evaluations of mHealth interventions is assessed and summarized.

Methods

Seven electronic bibliographic databases, grey literature, and relevant references were searched. Eligibility criteria included original articles, comparison of costs and consequences of interventions (one categorized as a primary mHealth intervention or mHealth intervention as a component of other interventions), health and economic outcomes and published in English. Full economic evaluations were appraised using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist and The PRISMA guidelines were followed.

Results

Searches identified 5902 results, of which 318 were examined at full text, and 39 were included in this review. The 39 studies spanned 19 countries, most of which were conducted in upper and upper-middle income countries (34, 87.2%). Primary mHealth interventions (35, 89.7%), behavior change communication type interventions (e.g., improve attendance rates, medication adherence) (27, 69.2%), and short messaging system (SMS) as the mHealth function (e.g., used to send reminders, information, provide support, conduct surveys or collect data) (22, 56.4%) were most frequent; the most frequent disease or condition focuses were outpatient clinic attendance, cardiovascular disease, and diabetes. The average percent of CHEERS checklist items reported was 79.6% (range 47.62–100, STD 14.18) and the top quartile reported 91.3–100%. In 29 studies (74.3%), researchers reported that the mHealth intervention was cost-effective, economically beneficial, or cost saving at base case.

Conclusions

Findings highlight a growing body of economic evidence for mHealth interventions. Although all studies included a comparison of intervention effectiveness of a health-related outcome and reported economic data, many did not report all recommended economic outcome items and were lacking in comprehensive analysis. The identified economic evaluations varied by disease or condition focus, economic outcome measurements, perspectives, and were distributed unevenly geographically, limiting formal meta-analysis. Further research is needed in low and low-middle income countries and to understand the impact of different mHealth types. Following established economic reporting guidelines will improve this body of research.

Introduction

Globally, mobile phone subscribers have grown from less than 1 billion in 2000 to more than 7 billion in 2015, corresponding to a penetration rate of 97% worldwide [1]. Capitalizing on this widespread use of mobile phones, researchers and implementers have used them as a catalyst for healthcare change to address disparities and inequities in health service access and delivery, geographic barriers, shortage of health care providers, and high health care costs [2, 3]. Mobile health (mHealth) is defined as the “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs), and other wireless devices” [4]. Ideally, mHealth improves health outcomes by, for example, efficiently and effectively increasing patient knowledge about a disease/condition, providing social support to those undergoing challenging treatment regimens of stigmatizing diseases, enhancing patient-provider communication, or improving communication and coordination across multidisciplinary care teams thereby improving quality of care delivery [5].

In addition to establishing the effectiveness of these interventions it is also crucial to understand their economic impact given the growing recognition across the globe that resources are finite. Economic evaluations can guide policymakers and funders in determining whether evidence supports wider adoption of mHealth interventions [6]. Such evaluations identify and compare alternative interventions, and assess incremental impact on health outcomes and their costs (differences between intervention under study and comparator intervention) [7].

mHealth is often assumed to be or described as cost-effective or cost-saving, yet the strength of the evidence supporting this assertion has been limited [810]. Despite optimism, unknown cost-effectiveness has been listed as one of the top six barriers to mHealth implementation [4] and as a key factor in limited mHealth policy investment [11]. There are a number of ways in which mHealth interventions may reduce health care costs including, but not limited to, decreasing transportation costs for patients or healthcare workers, addressing inefficient practices, decreasing time to diagnosis, keeping patients in their home longer versus costly health care facilities, or reducing hospital visits [12]. Prior systematic reviews of economic evidence of technology-based interventions have focused on telehealth/telemedicine, [13, 14] electronic health (eHealth) and/or specific diseases or populations,[1518] or a combination of telehealth, eHealth, mHealth [10]. Therefore, to our knowledge, no studies focused on understanding the potential economic impact of mHealth interventions broadly. The aim of this review was to summarize and assess the body of evidence related to economic evaluations of mHealth interventions.

Methods

The methods for reviewing economic evaluations included (1) applying explicit inclusion criteria to select studies; (2) developing a data abstraction form and characteristic categories to record individual study characteristics; (3) evaluating the quality of reporting of each study using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement checklist [19]; and (4) interpreting and summarizing the identified economic evaluations based on study questions. Questions included: What economic evaluation of mHealth interventions evidence is currently available? What types of mHealth interventions have undergone economic evaluations? What economic evaluation methods were used? What patient population, disease, or health outcomes are targeted? In what settings is research being conducted? Is there any evidence of the cost-effectiveness of mHealth interventions? If so, what is the quality of reporting the economic evidence? Of the identified studies, how many report that the intervention economic outcomes were positive (e.g., cost-effective, economically beneficial, or cost saving) at base case? The protocol for this systematic review was registered in PROSPERO (CRD42014014913) (www.crd.york.ac.uk/PROSPERO/). We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for this report [20]. The PRISMA checklist is in Supporting Information S1 Checklist.

Finding and selecting relevant studies

Search strategy.

Seven electronic bibliographic databases including MEDLINE (Ovid), EMBASE (EMBASE.com), CINAHL, The Cochrane Center Register of Controlled Trials (CENTRAL), NHS Economic Evaluation Database (NHS EED), Database of Abstracts of Reviews of Effects (DARE), and PsycInfo (Ovid) were searched to April, 2016. Subject headings and text words that encompassed the concepts of mobile health and economic evaluations were used. Search strategies were developed and run by an information specialist (LF). The search strategies are provided in Supporting Information S1 Appendix. Clinicaltrials.gov and WHO International Clinical Trials Registry Platform were searched for ongoing trials and sites such as mHealth Evidence, OpenGrey, HIMSS website were searched for grey literature from October—December, 2015. We further searched the cited references and reference lists of included studies (through ISI Web of Science) and relevant systematic reviews by hand to identify additional relevant studies.

Inclusion and exclusion criteria.

We included original articles published in English that compared the costs and consequences of at least two interventions, one of which was a mHealth intervention. mHealth interventions could be either the primary intervention or a component of an intervention. We excluded studies not meeting mHealth criteria such as telehealth with stationary devices (e.g., desktop videophone, desktop computer, videoconferencing equipment) unless they reported also using mobile devices (e.g., mobile phone or sensors), were web-based only, or were devices for clinical diagnosis (e.g., EEG) that did not report on a health outcome. We excluded reviews or commentaries of economic evaluations. Protocols for planned economic evaluations of mHealth interventions were identified and included in summary of study characteristics but excluded from the full analysis. While we classified and reported the number of partial economic evaluations identified, which measure only costs of an intervention without comparator (e.g., cost accounting of intervention, cost per patient or cost per an event), these provide limited insight [20].

Study selection process.

We used a web-based database management system, Early Review Organizing Software (EROS) for data management, developed by the Institute for Clinical Effectiveness in Argentina, to facilitate screening, identify and resolve discrepancies, and to produce the flow diagram. Two reviewers (SI, KC) screened the titles and abstracts of the retrieved records independently and excluded obviously irrelevant records. Full text articles were obtained for studies of possible inclusion for further evaluation. Final decisions were based on the consensus of both screeners. A third reviewer (PS) was included when uncertainty regarding eligibility arose. Authors of published research protocols and conference proceedings were contacted for final report, economic data, or estimated date of publication.

Development of data abstraction form and study characteristic categories

The data extraction form was developed in MS Excel to capture data for 48 study characteristics based on data points of reviews of economic evaluation [13, 15] and study questions. The data points included the type of economic evaluation, mHealth intervention type, mHealth as the primary intervention versus combined with other intervention strategies, target disease/condition, country, economic outcomes, costs, effectiveness and funding.

mHealth applications have been categorized in different ways. For example, the mHealth Compendium outlines five main categories of mHealth types or applications (e.g., behavior change communication, data collection, finance, logistics, and service delivery), [21] while others use six [22] or twelve categories [5]. To code the mHealth intervention type we used five categories of mHealth interventions and provide definitions and examples of what each include in Table 1. We selected the primary mHealth type with the understanding that other application categories may also be applicable. Often interventions integrate two or more types of mHealth applications (e.g., text message, app) to address a health need or health system constraints [22]. Country income level was classified according to the 2015 World Bank 4 ratings categories (Low income (LIC), lower-middle-income (LMIC), upper-middle-income economies (UMIC), upper income country (UIC)) [23].

There are several types of comprehensive economic evaluations methods including: cost minimization, cost-consequence, cost-effectiveness, cost-utility (i.e., a special type of cost-effectiveness analysis) and cost-benefit analysis [7, 24]. Each type of evaluation compares the costs of alternative strategies but vary in how effectiveness is measured [24]. Outcomes of full economic evaluations include estimates of cost and effectiveness, incremental cost effectiveness ratio (ICER), cost per life saved, disability adjusted life year (DALY), quality adjusted life year (QALY), time-savings gained, and measurement and comparisons of healthcare costs (e.g., costs for buying, implementing, running, representative monetary conversion factors, cost of mobile phone access and provision, and healthcare utilization) [8]. Consequences of health interventions can be evaluated using a number of approaches, for example, a single analytical study, a synthesis of studies, mathematical modeling, or a combination can be used to estimate health consequences [19]. Interpretation of results should reflect the constituents represented and is influenced by assumptions and values used to conduct the evaluation [25]. These assumptions include, for example, the perspective, time horizon, data source, and at which ‘threshold’ an intervention may be considered cost-effective in a given country or setting—such as $50,000 per Quality Adjusted Life Year (QALY) or considering the prevalence or severity of the condition studied. Study outcomes reporting intervention at base case as cost-effective, economically beneficial, or cost saving were coded as having a positive costing outcome or not [Y/N].

Quality of reporting assessment

To optimize the reporting of economic evaluations, a relatively new checklist of 24 items was developed through expert consensus, the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement [19]. Items include reporting target population, time horizon, discount rate, source of effectiveness and cost data, and currency, for example. This checklist has been endorsed by the ISPOR Health Economic Evaluations Publication Guidelines Task Force and co-published across 10 health economics and medical journals to improve reporting and, in turn, health and health care decisions. Full economic evaluations were evaluated for quality of reporting using the CHEERS statement by two authors independently (SI, KC) as a measure to assess risk of bias of reporting economic outcomes [19]. Any discrepancies were resolved with a third reviewer (PS). Although there is no standard, universally accepted method of critical appraisal of economic evaluations, there are several points of methodological quality that can be considered [25]. We used the quantity of reported CHEERS items and considerations described by Henrikson et al (2013) to discuss study quality.

Interpreting and summarizing economic evaluations

Findings were summarized and reported based on our study questions: number of economic evaluation of mHealth interventions; mHealth interventions types; economic evaluation methods; disease/condition focus; country where conducted; reporting quality; and evidence of the positive costing outcomes (e.g., cost-effective, economically beneficial, or cost saving) at base case. Disparate study designs, intervention type, study context, patient population, and types of economic analysis meant that formal meta-analysis was inappropriate. For this reason, we present a descriptive analysis of the studies included. For data points without predefined categories, as described above, we first free texted study characteristics and then recoded by grouping similar themes (e.g., similar disease/condition focus areas, similar costing perspectives). To summarize reporting quality we (1) categorized highest quality studies as the top 25th percentile (reporting 90–100% of the recommended items CHEERS guideline items); (2) reported most items missing from being reported, and (3) discussed quality evaluation based on the CHEERS assessment domains. To gain insight into what mHealth interventions showed promise, studies categorized as reporting positive costing outcome (eg. cost-effective, economically beneficial, or cost saving) at base case were reported by category in study characteristic summary table.

Results

Literature search and evaluation for study inclusion

The searches yielded a total of 8826 results. Subsequent searches in the grey literature and screening through Web of Science resulted in an additional 459. Of the 5902 studies screened after deduplication and excluding those clearly outside inclusion criteria, 318 were screened full text, and 39 economic evaluations were included. Of those excluded from final analysis were 30 protocols with planned economic evaluations and 18 classified as partial economic evaluations. Other reasons for exclusion are described in the PRISMA flow diagram (Fig 1).

Study characteristics

The number of published mHealth economic evaluations increased since 2012 (Fig 2). No full economic evaluations were identified up to the 2016 screening date. Of the 30 identified protocols 6 were published in 2013, 8 in 2014 and 16 in 2015. Eight corresponding authors of published protocols reported that their study was in progress and/or expected publication late 2016–2017.

Table 2 provides a summary of the study characteristics and corresponding number of studies with reported costing outcomes categorized as positive (e.g., cost-effective, economically beneficial, or cost saving). Most reported positive costing outcomes at base case (26, 74.3%). The 39 studies spanned 19 countries, the majority of which were conducted in upper and upper-middle income countries (34, 87.2%) compared to 5 in lower-middle or low-income countries (12.8%). Of these 70.6% and 100% reported positive costing outcomes, respectively. The majority evaluated an mHealth intervention as the primary intervention (35, 89.7%) versus as a component of an intervention (4, 10.3%), both categories with about three quarters reporting positive costing outcomes. Twenty seven of the 39 studies used a behavior change communication type interventions (e.g., improve attendance rates, medication adherence) (27, 69.2%) with high rates of reported positive costing outcomes (20, 74.1%). SMS was the mHealth function most often used in the interventions (e.g., used to send reminders, information, provide support, conduct surveys or collect data) (22, 56.4%) with 17 (77.3%) resulting in reported positive economic outcomes. The most frequent disease or condition focuses were outpatient clinic attendance, cardiovascular disease, and diabetes.

thumbnail
Table 2. Study characteristics summary of economic evaluations with reported positive costing outcomes.

https://doi.org/10.1371/journal.pone.0170581.t002

Cost-effectiveness analysis (CEA) was the predominant economic evaluation method (25, 64.1%) and cost utility analysis (CUA) had the highest within group positive costing outcomes (10, 83.3%). The costing perspective was mostly from the payer/health service provider/program/employer perspective; however, a large number did not clearly report the costing perspective (16, 41%).

Quality of reporting of full economic evaluations (CHEERS)

Supporting Information S2 Appendix provides the evaluation of each study by CHEERS item checklist. Table 3 summarizes the CHEERS items missing when expected for all studies. The average percent of items reported in the studies was 79.6% (range 47.62–100, STD 14.18). The five items most likely not to be reported were: characterization of heterogeneity (29, 74.4%); characterizes uncertainty—sensitivity of incremental costs (single study-based) (17, 61.7%); identifying the study as an economic evaluation in the title (19, 48.7%); stating currency, price date, conversion (18, 46.2%); and stating study perspective (16, 41%).

Twelve (30.7%) of the full economic evaluations reported greater than 90% of the CHEERS items, ranking them in the 25th percentile, thus are considered of higher reporting quality [2637]. Supporting Information S3 Appendix provides a summary of these studies intervention comparators, time horizon, discount rate, outcomes and findings. These studies were all published between 2012 and 2016, whereas the publication dates of those not consistent with CHEERS items ranged from 2005 to 2015. Of the economic evaluations reporting the highest CHEERS items, nine (75%) were behavior change communication type based interventions using text messaging as the primary intervention to send reminders or support [2832, 34, 36, 37] or an app [35]. Two were service delivery based using an app to replace surgical follow-up visits [27] and an iPhone based sensor for screening [33, 27]. One was primarily for data collection using a personal digital assistant (PDA) [26]. There was a range of disease/condition focuses, for example, tuberculosis control,[31] diabetes prevention and management, [26, 36] and malaria management [37]. Most were CUA (8, 66.7%), the remainder were CEA. Of these studies, 4 used primary RCT as their effectiveness data source [28, 30, 34, 37] and 5 drew from prior RCT or multiple studies [27, 29, 31, 32, 36]. Eleven used usual care, current practice or a control group not receiving intervention as the comparator intervention and one used medication self-administration as the comparator group [31]. The interventions and comparators for top 25th percentile reporting are listed in Appendix B and Appendix C summarizes studies by mHealth type. Five applied a lifetime time horizon [2931, 33, 36] while others used four years or less. The majority applied a discount rate of 3–5% (Add definition of what a discount rate is) (9, 75%) [26, 2833, 36]. Only two calculated costs based on a societal perspective [27, 30]. All conducted sensitivity analyses.

In contract, in the lowest 25th percentile (n = 11), 5 used RCT data [3841] while others used effectiveness data from pilot or observational studies [4247]. However, none reported the study perspective or characterized heterogeneity, and only one [48] included sensitivity analyses. Study duration for all were reported as under 6 months, therefore, not requiring discount rates [48].

Protocols with planned economic evaluations

Twenty-eight of the protocols of mHealth based interventions describe full economic evaluations (five from the grey literature). Of these, one is a primary CEA, one is a prospective cohort study with planned CEA, another is a fractional factorial design with CEA and the remaining are RCTs with CEA. Two were categorized as partial economic evaluations (e.g., cost accounting, and partial and direct costs). One focuses on data collection mHealth intervention type, 7 focus on service delivery and the majority focus on behavior change communication based interventions (26, 87%). The focus conditions vary widely. For example, six will focus on HIV and four each on diabetes, physical activity, and pulmonary issues. Other topics include risk reduction of binge drinking, self-harm, and injury prevention.

Discussion

Overview

Economic evaluations facilitate the comparison between interventions in terms of their costs and consequences and can be used to guide decision makers or funders in determining if mHealth-based interventions improve health outcomes relative to other existing interventions and if the cost to adopt and maintain the intervention in a system or setting is justified [49]. mHealth guidelines recommend the use of economic evaluation tailored reporting standards, such as the CHEERS checklist, for full economic evaluations and support the reporting of basic costs assessment of the mHealth intervention from varying perspectives [50]. In our systematic review, we provide a summary of the economic evidence of mHealth and confirm the common criticism that cost-effectiveness is often assumed, without evidence to support it. In fact, of the excluded studies during screening, 57% included statements of cost-effectiveness or cost in the abstract or title but upon further evaluation did not provide enough detail to be considered a partial or full economic evaluation. However, we did identify more economic evaluations of mHealth than expected given findings from prior reviews [10, 15]. Our review provides an overview of full economic evaluations and highlights a growing number of published planned economic evaluations. Findings show a diverse range of mHealth interventions, focus conditions, and types of mHealth tools used in the interventions evaluated for economic impact. The majority evaluated mHealth as the primary intervention (versus a component of the intervention) and were conducted in upper and upper-middle income countries.

All studies included a comparison of effectiveness of a health-related outcome and reported economic data. However, many did not report all recommended economic outcome items, were not titled/reported as a full economic evaluation, or did not calculate a summary measure. To ensure transparency the authors should provide detail of data sources, assumptions made regarding modeling of data, funding source, and the role of the funder in the analysis and reporting of the study [25]. Over half reported 80% or less of the recommended criteria. Regardless of the quality of reporting, overall there was consistent reporting of positive economic outcomes (e.g., increase in life years gained, cost savings, cost-effectiveness) across mHealth type and cost calculation perspectives. Although findings from this review support cost effectiveness of mHealth interventions, this result must be considered with caution. It is important to evaluate case by case and additional research is needed to identify mHealth components that contribute most to positive outcomes.

Selecting the most appropriate methodology and data collection strategy is important to increase the transferability of findings across economic evaluations [6]. The source of data from randomized control trials or a rigorous prospective cohort study are considered high quality while expert opinion is of low quality due to risk of bias [25]. How outcomes are measured is another quality consideration. High quality outcomes use reliable and validated instruments and/or clinical endpoints (e.g., disease specific) and state how they were calculated and ideally sample people affected by condition or from a general community population [25]. Research designs, sample sizes, and economic reporting quality varied. Three of the studies were pilot study designs,[42, 43, 46] and had small sample size which lack economies of scale (which would indicate the intervention may be even more cost saving) or lack volume (which would indicate more utilization that may increase costs). Some of these studies did not self-report as formal cost-effectiveness studies and instead, for example, reported methods of cost outcome analyses [32, 51]. Additionally, others reported outcomes of effectiveness and difference in costs compared with other interventions or modeled multiple scenarios. Various assumptions about modeling may introduce risk such as using poor quality studies or data that does not reflect current practice that may favor one intervention over the other [25]. For example, Moore et al., assessed effectiveness of a technology supported intervention to support management of hypertension and compared costs between intervention and standard care, identifying cost savings [52]. Similarly, O’Leary et al., evaluated text messaging to increase vaccination outcomes and reported cost scenarios [53]. However, neither calculated a summary measure such as an incremental cost-effectiveness. In another example, Chang et al., calculated costs per outcome averted (e.g., virologic failure and patient lost to follow-up averted) of a peer health worker intervention compared with mHealth supported peer health workers intervention to report patient clinical data to centralized staff. Their findings were based on threshold analyses to identify costs to avert an unwanted outcome and the associated cost savings [54]. Costs and savings were also calculated for implementation at large-scale based on pilot study findings [51]. Efficiency was used as a measure of cost-effectiveness. For example, Bingna et al., defined efficiency as improvements in the primary efficacy outcome relative to the staff working time used and the direct financial costs of the intervention [38]. Cost-effectiveness evaluations were often secondary outcomes or reported in results. Koshy [43] and Perron et al., [40] reported the cost-effectiveness methods within results and discussion sections. Similarly, Loranzo-Fuentes et al., reported cost-effectiveness as a primary study aim, but briefly described the cost analysis methods in the result section [45]. Downer et al., modeled financial benefits that could result from increasing outpatient attendance using text messaging and although the authors report the methods as cost-effectiveness, it was classified as a cost benefit analysis because the outcome was measured as cost per success and the reported cost difference was in monetary units [55]. However, several of the CHEERS quality items were not reported in this study. Joo et al., reported the short intervention period prohibited the calculation of QALYs as a limitation [56].

To be conservative, we reported results as positive or probable cost-effectiveness if one or more of the primary outcomes showed a positive economic outcome. There were cases where secondary outcomes had increased likelihood of cost-effectiveness for an intervention. For example, Maddison et al., identified a mobile phone intervention as not cost-effective compared with usual care for the primary outcome of exercise capacity; however, there were results of cost-effectiveness for secondary outcomes [34]. Similarly, some studies reported the intervention as not cost-effective at base case. However, as the number of patients treated increased, the treatment became less expensive. For example, at approximately 1600 users the app evaluated by Luxton et al., became less expensive than in-office treatment [57]. Furthermore, because of the large number of potential app users, an estimated $USD 2.7–2.9 million societal savings was calculated. The estimates of probability falling below a recognized value threshold for the cost per QALY gained varied based on access to existing software [58].

A main objective of medical research is to improve the health of a population; therefore, conducting economic evaluations from a societal perspective is preferred [59]. The study perspective, such as patient, payer, provider, health systems each take into consideration differing costs and outcomes. A societal perspective is considered high quality because it includes both full direct and indirect costs, such as direct costs to patients and opportunity costs regardless of who bears the costs or receives the effects [25]. Since research is expensive and exposes patients to the risk of experimentation, it is important to think beyond effectiveness of the intervention and to take into consideration the potential to incur opportunity costs [49]. Only two full economic evaluations reported analyses from a societal perspective [27, 30]. Both of these studies also ranked within the top 25th percentile of CHEERS items reported, representing good quality economic evaluations. Those with lower quality score often did not describe the perspective. In such cases from the costs calculated it can often be assumed that the perspective is from the intervention implementer and sometimes the patient or service user costs are also included. Some studies included a mix of perspectives (e.g., provider or health care sector and service user) or at various levels of implementation (e.g., start up, regional level, national level). Similarly, although the majority of the studies included the currency used in their analyses, of those categorized as not reporting the CHEERS item ‘currency, price date, and conversion,’ most failed to include the date or year the costs were calculated. The lack of this information limits reviewers’ ability to convert and compare to other similar studies. In addition, time horizon was not reported in about a quarter of the studies. A time horizon takes into account preferences for future benefits over immediate benefits and applies a discount rate, typically 3 and 5%. A high-quality study will use a time horizon of over ten years [25]. In evaluations of response rate or clinic attendance, it seems appropriate to use a short time horizon. However, for behavior change interventions, e.g., smoking cessation or adherence to medication for a chronic disease, longer time horizons may be necessary.

As technology based interventions and mHealth, in particular, are relatively new it is not feasible to wait for lifetime data to validate cost-effectiveness. Consequently, it is not surprising that many of the identified economic evaluations used modeling techniques to simulate disease projections over a lifetime or long period while incorporating effectiveness and cost evidence. In so doing it is important to calculate and represent uncertainty [60]. For a study to be considered high quality sensitivity analyses are needed to evaluate factors that most influence results and failing to account for the range of adverse events can induce bias [25]. There are a number of methods used to explore uncertainty in economic evaluations (e.g., one-way and multiway sensitivity analyses, threshold analyses, analyses of extremes and probabilistic sensitivity analyses) [49]. For the studies included in this review that applied a model-based economic evaluation, all described the calculation and reported of uncertainty/sensitivity of incremental costs and most used a Monte Carlo simulation for probabilistic sensitivity analysis. In contrast, for single study-based economic evaluations, the uncertainty of sampling together with the impact of methodological assumptions (e.g., discount rate, study perspective) was described in less than half.

Behavior change communication interventions were the most represented mHealth intervention type identified in our review. This finding is consistent with others reporting behavior change communication interventions as the predominant and most successful of all mHealth interventions [21]. No economic evaluations were found for three of the domains (human resource management, financial transactions and incentives, and logistics/supply management). Similarly, a review of mHealth in low- and middle-income countries by Hall et al., showed no studies were identified as human resource management nor financial transactions and incentives based interventions [61]. In addition, although mHealth is recognized as drawing from a range of tools and often combined with other strategies,[5] the majority in our study assessed mHealth tools as the primary intervention type and fewer as a secondary or combined component. One exception was Joo et al., who assessed a remote type Internet based intervention with twice weekly SMS prompts for behavior modifications in combination with other intervention components [56].

Text messaging was the predominant tool assessed in the studies, consistent with mHealth literature [21, 62]. Text-messaging interventions are popular because they can be sent, stored, answered and retrieved at the user’s convenience; they are relatively inexpensive; and they are available for any type of phone [61, 6365]. Increasingly, SMS reminder systems are being used by healthcare systems to counter the negative impacts of missed appointments, such as lost revenue to the health care system, potential poor impact on patient health and treatment outcomes, and system efficiency [66]. The largest number of identified economic evaluation studies focused on assessing cost outcomes of SMS appointment reminders to increase outpatient clinic attendance. All but one found positive economic outcomes [43]. In our review, four focused on diabetes management or prevention, and each of these identified the intervention as cost-effective. However, a systematic review of text messaging interventions concluded that text messaging benefits are still unclear because most studies have used self-reported adherence measures, omitted the measurement of clinical outcomes, and neglected to evaluate beyond the active intervention period to determine lasting effects or to assess baseline adherence difference [65, 67].

Apps are reported to be an ideal platform for behavior change because of their popularity, connectivity, and increased sophistication [68]. Apps can support added functionalities beyond, text messaging, for example. They have the potential for real-time data collection, graphic feedback, interactivity, and links to social functionalities. In our review there were apps identified as cost-effective for follow-up care for low-risk postoperative ambulatory patients [27]. Telephone follow-up may decrease costs and time for follow-up care compared with in-person, but requires synchronous communication and often multiple calls. In contrast, mobile app follow-up can collect and relay data asynchronously and to those in need of evaluating the information, such as surgeons [27].

Interventions to improve disease detection or point of care testing using mobile phone based technologies is an area of mHealth considered to have high potential to increase access to rapid testing and be cost savings [69, 70]. In our study, we identified an app to screen for atrial fibrillation [33]. Authors have noted that widespread implementation remains subject to several challenges and pending issues [70].

Implications for researchers, health professionals, and policy makers

The increasing focus of economic evaluations of mHealth interventions means there is a need for careful reporting and rigorous evaluation. This review highlights that there is moderate to high quality of economic reporting, but that there is a lack of evaluations in low-middle and low-income counties. And there is significant heterogeneity in terms of settings, costing strategies, and length of follow-up periods, limiting the conclusions to be drawn. The high-quality studies clearly described what was included in the costs and how they were calculated. We encourage the use of the CHEERS checklist for reporting of economic evaluations and for authors to refrain from including statements of cost-effectiveness in their findings in the absence of an economic evaluation.

Limitations

We systematically searched mHealth research with economic evaluations and included multiple cost outcomes in the developed search criteria. However, as reflected in the CHEERS evaluation, less than half reported economic evaluation in the title and a clear cost outcome was not always identified in the abstract. In abstracts including statements about a cost outcome, we evaluated the full text for inclusion. Nonetheless, economic evaluations not mentioned in the title or abstract may have been missed. Because we found a high percentage of the studies reporting a positive outcome, it is possible that there may be a publication bias and fewer negative findings are being published. Articles not published in English were not included, which is another limitation.

Conclusions

The body of economic evaluations of mHealth interventions is growing, evidencing researchers’ response to the call as one of mHealth’s major gaps in further implementation and scale up. A number of the studies were rated as reporting high quality evidence and findings suggest high rates of reporting positive costing outcomes using mHealth interventions compared with usual care or other comparators. Although findings from this review support cost effectiveness of mHealth interventions, this result must be considered with caution. It is important to evaluate case by case. All studies compared intervention effectiveness on a health-related outcome and reported economic data, however many did not report all recommended economic outcome items and were lacking in comprehensive analysis. Further attention is needed to follow established economic reporting guidelines to improve the body of evidence. Due to few similarities in the interventions which precluded any quantitative synthesis such as, varied target disease/condition focus, intervention comparators, economic outcomes measures and an uneven geographical distribution, caution is needed in drawing a conclusion of economic evidence of mHealth interventions to date. Further research is needed in low and low-middle income countries to understand the impact of mHealth components that contribute most to positive outcomes. The growing number of planned economic evaluations, along with improved reporting and further targeted synthesis of economic evaluations, will help guide policymakers and funders.

Supporting information

S2 Appendix. Study evaluation by CHEERS item checklist.

https://doi.org/10.1371/journal.pone.0170581.s003

(DOCX)

S3 Appendix. Summary of high quality economic evaluations.

https://doi.org/10.1371/journal.pone.0170581.s004

(DOCX)

Acknowledgments

We would like to thank Sean Broomhead and Tom Jones at tinTree (tintree.org) for their review and guidance for this project and Kristine Kulage for her manuscript drafting assistance.

Author Contributions

  1. Conceptualization: SI KC PS LF.
  2. Data curation: SI KC LF.
  3. Formal analysis: SI KC PS.
  4. Funding acquisition: PS.
  5. Investigation: SI KC.
  6. Methodology: SI KC PS.
  7. Project administration: SI.
  8. Supervision: PS.
  9. Validation: SI KC PS.
  10. Visualization: SI KC LF.
  11. Writing – original draft: SI KC LF PS.
  12. Writing – review & editing: SI KC PS LF.

References

  1. 1. International telecommunication Union. The World in 2015: ICT Facts & Figures. 2015.
  2. 2. Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. Journal of medical Internet research. 2013;15(5):e95. Epub 2013/05/24. pmid:23697600
  3. 3. Mendoza G, Okoko L, Konopka S, Jonas E. mHealth Compendium, Volume Three. Arlington, VA: African Strategies for Health project, Management Sciences for Health; 2013.
  4. 4. World Health Organization. mHealth: New horizons for health through mobile technologies. Geneva, Switzerland http://www.who.int/goe/publications/goe_mhealth_web.pdf, 2011.
  5. 5. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Health Sci Pract. 2013;1(2):160–71. Epub 2013/08/01. pmid:25276529
  6. 6. Bergmo TS. How to Measure Costs and Benefits of eHealth Interventions: An Overview of Methods and Frameworks. Journal of medical Internet research. 2015;17(11):e254. pmid:26552360
  7. 7. Drummond MF S M, Torrance G, O’Brien J, Stoddart GL. Methods for the economic evaluation of health care programmes 3rd ed. Oxford University Press 2005.
  8. 8. Schweitzer J, Synowiec C. The economics of eHealth and mHealth. Journal of health communication. 2012;17 Suppl 1:73–81.
  9. 9. Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC public health. 2014;14:188. pmid:24555733
  10. 10. de la Torre-Diez I, Lopez-Coronado M, Vaca C, Aguado JS, de Castro C. Cost-utility and cost-effectiveness studies of telemedicine, electronic, and mobile health systems in the literature: a systematic review. Telemedicine journal and e-health: the official journal of the American Telemedicine Association. 2015;21(2):81–5.
  11. 11. Chib A, van Velthoven MH, Car J. mHealth adoption in low-resource environments: a review of the use of mobile healthcare in developing countries. Journal of health communication. 2015;20(1):4–34. pmid:24673171
  12. 12. Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? Jama. 2013;310(22):2395–6. pmid:24158428
  13. 13. Whitten PS, Mair FS, Haycox A, May CR, Williams TL, Hellmich S. Systematic review of cost effectiveness studies of telemedicine interventions. Bmj. 2002;324(7351):1434–7. pmid:12065269
  14. 14. Zhai YK, Zhu WJ, Cai YL, Sun DX, Zhao J. Clinical- and cost-effectiveness of telemedicine in type 2 diabetes mellitus: a systematic review and meta-analysis. Medicine (Baltimore). 2014;93(28):e312.
  15. 15. Elbert NJ, van Os-Medendorp H, van Renselaar W, Ekeland AG, Hakkaart-van Roijen L, Raat H, et al. Effectiveness and cost-effectiveness of ehealth interventions in somatic diseases: a systematic review of systematic reviews and meta-analyses. Journal of medical Internet research. 2014;16(4):e110. pmid:24739471
  16. 16. Naslund JA, Marsch LA, McHugo GJ, Bartels SJ. Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature. Journal of mental health. 2015;24(5):321–32. pmid:26017625
  17. 17. Badawy SM, Kuhns LM. Economic Evaluation of Text-Messaging and Smartphone-Based Interventions to Improve Medication Adherence in Adolescents with Chronic Health Conditions: A Systematic Review. JMIR Mhealth Uhealth. 2016;4(4):e121. pmid:27780795
  18. 18. Chen YF, Madan J, Welton N, Yahaya I, Aveyard P, Bauld L, et al. Effectiveness and cost-effectiveness of computer and other electronic aids for smoking cessation: a systematic review and network meta-analysis. Health technology assessment. 2012;16(38):1–205, iii–v. pmid:23046909
  19. 19. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Value in health: the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2013;16(2):e1–5.
  20. 20. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009;6(7):e1000097. pmid:19621072
  21. 21. Levine R, Corbacio A, Konopka S, Saya U, Gilmartin C, Paradis J, et al. mHealth Compendium, Volume Five. Arlington, VA African Strategies for Health, Management Sciences for Health, 2015.
  22. 22. Global Health eLearning Center. mHealth Basics: Introduction to Mobile Technology for Health. In: Lee CK, Raney L, L'Engle K, editors. 2014.
  23. 23. Country and Lending Groups [Internet]. 2016 [cited July 18].
  24. 24. Frick K, Cohen C, Stone PW. Economic outcomes and analyses in advanced practice nursing in Kleinpell R. (Ed.): Outcome Assessment in Advanced Practice Nursing. (3rd ed). 3rd ed. Kleinpell R, editor 2013.
  25. 25. Henrikson NB, Skelly AC. Economic studies part 2: evaluating the quality. Evid Based Spine Care J. 2013;4(1):2–5. pmid:24436693
  26. 26. Adepoju OE, Bolin JN, Ohsfeldt RL, Phillips CD, Zhao H, Ory MG, et al. Can chronic disease management programs for patients with type 2 diabetes reduce productivity-related indirect costs of the disease? Evidence from a randomized controlled trial. Popul Health Manag. 2014;17(2):112–20. pmid:24152055
  27. 27. Armstrong KA, Semple JL, Coyte PC. Replacing ambulatory surgical follow-up visits with mobile app home monitoring: modeling cost-effective scenarios. Journal of medical Internet research. 2014;16(9):e213. pmid:25245774
  28. 28. Burn E, Marshall AL, Miller YD, Barnett AG, Fjeldsoe BS, Graves N. The cost-effectiveness of the MobileMums intervention to increase physical activity among mothers with young children: a Markov model informed by a randomised controlled trial. BMJ Open. 2015;5(4):e007226. pmid:25926145
  29. 29. Guerriero C, Cairns J, Roberts I, Rodgers A, Whittaker R, Free C. The cost-effectiveness of smoking cessation support delivered by mobile phone text messaging: Txt2stop. Eur J Health Econ. 2013;14(5):789–97. pmid:22961230
  30. 30. Hagberg LA, Brekke HK, Bertz F, Winkvist A. Cost-utility analysis of a randomized controlled weight loss trial among lactating overweight/obese women. BMC public health. 2014;14:38. pmid:24428802
  31. 31. Hunchangsith P, Barendregt JJ, Vos T, Bertram M. Cost-effectiveness of various tuberculosis control strategies in Thailand. Value in health: the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2012;15(1 Suppl):S50–5.
  32. 32. Larsen-Cooper E, Bancroft E, Rajagopal S, O'Toole M, Levin A. Scale Matters: A Cost-Outcome Analysis of an m-Health Intervention in Malawi. Telemedicine journal and e-health: the official journal of the American Telemedicine Association. 2016;22(4):317–24.
  33. 33. Lowres N, Neubeck L, Salkeld G, Krass I, McLachlan AJ, Redfern J, et al. Feasibility and cost-effectiveness of stroke prevention through community screening for atrial fibrillation using iPhone ECG in pharmacies. The SEARCH-AF study. Thromb Haemost. 2014;111(6):1167–76. pmid:24687081
  34. 34. Maddison R, Pfaeffli L, Whittaker R, Stewart R, Kerr A, Jiang Y, et al. A mobile phone intervention increases physical activity in people with cardiovascular disease: Results from the HEART randomized controlled trial. Eur J Prev Cardiol. 2015;22(6):701–9. pmid:24817694
  35. 35. Ryan D, Price D, Musgrave SD, Malhotra S, Lee AJ, Ayansina D, et al. Clinical and cost effectiveness of mobile phone supported self monitoring of asthma: multicentre randomised controlled trial. Bmj. 2012;344:e1756. pmid:22446569
  36. 36. Wong CK, Jiao FF, Siu SC, Fung CS, Fong DY, Wong KW, et al. Cost-Effectiveness of a Short Message Service Intervention to Prevent Type 2 Diabetes from Impaired Glucose Tolerance. J Diabetes Res. 2016;2016:1219581. pmid:26798647
  37. 37. Zurovac D, Larson BA, Sudoi RK, Snow RW. Costs and cost-effectiveness of a mobile phone text-message reminder programmes to improve health workers' adherence to malaria guidelines in Kenya. PloS one. 2012;7(12):e52045. pmid:23272206
  38. 38. Bigna JJ, Noubiap JJ, Kouanfack C, Plottel CS, Koulla-Shiro S. Effect of mobile phone reminders on follow-up medical care of children exposed to or infected with HIV in Cameroon (MORE CARE): a multicentre, single-blind, factorial, randomised controlled trial. Lancet Infect Dis. 2014;14(7):600–8. pmid:24932893
  39. 39. Chen ZW, Fang LZ, Chen LY, Dai HL. Comparison of an SMS text messaging and phone reminder to improve attendance at a health promotion center: a randomized controlled trial. J Zhejiang Univ Sci B. 2008;9(1):34–8. pmid:18196610
  40. 40. Junod Perron N, Dao MD, Righini NC, Humair JP, Broers B, Narring F, et al. Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Serv Res. 2013;13:125. pmid:23557331
  41. 41. Moore J, Marshall M, Judge M, Moss F, Gilroy S, Crocker B, Zusman R. Technology-Supported Apprenticeship in the Management of Hypertension: A Randomized Controlled Trial. JCOM. 2014;21(3).
  42. 42. Farley JE, Srinivasan A, Richards A, Song X, McEachen J, Perl TM. Handheld computer surveillance: shoe-leather epidemiology in the "palm" of your hand. Am J Infect Control. 2005;33(8):444–9. pmid:16216657
  43. 43. Koshy E, Car J, Majeed A. Effectiveness of mobile-phone short message service (SMS) reminders for ophthalmology outpatient appointments: observational study. BMC ophthalmol. 2008;8:9. pmid:18513438
  44. 44. Li Y, Wang W, van Velthoven MH, Chen L, Car J, Rudan I, et al. Text messaging data collection for monitoring an infant feeding intervention program in rural China: feasibility study. Journal of medical Internet research. 2013;15(12):e269. pmid:24305514
  45. 45. Lozano-Fuentes S, Wedyan F, Hernandez-Garcia E, Sadhu D, Ghosh S, Bieman JM, et al. Cell phone-based system (Chaak) for surveillance of immatures of dengue virus mosquito vectors. J Med Entomol. 2013;50(4):879–89. pmid:23926788
  46. 46. Mahmud N, Rodriguez J, Nesbit J. A text message-based intervention to bridge the healthcare communication gap in the rural developing world. Technol Health Care. 2010;18(2):137–44. pmid:20495253
  47. 47. Nundy S, Dick JJ, Chou CH, Nocon RS, Chin MH, Peek ME. Mobile phone diabetes project led to improved glycemic control and net savings for Chicago plan participants. Health Aff (Millwood). 2014;33(2):265–72.
  48. 48. Au-Yeung KY, DiCarlo L. Cost comparison of wirelessly vs. directly observed therapy for adherence confirmation in anti-tuberculosis treatment. Int J Tuberc Lung Dis. 2012;16(11):1498–504. pmid:23006834
  49. 49. Edlin R, McCabe C, Hulme C, Hall P, Wright J. Cost Effectiveness Modelling for Health Technology Assessment. Cham: Adis Adis; 2015.
  50. 50. Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. Bmj. 2016;352:i1174. pmid:26988021
  51. 51. McInnes DK, Petrakis BA, Gifford AL, Rao SR, Houston TK, Asch SM, et al. Retaining homeless veterans in outpatient care: a pilot study of mobile phone text message appointment reminders. Am J Public Health. 2014;104 Suppl 4:S588–94.
  52. 52. Moore JO. Technology-supported apprenticeship in the management of chronic disease. JCOM. 2014;21(3).
  53. 53. O'Leary ST, Lee M, Lockhart S, Eisert S, Furniss A, Barnard J, et al. Effectiveness and Cost of Bidirectional Text Messaging for Adolescent Vaccines and Well Care. Pediatrics. 2015;136(5):e1220–7. pmid:26438703
  54. 54. Chang LW, Kagaayi J, Nakigozi G, Serwada D, Quinn TC, Gray RH, et al. Cost analyses of peer health worker and mHealth support interventions for improving AIDS care in Rakai, Uganda. AIDS care. 2013;25(5):652–6. pmid:22971113
  55. 55. Downer SR, Meara JG, Da Costa AC, Sethuraman K. SMS text messaging improves outpatient attendance. Aust Health Rev. 2006;30(3):389–96. pmid:16879098
  56. 56. Joo NS, Park YW, Park KH, Kim CW, Kim BT. Cost-effectiveness of a community-based obesity control programme. J Telemed Telecare. 2010;16(2):63–7. pmid:20008053
  57. 57. Luxton DD, Hansen RN, Stanfill K. Mobile app self-care versus in-office care for stress reduction: a cost minimization analysis. J Telemed Telecare. 2014;20(8):431–5. pmid:25316037
  58. 58. Snooks HA, Carter B, Dale J, Foster T, Humphreys I, Logan PA, et al. Support and Assessment for Fall Emergency Referrals (SAFER 1): cluster randomised trial of computerised clinical decision support for paramedics. PloS one. 2014;9(9):e106436. pmid:25216281
  59. 59. Jonsson B. Ten arguments for a societal perspective in the economic evaluation of medical innovations. Eur J Health Econ. 2009;10(4):357–9. pmid:19618224
  60. 60. Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation (Handbooks in Health Economic Evaluation). 1st ed 2006.
  61. 61. Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health. 2015;36:393–415. pmid:25785892
  62. 62. Forrest JI, Wiens M, Kanters S, Nsanzimana S, Lester RT, Mills EJ. Mobile health applications for HIV prevention and care in Africa. Curr Opin HIV AIDS. 2015;10(6):464–71. pmid:26352394
  63. 63. Finitsis DJ, Pellowski JA, Johnson BT. Text message intervention designs to promote adherence to antiretroviral therapy (ART): a meta-analysis of randomized controlled trials. PloS one. 2014;9(2):e88166. pmid:24505411
  64. 64. Mbuagbaw L, Mursleen S, Lytvyn L, Smieja M, Dolovich L, Thabane L. Mobile phone text messaging interventions for HIV and other chronic diseases: an overview of systematic reviews and framework for evidence transfer. BMC Health Serv Res. 2015;15(1):33. Epub 2015/01/23.
  65. 65. Thakkar J, Kurup R, Laba TL, Santo K, Thiagalingam A, Rodgers A, et al. Mobile Telephone Text Messaging for Medication Adherence in Chronic Disease: A Meta-analysis. JAMA Intern Med. 2016;176(3):340–9. pmid:26831740
  66. 66. McLean SM, Booth A, Gee M, Salway S, Cobb M, Bhanbhro S, et al. Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence. 2016;10:479–99. pmid:27110102
  67. 67. Nieuwlaat R, Mistry N, Haynes RB. Mobile Text Messaging and Adherence of Patients to Medication Prescriptions: A txt a dA keeps da doctR awA? JAMA Intern Med. 2016;176(3):350–1. pmid:26831149
  68. 68. Hale K, Capra S, Bauer J. A Framework to Assist Health Professionals in Recommending High-Quality Apps for Supporting Chronic Disease Self-Management: Illustrative Assessment of Type 2 Diabetes Apps. JMIR Mhealth Uhealth. 2015;3(3):e87. pmid:26369346
  69. 69. St John A, Price CP. Existing and Emerging Technologies for Point-of-Care Testing. Clin Biochem Rev. 2014;35(3):155–67. pmid:25336761
  70. 70. Vashist SK, Luppa PB, Yeo LY, Ozcan A, Luong JH. Emerging Technologies for Next-Generation Point-of-Care Testing. Trends Biotechnol. 2015;33(11):692–705. pmid:26463722