Effectiveness of eHealth interventions for reducing mental health conditions in employees: A systematic review and meta-analysis

Background Many organisations promote eHealth applications as a feasible, low-cost method of addressing mental ill-health and stress amongst their employees. However, there are good reasons why the efficacy identified in clinical or other samples may not generalize to employees, and many Apps are being developed specifically for this group. The aim of this paper is to conduct the first comprehensive systematic review and meta-analysis evaluating the evidence for the effectiveness and examine the relative efficacy of different types of eHealth interventions for employees. Methods Systematic searches were conducted for relevant articles published from 1975 until November 17, 2016, of trials of eHealth mental health interventions (App or web-based) focused on the mental health of employees. The quality and bias of all identified studies was assessed. We extracted means and standard deviations from published reports, comparing the difference in effect sizes (Hedge’s g) in standardized mental health outcomes. We meta-analysed these using a random effects model, stratified by length of follow up, intervention type, and whether the intervention was universal (unselected) or targeted to selected groups e.g. “stressed”. Results 23 controlled trials of eHealth interventions were identified which overall suggested a small positive effect at both post intervention (g = 0.24, 95% CI 0.13 to 0.35) and follow up (g = 0.23, 95% CI 0.03 to 0.42). There were differential short term effects seen between the intervention types whereby Mindfulness based interventions (g = 0.60, 95% CI 0.34 to 0.85, n = 6) showed larger effects than the Cognitive Behaviour Therapy (CBT) based (g = 0.15, 95% CI 0.02 to 0.29, n = 11) and Stress Management based (g = 0.17, 95%CI -0.01 to 0.34, n = 6) interventions. The Stress Management interventions however differed by whether delivered to universal or targeted groups with a moderately large effect size at both post-intervention (g = 0.64, 95% CI 0.54 to 0.85) and follow-up (g = 0.69, 95% CI 0.06 to 1.33) in targeted groups, but no effect in unselected groups. Interpretation There is reasonable evidence that eHealth interventions delivered to employees may reduce mental health and stress symptoms post intervention and still have a benefit, although reduced at follow-up. Despite the enthusiasm in the corporate world for such approaches, employers and other organisations should be aware not all such interventions are equal, many lack evidence, and achieving the best outcomes depends upon providing the right type of intervention to the correct population.

Introduction Mental health problems in the workforce are common and have a substantial impact on employee wellbeing, productivity, absenteeism, compensation claims, and the social welfare systems [1][2][3]. Reported annual productivity losses to organisations and workforces caused by mental health conditions are estimated at $225US billion in the USA [4] and £13 billion in the UK [5].
The majority of mental ill-health seen in the workforce is due to common mental disorders, most notably, depression and anxiety [6][7][8], accounting for up to 12% of the Australian working population [9]. However, common mental disorders may go unnoticed in the workplace, as they can often be characterized as work stress or other stress related conditions [7]. More severe mental illness is less common in the workplace, due to lower levels of unemployment in those with severe mental illness [10].
Over the recent decades, organisations have increasingly recognised the importance of maximizing employee health, both for ethical reasons, to improve productivity, meet legislation changes and reduce their cost burden [11,12]. While physical health promotion interventions are well established in many workplaces [13], workplace mental health interventions have lagged. Despite mental illness now being the leading cause of work incapacity and sickness absence [14], and depression estimated to be the leading cause of disability at work by 2020 [15], a recent systematic meta-review found that there no workplace mental health interventions that could be considered evidence-based [16].
The past decade has seen an explosion in the delivery of eHealth interventions targeting common mental illnesses such as depression and anxiety, as well as those focused upon broader concepts of stress and distress [17] through Apps and via the Internet. The components of these interventions have included Cognitive Behavioural Therapy, Stress Management, Mindfulness approaches and Cognitive training. Some eHealth interventions offer fully unguided self-help programs while others involve supported guidance throughout and are thus more costly [18]. Their effectiveness has been examined in both general population and in clinical settings. Two previous systematic review and meta-analysis found that eHealth interventions delivered to general population could both reduce and prevent depression and anxiety [11,19]. Similar results were found in a systematic review and meta-analysis of eHealth interventions in a clinical setting [20].
Thus there is currently ample evidence for the short and long term benefits for eHealth delivered Cognitive Behavioural Therapy (CBT) for treating anxiety and depressive conditions in both general population and clinical settings [21][22][23], with greater improvement shown for those interventions that include guidance throughout the intervention period [21,24]. Evidence has also emerged for the effectiveness of Mindfulness based eHealth interventions in improving symptoms in both unselected and symptomatic people [25].
However, we know employed people differ systemically from both general or clinical populations used in most eHealth studies (in terms of symptom profile, risks, function and response [26]) Therefore, this evidence may not be generalisable to the occupational setting.
Many workplaces now provide employees with access to mental health assistance, usually through Employee Assistance Programs (EAPs), providing employees with a range of integrated services including short term counseling, management consultations, organisational and team interventions, often with aggregate reporting to the organisation [27]. EAPs are estimated to cost US workplaces up to US$40.00 per person [28]. To mitigate the cost of these, many organisations are now providing eHealth interventions and applications into their workplace health management systems for employees, either by creating their own programs for internal use or buying commercially available products [29,30]. However there is no guidance on whether eHealth interventions are effective in the workplace, or even whether they may cause harm [30]. Given this gap in the evidence we undertook a systematic review of the effectiveness of eHealth interventions delivered to employees through the workplace with the aim of informing organisations and employees that eHealth interventions targeting mental illhealth and work stress are safe and effective.

Methods
This work complies with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [31].
We aimed to identify all published, peer reviewed, clinical trials including randomized controlled trials (RCTs), controlled trials & pre/post trials using an eHealth intervention targeted at employees that reported outcomes on a standardized mental health measure of depression, anxiety and/or stress.

Inclusion
We included only studies evaluating the outcome of an eHealth based intervention, defined as a therapeutic intervention delivered through a website, smartphone or tablet App, focused on the mental health of employees. Participants of the studies had to be in current paid employment and working age adults between 18 to 65 years of age. Included studies were peerreviewed articles published in English.

Exclusion
Studies were excluded from the review if they focused on volunteer workers, unemployed participants, general or clinical population studies that did not mention being targeted to employees or in a workplace setting, examined non-mental health outcomes, or used telephone and/ or email interventions only. Studies without a control group were also excluded from the meta-analysis.

Search strategy
A thorough literature search was conducted using the electronic databases MEDLINE, Psy-cINFO, Cochrane Register of Controlled Trials (CENTRAL) and EMBASE for relevant articles published from 1975 (the first eHealth intervention for employees was identified in 2004 [32]) until 17 November 2016. Keywords were searched relating to 'workplace' and 'intervention' and 'outcome' and 'study design' and an example of a search strategy for these databases is displayed in S1 Table. To increase coverage for relevant databases, we also manually screened the table of contents of the Journal of Medical Internet Research, Journal of Internet Interventions, Occupational and Environmental medicine and the Journal of Occupational Environmental Medicine, as well as the reference list of included studies.

Study selection criteria
Identification. After duplicates were removed, authors (ES) screened all titles and abstracts to identify potentially relevant studies. Abstracts and Full-text versions of potentially eligible studies were independently assessed by two investigators (ES and IC). Disagreements were adjudicated in conjunction with the senior author (NG).
Data collection and coding of outcomes. We collected the mean, and standard deviation (SD) of standardized measures of (dis)stress, depression and/or anxiety, and sample size (n) of participants in each arm (intervention and control) at baseline and each follow-up. Where studies used a number of measures, these were pooled as a single composite Mental Health outcome. When SDs were not reported, we contacted the authors. Three authors were contacted, and two of those responded with relevant data [33,34]. In one study [35] the SD could not be obtained from the paper or the authors; effect sizes for this study were derived from 2-tailed p-value for Group × Time interaction and sample sizes. One author did not respond and we were unable to obtain data so this study was excluded from the meta-analysis [36].
Where studies reported results of the same intervention in different papers (e.g., a post intervention and follow-up study), papers were grouped together and treated as one study and we used the first follow-up outcome point in the analysis.
We then categorized the studies by intervention type, in which three main types were identified from the primary paper as to the main component of the intervention: Cognitive Behavioural Therapy (CBT), Stress Management, and Mindfulness-based treatments, and by whether the intervention was given to unselected employees (universal) or targeted to those selected for higher symptom scores or another indication of poorer mental health (indicated). Sensitivity analysis results were also assessed by whether the intervention was unguided (selfhelp) or guided (feedback, rather than just technical support, as this is known to affect the outcome [37]. Quality assessment and risk of bias within studies. The quality of the identified studies was assessed using the Downs and Black checklist [38]. Specifically developed for public health, and used previously in similar reviews in the field [16], the Downs and Black checklist demonstrates good inter-rater reliability (r = 0.75) and strong criterion validity (r = 0.90) [11]. The checklist has 27-items to score within five sub categories: reporting, external validity, internal validity -assessing risk of bias & confounding, and power. The checklist was modified to simplify the power item to a score of either zero = no or one = yes based on whether the studies reported sufficient power to detect a clinically significant effect, as reported in previous studies [11,39]. The maximum score for the modified checklist was 28 with most items reporting on yes = one, no = zero or unable to determine = zero, with the exception of one question that used yes = two, partially = one, and no = zero. The first author (ES) individually scored each included paper. Scores were pooled into four categories as used in other reviews [11]: Excellent = 26 to 28, Good = 20 to 25, Fair = 15 to 19 and Poor = 14 and less. Any rated poor were reviewed by an independent research assistant and discrepancies adjudicated by the senior author (NG) and excluded.
As a further standardized measure we assessed Risk of Bias using the Cochrane Guidelines [40]. For this analysis five of the six items (sequence generation, allocation concealment, assessor blinding, incomplete outcome data and selective outcome reporting) were used to assess bias as all of the trials included in the Meta-analysis had only a wait list control group, and it would be impossible for participants and personnel to be blinded to the arm that they were randomized to (S1 Fig).

Statistical analysis
The summary measure was the standardized mean difference (SMD, calculated as Hedge's g), of change in the effect sizes between baseline and each follow-up. A positive effect size (SMD) indicated that the intervention group had superior effects to the control group. Using standard measures of effect size, SMDs of 0.2 were considered small, 0.5 moderate, and 0.8 large [41]. Precision of effect sizes was estimated using 95% confidence interval (CI). Pooling of effect sizes across studies was done using random effects model in Comprehensive Meta-Analysis V3. Random effects analysis was used as we could not make the assumption that these studies represented a common effect as the different types of interventions delivered may result in large heterogeneity [42].
Analyses were conducted in three stages, first, we combined results of all studies to achieve a combined effect size at each time point, immediately post-intervention and at follow-up. We then estimated pooled effect sizes for each of the three intervention types (CBT, Stress Management, and Mindfulness based interventions). Studies targeting unselected employees were analysed separately from those targeting 'symptomatic' (selected) employees such as those having reported increased 'stress' or having a mental health condition at baseline measures indicated for intervention. Studies were then stratified using the post-intervention and follow-up combined SMD for each study in each sub-group.
Heterogeneity of real effects across studies was quantified using the I 2 statistic to estimate the proportion observed variance not due to sampling error.
To assess small study effect (in the case where at least 10 studies were available for analysis) we used a funnel plot for the overall effects and each of the grouped intervention types which compared the outcome effects with their standard errors. We used Egger regression test to further examine asymmetry of the funnel plot [43] with a statistical significance based on a Pvalue less than 0.05. In the case where at least 10 studies were available for analysis and a small study effect was found we used a Duval and Tweedie trim and fill to quantify the magnitude of small study effect [44].

Search results
The search strategy identified a total of 1147 titles (Fig 1). After the removal of duplicates (n = 276), 871 titles and abstracts were reviewed by author (ES). Of those, 723 studies were excluded on the basis of the inclusion/exclusion criteria, leaving 108 studies potentially relevant to the research question. The 108 abstracts were examined by the two independent researchers (ES and IC), and 68 articles were excluded. Both researchers independently examined the full text of 40 articles, where a further 8 articles were excluded. The 32 remaining articles were identified as meeting criteria for quality assessment [32][33][34][35][36][37]. Four papers were identified as reporting on different follow-up periods and were merged [48,49] and [50,53].
After the quality assessment of all studies using the Down' & Black Checklist, two studies were excluded from the analysis as they were found to be of 'poor' quality [45,55]. Two papers did not have a control group [56,57], and one study used a face-to-face control group [68]. One study [58] included two intervention groups, which were treated as two studies in the metaanalysis. The two intervention groups' SMDs were computed and the mean (n) of the control group was evenly divided among each intervention group to ensure that control participants were only included once in the analysis. The two studies using cognitive training [59,60] were not included for the meta-analysis as the outcomes were not standardized mental health outcomes. One paper was excluded due to missing data [36]. This approach left 23 studies to be analysed in the meta-analysis, 22 of which were randomized controlled trials and a "controlled" trial [33]. Table 1 shows the details of each included intervention. Eleven studies were identified to have a CBT intervention [32,33,35,[45][46][47][48][49][50][51][52][53]56,61], six studies described themselves as Stress Management [34,36,[62][63][64][65][66], and six studies used Mindfulness-based approaches [57,58,67,[69][70][71]. Of the CBT interventions, nine were web-based interventions while two were smartphone Apps. All of the Stress Management interventions were web-based interventions. Of the Mindfulness based interventions, five were web-based and one was on a smartphone App (Table 1).
However, the effect of Stress Management interventions differed between selected and unselected employees at both post-intervention and follow-up. Interventions delivered to an unselected group (n = 4) showed a very small non-significant negative post-intervention effect (g = -0.03, 95% CI -0.16 to 0.09, p = 0.57, I 2 = 0.0%) whereas, interventions given to a selected group showed a moderate to large significant effect reflected in the significant difference between these two groups at post-intervention (Q = 29.63 (df1), p< = 0.01) and high heterogeneity (g = 0.64, 95% CI 0.43 to 0.85, p = 0.00, I 2 = 70.1%), we can also see that At follow-up there was only study in an unselected group showing a small non-significant negative effect (g = -0.16, 95% CI -0.41 to 0.09, p = 0.21, I 2 = 0.0%). In contrast interventions in selected groups continued to show a significant moderate to large effect size, however large heterogeneity was detected (g = 0.69, 95% CI 0.06 to 1.33, p = 0.00, I 2 = 89.5%). All Mindfulness-based interventions were aimed at unselected employees in the studies.
Small study effect. The funnel plots (S2 Fig Overview of efficacy of intervention type and moderators of efficacy across each mental health outcome. A summary of all of the outcomes is visually presented in Table 2, a matrix showing each Hedge's g effect size for each intervention and each moderator. Overall, there is evidence for the unselected and guided CBT in reducing overall symptoms, which appears to be due to the effects on depression and stress, but not anxiety. The overall effects of unselected and unguided CBT were maintained at follow up, mainly due to the follow up effects of CBT on stress. Mindfulness in unselected and unguided formats had a moderate to large effect size (g = 0.59) in reducing overall symptoms, a small to moderate effect in reducing depression (g = 0.34), and a large effect size (g = 0.68) in reducing stress. Unguided and unselected Effectiveness of eHealth interventions for reducing mental health conditions in employees Mindfulness had a small to moderate effect on overall symptoms at follow up, with a large effect on stress. Stress Management in selected formats had a large effect size (g = 0.64) and moderate effects when guided (g = 0.37) in reducing overall symptoms, a large effect in reducing depression (g = 0.63), anxiety (g = 0.83) and stress (g = 0.64). These results continued at post-intervention however the effect on Stress Management was no longer significant.

Discussion
This study provides the first comprehensive review of the published evidence for the effectiveness of a range of work-based eHealth interventions addressing the mental health of employees.
Our results were drawn from 32 trials, testing a range of online and mobile applications utilizing Cognitive Behavioural techniques, Stress Management based approaches, and Mindfulness based interventions, 23 studies were suitable for the meta-analysis.
Overall pooled eHealth interventions showed a small significant positive effect at post intervention and follow-up for reducing overall mental health (depression, anxiety and stress) in both universal and selected employees which is similar to a recent review of eHealth interventions for general population [19], providing further evidence for potential use on eHealth interventions as a low cost, alternative to face-to-face interventions for employees [4]. However, this result showed large heterogeneity, which is examined in our sub-group analysis. There is evidence of a small study effect in the overall 23 studies at post-intervention, a possible cause for this is the clinical heterogeneity between the participants in the large and small studies, the smaller studies were targeted at the selected group (n = 1328 participants) so that a favourable outcome of the selected group may be expected compared to the larger amount of participants in the universal studies (n = 2922) [72].
As planned, sub-group analyses were performed on selected and universal interventions. Mindfulness interventions were all universal and unguided interventions, and resulted in a moderately large and statistically significant effect size at post intervention, this result has a potential to make a meaningful change when considered on a population level and is a contrary result to previous findings for interventions delivered to a universal population. Previous research indicates that large effect sizes are unlikely to be found in interventions delivered to general populations who may not be unwell [19,73]. However, this appeared to be relatively short lived with a halved effect size at follow-up.
There was no difference between universal and targeted CBT interventions, although, previous evidence suggests that selected interventions may be more effective than universal interventions [20,74,75]. Both groups showed very small if any effect at post-intervention or followup, indicating that CBT interventions appeared not to have any statistical benefits either universal or targeted working populations which seems to differ from large results seen in both universal [76] and clinical [77], populations. This could be due to recent evidence that shows employed people differ systemically from the general or clinical in terms of symptom profile, function and response [26]. It may also be that the setting, with propensity for perverse incentives, or perceived coercion by employers may not be an effective therapeutic milieu.
In contrast, eHealth Stress Management interventions did appear to have benefits when used in a selected fashion e.g. amongst those reporting increased stress, showing a moderately large effect at post-intervention and a moderate effect size follow-up however, large heterogeneity was detected showing that the two selected studies had a significant difference in what can assume is the sample size as the follow-up length and intervention designs were both very similar. Importantly, when the same type of Stress Management based eHealth interventions were used in unselected (universal) employees, there was no effect compared to wait list controls. Only one study reported a follow up and suggested psychological harm, which may, if replicated, be another example of well-meaning intentions going awry, as some believe to occur with psychological debriefing [78]. It is not clear why a Stress Management intervention would not be effective when used on an unselected population, though it may relate to the dangers of providing non-help seeking populations with too much information, leading them to feel more anxious and vulnerable, as suggested by the authors of these studies [79], or could reflect intra-study effects e.g. differential drop out.
The outcome measures were then analysed separately by means of mental health condition.
We can see evidence for workplace eHealth interventions that are targeted and guided with low risk of bias reducing depression overall at post-intervention. There is no evidence to suggest that any eHealth intervention in the workplace have any effect on reducing anxiety level or symptoms at either post-intervention or follow-up. For stress levels it is evident that Stress Management interventions are most effective at post-intervention. It is evident that where a high risk of bias is found: in overall interventions, CBT and Mindfulness are all due to stress outcomes, it is unclear why, but this may be due to the different self-reporting measures used to report stress.

Strengths and limitations
The main strengths of this review are the detailed systematic search strategy, the clearly defined inclusion and exclusion criteria and the assessment of the quality of the studies. There are a number of potential limitations, first, despite widespread use, the number of studies examining eHealth interventions in the workplace was surprisingly small with only 23 studies able to be included in the meta-analysis and for some of the subgroups there were too few to draw strong conclusions. Second, although some of the studies had clearly defined the intervention type, e.g. reporting only using CBT approaches, other interventions prioritised one approach in a more eclectic intervention, e.g. using "Mindfulness with aspects of Stress Management", and we had to group these studies into that dominant intervention basis. However the high heterogeneity of differences of true effects when all these eHealth interventions were combined was reduced substantially when analysed by intervention type, sometimes to zero, supporting this grouping. Where a study had several outcome measures, we decided to pool them into one 'overall mental health' combined category, however Table 2 shows the outcome specific moderators and each level of efficacy in the intervention groups and overall. As self-reported measures were used in all studies, our conclusions are limited to self-reported reduction of symptoms rather than a clinical diagnosis. Finally, initial screening of some titles for exclusion was only completed by one author.

Conclusion
This review demonstrated the evidence that certain types of eHealth interventions delivered to employees via their workplace can be effective at reducing mental health and stress symptoms although the evidence base is affected by a small study effect that seems to inflate effectiveness. If an eHealth intervention is to be offered to all employees, in a universal fashion, then Mindfulness approaches appear to have a stronger effect than the other types. There is little to recommend Stress Management approaches delivered to the whole workforce, and one study suggests that providing this type of intervention may even be harmful. In contrast the use of Stress Management eHealth interventions by workers who are reporting high levels of stress may have a positive effect, despite few studies. Thus although eHealth interventions are now popular and widely promoted in workplaces it appears that some caution is required in their advocacy and uptake and more evidence is needed on the effectiveness, targeting, implementation and potential risks of these new technologies.
Several considerations for future research and practice for delivering eHealth interventions to employees are evident. First, it is important to consider what the intended populations needs from the interventions to ensure the greatest potential for benefit, for example, the diverse mental health status of the employees should be considered, and where possible interventions should be offered that best fit a range of symptoms which may entail more than one application. Second, it is important to select interventions on the basis of available evidence, as some intervention types may have little or no effect on the intended outcome or may even be potentially harmful. Third, it is important for organisations that are providing interventions to their employees consider both the effectiveness of the interventions and design features to ensure that they can be applied to both universal and selected groups of people, and to be aware that not all employees would benefit from the same type of intervention. Fourth, more research is needed in the form of randomised controlled trials to understand which type of interventions would best benefit different types of employees and mental health conditions, especially Mindfulness interventions that are guided and targeting symptomatic employees, as there is no current evidence for this group.
Supporting information S1