The authors have declared that no competing interests exist.
Conceived and designed the experiments: SC-M AWL LK RJS EHT BDT. Performed the experiments: SC-M AWL LK BDT. Analyzed the data: SC-M AWL BDT. Wrote the paper: SC-M BDT. Contributed critical revision of manuscript and approved submission: SC-M AWL LK RJS EHT BDT.
A large proportion of mindfulness-based therapy trials report statistically significant results, even in the context of very low statistical power. The objective of the present study was to characterize the reporting of “positive” results in randomized controlled trials of mindfulness-based therapy. We also assessed mindfulness-based therapy trial registrations for indications of possible reporting bias and reviewed recent systematic reviews and meta-analyses to determine whether reporting biases were identified.
CINAHL, Cochrane CENTRAL, EMBASE, ISI, MEDLINE, PsycInfo, and SCOPUS databases were searched for randomized controlled trials of mindfulness-based therapy. The number of positive trials was described and compared to the number that might be expected if mindfulness-based therapy were similarly effective compared to individual therapy for depression. Trial registries were searched for mindfulness-based therapy registrations. CINAHL, Cochrane CENTRAL, EMBASE, ISI, MEDLINE, PsycInfo, and SCOPUS were also searched for mindfulness-based therapy systematic reviews and meta-analyses.
108 (87%) of 124 published trials reported ≥1 positive outcome in the abstract, and 109 (88%) concluded that mindfulness-based therapy was effective, 1.6 times greater than the expected number of positive trials based on effect size d = 0.55 (expected number positive trials = 65.7). Of 21 trial registrations, 13 (62%) remained unpublished 30 months post-trial completion. No trial registrations adequately specified a single primary outcome measure with time of assessment. None of 36 systematic reviews and meta-analyses concluded that effect estimates were overestimated due to reporting biases.
The proportion of mindfulness-based therapy trials with statistically significant results may overstate what would occur in practice.
Mindfulness-based therapies (MBT), which include mindfulness-based stress reduction (MBSR) programs and mindfulness-based cognitive therapy (MBCT), have been described as feasibly delivered, low-cost, evidence-based options for managing stress, reducing mental health symptoms, and preventing relapse of depression [
MBSR and MBCT have been reported to improve mental health outcomes among patients with psychiatric conditions (e.g., depression [
A concern, however, is that the overwhelmingly statistically significant results in favor of MBSR and MBCT interventions that can be seen in the published literature, despite very low power in many studies, may be influenced by reporting biases. Reporting biases are said to occur when statistically significant or “positive” outcomes have been preferentially published compared to non-significant or “negative” outcomes [
Meta-analyses of MBT have either not assessed reporting biases [
Some studies of MBT [
The authors of a recent meta-analysis of 47 trials of MBT for psychological stress and well-being [
We have observed anecdotally that there seem to be few examples of published MBT trials without statistically significant results, even though many existing trials appear to have been conducted with very low statistical power. Thus, our objectives were to (1) characterize the degree to which published MBT trials report statistically significant results in favor of MBT interventions; (2) attempt to evaluate the plausibility of the number of statistically significant results; (3) evaluate MBT trial registrations and subsequent publication status to assess the potential influence of study publication bias and selective outcome reporting bias on the number of positive trials; and (4) evaluate systematic reviews and meta-analyses on MBT to determine whether reporting bias has been assessed and, if so, what conclusions have been drawn.
The CINAHL, Cochrane CENTRAL, EMBASE, ISI, MEDLINE, PsycInfo, and SCOPUS databases were searched on July 4, 2013. See
Our main objective was to characterize the degree to which published MBT trials have reported statistically significant results, not comparative effectiveness. Thus, RCTs published in any language, including dissertations that appeared in indexed databases, were eligible if they evaluated the effect of MBT versus usual care, placebo, or other inactive controls (e.g., waitlist, sham control) on mental health outcomes in any population. MBT was defined as a group-based intervention in which standard MBT components comprised the core of the intervention [
Search results were downloaded into the citation management database RefWorks (RefWorks, RefWorks-COS, Bethesda, MD, USA) and duplicates were removed using the RefWorks duplication check and manual searching. Two investigators independently reviewed articles for eligibility. If either deemed an article potentially eligible based on title/abstract review, then a full-text review was completed. Disagreements after full-text review were resolved by consensus. Translation was done for non-English articles.
Two investigators independently extracted and entered data items from eligible RCTs into a standardized spreadsheet; discrepancies were resolved by consensus. When there was more than one publication on the same RCT, we extracted data for the RCT as a unit, incorporating information from all publications together. Identification of multiple publications from the same RCT was done by cross-referencing authors and co-authors, patient characteristics, and countries. In cases where it was not clear whether publications reported data from the same RCT, we contacted study authors.
For most included trials it was not possible to identify a pre-defined, single primary outcome variable, and, in many trials, it was not possible to even identify a single primary outcome, whether or not pre-defined. Most trial reports included multiple outcome variables with no indication of primacy or included multiple “primary” outcome variables with no statistical adjustment. Since we could not identify a single primary outcome variable in most cases, in order to attempt to determine if a trial had been reported as a positive trial, we used a classification method based on a method published by Kyzas et al. [
Based on our primary classification method, negative RCTs were further coded to indicate whether results were presented with a caveat, defined as a statement made by investigators to mitigate the lack of statistical significance [
The effectiveness of mental health therapies in trials may depend on whether they are delivered by highly trained professionals, as in MBCT, versus professionals with less clinical training, as in MBSR; whether the patient sample is a symptomatic clinical sample versus a non-clinical sample; and whether a minimum symptom threshold is required for enrollment [
We initially intended to evaluate the plausibility of the number of RCTs with positive results using the test for excess significance, which was developed by Ioannidis and Trikalinos [
We encountered three substantive barriers, however, to applying the test for excess significance in this group of studies, as we had originally intended. First, we were not able to identify a single primary outcome variable in most studies on which to base a study-specific effect size estimate. Second, related to this, we did not believe that we could reasonably estimate an unbiased “true” effect size upon which to base an estimate of statistical power for all MBT trials, which is needed for the text of excess significance. In the context of substantial selective reporting, a meta-analysis based effect estimate would exaggerate actual effectiveness and underestimate excess statistical significance. Third, given the clinical heterogeneity of the included studies, it was reasonable to assume that there was substantial heterogeneity of effects across studies, and substantive heterogeneity can also lead to a greater number of statistically significant results, beyond reporting biases.
Thus, we did not conduct a statistical test for excess significance. Instead, we presented the overall number of positive trials and the number of positive trials in each subgroup. For comparison purposes, we also presented the number that would have been expected if the true effect size were the same as the effect size reported in a recent meta-analysis of trials of individual psychotherapy for adult depression, d = 0.55 [
Power for each study that we included was calculated using the pwr package in R [
We examined MBT trial registrations to assess the degree to which publication bias and selective outcome reporting may have influenced the number of positive trials we encountered.
We searched 3 trial registries: ClinicalTrials.gov, the Standard Randomized Controlled Trial Number Register, and the World Health Organization’s (WHO) International Clinical Trials Registry Platform, which is a central database that provides access to multiple national or region-specific registries (see
Results were downloaded into an Excel database. Duplicate registrations were identified in the WHO registry platform automatically, and any additional duplicates across registries were identified by manual search. Two investigators independently reviewed trial registrations for eligibility with any disagreements resolved by consensus.
Two investigators independently reviewed each trial registration for listed publications of trial results in peer-reviewed journals. If none were listed, search results from the electronic database search were reviewed for published RCTs (see above section regarding eligible RCTs) to attempt to identify published results. If none were found, MEDLINE and PsycInfo were additionally searched for results published in peer-reviewed journals using the trial registration number and, if unsuccessful, using the intervention, condition studied, and the name of the listed principal investigator (e.g., Bremner AND mindful*). Each trial registration was classified as having published or not published trial results in a peer-reviewed journal or indexed doctoral dissertation within 30 months of completion. For trials published online ahead of print, the date when the trial was made available electronically was used as the publication date [
The risk of selective outcome reporting increases when, prior to data collection, there is no clear declaration of a single primary trial outcome or, in the case of multiple primary trial outcomes, when there is no clear declaration of those outcomes with a plan to adjust for multiple analyses [
The CINAHL, Cochrane CENTRAL, EMBASE, ISI, MEDLINE, PsycInfo, and SCOPUS databases were searched on August 26, 2013 for recent systematic reviews and meta-analyses, published since January 1, 2011, on the effectiveness of MBT for improving mental health outcomes (see
Systematic reviews and meta-analyses published in any language were eligible if they reviewed the effectiveness of MBT on mental health outcomes. The same review procedure was used as for individual published RCTs of MBT, described above.
Two investigators independently extracted and entered data items into a standardized spreadsheet, with discrepancies resolved by consensus. For each systematic review or meta-analysis, they determined whether authors conducted a statistical test (e.g., asymmetry test, fail-safe N, regression analysis) or a visual inspection of funnel plots to assess possible reporting bias. They also reviewed the abstract and discussion section of each systematic review or meta-analysis to determine whether authors mentioned the possibility that reporting biases could have influenced results.
The electronic database search yielded 1,183 unique publications for review, of which 830 were excluded after review of titles and abstracts and 193 after full-text review. There were 36 related publications that reported on the same RCT in multiple publications, leaving 124 unique MBT RCTs (see
PRISMA flow diagram of selection of published randomized controlled trials of mindfulness-based therapies on mental health outcomes, including reasons for and number of excluded trials.
Of the 124 included RCTs, there were 62 RCTs (50%) from North America, 42 (34%) from Europe, and 20 (16%) others. There were 4 RCTs (3%) published before 2000, 40 (32%) between 2000 and 2009, and 80 (65%) in 2010 or later. The total number of patients analyzed in the combined intervention and control groups ranged from 10 to 357 participants per RCT. There were 58 trials (47%) with 10–49 total participants analyzed, 40 (32%) with 50–99, and 26 (21%) with 100–357. The mean number of patients analyzed in each trial was 70.3, and the median was 54.5. There were 13 trials that were published only as dissertations and retrieved via electronic databases, but were not published in a peer-reviewed journal.
According to the three key subgroup classifications, there were 28 RCTs (23%) of MBCT versus 96 (77%) of other MBTs; 41 RCTs (33%) that required a minimum symptom threshold for trial eligibility versus 83 (67%) that did not; and 83 RCTs (67%) with clinical samples versus 41 (33%) with non-clinical samples. Of the 83 RCTs with clinical populations, there were 36 with psychiatric patients, 12 with chronic pain patients, 11 with cancer patients, 7 with obese or diabetic patients, and 17 with patients with other conditions. See
Of the 124 RCTs, 26 (21%) had a registration record, including 21 (17%) that were registered prior to data collection. Of these, 12 were listed as completed by December 31, 2010 and included in our analysis of trial registrations and publication status (see section below, Evaluation of MBT Registrations); 1 was registered in the Centre for Clinical Trials registry (
Of the 124 included RCTs, 108 (87%) were classified as positive and 16 (13%) as negative based on reporting at least one positive outcome in the abstract. When classifications were instead based on study conclusions, there were 109 (88%) clearly positive studies that concluded that MBT was effective, 11 (9%) with mixed conclusions, and 4 (3%) negative studies that concluded that MBT had not been effective. There was a 91% rate of agreement between the two methods (113 of 124 trials). Of the 13 RCTs published only as dissertations, 8 (62%) were classified as positive based on both methods.
Of the 108 positive RCTs based on our primary classification method, 94 reported at least one significant between-groups mental health outcome, and 14 did not report any between-groups mental health outcomes, but reported at least one significant within-group mental health outcome. In the abstracts of the 16 negative RCTs, 11 reported positive pre-post changes for the MBT group in addition to negative between-groups results. Additionally, 5 of the 16 negative RCTs included a caveat. Only 3 reported negative between-groups results without highlighting significant within-group findings or providing a caveat.
For an assumed effect size of d = 0.55, the expected number of positive RCTs was 65.7. As shown in
Statistical Power (1 − β) | Total Number of RCTs | Total Number of Patients Analyzed in RCTs |
Observed Number (%) of Positive Studies | Expected Number (%) of Positive Studies |
Ratio of Observed Number / Expected Number of Positive Studies |
---|---|---|---|---|---|
0 ≤ (1 − β) < 0.25 | 15 | 10–29 | 10 (67%) | 2.8 (19%) | 3.6 |
0.25 ≤ (1 − β) < 0.50 | 45 | 24–51 | 39 (87%) | 15.7 (35%) | 2.5 |
0.50 ≤ (1 − β) < 0.75 | 34 | 54–92 | 33 (97%) | 21.1 (62%) | 1.6 |
0.75 ≤ (1 − β) ≤ 1.00 | 30 | 95–357 | 26 (87%) | 26.1 (87%) | 1.0 |
Total | 124 | 10–357 | 108 (87%) | 65.7 (53%) | 1.6 |
Abbreviation: RCTs = Randomized controlled trials
a See
b Statistical power is influenced by the distribution of patients between treatment and control groups. Thus, total sample size is not directly indicative of statistical power.
c Expected number of positive studies calculated by summing the power for all studies.
Key Subgroups | Total Number of RCTs | Total Number of Patients Analyzed in RCTs | Observed Number (%) of Positive Studies | Expected Number (%) of Positive Studies |
Ratio of Number Observed / Expected Number of Positive Studies |
---|---|---|---|---|---|
MBCT | 28 | 16–205 | 26.0 (93%) | 14.7 (53%) | 1.8 |
Other MBT | 96 | 10–357 | 82.0 (85%) | 51.1 (53%) | 1.6 |
Clinical | 83 | 10–282 | 72.0 (87%) | 46.7 (56%) | 1.5 |
Non-clinical | 41 | 17–357 | 36.0 (88%) | 19.0 (46%) | 1.9 |
Yes | 41 | 10–205 | 39.0 (95%) | 22.3 (54%) | 1.7 |
No | 83 | 12–357 | 69.0 (83%) | 43.5 (52%) | 1.6 |
Abbreviations: RCTs = Randomized controlled trials; MBCT = Mindfulness-based cognitive therapy; MBT = Mindfulness-based therapy
a See
b Expected number of positive studies calculated by summing the power for all studies.
To obtain an expected number of positive studies of 108, which was the number of observed positive studies, the true effect size would have needed to be d = 1.03.
The trial registry search yielded 313 unique registrations, of which 292 were excluded, leaving 21 eligible trial registrations of MBT RCTs (see
PRISMA flow diagram of trial registrations of completed randomized controlled trials of mindfulness-based therapy on mental health outcomes, including reasons for and number of excluded trial registrations.
Of the 21 registered trials, 8 (38%) were published within 30 months of trial completion. All 8 reported positive outcomes in the published abstract and were classified as positive studies based on their conclusions.
None of the 21 registered trials adequately specified a single primary outcome, including the outcome measure, the assessment time, and the metric. When metric specification was not required, there were 2 (10%) adequate trial registrations and 19 trials (90%) not adequately registered. These 19 registrations were classified as inadequate because multiple outcomes were listed without specifying a primary outcome or plan to adjust statistically for multiple outcomes (n = 16), because a specific measure was not listed for the primary outcome (n = 2), or because a time point was not specified for the primary outcome (n = 1). See
The search for systematic reviews and meta-analyses yielded 93 unique articles for review, of which 29 were excluded after review of titles and abstracts and 28 after full-text review, leaving 36 systematic reviews and meta-analyses eligible for evaluation (see
PRISMA flow diagram of recent meta-analyses and systematic reviews of mindfulness-based therapy on mental health outcomes, including reasons for and number of excluded reviews.
As shown in
The main finding of this study was that of 124 MBT RCTs that were reviewed, almost 90% were presented as positive studies when published. Furthermore, there were only 3 trials that were presented unequivocally as negative trials without alternative interpretations or caveats to mitigate the negative results and suggest that the treatment might still be an effective treatment.
For a point of reference, we compared the number of positive trials that we found to the number that would have been generated by a group of heterogeneous studies with a true effect size of d = 0.55, which is the effect size obtained from a recent meta-analysis of individual therapy for depression [
Although there is reason to believe that the effect estimate we used as a reference point may have been too large and, thus, overestimated the expected number of positive studies, we cannot rule out several different explanations for why we found so many positive trials. One explanation is simply that we cannot be sure that the effect size that we used as a reference point was indeed an accurate estimate or that it overstated likely effectiveness, as we believe. Second, it may be the case that heterogeneity in study effects could have contributed to the high number of positive studies. Finally, it may be the case that reporting biases played an important role in this. This idea is supported by the fact that the tendency to generate more positive studies that would be expected was concentrated in smaller studies, although it is also possible that lower quality in smaller studies could have played a role.
Our review of trial registration records also suggest the possibility that reporting biases may have been an important factor. Of the 124 RCTs reviewed, only 21 (17%) were registered prior to data collection, even though 80 of the eligible RCTs were published recently (since 2010). When we examined trial registries, we identified 21 registrations of MBT trials listed as completed by 2010 and found that 13 (62%) remained unpublished 30 months after completion; of the published trials, all conveyed a positive conclusion. None of the 21 registrations, however, adequately specified a single primary outcome (or multiple primary outcomes with an appropriate plan for statistical adjustment) and specified the outcome measure, the time of assessment, and the metric (e.g., continuous, dichotomous). When we removed the metric requirement, only 2 (10%) registrations were classified as adequate. We evaluated more than 30 published systematic reviews and meta-analyses of MBTs, and none concluded that reporting biases likely exaggerated estimates of effect (see
Ross et al. [
Thus, it may be the case that selective outcome reporting, as well as “data dredging” [
Mathieu et al. [
The very small number of trials that clearly declared negative results in the present study without caveats or “spin” also reminds us that when negative results are reported, they are often “spun” so that they appear to be equivocal or even positive findings [
In the present study, we found that most existing evidence syntheses either did not evaluate reporting biases or concluded that they were not present. The majority of these systematic reviews, which focused on a wide range of applications of MBT, included very small numbers of RCTs, which did not permit a statistical assessment of reporting biases. However, that would not have precluded approaches such as reviewing trial registries, in order to better understand the likelihood that completed trials of MBT may go unreported or that outcomes in published trials may be selectively reported. A meta-analysis, which was published subsequent to our search period and not included in our analysis, for instance, did not assess publication or other forms of reporting bias with statistical methods, but did identify patterns of non-publication and likely selective outcome reporting by reviewing MBT trial registrations [
There are a number of limitations that should be considered in interpreting the results of the present study. First, we were not able to conduct a statistical test to determine if there was excess significance bias. The results that we presented in comparison to a reference point effect size cannot rule determine the relative contributions to the relatively high number of positive results to the use of an inaccurate effect estimate as a reference point, to heterogeneity across studies, or to reporting biases. Generally, risk of reporting bias will be higher in fields with small, underpowered trials [
In summary, MBT appears to be a low-cost and easily implemented treatment that may be useful for providing effective mental health care to the large number of patients who are currently under-served [
(DOCX)
(DOCX)
(DOCX)
(DOCX)
(DOCX)
(DOCX)
(DOC)
The authors would like to thank Shervin Assassi, MD, MSc of the University of Texas Health Science Center, Houston, Texas, USA and Shadi Gholizadeh, MSc of the San Diego State University/University of California, San Diego Joint Doctoral Program in Clinical Psychology, San Diego, California, USA for assistance with translation. They were not compensated for their contribution.