Advertisement
  • Loading metrics

Reporting bias in the literature on the associations of health-related behaviors and statins with cardiovascular disease and all-cause mortality

  • Leandro Fórnias Machado de Rezende,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft

    Affiliation Departamento de Medicina Preventiva, Faculdade de Medicina FMUSP, Universidade de São Paulo, São Paulo, São Paulo, Brasil

  • Juan Pablo Rey-López,

    Roles Data curation, Formal analysis, Methodology, Writing – review & editing

    Affiliation Prevention Research Collaboration, School of Public Health, The University of Sydney, Sydney, Australia

  • Thiago Hérick de Sá,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Núcleo de Pesquisas Epidemiológicas em Nutrição e Saúde, Universidade de São Paulo, São Paulo, São Paulo, Brasil

  • Nicholas Chartres,

    Roles Data curation, Methodology, Writing – review & editing

    Affiliation Charles Perkins Centre, Faculty of Pharmacy, The University of Sydney, Sydney, New South Wales, Australia

  • Alice Fabbri,

    Roles Data curation, Investigation, Methodology, Writing – review & editing

    Affiliation Charles Perkins Centre, Faculty of Pharmacy, The University of Sydney, Sydney, New South Wales, Australia

  • Lauren Powell,

    Roles Data curation, Investigation, Methodology, Writing – review & editing

    Affiliation Prevention Research Collaboration, School of Public Health, The University of Sydney, Sydney, Australia

  • Emmanuel Stamatakis ,

    Roles Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Writing – review & editing

    ‡ These authors are joint senior authors on this work.

    Affiliations Prevention Research Collaboration, School of Public Health, The University of Sydney, Sydney, Australia, Charles Perkins Centre, Epidemiology Unit, The University of Sydney, Sydney, Australia

  • Lisa Bero

    Roles Conceptualization, Investigation, Methodology, Supervision, Writing – review & editing

    lisa.bero@sydney.edu.au

    ‡ These authors are joint senior authors on this work.

    Affiliation Charles Perkins Centre, Faculty of Pharmacy, The University of Sydney, Sydney, New South Wales, Australia

Reporting bias in the literature on the associations of health-related behaviors and statins with cardiovascular disease and all-cause mortality

  • Leandro Fórnias Machado de Rezende, 
  • Juan Pablo Rey-López, 
  • Thiago Hérick de Sá, 
  • Nicholas Chartres, 
  • Alice Fabbri, 
  • Lauren Powell, 
  • Emmanuel Stamatakis, 
  • Lisa Bero
PLOS
x

Abstract

Reporting bias in the literature occurs when there is selective revealing or suppression of results, influenced by the direction of findings. We assessed the risk of reporting bias in the epidemiological literature on health-related behavior (tobacco, alcohol, diet, physical activity, and sedentary behavior) and cardiovascular disease mortality and all-cause mortality and provided a comparative assessment of reporting bias between health-related behavior and statin (in primary prevention) meta-analyses. We searched Medline, Embase, Cochrane Methodology Register Database, and Web of Science for systematic reviews synthesizing the associations of health-related behavior and statins with cardiovascular disease mortality and all-cause mortality published between 2010 and 2016. Risk of bias in systematic reviews was assessed using the ROBIS tool. Reporting bias in the literature was evaluated via small-study effect and excess significance tests. We included 49 systematic reviews in our study. The majority of these reviews exhibited a high overall risk of bias, with a higher extent in health-related behavior reviews, relative to statins. We reperformed 111 meta-analyses conducted across these reviews, of which 65% had statistically significant results (P < 0.05). Around 22% of health-related behavior meta-analyses showed small-study effect, as compared to none of statin meta-analyses. Physical activity and the smoking research areas had more than 40% of meta-analyses with small-study effect. We found evidence of excess significance in 26% of health-related behavior meta-analyses, as compared to none of statin meta-analyses. Half of the meta-analyses from physical activity, 26% from diet, 18% from sedentary behavior, 14% for smoking, and 12% from alcohol showed evidence of excess significance bias. These biases may be distorting the body of evidence available by providing inaccurate estimates of preventive effects on cardiovascular and all-cause mortality.

Author summary

In the scientific literature, reporting bias occurs when communication and publication of results are influenced by the direction of findings. Reporting bias can distort scientific evidence and may misguide subsequent clinical and public health efforts. Our study provided an assessment of the degree of reporting bias in the literature on health-related behavior (smoking, alcohol, diet, physical activity, and sedentary behavior) and statins and their association with cardiovascular disease and mortality. We analyzed recently published systematic reviews. Most of the systematic reviews (90%) had a high risk of bias related to study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings. We found evidence of reporting bias in about one-fifth of health-related behavior meta-analyses but none of the statin-related meta-analyses. Readers should be aware of the extent of reporting bias in these research areas when interpreting meta-analytical results.

Introduction

The literature on the association between behavioral risk factors (e.g., smoking, alcohol, physical inactivity, and unhealthy diet) and cardiovascular diseases—the single largest cause of death globally [1]—has grown exponentially in the last decades [239]. Observational epidemiological studies are the dominant design assessing these associations, since clinical trials cannot always be ethically or logistically conducted [40]. Systematic review methods are used to synthesize and evaluate this growing body of evidence. It is important to evaluate the methodological risks of bias in systematic reviews [41], as well as the impact that reporting bias can have on the findings of reviews [42, 43].

Reporting bias is one of the most common biases identified in the literature. It includes selective publication of studies or outcomes of studies [44, 45] based on factors other than the study quality, such as nominally statistically significant results (P < 0.05) [46, 47] or authors’ “pedigree” [44, 45, 48]. These practices threaten the completeness and validity of scientific evidence [46] by distorting the estimates of causal effects of interventions or exposures on diseases [49]. The extent of reporting bias could differ between bodies of evidence consisting of randomized trials, such as drug studies, compared to observational studies, such as studies of health behavior. Different levels of reporting bias in the literature on health behavior may lead to inaccurate estimates of preventive effects on cardiovascular and all-cause mortality and therefore offer incorrect guidance for policymaking.

To gain a better understanding of the potential reporting bias in the literature on health-related behavior and cardiovascular disease mortality and all-cause mortality, we examined reporting and other risks of bias in a sample of systematic reviews published between 2010 to 2016. Our analysis also provided a comparative assessment of the reporting bias between health-related behavior and statins used in primary prevention.

Results

Of the 5,511 records identified while searching the databases, we selected 49 systematic reviews. All research areas (tobacco, alcohol, diet, physical activity, sedentary behavior, and statins) presented fewer than 20 eligible systematic reviews; therefore, we included all the systematic reviews within each area that met our inclusion criteria (Fig 1). Lists of excluded reviews and reasons for exclusions are described in S1 and S2 Tables. Studies were excluded most frequently for not including one of the exposures (28%) or outcomes (29%) of interest and for utilizing clinical samples (16%).

Most of the included systematic reviews (n = 35, 71.4%) analyzed only one outcome eligible for our study (cardiovascular disease mortality and all-cause mortality), whereas 9 (18.4%), 4 (8.2%), and 1 (2.0%) analyzed two, three, and four outcomes, respectively. All-cause mortality (69%), cardiovascular disease mortality (29%), and stroke mortality (14%) were the most frequent outcomes investigated (Table 1).

thumbnail
Table 1. Risk of bias in systematic reviews of the associations of health-related behavior and statins with cardiovascular and all-cause mortality—ROBIS assessment.

https://doi.org/10.1371/journal.pbio.2005761.t001

Risk of bias in systematic reviews—ROBIS results

The majority of the systematic reviews exhibited a high overall risk of bias (n = 44, 90%) (Fig 2). Among the four ROBIS domains, domain 1 (study eligibility criteria) presented the best scores, with 32 (65%) out of 49 reviews showing a low risk of bias. In domain 2 (identification and selection of studies), 2 (4%) reviews were scored as unclear, 40 (82%) showed a high risk, and 7 (14%) a low risk of bias. Whereas, in domain 3 (data collection and study appraisal), 7 (14%) reviews were scored as unclear, 28 (57%) scored with high risk, and 14 (29%) with low risk of bias. Finally, in domain 4 (synthesis and findings), 2 (4%) review was scored as unclear, 30 (61%) with high risk, and 17 (35%) with low risk of bias (Fig 2 and S3 Table).

thumbnail
Fig 2. Risk of bias in systematic reviews of the associations of health behavior and statins with cardiovascular disease mortality and all-cause mortality—ROBIS results.

Underlying data can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.2005761.g002

Comparing risk of bias in the reviews across research areas, sedentary behavior performed worst in domain 1 (study eligibility criteria; 70% of reviews were regarded as having high risk of bias). All research areas performed poorly in domain 2 (identification and selection of studies), with high risk of bias ranging from 70% in smoking reviews to 90% in both sedentary behavior and statin reviews. Alcohol (70%) and diet (60%) reviews presented high risk of bias in domain 3 (data collection and study appraisal). Sedentary behavior (90%), smoking (70%), and diet (70%) reviews presented high risk of bias in domain 4 (synthesis and findings). Overall, statin reviews presented the best scores in the ROBIS assessment compared to other research areas. Among statin reviews, a low risk of bias was identified in 60% in domain 1, 10% in domain 2, 50% in domain 3, and 60% in domain 4 (Table 1 and Fig 3).

thumbnail
Fig 3. Risk of bias in systematic reviews of the associations of health behavior and statins with cardiovascular disease mortality and all-cause mortality, by research area—ROBIS results.

Risk of bias in systematic reviews of the associations of physical activity (A), smoking (B), sedentary behavior (C), diet (D), alcohol (E), and statins (F) with cardiovascular disease mortality and all-cause mortality. Underlying data can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.2005761.g003

Risk of reporting bias in the body of evidence

We identified 111 meta-analyses (exposure–outcomes associations) that were performed across the 49 included reviews. On average, each meta-analysis synthesized results from 9 primary studies (ranging from 2 to 81), including 331,688 participants (ranging from 595 to 3,674,042) and 19,012 deaths (ranging from 33 to 320,252) (Table 2 and S4 Table). Of the 111 meta-analyses, 72 (65%) showed a nominally statistically significant result at P < 0.05.

thumbnail
Table 2. Relative and absolute frequency of meta-analyses with nominally statistically significant results, small-study effect, and excess significance, by research area.

https://doi.org/10.1371/journal.pbio.2005761.t002

Nominally statistically significant results (P < 0.05) were found in 92% of the meta-analyses from sedentary behavior and 100% of the meta-analyses from physical activity and smoking. Alcohol and statin reviews had 38% and 45% of meta-analyses with P < 0.05 results, respectively (Table 2 and S4 Table).

Small-study effect.

Indication of small-study effect was found in 17 (18%) of the 96 meta-analyses. Physical activity and the smoking research areas had more than 40% of meta-analyses with small-study effect. Sedentary behavior had less than 10% of meta-analyses with small-study effect (Table 2 and S4 Table). Overall, 17 (22%) out of 79 meta-analyses of health-related behavior presented small-study effect, as compared to zero-statin meta-analyses.

Excess significance.

More than half (56%; 53/95) of the meta-analyses displayed a greater number of observed primary studies with P < 0.05 results (O) than the number number of expected primary studies with P < 0.05 results (E). Of those, 19 meta-analyses (20% of the total meta-analyses and 36% of the meta-analyses with O > E) showed evidence of excess significance bias (P < 0.10). Half of the meta-analyses from physical activity, 26% from diet, 18% from sedentary behavior, 14% from smoking, and 12% from alcohol showed evidence of excess significance (Table 2 and S4 Table). Overall, 24% of the meta-analyses of health-related behavior showed excess significance, as compared to zero-statin meta-analyses.

Sensitivity analyses

We conducted a sensitivity analysis by restricting the sample in each research area to meta-analyses with ≥10 primary studies. In this subsample (n = 29), 86% of the meta-analyses showed statistically significant results at P < 0.05, as compared to 65% in the entire sample of meta-analyses. These results varied by research area, ranging from 60% in statin meta-analyses to 100% in physical activity, sedentary behavior, and smoking meta-analyses (Table 3).

thumbnail
Table 3. Sensitivity analysis: Relative and absolute frequency of meta-analyses with ≥10 primary studies showing nominally statistically significant results, small-study effect, and excess significance, by research area.

https://doi.org/10.1371/journal.pbio.2005761.t003

Small-study effect was present in 31% of the meta-analyses. The proportions of meta-analyses in the sensitivity analysis with small-study effect were 80% for physical activity, 50% for sedentary behavior, 29% for alcohol, and 33% for smoking. Diet and statin meta-analyses had no evidence of small-study effect. Around 38% of the health-related behavior meta-analyses with ≥10 primary studies presented small-study effect, as compared to zero in statin meta-analyses (Table 3).

Excess significance was identified in 27% of the meta-analyses with ≥10 primary studies: 100% of the meta-analyses for sedentary behavior, 50% for diet, 40% for physical activity, 20% for alcohol, and 17% for smoking. Around 33% of the health-related meta-analyses with ≥10 primary studies showed evidence of excess significance, as compared to zero in statin meta-analyses (Table 3).

Overall, after excluding small individual studies (with <200 deaths) from meta-analyses, results from small-study effect and excess significance tests did not change (S5 Table).

Discussion

This study aimed to assess the extent of reporting bias among recent meta-analyses that examined the associations of health behavior and statins with cardiovascular and all-cause mortality. We found evidence of reporting bias across all health-related behavior areas. The degree of reporting bias varied by the method used to assess it. Reporting bias was present in 20% (according to excess significance test) or 18% (according to small-study effect test) of all meta-analyses included (health behavior and statins). Evidence of reporting bias was found in between a quarter and one-fifth of health-related behavior meta-analyses (22% small-study effect and 24% excess significance) but in none of the statin meta-analyses (0%).

In lifestyle epidemiology, the interpretation of evidence for researchers and policymakers is challenging for several reasons [61]. As observational studies are the dominant designs in this area, spurious associations can arise due to confounding or several sources of bias. The impact of such biases on statistical findings and interpretation of findings has been poorly reported and discussed [62]. Therefore, meta-analytical synthesis of the evidence in lifestyle health behavior epidemiology may provide precise but spurious results [63].

Reporting bias is a major threat to the validity of the relevant body of evidence. Our results suggest that around 20% of the meta-analyses on health-related behavior and cardiovascular disease mortality and all-cause mortality may be susceptible to reporting biases. The existence of reporting bias in the literature has several explanations. Failure to submit manuscripts of analyses that did not produce statistically significant results (“the file-drawer problem” [46]) and the low likelihood of publication of small studies (regardless of statistical significance) [44] are two possible reasons. The selective reporting of certain analyses with statistically significant results is another likely source of reporting bias [44, 46, 47]. Each of the research areas we examined is likely to be linked to variable levels of reporting bias due to the different economics, dynamics, and conflicts of interest in each discipline [64, 65]. Interpreting the literature as a whole is challenging, considering the numerous biases that may affect the reliability and integrity of the scientific enterprise [66, 67].

To obtain a complete picture of the evidence (i.e., without reporting bias), it is important to know the results from all conducted studies on a given research question [68]. In our study, results from meta-analyses of health-related behavior and cardiovascular disease mortality and all-cause mortality were more likely to be affected by reporting bias compared to statin meta-analyses (22% and 24% versus 0%, respectively). The literature of health-related behavior is almost exclusively composed by observational studies, whereas statins are most often studied using randomized controlled trials. Reporting bias may be less frequent among trials than observational studies because several efforts to increase transparency and reproducibility of results have been adopted over the history of randomized controlled trials [69]. These include the mandatory registration of all clinical trials in humans and disclosure of all results [70]. As of more recently, data sharing statements of clinical trials are also required [71]. Observational epidemiologic studies should embrace these reproducible research practices to reduce reporting bias in the literature [6870, 72]. These practices could involve key elements of the scientific process, including (a) methods (e.g., rigorous training in statistics), (b) reporting and dissemination (e.g., disclosure of conflicts of interest), (c) reproducibility (e.g., open data), (d) evaluation (e.g., pre- and postpublication peer review), and (e) incentives (e.g., funding replication studies) [72]. Improving methodological training involves aspects of both research design and statistical analyses—for example, correct interpretation of P values [73], acknowledging the importance of statistical power, and improving the accuracy of effect sizes [72]. Protecting against cognitive biases is another major issue that has been overlooked [72]. Protecting against conflict of interests, especially financially related, is an imperative to achieve reproducible science. In addition to disclosure of potential conflicts of interest, promoting preregistration of study procedures and analytical plan may prevent reporting bias favoring positive results [72]. Funding replication of studies and encouraging openness in science and reproducibility practices by making datasets, scripts, and software publicly available may increase transparency and credibility of scientific claims [72]. For instance, food industry–sponsored studies are more likely to report conclusions favorable to the sponsors [74] but frequently lack transparency on acknowledgment of the funding source [75]. Further examples of reproducibility practices have been described and discussed by Munafò and colleagues [72].

To our knowledge, our analysis is the first comparative assessment of reporting bias across different fields of health-related behavior and statins. Our findings were based on well-established statistical tests developed to detect different aspects of reporting bias, as well as a complementary assessment of the risk of bias of systematic reviews using the ROBIS tool. We selected the ROBIS tool as it has greater specification to assess risk of bias compared to other tools. For instance, the “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) that has been used to evaluate the methodological quality of systematic reviews has constructs that are more related to quality of reporting than risk of bias [76, 77]. Risk of bias is linked to methodological quality of systematic reviews but provides further evaluation on how methodological limitations were considered to form conclusions. In this sense, the ROBIS tool is increasingly being used to assess risk of bias not only in systematic reviews [41, 76, 78] but also in guideline committees that evaluate evidence level (e.g., Australian government, National Health and Medical Research Council). Our ROBIS tool results showed that most of the systematic reviews had high risk of bias. Similar findings have been observed in previous studies appraising risk of bias in other research areas using the ROBIS tool [76, 78]. For instance, 18 (58%) out of 31 systematic reviews evaluating the effectiveness of intra-articular hyaluronic acid injection in treating knee osteoarthritis had high (n = 16) or unclear (n = 2) risk of bias [78]. Another survey assessing systematic reviews about psoriasis found that most reviews (86%) were classified as high risk of bias [76]. It is noteworthy that high risk of bias was found even for systematic reviews exhibiting high methodological quality as assessed through AMSTAR [76].

Our ROBIS assessment indicated that identification and selection of studies (i.e., appropriate range of databases, terms and filters used, and efforts to minimize errors in selection of studies) are major concerns. These biases in the review process could explain, at least in part, reporting bias results obtained from small-study effect and excess significance tests. The synthesis and findings domain also revealed potential risk of bias due to insufficient inclusion of studies and appropriate synthesis of estimates. This domain also reflects between-study variation, robustness of findings (e.g., sensitivity analyses), and biases in synthesis findings (i.e., if evaluated by systematic reviews).

We used small-study effect and excess significance tests to appraise reporting bias in the literature, which are the most commonly recommended and used methods [79]. However, results from these tests might also reflect methodological and clinical heterogeneity, or even chance [42]. In fact, most meta-analyses contained moderate to high heterogeneity (based on I2 statistic; S4 Table). Results from an Egger test (small-study effect) can give spurious false positive results due to correlation between log of effect size and its variance, especially in the presence of heterogeneity between studies in a meta-analysis. An alternative better-performing test has been proposed by Peters to identify reporting bias in meta-analyses, but it requires data from a 2 × 2 table [80]. Such data were rarely reported in individual studies in the meta-analyses of observational studies. As also noted by Tsilidis and colleagues [81], meta-analyses commonly use maximally adjusted relative risks rather than unadjusted relative risks calculated from 2 × 2 tables. For such data, the use of the Egger test is appropriate.

The egger test and excess significance test have low power to detect reporting bias and do not give indication about what the sources of bias are. Therefore, we performed sensitivity analyses, retaining only meta-analyses with ≥10 primary studies. In this subsample of meta-analyses, evidence of reporting bias was higher than the entire sample (small-study effect: 31% versus 18%; excess significance: 27% versus 20%). Differences between primary results and sensitivity analyses are likely related to low power of reporting bias tests, which could lead to false negative results in the former group of meta-analyses. Therefore, our estimates of reporting bias in the meta-analyses are possibly conservative. The ranking of research areas according to levels of reporting bias was also different between the main analysis and the sensitivity analysis (i.e., meta-analyses with ≥10 primary studies). For instance, meta-analyses of sedentary behavior appeared most sensitive to this restriction, as the estimated proportion of reporting bias increased when calculated with either the small-study effects (from 9% to 50%) or excess significance tests (from 18% to 100%). A possible explanation could be the small fraction of meta-analyses with ≥10 primary studies (2 out of 12) in this relatively new research field [82].

It is important to acknowledge that certain methodological decisions we made may have introduced bias in the sample of reviews selected or may compromise the generalizability of our findings. We excluded systematic reviews on alcohol published in Chinese language (n = 2), which potentially have high risk of bias [83]. In addition, we restricted our analyses to systematic reviews published in this decade only (2010–2016), which explains the small number of included meta-analyses in some research areas. This may have limited comparisons of the extent of reporting bias between research areas investigated. Our results may not provide a complete historical assessment of reporting bias in these areas. Nevertheless, our results reflect reporting bias in the literature of recent and relevant public health topics and from a time period when reporting standards have been improving due to, e.g., the widespread use of various manuscript reporting checklists [84]. Recent systematic reviews contain a higher number of primary studies than older systematic reviews and synthesize evidence of emerging fields that have flourished only recently (i.e., sedentary behavior).

In conclusion, we found evidence of reporting bias in approximately one-fifth of recent meta-analyses of observational studies of health-related behavior (physical activity, sedentary behavior, smoking, alcohol consumption, diet) and cardiovascular and all-cause mortality. Such a level of reporting bias may, to some extent at least, distort conclusions arising from this body of evidence. Contrarily, we found no evidence of reporting bias in meta-analyses of randomized controlled trials of statins.

Materials and methods

Identification and selection of relevant systematic reviews

We searched Medline (through PubMed), Embase (i.e., excluding Medline), Cochrane Methodology Register Database, and Web of Science for systematic reviews published between 2010 and 2016. We restricted our search to recent systematic reviews for several reasons. These systematic reviews belong to a “birth cohort” of systematic reviews published after the launch of the Meta-analysis of Observational Studies in Epidemiology (MOOSE) [85] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [86] guidelines and are expected to have lower risk of bias. As we were interested in comparing levels of bias across different research areas, this restriction may have reduced confounding due to date of publication. We restricted the search, as well as the successive phases of our study, to systematic reviews aiming to investigate the associations of health-related behavior (tobacco, alcohol, diet [fat, fruits and vegetables, salt, and sugar], physical activity, and sedentary behavior) and statins with cardiovascular disease mortality (overall cardiovascular mortality and cause-specific deaths from cardiovascular disease) and all-cause mortality. We accepted any definition for the exposures and the outcomes as defined in the original systematic reviews. The keywords used in the search are described in S1 Text, and files exported from databases during search strategy with all studies screened and selected are available at https://osf.io/wpb69/.

Systematic reviews were screened and selected (by two reviewers, and disagreements solved by a third reviewer) based on the following eligibility criteria: (i) sought to investigate an exposure–outcome association in a nonclinical population; (ii) systematically searched for primary studies and performed a meta-analysis (i.e., weighted summary effect size) using results from primary studies; (iii) selected only observational studies (cohort and case-control studies) if a health-related behavior review and only randomized controlled trials if a statin review; (v) reported data from each primary study included in the meta-analysis (S1 Text).

We decided a priori that a random sample of up to 20 systematic reviews per research area (tobacco, alcohol, diet, physical activity, sedentary behavior, and statins) would be included to compare levels of reporting bias in the relevant literature. If our search retrieved fewer than 20 meta-analyses in a given research area, we included them all. A similar study-selection strategy was recently used in a study evaluating publication bias in meta-analyses of individual studies [87]. These methods were decided a priori as described in the analysis plan available at https://osf.io/wpb69/ (not published prior to the identification and selection of systematic reviews).

Risk of bias in systematic reviews

Reporting bias could be related to overall risk of bias in a review. Therefore, four reviewers (JPRL, NC, AF, LP), working in pairs, independently assessed the risk of bias in the included systematic reviews using the ROBIS tool [41]. ROBIS comprises three phases: (1) assess relevance; (2) identify concerns with review process; (3) judge risk of bias in the review. To assess relevance, we extracted the target question from each review using the PICOS acronym (participants, interventions, comparisons, outcomes) or equivalents for etiological questions (participants, exposure, comparisons, outcomes). In phase 2, we assessed the risk of bias in four domains related to the review process: (1) study eligibility criteria; (2) identification and selection of studies; (3) data collection and study appraisal; and (4) synthesis and findings. Questions included in each of the four domains are available in S3 Table. Questions were answered as “Yes,” “Probably Yes,” “Probably No,” “No,” and “No Information,” with “Yes” indicating low risk of bias. In phase 3, we summarized the concerns identified in each domain during phase 2 and risk of bias in the review as low, high, or unclear. Further details on the ROBIS tool are described elsewhere [41].

Risk of reporting bias in the body of evidence

For each meta-analysis performed in the selected systematic reviews, we assessed the extent of reporting bias in the included literature via small-study effect [42] and excess significance tests [88]. To perform these tests, we extracted necessary data (e.g., effect size, confidence intervals, sample size, and number of events [deaths]) for each primary study included in the main meta-analysis performed in the systematic reviews. We also used these data to reperform the meta-analyses (i.e., using random effect models, which was used in the majority of the original meta-analyses). We did this to describe the number of meta-analyses with nominally statistically significant results at P < 0.05 (S1 Text).

Small-study effect test (also known as regression asymmetry test, proposed by Egger and colleagues) evaluates whether smaller studies tend to overestimate the effect size estimates compared to larger studies. For this matter, the test evaluates whether the association between effect size (e.g., relative risk, odds ratio) and precision (standard error) is greater than might be expected by chance. We considered a P value < 0.10 as a statistical significance threshold for small-study effect bias (i.e., suggesting evidence of reporting bias), as initially proposed by Egger and colleagues [42, 89] and consistently used in the literature [42, 66, 81, 87, 90, 91].

Excess significance test evaluates whether the O differs from the E. The E in each meta-analysis was obtained from the sum of power estimates of each primary study. The power estimate of each primary study depends on the plausible causal effect of each research area (e.g., smoking and cardiovascular mortality), which was assumed to be the effect of the most precise primary study (smaller standard error) in each meta-analysis [88]. We considered P < 0.10 (one-side P < 0.05 for O > E) as a statistical significance threshold for excess significance bias [43, 88]. The excess significance is reported as a proportion of studies, with the higher proportion indicating more excess significance (O > E) and thus more evidence of reporting bias.

Due to the low power of these bias tests, we performed a sensitivity analysis excluding meta-analyses with fewer than 10 studies to analyze the impact in the results. We also performed a sensitivity analysis excluding small individual studies (fewer than 200 deaths) within meta-analyses to evaluate whether results reflect reporting bias among small studies only. We performed all statistical analyses using Stata version 15.0 (College Station, TX).

Supporting information

S1 Text. Supporting materials and methods.

https://doi.org/10.1371/journal.pbio.2005761.s002

(DOC)

S1 Table. List of excluded studies and reasons for exclusion, by research area.

https://doi.org/10.1371/journal.pbio.2005761.s003

(DOC)

S2 Table. Summary of reasons for excluding studies during full-text screening, by research area.

https://doi.org/10.1371/journal.pbio.2005761.s004

(DOC)

S3 Table. Risk of bias in systematic reviews of physical activity, sedentary behavior, diet, alcohol, smoking, and statins, using ROBIS tool.

https://doi.org/10.1371/journal.pbio.2005761.s005

(DOC)

S4 Table. Meta-analyses of cardiovascular disease and all-cause mortality, by research area.

https://doi.org/10.1371/journal.pbio.2005761.s006

(DOC)

S5 Table. Sensitivity analysis excluding individual studies with fewer than 200 deaths.

Relative and absolute frequencies of meta-analyses with nominally statistically significant results, small-study effects, and excess significance, by research area.

https://doi.org/10.1371/journal.pbio.2005761.s007

(DOC)

References

  1. 1. World Health Organization. The top 10 causes of death 2017 [cited 2017 November 27]. http://www.who.int/mediacentre/factsheets/fs310/en/.
  2. 2. Kelly P, Kahlmeier S, Gotschi T, Orsini N, Richards J, Roberts N, et al. Systematic review and meta-analysis of reduction in all-cause mortality from walking and cycling and shape of dose response relationship. Int J Behav Nutr Phys Act. 2014;11:132. pmid:25344355
  3. 3. Samitz G, Egger M, Zwahlen M. Domains of physical activity and all-cause mortality: systematic review and dose-response meta-analysis of cohort studies. Int J Epidemiol. 2011;40(5):1382–400. pmid:22039197
  4. 4. Woodcock J, Franco OH, Orsini N, Roberts I. Non-vigorous physical activity and all-cause mortality: systematic review and meta-analysis of cohort studies. Int J Epidemiol. 2011;40(1):121–38. pmid:20630992
  5. 5. Hupin D, Roche F, Gremeaux V, Chatard JC, Oriol M, Gaspoz JM, et al. Even a low-dose of moderate-to-vigorous physical activity reduces mortality by 22% in adults aged >/ = 60 years: a systematic review and meta-analysis. Br J Sports Med. 2015;49(19):1262–7. pmid:26238869
  6. 6. Biswas A, Oh PI, Faulkner GE, Bajaj RR, Silver MA, Mitchell MS, et al. Sedentary time and its association with risk for disease incidence, mortality, and hospitalization in adults: a systematic review and meta-analysis. Ann Intern Med. 2015;162(2):123–32. pmid:25599350
  7. 7. Chau JY, Grunseit AC, Chey T, Stamatakis E, Brown WJ, Matthews CE, et al. Daily sitting time and all-cause mortality: a meta-analysis. PLoS ONE. 2013;8(11):e80000. pmid:24236168
  8. 8. Grontved A, Hu FB. Television viewing and risk of type 2 diabetes, cardiovascular disease, and all-cause mortality: a meta-analysis. JAMA. 2011;305(23):2448–55. pmid:21673296
  9. 9. Wilmot EG, Edwardson CL, Achana FA, Davies MJ, Gorely T, Gray LJ, et al. Sedentary time in adults and the association with diabetes, cardiovascular disease and death: systematic review and meta-analysis. Diabetologia. 2012;55(11):2895–905. pmid:22890825
  10. 10. Ford ES, Caspersen CJ. Sedentary behaviour and cardiovascular disease: a review of prospective studies. Int J Epidemiol. 2012;41(5):1338–53. pmid:22634869
  11. 11. Pandey A, Salahuddin U, Garg S, Ayers C, Kulinski J, Anand V, et al. Continuous Dose-Response Association Between Sedentary Time and Risk for Cardiovascular Disease: A Meta-analysis. JAMA cardiology. 2016;1(5):575–83. pmid:27434872
  12. 12. Sun JW, Zhao LG, Yang Y, Ma X, Wang YY, Xiang YB. Association Between Television Viewing Time and All-Cause Mortality: A Meta-Analysis of Cohort Studies. American journal of epidemiology. 2015;182(11):908–16. pmid:26568572
  13. 13. Costanzo S, Di Castelnuovo A, Donati MB, Iacoviello L, de Gaetano G. Wine, beer or spirit drinking in relation to fatal and non-fatal cardiovascular events: a meta-analysis. Eur J Epidemiol. 2011;26(11):833–50. pmid:22076059
  14. 14. Jayasekara H, English DR, Room R, MacInnis RJ. Alcohol consumption over time and risk of death: a systematic review and meta-analysis. Am J Epidemiol. 2014;179(9):1049–59. pmid:24670372
  15. 15. Roerecke M, Rehm J. Ischemic heart disease mortality and morbidity rates in former drinkers: a meta-analysis. Am J Epidemiol. 2011;173(3):245–58. pmid:21156750
  16. 16. Roerecke M, Rehm J. Alcohol consumption, drinking patterns, and ischemic heart disease: a narrative review of meta-analyses and a systematic review and meta-analysis of the impact of heavy drinking occasions on risk for moderate drinkers. BMC Med. 2014;12:182. pmid:25567363
  17. 17. Ronksley PE, Brien SE, Turner BJ, Mukamal KJ, Ghali WA. Association of alcohol consumption with selected cardiovascular disease outcomes: a systematic review and meta-analysis. BMJ. 2011;342:d671. pmid:21343207
  18. 18. Park JE, Choi TY, Ryu Y, Cho SI. The relationship between mild alcohol consumption and mortality in Koreans: a systematic review and meta-analysis. BMC Public Health. 2015;15:918. pmid:26385795
  19. 19. Stockwell T, Zhao J, Panwar S, Roemer A, Naimi T, Chikritzhs T. Do "Moderate" Drinkers Have Reduced Mortality Risk? A Systematic Review and Meta-Analysis of Alcohol Consumption and All-Cause Mortality. J Stud alcohol and drugs. 2016;77(2):185–98.
  20. 20. Zheng YL, Lian F, Shi Q, Zhang C, Chen YW, Zhou YH, et al. Alcohol intake and associated risk of major cardiovascular outcomes in women compared with men: a systematic review and meta-analysis of prospective observational studies. BMC Public Health. 2015;15:773. pmid:26264040
  21. 21. Roerecke M, Rehm J. Irregular heavy drinking occasions and risk of ischemic heart disease: a systematic review and meta-analysis. Am J Epidemiol. 2010;171(6):633–44. pmid:20142394
  22. 22. Roerecke M, Rehm J. Chronic heavy drinking and ischaemic heart disease: a systematic review and meta-analysis. Open Heart. 2014;1(1):e000135. pmid:25332827
  23. 23. Gellert C, Schottker B, Brenner H. Smoking and all-cause mortality in older people: systematic review and meta-analysis. Arch Intern Med. 2012;172(11):837–44. pmid:22688992
  24. 24. Lv X, Sun J, Bi Y, Xu M, Lu J, Zhao L, et al. Risk of all-cause mortality and cardiovascular disease associated with secondhand smoke exposure: a systematic review and meta-analysis. Int J Cardiol. 2015;199:106–15. pmid:26188829
  25. 25. Sinha DN, Suliankatchi RA, Gupta PC, Thamarangsi T, Agarwal N, Parascandola M, et al. Global burden of all-cause and cause-specific mortality due to smokeless tobacco use: systematic review and meta-analysis. Tob Control. 2018;27(1):35–42. pmid:27903956
  26. 26. Farvid MS, Ding M, Pan A, Sun Q, Chiuve SE, Steffen LM, et al. Dietary linoleic acid and risk of coronary heart disease: a systematic review and meta-analysis of prospective cohort studies. Circulation. 2014;130(18):1568–78. pmid:25161045
  27. 27. Graudal N, Jurgens G, Baslund B, Alderman MH. Compared with usual sodium intake, low- and excessive-sodium diets are associated with increased mortality: a meta-analysis. Am J Hypertens. 2014;27(9):1129–37. pmid:24651634
  28. 28. Hu D, Huang J, Wang Y, Zhang D, Qu Y. Fruits and vegetables consumption and risk of stroke: a meta-analysis of prospective cohort studies. Stroke. 2014;45(6):1613–9. pmid:24811336
  29. 29. Li XY, Cai XL, Bian PD, Hu LR. High salt intake and stroke: meta-analysis of the epidemiologic evidence. CNS Neurosci Ther. 2012;18(8):691–701. pmid:22742770
  30. 30. Musa-Veloso K, Binns MA, Kocenas A, Chung C, Rice H, Oppedal-Olsen H, et al. Impact of low v. moderate intakes of long-chain n-3 fatty acids on risk of coronary heart disease. Br J Nutr. 2011;106(8):1129–41. pmid:21736820
  31. 31. Pan A, Chen M, Chowdhury R, Wu JH, Sun Q, Campos H, et al. alpha-Linolenic acid and risk of cardiovascular disease: a systematic review and meta-analysis. Am J Clin Nutr. 2012;96(6):1262–73. pmid:23076616
  32. 32. Poggio R, Gutierrez L, Matta MG, Elorriaga N, Irazola V, Rubinstein A. Daily sodium consumption and CVD mortality in the general population: systematic review and meta-analysis of prospective studies. Public Health Nutr. 2015;18(4):695–704. pmid:24848764
  33. 33. Schwingshackl L, Hoffmann G. Monounsaturated fatty acids, olive oil and health status: a systematic review and meta-analysis of cohort studies. Lipids Health Dis. 2014;13:154. pmid:25274026
  34. 34. Wang X, Ouyang Y, Liu J, Zhu M, Zhao G, Bao W, et al. Fruit and vegetable consumption and mortality from all causes, cardiovascular disease, and cancer: systematic review and dose-response meta-analysis of prospective cohort studies. BMJ. 2014;349:g4490. Epub 2014/07/31. pmid:25073782
  35. 35. Chen GC, Yang J, Eggersdorfer M, Zhang W, Qin LQ. N-3 long-chain polyunsaturated fatty acids and risk of all-cause mortality among general populations: a meta-analysis. Scientific reports. 2016;6:28165. pmid:27306836
  36. 36. Cheng P, Huang W, Bai S, Wu Y, Yu J, Zhu X, et al. BMI Affects the Relationship between Long Chain N-3 Polyunsaturated Fatty Acid Intake and Stroke Risk: a Meta-Analysis. Sci Rep. 2015;5:14161. pmid:26369699
  37. 37. Cheng P, Wang J, Shao W, Liu M, Zhang H. Can dietary saturated fat be beneficial in prevention of stroke risk? A meta-analysis. Neurol Sci. 2016;37(7):1089–98. pmid:26979840
  38. 38. de Souza RJ, Mente A, Maroleanu A, Cozma AI, Ha V, Kishibe T, et al. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies. BMJ. 2015;351:h3978. pmid:26268692
  39. 39. Narain A, Kwok CS, Mamas MA. Soft drinks and sweetened beverages and the risk of cardiovascular disease and mortality: a systematic review and meta-analysis. Int J Clin Pract. 2016;70(10):791–805. pmid:27456347
  40. 40. Grisso JA. Making comparisons. Lancet. 1993;342(8864):157–60. pmid:8101260
  41. 41. Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34. pmid:26092286
  42. 42. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. pmid:21784880
  43. 43. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clinical Trials. 2007;4:245–53. pmid:17715249
  44. 44. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263(10):1385–9. pmid:2406472
  45. 45. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65. pmid:15161896
  46. 46. Kennedy D. The old file-drawer problem. Science (New York, NY). 2004;305(5683):451. pmid:15273363
  47. 47. Newcombe RG. Towards a reduction in publication bias. BMJ. 1987;295(6599):656–9. pmid:3117278
  48. 48. Colonna G, Ragone R. Referee bias. Nature. 1994;367(6459):108. pmid:8114900
  49. 49. Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med. 2012;9(3):e1001189. pmid:22448149
  50. 50. Bukkapatnam RN, Gabler NB, Lewis WR. Statins for primary prevention of cardiovascular mortality in women: a systematic review and meta-analysis. Prev Cardiol. 2010;13(2):84–90. pmid:20377811
  51. 51. Kizer JR, Madias C, Wilner B, Vaughan CJ, Mushlin AI, Trushin P, et al. Relation of different measures of low-density lipoprotein cholesterol to risk of coronary artery disease and death in a meta-regression analysis of large-scale trials of statin therapy. Am J Cardiol. 2010;105(9):1289–96. pmid:20403481
  52. 52. Kostis WJ, Cheng JQ, Dobrzynski JM, Cabrera J, Kostis JB. Meta-analysis of statin effects in women versus men. J Am Coll Cardiol. 2012;59(6):572–82. pmid:22300691
  53. 53. Lv HL, Jin DM, Liu M, Liu YM, Wang JF, Geng DF. Long-term efficacy and safety of statin treatment beyond six years: a meta-analysis of randomized controlled trials with extended follow-up. Pharmacol Res. 2014;81:64–73. pmid:24602799
  54. 54. Ray KK, Seshasai SR, Erqou S, Sever P, Jukema JW, Ford I, et al. Statins and all-cause mortality in high-risk primary prevention: a meta-analysis of 11 randomized controlled trials involving 65,229 participants. Arch Intern Med. 2010;170(12):1024–31. pmid:20585067
  55. 55. Savarese G, Gotto AM Jr., Paolillo S, D’Amore C, Losco T, Musella F, et al. Benefits of statins in elderly subjects without established cardiovascular disease: a meta-analysis. J Am Coll Cardiol. 2013;62(22):2090–9. pmid:23954343
  56. 56. Taylor F, Ward K, Moore TH, Burke M, Davey Smith G, Casas JP, et al. Statins for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2011;(1):CD004816. pmid:21249663
  57. 57. Tonelli M, Lloyd A, Clement F, Conly J, Husereau D, Hemmelgarn B, et al. Efficacy of statins for primary prevention in people at low cardiovascular risk: a meta-analysis. CMAJ. 2011;183(16):E1189–202. pmid:21989464
  58. 58. Chou R, Dana T, Blazina I, Daeges M, Jeanne TL. Statins for Prevention of Cardiovascular Disease in Adults: Evidence Report and Systematic Review for the US Preventive Services Task Force. JAMA. 2016;316(19):2008–24. pmid:27838722
  59. 59. Preiss D, Campbell RT, Murray HM, Ford I, Packard CJ, Sattar N, et al. The effect of statin therapy on heart failure events: a collaborative meta-analysis of unpublished data from major randomized trials. Eur Heart J. 2015;36(24):1536–46. pmid:25802390
  60. 60. Teng M, Lin L, Zhao YJ, Khoo AL, Davis BR, Yong QW, et al. Statins for Primary Prevention of Cardiovascular Disease in Elderly Patients: Systematic Review and Meta-Analysis. Drugs Aging. 2015;32(8):649–61. pmid:26245770
  61. 61. Siontis GC, Ioannidis JP. Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011;40(5):1292–307. pmid:21737403
  62. 62. Hemkens LG, Ewald H, Naudet F, Ladanie A, Shaw JG, Sajeev G, et al. Interpretation of epidemiological studies very often lacked adequate consideration of confounding. Journal of clinical epidemiology. 2017. pmid:28943377
  63. 63. Egger M, Schneider M, Davey Smith G. Spurious precision? Meta-analysis of observational studies. BMJ. 1998;316(7125):140–4. pmid:9462324
  64. 64. Stamatakis E, Weiler R, Ioannidis JP. Undue industry influences that distort healthcare research, strategy, expenditure and practice: a review. Eur J Clin Invest. 2013;43(5):469–75. pmid:23521369
  65. 65. Moodie R, Stuckler D, Monteiro C, Sheron N, Neal B, Thamarangsi T, et al. Profits and pandemics: prevention of harmful effects of tobacco, alcohol, and ultra-processed food and drink industries. Lancet. 2013;381(9867):670–9. pmid:23410611
  66. 66. Fanelli D, Costas R, Ioannidis JP. Meta-assessment of bias in science. Proc Natl Acad Sci U S A. 2017;114(14):3714–9. pmid:28320937
  67. 67. Fanelli D. Redefine misconduct as distorted reporting. Nature. 2013;494(7436):149. pmid:23407504
  68. 68. Ioannidis JP. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012;308(6):575–6. pmid:22871867
  69. 69. Bothwell LE, Greene JA, Podolsky SH, Jones DS. Assessing the Gold Standard—Lessons from the History of RCTs. N Engl J Med. 2016;374(22):2175–81. pmid:27248626
  70. 70. Zarin DA, Tse T, Sheehan J. The Proposed Rule for U.S. Clinical Trial Registration and Results Submission. N Engl J Med. 2014;372(2):174–80. pmid:25539444
  71. 71. Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data Sharing Statements for Clinical Trials—A Requirement of the International Committee of Medical Journal Editors. N Engl J Med. 2017;376(23):2277–9. pmid:28581902
  72. 72. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, et al. A manifesto for reproducible science. Nature Human Behaviour. 2017;1:0021.
  73. 73. Ioannidis JA. The proposal to lower p value thresholds to .005. JAMA. 2018;319(14):1429–30. pmid:29566133
  74. 74. Chartres N, Fabbri A, Bero LA. Association of industry sponsorship with outcomes of nutrition studies: A systematic review and meta-analysis. JAMA Internal Medicine. 2016;176(12):1769–77. pmid:27802480
  75. 75. Serodio PM, McKee M, Stuckler D. Coca-Cola—a model of transparency in research partnerships? A network analysis of Coca-Cola’s research funding (2008–2016). Public Health Nutr. 2018:1–14. pmid:29560842
  76. 76. Gomez-Garcia F, Ruano J, Gay-Mimbrera J, Aguilar-Luque M, Sanz-Cabanillas JL, Alcalde-Mellado P, et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool. J Clin Epidemiol. 2017;92:79–88. pmid:28893571
  77. 77. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC medical research methodology. 2007;7:10. pmid:17302989
  78. 78. Xing D, Wang B, Zhang W, Yang Z, Hou Y, Chen Y, et al. Intra-articular hyaluronic acid injection in treating knee osteoarthritis: assessing risk of bias in systematic reviews with ROBIS tool. Int J Rheum Dis. 2017;20(11):1658–73. pmid:29044993
  79. 79. Higgins JPT, G S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011.
  80. 80. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676–80. pmid:16467236
  81. 81. Tsilidis KK, Kasimis JC, Lopez DS, Ntzani EE, Ioannidis JP. Type 2 diabetes and cancer: umbrella review of meta-analyses of observational studies. BMJ. 2015;350:g7607. pmid:25555821
  82. 82. Owen N, Sparling PB, Healy GN, Dunstan DW, Matthews CE. Sedentary behavior: emerging evidence for a new health risk. Mayo Clin Proc. 2010;85(12):1138–41. pmid:21123641
  83. 83. Ioannidis JP. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. The Milbank quarterly. 2016;94(3):485–514. pmid:27620683
  84. 84. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. pmid:17941714
  85. 85. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283(15):2008–12. pmid:10789670
  86. 86. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. pmid:19622551
  87. 87. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2011;344:d7762. pmid:22214758
  88. 88. Ioannidis JPA. Clarifications on the application and interpretation of the test for excess significance and its extensions. J Math Psychol. 2013;57(5):184–7. http://dx.doi.org/10.1016/j.jmp.2013.03.002.
  89. 89. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34. pmid:9310563
  90. 90. Poole R, Kennedy OJ, Roderick P, Fallowfield JA, Hayes PC, Parkes J. Coffee consumption and health: umbrella review of meta-analyses of multiple health outcomes. BMJ. 2017;359:j5024. pmid:29167102
  91. 91. Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53(11):1119–29. pmid:11106885