https://orcid.org/0000-0002-0417-1988
Figures
Abstract
Introduction
The number of umbrella reviews (URs) that compiled systematic reviews and meta-analysis (SR-MAs) has increased dramatically over recent years. No formal guidance for assessing the certainty of evidence in URs of meta-analyses exists nowadays. URs of non-interventional studies help establish evidence linking exposure to certain health outcomes in a population. This study aims to identify and describe the methodological approaches for assessing the certainty of the evidence in published URs of non-interventions.
Methods
We searched from 3 databases including PubMed, Embase, and The Cochrane Library from May 2010 to September 2021. We included URs that included SR-MAs of studies with non-interventions. Two independent reviewers screened and extracted data. We compared URs characteristics stratified by publication year, journal ranking, journal impact factor using Chi-square test.
Results
Ninety-nine URs have been included. Most were SR-MAs of observational studies evaluating association of non-modifiable risk factors with some outcomes. Only half (56.6%) of the included URs assessed the certainty of the evidence. The most frequently used criteria is credibility assessment (80.4%), followed by GRADE approach (14.3%). URs published in journals with higher journal impact factor assessed certainty of evidence than URs published in lower impact group (77.1 versus 37.2% respectively, p < 0.05). However, criteria for credibility assessment used in four of the seven URs that were published in top ranking journals were slightly varied.
Conclusions
Half of URs of MAs of non-interventional studies have assessed the certainty of the evidence, in which criteria for credibility assessment was the commonly used method. Guidance and standards are required to ensure the methodological rigor and consistency of certainty of evidence assessment for URs.
Citation: Sadoyu S, Tanni KA, Punrum N, Paengtrai S, Kategaew W, Promchit N, et al. (2022) Methodological approaches for assessing certainty of the evidence in umbrella reviews: A scoping review. PLoS ONE 17(6): e0269009. https://doi.org/10.1371/journal.pone.0269009
Editor: Muhammad Shahzad Aslam, Xiamen University - Malaysia Campus: Xiamen University - Malaysia, MALAYSIA
Received: January 30, 2022; Accepted: May 12, 2022; Published: June 8, 2022
Copyright: © 2022 Sadoyu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the article and its Supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: AMSTAR, A MeaSurement Tool to Assess systematic Reviews; GRADE, Grading of Recommendations, Assessment, Development and Evaluations; IQR, Interquartile range; JBI, Joanna Bring Institute; JIF, journal impact factors; MA, meta-analysis; OQAQ, Oxman and Guyatt Overview Quality Assessment Questionnaire; RCT, randomized controlled trials; ROBIS, A Risk of Bias Assessment Tool for Systematic Reviews; SR, systematic reviews; UR, umbrella review
Introduction
The number of systematic review (SR) and umbrella reviews (UR) and meta-analysis (MAs) has increased dramatically over recent years [1]. Most SRs and MAs focus on answering a question such as the effect of a single treatment comparison on an outcome, this leaves a big gap of lacking an overall summary of evidence addressing broader related questions. URs, also known as overview or review of reviews, evolved in the last decade, can summarize and even synthesize the findings into single comprehensive evidence answering the broader picture of all existing findings [1, 2]. As a result, URs are considered as one of the highest levels of evidence summary in biomedical literature [2]. URs of epidemiological investigations and non-interventional studies help establish evidence linking exposure to certain health outcomes in a population. Therefore, these studies are expected to play a key role in gauging the burden of diseases, understand the risk or protective factors, delineating guideline for prevention as well as streamlining the treatment development process [3, 4].
Some steps for performing URs are generally similar to SR-MAs (e.g., search strategy, study selection, data extraction), yet some others especially assessment of uncertainty methods are not applicable owing to the difference in types of included studies [5]. Recently, there is a methodological guidance focusing on conducting and reporting an UR [6]. However, it does not cover every single aspect of the process including assessment methods for certainty of the evidence.
The certainty of the evidence from URs is an essential component as it demonstrates the confidence of the findings found across studies leading to support a decision or recommendations. Several approaches have been used in literature. For instance, some URs adopted Grading of Recommendation, Assessment, Development and Evaluation (GRADE) approach, which was originally designed for assessing the certainty of evidence of primary studies included in SRs, not URs [7]. In contrast, some URs reported the certainty of included SRs and MAs as originally reported from each study without further assessment [8–10]. Furthermore, the UR of MAs is more challenging as they usually report summary statistical data as one of the objective criteria to grade the certainty of evidence. Recently, the relatively strict criteria for stratifying the certainty of evidence using several statistical parameters (i.e., degree of statistical significance, predictive interval, small-study effects, and excess significance bias) have also been used and suggested as the good practical tips for conducting good URs [2]. Currently, there is no formal guidance for assessing the certainty of evidence in URs. Therefore, this scoping review aims to identify and describe the methodological approaches for assessing the certainty of the evidence in published URs that included MAs of non-interventions.
Methods
This review was conducted according to the methods pre-specified in a registered protocol (PROSPERO registration: CRD42020203273), following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews (S1 Table in S1 File).
Search strategy and selection criteria
We searched three databases including PubMed, Embase, and The Cochrane Library from May 2010 to Sep 2021. The keyword ‘umbrella review’ was used. The full search strategies without language restriction are described in S2 Table in S1 File. Manual searches of the reference lists of the eligible articles were also performed. We defined an UR as the review that is designed to summarize the evidence from multiple SR-MAs that were labeled as ‘umbrella review’ in the title or abstract of the article.
At least two reviewers (SS, KT, NP, SN, WK, NP, and SP) independently reviewed the titles, abstracts, and full texts for their potential inclusion against the eligibility criteria. Any disagreement was resolved by consensus with a third reviewer (NC). No language restriction was applied. URs, overview of SR-MAs, review of SR-MAs were selected if they included (a) MAs of interventions and (b) MAs of non-interventions involving diagnostic/prognostic factors and non-modifiable risk factors of diseases or health conditions, disease etiology, prevalence or incidence; in which most studies were observational studies, e.g., cohort, case-control, and cross-sectional studies. In this current study, we focus only umbrella reviews that included MAs of non-intervention studies.
Other types of studies or reviews (e.g., handbooks, guidelines, commentaries, editorials, and methodological studies), materials for poster presentations, UR of SRs without MAs, and protocols of URs were excluded. URs with network MAs were also excluded because these studies might use different methodological approaches.
Data extraction
At least two reviewers (SS, NP, SN, KT, WK, MP, and SP) independently extracted the data from each UR into a customized data extraction table. Any disagreement was resolved by consensus with a third reviewer (NC). Details of data extraction are described in S1 File.
The assessment of the certainty of evidence was defined as any of evaluation of the totality or strength of the evidence such as the GRADE approach, criteria for credibility assessment, and other approaches used to grade the overall body of the UR evidence.
Data synthesis and analysis
A descriptive analysis of the methodological approaches for assessing the certainty of evidence in the URs was performed by frequencies and percentage. The included URs were classified into high and low impact sources based on the journal impact factors (JIF) reported by the Institute of Scientific Information’s Journals Citation Report in 2021 accessed on October 21, 2021. The journals reported as the top 100 highest ranking were defined as the high, otherwise they were classified as low impact groups. In addition, we also classified based on a median JIF, i.e., high if the URs were published in JIF ≥ median, otherwise the URs were classified as lower impact groups. According to the previous study [11] that suggested the usefulness of URs according to the higher number of citations after 2015 and the release of tools for the methodological quality assessment in 2016 [12]. Thus, we further compared URs published between 2010 to 2016 with those published from 2017 to 2021, when feasible. Chi-square or Fisher’s exact test where appropriated was applied to compare characteristics of URs between groups. All analyses were performed using STATA version 15.0 (College Station, TX), p-value ≤ 0.05 was considered as statistical significance.
Results
Search results
We identified 2405 articles, of which 302 and 1573 articles were excluded due to duplicates and during screening titles/abstracts, respectively: leaving 530 studies for the full-text review. A total of 447 URs matched with the eligibility criteria. Finally, 348 and 99 URs of intervention and non-therapy/non-intervention studies were eligible but only 99 URs were focused and reported in our scoping review (Fig 1) [13–111]. The reasons for exclusion of the articles after full-text review were described in detail in S3 Table in S1 File.
Characteristics of included URs
Table 1 summarize the characteristics of the 99 included URs. The number of URs increased over time from 1 in 2015 to 32 in 2021, with the majority being published in the recent 5 years (2017–2021), (N = 90, 90.9%).
Most URs (n = 78, 76.8%) included individual SR-MAs of observational studies evaluating risk or protective effects of risk factors, association, and non-modifiable risk factors of a disease or health condition, followed by URs that focused on prevalence/ incidence (n = 11, 11.1%), and etiology, diagnosis/prognostic biomarkers (n = 6, 6.1%). The median number of MAs included in these corresponding URs were 12 (Interquartile range (IQR): 5–42), The median of total number of primary studies included were 243 (IQR: 174–683). The median JIF of URs that included in this study was 4.45 (3.01–7.72). Seven of the 99 URs (7.1%) were published in top-100 ranking journals [30, 31, 41, 71, 95, 100, 103] according to the JIF in 2021.
Methodological approaches for assessing certainty of the evidence
Of 99 URs, only half of them (N = 56, 56.6%) assessed the certainty of the evidence, see Table 1, S4 and S5 Tables in S1 File. Criteria for credibility assessment was the most frequently used method (N = 45, 80.4%) followed by GRADE approach (N = 8, 14.3%), see Fig 2. Almost all URs used one tool (N = 55, 98.2%), only 1 URs (1.8%) used both criteria for credibility assessment and GRADE approach [61].
(A) Methodological approaches for certainty of evidence assessment. (B) Methodological approaches for methodological quality assessment.
Table 2 showed the certainty and methodological quality assessment in URs. Based on the median of JIFs, the percent URs that assessed the certainty of evidence was significantly higher in the high impact group (JIF > 4.45) than the lower impact group (JIF ≤ 4.45), i.e., 77.1% vs 37.3%, p < 0.0001. The number of URs that published in top-100 journals group also assessed the certainty of evidence more than the lower impact group, although it was not significant (71.4% vs 55.4%, p = 0.70). Comparing the period of 2010–2016 with 2017–2021, the percent of URs with certainty of the evidence assessments was higher in a period of 2010–2016 in than 2017–2021 but no statistically significant was found (66.7% vs 55.6%, p = 0.73). In addition, Fig 3 showed the direction that the proportion of URs that performed the certainty of evidence assessment was increase over time. Except that only 1 UR that published in 2015 meet our criteria and the author performed certainty of evidence assessment, thus, the proportion of URs that assess the certainty was 100%.
Of 7 URs published in top-100 ranking journals group [30, 31, 41, 71, 95, 100, 103], the assessment of certainty of the evidence was performed in 5 studies. The most used tools were criteria for credibility assessment (n = 4) followed by 1 study using GRADE approach. However, criteria for credibility assessment used in these URs were varied across studies, as shown in Table 3. For instance, 2 URs used the retained statistical significance in 10% credibility ceiling and the largest study with statistically significant effect as criteria for convincing class (the highest certainty). Four URs classified the certainty of evidence into 5 levels including “Convincing,” Highly suggestive,” Suggestive,” “Weak,” or "Not significant" but 1 UR used this tool to identified associations that had the strongest validity and were not suggestive of bias. Moreover, convincing evidence was graded based on the number of cases included in each MA of ≥ 350 to ≥ 5000 or p-values of < 10−6 or even < 0.001, as shown in S6 Table in S1 File.
Methodological quality assessment
Most of the included URs performed the methodological quality assessment of included MAs (n = 74, 74.8%). Of these, the most frequently used tool was AMSTAR 2 (n = 34, 46%), followed by its old version called AMSTAR (n = 20, 27%), and Joanna Bring Institute (JBI) critical appraisal checklist for SRs (n = 13, 17.6%), as shown in Table 1 and Fig 2.
Among 7 URs that published in top-100 journal ranking group [30, 31, 41, 71, 95, 100, 103], 5 of them (71.4%) assessed the methodological quality using AMSTAR [100, 103] and AMSTAR 2 [30, 41, 95]. The proportion of URs in the high-impact journal group, based on the median of journal impact factor, that performed the methodological quality assessment was lower than those published in the lower impact journal group, but it was not statistically significant (72.9% vs 74.6%, p = 0.69), as shown in Table 3. The more recent URs published in 2017–2021, performed a methodological quality assessment more often than those published in 2010–2016 significantly, (78.9% vs 33.3%, p < 0.05). Furthermore, Fig 3 also showed the inclination of the proportion of URs that performed the methodological assessment over time.
Discussion
To the best of our knowledge, this is the first study identifying the methodological approaches for assessing the certainty of evidence in URs of SR-MAs considering 99 URs of non-intervention studies. Our finding suggested that only nearly half of the included URs assessed certainty of evidence, in which the criteria of credibility assessment was a mainly used tool. URs that were published in high JIFs and high-ranking journals are more likely to assess the certainty of evidence than URs published in lower JIFs and lower-ranking journals. Nearly 80% of the URs performed a methodological quality assessment and the AMSTAR 2 was the most frequently used tool for this process.
URs were increasingly published over the last decade to compile evidence and provide broad pictures of information from SR-MAs [2]. URs of epidemiological investigations and non-interventional studies also help in establish evidence linking exposure to the incidence of certain health condition in a population. Consequently, these studies are expected to play a key role in gauging the burden of diseases, understand the risk or protective factors, delineating guidelines for prevention as well as streamlining the treatment development process [3, 4]. The certainty of the evidence from these URs, which is the extent of confidence to support a decision or recommendations, may further be used as supportive evidence in develop clinical practice guidelines and recommendations. High certainty in evidence means that the investigators are very confident that the effect they found across studies is close to the true effect and vice versa [111]. URs of these studies should aim to provide the highest certainty of evidence to facilitate better health outcomes. Despite the necessity of assessing the certainty of the evidence in URs, there is no consensus that which approach should be the method of choice. Although Aromataris et al.—a methodology working group formed by the JBI (formerly named the URs Working Group)—published the guidance on how to conduct and report an UR [6], the methodology for the certainty assessment was not provided. Hence, we included URs of non-interventional studies in this scoping review to provide information the methodology for the certainty assessment that used these days.
According to our findings, approximately half of the included URs assessed the certainty of evidence, with the criteria of credibility assessment being the most commonly utilized tool. In contrast to the results from a previous study by Hartling et al [1] indicating that described the methodological approaches in overviews of interventions. They indicated that only 16% of the overview of reviews published between 2000 and 2011 assessed the certainty of the evidence and the most frequently used method is the GRADE approach. One of the reasons could be that the GRADE approach is a well-established tool developed to determine the certainty of evidence-based on several factors namely risk of bias, imprecision, indirectness, inconsistency, and publication bias [111]. However, GRADE approach was primarily designed for assessing the quality of the evidence from primary studies. Further guidance is needed to ensure appropriate use and interpretation of the GRADE tool when it is applied to assess the quality of evidence of SRs, instead of primary studies [1]. The other reasons that our study differs from the previous study likely because we specifically considered the URs that included MAs of non-interventions. The criteria for credibility assessment classified the certainty of the evidence according to several statistical criteria, which usually reported in MAs. This method was recently released [6, 112], might be specific to URs of MAs of non-interventions, and was being used more commonly.
Because most researchers generally aspire to publish their research findings in the top journal publishers. We then classified URs that included in our study into high and lower impact groups using journal impact factors (JIFs) to find the impact of methodological approaches on journal publication. The JIFs help reflect several factors such as the high frequency of citations, media promotion of articles and journals, and the increase in speed of the review and publication process [113, 114]. Although JIFs could not be used to reflect the full impact of journals on formal implications, it remains an acceptable objective and quantifiable measure of knowledge dissemination nowadays [113–115]. This study demonstrated that a higher number of URs with a certainty assessment was published in higher impact and higher-ranking journals. One of the reasons could be that the assessment helped to reflect the certainty of results and facilitate the translation of the evidence into guideline recommendations.
When focusing in the URs that published in top-100 ranking journal, criteria for credibility assessment was also the most used method. However, levels of evidence and a series of statistical tests in these URs using the arbitrary cut-off values and the cut point of each component in these criteria varied slightly. As shown in Table 3, UR that published earlier start using criteria for credibility assessment to identify associations that had the strongest certainty or not. Then, the URs published later used criteria for credibility assessment to categorize the certainty of evidence into several levels from convincing (the highest level) to weak (the lowest level). Their classification was obtained through strict criteria including number of cases and several statistical parameters. The degree of statistical significance (p-value) was used despite the lack of consensus on what might be and optimal threshold and p-value thresholds might need to be tailored to the specific research setting and even to a specific database [2, 116]. Some URs used p<0.05 or p<0.001, but several URs used the stricter level of P < 10−6 to be categorized as the highest certainty of evidence. Several other parameters also have been used including a heterogeneity, predictive interval, and small-study effect test. The excess significance bias, which evaluates whether the number of observed studies with statistically significant results differs from the expected number of positive studies [117], has also been used to be a criterion in some of the URs. The credibility ceiling is a method to test whether the observational studies can survive the specific level that the likelihood of summary effect being in a specific direction [118]. Two URs that published more recently used the retained statistical significance in 10% credibility ceiling as criteria for convincing class (the highest certainty) likely because this method was published more recently [118, 119]. Moreover, criteria of credibility assessment were generally reproduced following previous URs, where some of the published URs were repeatedly cited [71, 79, 80]. Therefore, our findings highlighted the importance of guidance for assessing the certainty of the evidence in URs to recommend the most appropriate tools to provide standards for those conducting URs.
This study also demonstrated that majority of the included URs performed a methodological quality assessment. This was more frequent than a previous study [1] that reported the assessment of methodological quality in only 37% of the overviews of reviews. One of the reasons could be that this process has been strongly recommended in the methodological guidance for producing URs [2] and has been implemented longer than the certainty assessment. This process is essential to ensure that the methodological quality of SR-MAs that included in URs are adequately assessed and incorporated into the results and conclusions. Besides, we found that the most often used tool for methodological quality assessment changed from the Oxman and Guyatt Overview Quality Assessment Questionnaire (OQAQ) to AMSTAR. The AMSTAR tool has been recommended since 2007 and the revised version-AMSTAR 2 was released in 2016 [12]. Given that the revised tool introduced recently, the method advocated in published guidance have evolved over time and the variation of tool used for methodological quality assessment reported in this study confirms the need for updated guidance for conducting URs. In addition, our findings also highlighted that many of the published URs of non-intervention studies performed the certainty of evidence and/or methodological quality assessments, particularly in the more recent published URs and URs that published in journals with higher impact factor. In the current time, the criteria for credibility assessments is the most commonly used methods for certainty assessment and AMSTAR-2 is the most used methods for methodological quality assessments. These results help emphasize the future researchers to apply these assessments in their studies.
Our study has some limitations. First, the definition of included studies was restricted to URs. This might not cover all types of other kinds of reviews for example- overview of reviews, and review of reviews. Therefore, our findings with regards to terminology used to describe “umbrella reviews” and methods used might not be comprehensive or wholly representative. However, there is no universally accepted technical term for this new type of reviews that summarize or synthesize findings from systematic reviews. The term URs has been used increasingly and studies that describe the methodological approach regarding the URs are sparse to date. Second, this study is confined to only URs of non-intervention studies. The methods used for assessing the certainty of evidence and methodological quality in URs that contained other study designs might be different from our findings and could be extended in future research. Third, our study focused on describing the method used in previously published URs and most of them did not provide the reasons for methods selection. Thus, we could not assess the reasons why each UR used different approaches for assessing the certainty of evidence and methodological quality. Although this study could not endorse which method is the best for the certainty of the evidence assessment in URs, a major strength of our study is that it provides a broad picture of the certainty assessment methods used in URs and the commonly used tools to perform this assessment.
Conclusions
This study revealed that only half of URs that included MAs of non-interventional studies have assessed the certainty of the evidence. The criteria of credibility assessment were the most used method. Moreover, URs that published in higher impact journals assessed the certainty of evidence more than the lower impact group. Therefore, guidance and standards are required to ensure the methodological rigor and consistency of certainty of evidence assessment for URs.
Acknowledgments
The authors gratefully acknowledge Asst.Prof. Peerawat Jinatongthai for his help on databases searching and search strategy suggestion.
References
- 1. Hartling L, Chisholm A, Thomson D, Dryden DM. A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One. 2012;7(11):e49667. pmid:23166744
- 2. Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evidence Based Mental Health. 2018;21(3):95–100. pmid:30006442
- 3. Mishra D, Vora J. Non interventional drug studies in oncology: Why we need them? Perspectives in clinical research. 2010;1(4):128–33. pmid:21350727
- 4.
Committee on the Analysis of Cancer Risks in Populations near Nuclear Facilities-Phase I, Nuclear, Radiation Studies B, Division on E, Life S, National Research C. Analysis of Cancer Risks in Populations Near Nuclear Facilities: Phase I. Washington (DC): National Academies Press (US) Copyright 2012 by the National Academy of Sciences. All rights reserved.; 2012.
- 5. Lunny C, Brennan SE, McDonald S, McKenzie JE. Evidence map of studies evaluating methods for conducting, interpreting and reporting overviews of systematic reviews of interventions: rationale and design. Systematic Reviews. 2016;5(1):4. pmid:26739283
- 6. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Int J Evid Based Healthc. 2015;13(3):132–40. pmid:26360830
- 7. Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence. Syst Rev. 2018;7(1):159. pmid:30314530
- 8. De Freitas L, Goodacre S, O’Hara R, Thokala P, Hariharan S. Interventions to improve patient flow in emergency departments: an umbrella review. Emerg Med J. 2018;35(10):626–37. pmid:30093379
- 9. Papageorgiou PN, Deschner J, Papageorgiou SN. Effectiveness and Adverse Effects of Deep Brain Stimulation: Umbrella Review of Meta-Analyses. J Neurol Surg A Cent Eur Neurosurg. 2017;78(2):180–90. pmid:27642815
- 10. Seifo N, Cassie H, Radford JR, Innes NPT. Silver diamine fluoride for managing carious lesions: an umbrella review. BMC Oral Health. 2019;19(1):145. pmid:31299955
- 11.
Hossain M. Umbrella Review as an Emerging Approach of Evidence Synthesis in Health Sciences: A Bibliometric Analysis. SSRN Electronic Journal. 2020.
- 12. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. pmid:28935701
- 13. Abou Ghayda R, Duck-Young Park D, Lee JY, Kim JY, Lee KH, Hong SH, et al. Body mass index and mortality in patients with cardiovascular disease: an umbrella review of meta-analyses. Eur Rev Med Pharmacol Sci. 2021;25(1):273–86. pmid:33506916
- 14. Arias-de la Torre J, Puigdomenech E, García X, Valderas JM, Eiroa-Orosa FJ, Fernández-Villa T, et al. Relationship Between Depression and the Use of Mobile Technologies and Social Media Among Adolescents: Umbrella Review. J Med Internet Res. 2020;22(8):e16388. pmid:32663157
- 15. Avşar TS, McLeod H, Jackson L. Health outcomes of smoking during pregnancy and the postpartum period: an umbrella review. BMC Pregnancy Childbirth. 2021;21(1):254. pmid:33771100
- 16. Balante J, Broek DVD, White K. How does culture influence work experience in a foreign country? An umbrella review of the cultural challenges faced by internationally educated nurses. Int J Nurs Stud. 2021;118:103930. pmid:33819641
- 17. Bertelsen AS, Florisson S, Aagesen EK, Rosholm J-U, Nielsen LP. Are older adults insufficiently included in clinical trials?-An umbrella review. Basic and Clinical Pharmacology and Toxicology. 2021;128(2):213–23. pmid:33210799
- 18.
Blume KS, Dietermann K, Kirchner-Heklau U, Winter V, Fleischer S, Kreidl LM, et al. Staffing levels and nursing-sensitive patient outcomes: Umbrella review and qualitative study. Health Serv Res. 2021.
- 19. Bonaccorsi G, Manzi F, Del Riccio M, Setola N, Naldi E, Milani C, et al. Impact of the Built Environment and the Neighborhood in Promoting the Physical Activity and the Healthy Aging in Older People: An Umbrella Review. Int J Environ Res Public Health. 2020;17(17).
- 20. Bramley P, McArthur K, Blayney A, McCullagh I. Risk factors for postoperative delirium: An umbrella review of systematic reviews. Int J Surg. 2021;93:106063. pmid:34411752
- 21. Burns CJ, Juberg DR. Cancer and occupational exposure to pesticides: an umbrella review. Int Arch Occup Environ Health. 2021;94(5):945–57. pmid:33495906
- 22. Carvalho AF, Solmi M, Sanches M, Machado MO, Stubbs B, Ajnakina O, et al. Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders. Transl Psychiatry. 2020;10(1):152. pmid:32424116
- 23. Fan J, Senthanar S, Macpherson RA, Sharpe K, Peters CE, Koehoorn M, et al. An Umbrella Review of the Work and Health Impacts of Working in an Epidemic/Pandemic Environment. Int J Environ Res Public Health. 2021;18(13).
- 24. Farazi M, Jayedi A, Shab-Bidar S. Dietary inflammatory index and the risk of non-communicable chronic disease and mortality: an umbrella review of meta-analyses of observational studies. Crit Rev Food Sci Nutr. 2021:1–10. pmid:34176394
- 25. Gatti M, Bianchin M, Raschi E, De Ponti F. Assessing the association between fluoroquinolones and emerging adverse drug reactions raised by regulatory agencies: An umbrella review. Eur J Intern Med. 2020;75:60–70. pmid:31983604
- 26. Hossain MM, Sultana A, Tasnim S, Fan Q, Ma P, McKyer ELJ, et al. Prevalence of mental disorders among people who are homeless: An umbrella review. Int J Soc Psychiatry. 2020;66(6):528–41. pmid:32460590
- 27. Iqbal S, Ali S, Ali I. Maternal pesticide exposure and its relation to childhood cancer: an umbrella review of meta-analyses. Int J Environ Health Res. 2021:1–19. pmid:33745400
- 28. Kelly MM, Griffith PB. The Influence of preterm birth beyond infancy: Umbrella review of outcomes of adolescents and adults born preterm. J Am Assoc Nurse Pract. 2020;32(8):555–62. pmid:31651585
- 29. Kelly MM, Griffith PB. Umbrella Review of School Age Health Outcomes of Preterm Birth Survivors. J Pediatr Health Care. 2020;34(5):e59–e76. pmid:32660808
- 30. Kim JH, Kim JY, Lee J, Jeong GH, Lee E, Lee S, et al. Environmental risk factors, protective factors, and peripheral biomarkers for ADHD: an umbrella review. Lancet Psychiatry. 2020;7(11):955–70. pmid:33069318
- 31. Kim MS, Kim WJ, Khera AV, Kim JY, Yon DK, Lee SW, et al. Association between adiposity and cardiovascular outcomes: an umbrella review and meta-analysis of observational and Mendelian randomization studies. Eur Heart J. 2021;42(34):3388–403. pmid:34333589
- 32. Lee KS, Choi YJ, Cho J, Lee H, Lee H, Park SJ, et al. Environmental and Genetic Risk Factors of Congenital Anomalies: an Umbrella Review of Systematic Reviews and Meta-Analyses. J Korean Med Sci. 2021;36(28):e183. pmid:34282604
- 33.
Li W, Pan J, Wei M, Lv Z, Chen S, Qin Y, et al. Non-ocular influence factors for primary glaucoma: an umbrella review of meta-analysis. Ophthalmic Res. 2021.
- 34. Lopez-Leon S, González-Giraldo Y, Wegman-Ostrosky T, Forero DA. Molecular genetics of substance use disorders: An umbrella review. Neurosci Biobehav Rev. 2021;124:358–69. pmid:33556390
- 35. Magnavita N, Chirico F, Garbarino S, Bragazzi NL, Santacroce E, Zaffina S. SARS/MERS/SARS-CoV-2 Outbreaks and Burnout Syndrome among Healthcare Workers. An Umbrella Systematic Review. Int J Environ Res Public Health. 2021;18(8).
- 36. Solmi M, Dragioti E, Arango C, Radua J, Ostinelli E, Kilic O, et al. Risk and protective factors for mental disorders with onset in childhood/adolescence: An umbrella review of published meta-analyses of observational longitudinal studies. Neurosci Biobehav Rev. 2021;120:565–73. pmid:32931804
- 37. Martinez-Calderon J, Flores-Cortes M, Morales-Asencio JM, Luque-Suarez A. Which Psychological Factors Are Involved in the Onset and/or Persistence of Musculoskeletal Pain? An Umbrella Review of Systematic Reviews and Meta-Analyses of Prospective Cohort Studies. Clin J Pain. 2020;36(8):626–37. pmid:32379072
- 38. Mjaess G, Chebel R, Karam A, Moussa I, Pretot D, Abi Tayeh G, et al. Prognostic role of neutrophil-to-lymphocyte ratio (NLR) in urological tumors: an umbrella review of evidence from systematic reviews and meta-analyses. Acta Oncol. 2021;60(6):704–13. pmid:33586577
- 39. Mohammed SH, Habtewold TD, Arero AG, Esmaillzadeh A. The state of child nutrition in Ethiopia: an umbrella review of systematic review and meta-analysis reports. BMC Pediatr. 2020;20(1):404. pmid:32847552
- 40. Obaid M, Douiri A, Flach C, Prasad V, Marshall I. Can we prevent poststroke cognitive impairment? An umbrella review of risk factors and treatments. BMJ Open. 2020;10(9):e037982. pmid:32912953
- 41. Okoth K, Chandan JS, Marshall T, Thangaratinam S, Thomas GN, Nirantharakumar K, et al. Association between the reproductive health of young women and cardiovascular disease in later life: umbrella review. Bmj. 2020;371:m3502. pmid:33028606
- 42. Palmer Kelly E, Paredes AZ, Tsilimigras DI, Hyer JM, Pawlik TM. The role of religion and spirituality in cancer care: An umbrella review of the literature. Surg Oncol. 2020:101389. pmid:34103240
- 43. Paraskevi L, Antigoni S, Kleanthi G. Stress and Anxiety Levels in Couples who Undergo Fertility Treatment: a Review of Systematic Reviews. Mater Sociomed. 2021;33(1):60–4. pmid:34012353
- 44. Pelletier R, Ng K, Alkabbani W, Labib Y, Mourad N, Gamble JM. The association of sodium-glucose cotransporter 2 inhibitors with cancer: An overview of quantitative systematic reviews. Endocrinol Diabetes Metab. 2020;3(3):e00145. pmid:32704566
- 45. Pelletier R, Ng K, Alkabbani W, Labib Y, Mourad N, Gamble JM. Adverse events associated with sodium glucose co-transporter 2 inhibitors: an overview of quantitative systematic reviews. Ther Adv Drug Saf. 2021;12:2042098621989134. pmid:33552467
- 46. Peruzzi M, Biondi-Zoccai G, Carnevale R, Cavarretta E, Frati G, Versaci F. Vaping Cardiovascular Health Risks: an Updated Umbrella Review. Curr Emerg Hosp Med Rep. 2020:1–7. pmid:32837803
- 47. Pittara T, Vyrides A, Lamnisos D, Giannakou K. Pre-eclampsia and long-term health outcomes for mother and infant: an umbrella review. Bjog. 2021;128(9):1421–30. pmid:33638891
- 48.
Poveda-Moral S, Falcó-Pegueroles A, Ballesteros-Silva MP, Bosch-Alcaraz A. Barriers to Advance Care Planning Implementation in Health care: An Umbrella Review with Implications for Evidence-Based Practice. Worldviews Evid Based Nurs. 2021.
- 49. Rahmanian V, Rahmanian K, Jahromi AS, Bokaie S. Seroprevalence of toxoplasma gondii infection: An umbrella review of updated systematic reviews and meta-analyses. J Family Med Prim Care. 2020;9(8):3848–55. pmid:33110778
- 50. Román-Gálvez RM, Martín-Peláez S, Martínez-Galiano JM, Khan KS, Bueno-Cavanillas A. Prevalence of Intimate Partner Violence in Pregnancy: An Umbrella Review. Int J Environ Res Public Health. 2021;18(2). pmid:33467538
- 51. Rydzewska E, Dunn K, Cooper SA. Umbrella systematic review of systematic reviews and meta-analyses on comorbid physical conditions in people with autism spectrum disorder. Br J Psychiatry. 2021;218(1):10–9. pmid:33161922
- 52. Sahebi A, Nejati-Zarnaqi B, Moayedi S, Yousefi K, Torres M, Golitaleb M. The prevalence of anxiety and depression among healthcare workers during the COVID-19 pandemic: An umbrella review of meta-analyses. Prog Neuropsychopharmacol Biol Psychiatry. 2021;107:110247. pmid:33476692
- 53.
Sahle BW, Reavley NJ, Li W, Morgan AJ, Yap MBH, Reupert A, et al. The association between adverse childhood experiences and common mental disorders and suicidality: an umbrella review of systematic reviews and meta-analyses. Eur Child Adolesc Psychiatry. 2021.
- 54. Shams Vahdati S, Ala A, Rahmanpour D, Sadeghi-Hokmabadi E, Tahmasbi F. Neurological manifestations of COVID-19 infection: an umbrella review. Egypt J Neurol Psychiatr Neurosurg. 2021;57(1):113. pmid:34483649
- 55. Solmi M, Civardi S, Corti R, Anil J, Demurtas J, Lange S, et al. Risk and protective factors for alcohol and tobacco related disorders: An umbrella review of observational studies. Neurosci Biobehav Rev. 2021;121:20–8. pmid:33248149
- 56. Solmi M, Dragioti E, Croatto G, Radua J, Borgwardt S, Carvalho AF, et al. Risk and protective factors for cannabis, cocaine, and opioid use disorders: An umbrella review of meta-analyses of observational studies. Neurosci Biobehav Rev. 2021;126:243–51. pmid:33737104
- 57. Solmi M, Radua J, Stubbs B, Ricca V, Moretti D, Busatta D, et al. Risk factors for eating disorders: an umbrella review of published meta-analyses. Braz J Psychiatry. 2021;43(3):314–23. pmid:32997075
- 58. Solmi M, Veronese N, Galvano D, Favaro A, Ostinelli EG, Noventa V, et al. Factors Associated With Loneliness: An Umbrella Review Of Observational Studies. J Affect Disord. 2020;271:131–8. pmid:32479308
- 59. Thang Le VN, Kim JG, Yang YM, Lee DW. Risk Factors for Early Childhood Caries: An Umbrella Review. Pediatr Dent. 2021;43(3):176–94. pmid:34172110
- 60. Townsend R, Sileo FG, Allotey J, Dodds J, Heazell A, Jorgensen L, et al. Prediction of stillbirth: an umbrella review of evaluation of prognostic variables. Bjog. 2021;128(2):238–50. pmid:32931648
- 61.
Trott M, Smith L, Xiao T, Veronese N, Koyanagi A, Jacob L, et al. Hearing impairment and diverse health outcomes: An umbrella review of meta-analyses of observational studies. Wien Klin Wochenschr. 2021.
- 62. Tsiros MD, Tian EJ, Shultz SP, Olds T, Hills AP, Duff J, et al. Obesity, the new childhood disability? An umbrella review on the association between adiposity and physical function. Obes Rev. 2020;21(12):e13121. pmid:32779327
- 63. Wang Y, Zhong X, Zhou L, Lu J, Jiang B, Liu C, et al. Prognostic Biomarkers for Pancreatic Ductal Adenocarcinoma: An Umbrella Review. Front Oncol. 2020;10:1466. pmid:33042793
- 64.
Weir NM, Preston K, Newham R, Bennie M. Development of a primary care pharmacy outcomes framework: An umbrella literature review. Res Social Adm Pharm. 2021.
- 65. Xia L, Zhao R, Wan Q, Wu Y, Zhou Y, Wang Y, et al. Sarcopenia and adverse health-related outcomes: An umbrella review of meta-analyses of observational studies. Cancer Med. 2020;9(21):7964–78. pmid:32924316
- 66. Xu W, He Y, Wang Y, Li X, Young J, Ioannidis JPA, et al. Risk factors and risk prediction models for colorectal cancer metastasis and recurrence: an umbrella review of systematic reviews and meta-analyses of observational studies. BMC Med. 2020;18(1):172. pmid:32586325
- 67. Zhang TN, Wu QJ, Liu YS, Lv JL, Sun H, Chang Q, et al. Environmental Risk Factors and Congenital Heart Disease: An Umbrella Review of 165 Systematic Reviews and Meta-Analyses With More Than 120 Million Participants. Front Cardiovasc Med. 2021;8:640729. pmid:33791351
- 68. Zhang X, Gill D, He Y, Yang T, Li X, Monori G, et al. Non-genetic biomarkers and colorectal cancer risk: Umbrella review and evidence triangulation. Cancer Med. 2020;9(13):4823–35. pmid:32400092
- 69. Apóstolo J, Cooke R, Bobrowicz-Campos E, Santana S, Marcucci M, Cano A, et al. Predicting risk and outcomes for frail older adults: an umbrella review of frailty screening tools. JBI Database System Rev Implement Rep. 2017;15(4):1154–208. pmid:28398987
- 70. Belbasis L, Bellou V, Evangelou E. Environmental Risk Factors and Amyotrophic Lateral Sclerosis: An Umbrella Review and Critical Assessment of Current Evidence from Systematic Reviews and Meta-Analyses of Observational Studies. Neuroepidemiology. 2016;46(2):96–105. pmid:26731747
- 71. Belbasis L, Bellou V, Evangelou E, Ioannidis JP, Tzoulaki I. Environmental risk factors and multiple sclerosis: an umbrella review of systematic reviews and meta-analyses. Lancet Neurol. 2015;14(3):263–73. pmid:25662901
- 72. Belbasis L, Dosis V, Evangelou E. Elucidating the environmental risk factors for rheumatic diseases: An umbrella review of meta-analyses. Int J Rheum Dis. 2018;21(8):1514–24. pmid:30146746
- 73. Belbasis L, Köhler CA, Stefanis N, Stubbs B, van Os J, Vieta E, et al. Risk factors and peripheral biomarkers for schizophrenia spectrum disorders: an umbrella review of meta-analyses. Acta Psychiatr Scand. 2018;137(2):88–97. pmid:29288491
- 74. Belbasis L, Mavrogiannis MC, Emfietzoglou M, Evangelou E. Environmental factors, serum biomarkers and risk of atrial fibrillation: an exposure-wide umbrella review of meta-analyses. Eur J Epidemiol. 2020;35(3):223–39. pmid:32180061
- 75. Belbasis L, Savvidou MD, Kanu C, Evangelou E, Tzoulaki I. Birth weight in relation to health and disease in later life: an umbrella review of systematic reviews and meta-analyses. BMC Med. 2016;14(1):147. pmid:27677312
- 76. Belbasis L, Stefanaki I, Stratigos AJ, Evangelou E. Non-genetic risk factors for cutaneous melanoma and keratinocyte skin cancers: An umbrella review of meta-analyses. J Dermatol Sci. 2016;84(3):330–9. pmid:27663092
- 77. Bellou V, Belbasis L, Konstantinidis AK, Evangelou E. Elucidating the risk factors for chronic obstructive pulmonary disease: an umbrella review of meta-analyses. Int J Tuberc Lung Dis. 2019;23(1):58–66. pmid:30674376
- 78. Bellou V, Belbasis L, Tzoulaki I, Evangelou E. Risk factors for type 2 diabetes mellitus: An exposure-wide umbrella review of meta-analyses. PLoS One. 2018;13(3):e0194127. pmid:29558518
- 79. Bellou V, Belbasis L, Tzoulaki I, Evangelou E, Ioannidis JP. Environmental risk factors and Parkinson’s disease: An umbrella review of meta-analyses. Parkinsonism Relat Disord. 2016;23:1–9. pmid:26739246
- 80. Bellou V, Belbasis L, Tzoulaki I, Middleton LT, Ioannidis JPA, Evangelou E. Systematic evaluation of the associations between environmental risk factors and dementia: An umbrella review of systematic reviews and meta-analyses. Alzheimers Dement. 2017;13(4):406–18. pmid:27599208
- 81. Bortolato B, Köhler CA, Evangelou E, León-Caballero J, Solmi M, Stubbs B, et al. Systematic assessment of environmental risk factors for bipolar disorder: an umbrella review of systematic reviews and meta-analyses. Bipolar Disord. 2017;19(2):84–96. pmid:28470927
- 82. Bouras E, Karakioulaki M, Bougioukas KI, Aivaliotis M, Tzimagiorgis G, Chourdakis M. Gene promoter methylation and cancer: An umbrella review. Gene. 2019;710:333–40. pmid:31202904
- 83. Campbell JM, Bateman E, Peters M, Bowen JM, Keefe DM, Stephenson MD. Fluoropyrimidine and platinum toxicity pharmacogenetics: an umbrella review of systematic reviews and meta-analyses. Pharmacogenomics. 2016;17(4):435–51. pmid:26894782
- 84. Campbell JM, Bateman E, Stephenson MD, Bowen JM, Keefe DM, Peters MD. Methotrexate-induced toxicity pharmacogenetics: an umbrella review of systematic reviews and meta-analyses. Cancer Chemother Pharmacol. 2016;78(1):27–39. pmid:27142726
- 85. Campbell JM, Stephenson MD, Bateman E, Peters MD, Keefe DM, Bowen JM. Irinotecan-induced toxicity pharmacogenetics: an umbrella review of systematic reviews and meta-analyses. Pharmacogenomics J. 2017;17(1):21–8. pmid:27503581
- 86. Carvalho AF, Köhler CA, Fernandes BS, Quevedo J, Miskowiak KW, Brunoni AR, et al. Bias in emerging biomarkers for bipolar disorder. Psychol Med. 2016;46(11):2287–97. pmid:27193198
- 87. Cupp M, Cariolou M, Tzoulaki I, Evangelos E, Berlanga-Taylor A. Neutrophil counts and cancer prognosis: An umbrella review of systematic reviews and meta-analyses of observational studies. British Journal of Cancer. 2018;119(1):26–7.
- 88. Fullana MA, Tortella-Feliu M, Fernández de la Cruz L, Chamorro J, Pérez-Vigil A, Ioannidis JPA, et al. Risk and protective factors for anxiety and obsessive-compulsive disorders: an umbrella review of systematic reviews and meta-analyses. Psychol Med. 2019:1–16. pmid:31172897
- 89. Giannakou K, Evangelou E, Papatheodorou SI. Genetic and non-genetic risk factors for pre-eclampsia: umbrella review of systematic reviews and meta-analyses of observational studies. Ultrasound Obstet Gynecol. 2018;51(6):720–30. pmid:29143991
- 90. Giannakou K, Evangelou E, Yiallouros P, Christophi CA, Middleton N, Papatheodorou E, et al. Risk factors for gestational diabetes: An umbrella review of meta-analyses of observational studies. PLoS One. 2019;14(4):e0215372. pmid:31002708
- 91. Grabovac I, Veronese N, Stefanac S, Haider S, Jackson SE, Koyanagi A, et al. Human Immunodeficiency Virus Infection and Diverse Physical Health Outcomes: An Umbrella Review of Meta-analyses of Observational Studies. Clin Infect Dis. 2020;70(9):1809–15. pmid:31401650
- 92. Hossain MM, Khan N, Sultana A, Ma P, McKyer ELJ, Ahmed HU, et al. Prevalence of comorbid psychiatric disorders among people with autism spectrum disorder: An umbrella review of systematic reviews and meta-analyses. Psychiatry Res. 2020;287:112922. pmid:32203749
- 93. Hossain MM, Purohit N, Sultana A, Ma P, McKyer ELJ, Ahmed HU. Prevalence of mental disorders in South Asia: An umbrella review of systematic reviews and meta-analyses. Asian J Psychiatr. 2020;51:102041. pmid:32315966
- 94. Hutchens BF, Kearney J. Risk Factors for Postpartum Depression: An Umbrella Review. J Midwifery Womens Health. 2020;65(1):96–108. pmid:31970924
- 95. Kim JY, Son MJ, Son CY, Radua J, Eisenhut M, Gressier F, et al. Environmental risk factors and biomarkers for autism spectrum disorder: an umbrella review of the evidence. Lancet Psychiatry. 2019;6(7):590–600. pmid:31230684
- 96. Li L, Tan J, Liu L, Li J, Chen G, Chen M, et al. Association between H. pylori infection and health Outcomes: an umbrella review of systematic reviews and meta-analyses. BMJ Open. 2020;10(1):e031951. pmid:31924635
- 97. Lucaroni F, Cicciarella Modica D, Macino M, Palombi L, Abbondanzieri A, Agosti G, et al. Can risk be predicted? An umbrella systematic review of current risk prediction models for cardiovascular diseases, diabetes and hypertension. BMJ Open. 2019;9(12):e030234. pmid:31862737
- 98. Lucaroni F, Morciano L, Rizzo G, F DA, Buonuomo E, Palombi L, et al. Biomarkers for predicting spontaneous preterm birth: an umbrella systematic review. J Matern Fetal Neonatal Med. 2018;31(6):726–34. pmid:28274163
- 99. Machado MO, Veronese N, Sanches M, Stubbs B, Koyanagi A, Thompson T, et al. The association of depression and all-cause and cause-specific mortality: an umbrella review of systematic reviews and meta-analyses. BMC Med. 2018;16(1):112. pmid:30025524
- 100. O’Sullivan JW, Muntinga T, Grigg S, Ioannidis JPA. Prevalence and outcomes of incidental imaging findings: umbrella review. Bmj. 2018;361:k2387. pmid:29914908
- 101. Parreira P, Maher CG, Steffens D, Hancock MJ, Ferreira ML. Risk factors for low back pain and sciatica: an umbrella review. Spine J. 2018;18(9):1715–21. pmid:29792997
- 102. Piovani D, Danese S, Peyrin-Biroulet L, Nikolopoulos GK, Lytras T, Bonovas S. Environmental Risk Factors for Inflammatory Bowel Diseases: An Umbrella Review of Meta-analyses. Gastroenterology. 2019;157(3):647–59.e4. pmid:31014995
- 103. Radua J, Ramella-Cravaro V, Ioannidis JPA, Reichenberg A, Phiphopthatsanee N, Amir T, et al. What causes psychosis? An umbrella review of risk and protective factors. World Psychiatry. 2018;17(1):49–66. pmid:29352556
- 104. Raglan O, Kalliala I, Markozannes G, Cividini S, Gunter MJ, Nautiyal J, et al. Risk factors for endometrial cancer: An umbrella review of the literature. Int J Cancer. 2019;145(7):1719–30. pmid:30387875
- 105. Smith L, Luchini C, Demurtas J, Soysal P, Stubbs B, Hamer M, et al. Telomere length and health outcomes: An umbrella review of systematic reviews and meta-analyses of observational studies. Ageing Res Rev. 2019;51:1–10. pmid:30776454
- 106. Tortella-Feliu M, Fullana MA, Pérez-Vigil A, Torres X, Chamorro J, Littarelli SA, et al. Risk factors for posttraumatic stress disorder: An umbrella review of systematic reviews and meta-analyses. Neurosci Biobehav Rev. 2019;107:154–65. pmid:31520677
- 107. Van Der Burg N, Bakker PR, Van Os J, Van Harten P, Al Hadithy A. The genetics of drug-related movement disorders an umbrella review of meta-analyses. Schizophrenia Bulletin. 2018;44:S295.
- 108. Veronese N, Demurtas J, Soysal P, Smith L, Torbahn G, Schoene D, et al. Sarcopenia and health-related outcomes: an umbrella review of observational studies. European Geriatric Medicine. 2019;10(6):853–62. pmid:34652767
- 109. Yang T, Li X, Montazeri Z, Little J, Farrington SM, Ioannidis JPA, et al. Gene-environment interactions and colorectal cancer risk: An umbrella review of systematic reviews and meta-analyses of observational studies. Int J Cancer. 2019;145(9):2315–29. pmid:30536881
- 110. Zhou C, Zhong X, Song Y, Shi J, Wu Z, Guo Z, et al. Prognostic Biomarkers for Gastric Cancer: An Umbrella Review of the Evidence. Front Oncol. 2019;9:1321. pmid:31850212
- 111. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6. pmid:21208779
- 112. Schlesinger S, Schwingshackl L, Neuenschwander M, Barbaresko J. A critical reflection on the grading of the certainty of evidence in umbrella reviews. Eur J Epidemiol. 2019;34(9):889–90. pmid:31222608
- 113. Ali MJ. Questioning the Impact of the Impact Factor. A Brief Review and Future Directions. Semin Ophthalmol. 2022;37(1):91–6. pmid:33969807
- 114. Rizkallah J, Sin DD. Integrative Approach to Quality Assessment of Medical Journals Using Impact Factor, Eigenfactor, and Article Influence Scores. PLOS ONE. 2010;5(4):e10204. pmid:20419115
- 115. Saginur M, Fergusson D, Zhang T, Yeates K, Ramsay T, Wells G, et al. Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis. Systematic Reviews. 2020;9(1):53. pmid:32164791
- 116. Gianfredi V, Nucci D, Amerio A, Signorelli C, Odone A, Dinu M. What Can We Expect from an Umbrella Review? Advances in Nutrition. 2022;13(2):684–5.
- 117. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4(3):245–53. pmid:17715249
- 118. Salanti G, Ioannidis JP. Synthesis of observational studies should consider credibility ceilings. J Clin Epidemiol. 2009;62(2):115–22. pmid:19131013
- 119. Papatheodorou SI, Tsilidis KK, Evangelou E, Ioannidis JP. Application of credibility ceilings probes the robustness of meta-analyses of biomarkers and cancer risk. J Clin Epidemiol. 2015;68(2):163–74. pmid:25433443