Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Justification of research using systematic reviews continues to be inconsistent in clinical health science—A systematic review and meta-analysis of meta-research studies

  • Jane Andreasen ,

    Roles Conceptualization, Data curation, Methodology, Project administration, Writing – original draft, Writing – review & editing

    jaan@rn.dk

    Affiliation Department of Physiotherapy and Occupational Therapy, Aalborg University Hospital, Denmark and Public Health and Epidemiology Group, Department of Health, Science and Technology, Aalborg University, Aalborg, Denmark

  • Birgitte Nørgaard,

    Roles Conceptualization, Data curation, Methodology, Writing – review & editing

    Affiliation Department of Public Health, University of Southern Denmark Odense, Denmark

  • Eva Draborg,

    Roles Conceptualization, Data curation, Methodology, Writing – review & editing

    Affiliation Department of Public Health, University of Southern Denmark Odense, Denmark

  • Carsten Bogh Juhl,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Writing – review & editing

    Affiliation Department of Sports Science and Clinical Biomechanics, University of Southern Denmark and Department of Physiotherapy and Occupational Therapy, Copenhagen University Hospital, Herlev and Gentofte, Herlev, Denmark

  • Jennifer Yost,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation M. Louise Fitzpatrick College of Nursing, Villanova University, Villanova, PA, United States of America

  • Klara Brunnhuber,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Digital Content Services, Elsevier, London, United Kingdom

  • Karen A. Robinson,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Johns Hopkins University School of Medicine, Baltimore, MD, United States of America

  • Hans Lund

    Roles Conceptualization, Data curation, Methodology, Project administration, Supervision, Writing – review & editing

    Affiliation Department of Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway

Abstract

Background

Redundancy is an unethical, unscientific, and costly challenge in clinical health research. There is a high risk of redundancy when existing evidence is not used to justify the research question when a new study is initiated. Therefore, the aim of this study was to synthesize meta-research studies evaluating if and how authors of clinical health research studies use systematic reviews when initiating a new study.

Methods

Seven electronic bibliographic databases were searched (final search June 2021). Meta-research studies assessing the use of systematic reviews when justifying new clinical health studies were included. Screening and data extraction were performed by two reviewers independently. The primary outcome was defined as the percentage of original studies within the included meta-research studies using systematic reviews of previous studies to justify a new study. Results were synthesized narratively and quantitatively using a random-effects meta-analysis. The protocol has been registered in Open Science Framework (https://osf.io/nw7ch/).

Results

Twenty-one meta-research studies were included, representing 3,621 original studies or protocols. Nineteen of the 21 studies were included in the meta-analysis. The included studies represented different disciplines and exhibited wide variability both in how the use of previous systematic reviews was assessed, and in how this was reported. The use of systematic reviews to justify new studies varied from 16% to 87%. The mean percentage of original studies using systematic reviews to justify their study was 42% (95% CI: 36% to 48%).

Conclusion

Justification of new studies in clinical health research using systematic reviews is highly variable, and fewer than half of new clinical studies in health science were justified using a systematic review. Research redundancy is a challenge for clinical health researchers, as well as for funders, ethics committees, and journals.

Introduction

Research redundancy in clinical health research is an unethical, unscientific, and costly challenge that can be minimized by using an evidence-based research approach. First introduced in 2009 and since endorsed and promoted by organizations and researchers worldwide [16], evidence-based research is an approach whereby researchers systematically and transparently take into account the existing evidence on a topic before embarking on a new study. The researcher thus strives to enter the project unbiased, or at least aware of the risk of knowledge redundancy bias. The key is an evidence synthesis using formal, explicit, and rigorous methods to bring together the findings of pre-existing research to synthesize the totality what is known [7]. Evidence syntheses provide the basis for an unbiased justification of the proposed research study to ensure that the enrolling of participants, resource allocation, and healthcare systems are supporting only relevant and justified research. Enormous numbers of research studies are conducted, funded, and published globally every year [8]. Thus, if earlier relevant research is not considered in a systematic and transparent way when justifying research, the foundation for a research question is not properly established, thereby increasing the risk of redundant studies being conducted, funded, and published resulting in a waste of resources, such as time and funding [1, 4]. Most importantly, when redundant research is initiated, participants unethically and unnecessarily receive placebos or receive suboptimal treatment.

Previous meta-research, defined as the study of research itself including the methods, reporting, reproducibility, evaluation and incentives of the research [9] have shown that there is considerable variation and bias in the use of evidence syntheses to justify research studies [1012]. To the best of our knowledge, a systematic review of previous meta-research studies assessing the use of systematic reviews to justify studies in clinical health research has not previously been conducted. Evaluating how evidence-based research is implemented in research practices across disciplines and specialties when justifying new studies will provide an indication of the integration of evidence-based research in research practices [9]. The present systematic review aimed to identify and synthesize results from meta-research studies, regardless of study type, evaluating if and how authors of clinical health research studies use systematic reviews to justify a new study.

Methods

Prior to commencing the review, we registered the protocol in the Open Science Framework (https://osf.io/nw7ch/). The protocol remained unchanged, but in this paper we have made adjustments to the risk-of-bias assessment, reducing the tool to 10 items and removing the assessment of reporting quality. The review is presented in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) guidelines [13].

Eligibility criteria

Studies were eligible for inclusion if they were original meta-research studies, regardless of study type, that evaluated if and how authors of clinical health research studies used systematic reviews to justify new clinical health studies. No limitations on language, publication status, or publication year were applied. Only meta-research studies of studies on human subjects in clinical health sciences were eligible for inclusion. The primary outcome was defined as the percentage of original studies within the included meta-research studies using systematic reviews of previous studies to justify a new study. The secondary outcome was how the systematic reviews of previous research were used (e.g., within the text to justify the study) by the original studies.

Information sources and search strategy

This study is one of six ongoing evidence syntheses (four systematic reviews and two scoping reviews) planned to assess the global state of evidence-based research in clinical health research. These are; a scoping review mapping the area broadly to describe current practice and identify knowledge gaps, a systematic review on the use of prior research in reports of randomized controlled trials specifically, three systematic reviews assessing the use of systematic reviews when justifying, designing [14] or putting results of a new study in context, and finally a scoping review uncovering the breadth and characteristics of the available, empirical evidence on the topic of citation bias. Further, the research group is working with colleagues on a Handbook for Evidence-based Research in health sciences. Due to the common aim across the six evidence syntheses, a broad overall search strategy was designed to identify meta-research studies that assessed whether researchers used earlier similar studies and/or systematic reviews of earlier similar studies to inform the justification and/or design of a new study, whether researchers used systematic reviews to inform the interpretation of new results, and meta-research studies that assessed if there were published redundant studies within a specific area or not.

The first search was performed in June 2015. Databases included MEDLINE via both PubMed and Ovid, EMBASE via Ovid, CINAHL via EBSCO, Web of Science (Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Arts & Humanities Citation Index (A&HCI), and the Cochrane Methodology Register (CMR, Methods Studies) from inception (Appendix 1 in S1 File). In addition, reference lists of included studies were screened for relevant articles, as well as the authors’ relevant publications and abstracts from the Cochrane Methodology Reviews.

Based upon the experiences from the results of the baseline search in June 2015, an updated and revised search strategy was conducted in MEDLINE and Embase via Ovid from January 2015 to June 2021 (Appendix 1 in S1 File). Once again, the reference lists of new included studies were screened for relevant references, as were abstracts from January 2015 to June 2021 in the Cochrane Methodology Reviews. Experts in the field were contacted to identify any additional published and/or grey literature. No restrictions were made on publication year and language. See Appendix 1 and Appendix 2 in S1 File for the full search strategy.

Screening and study selection

Following deduplication, the search results were uploaded to Rayyan (https://rayyan.qcri.org/welcome). The search results from the 1st search (June 2015) were independently screened by a pair of reviewers. Twenty screeners were paired, with each pair including an author very experienced in systematic reviews and a less experienced author. To increase consistency among reviewers, both reviewers initially screened the same 50 publications and discussed the results before beginning screening for this review. Disagreements on study selection were resolved by consensus and discussion with a third reviewer, if needed. The full-text screening was also performed by two reviewers independently. Disagreements on study selection were resolved by consensus and discussion. There were also two independent reviewers who screened following the last search, using the same procedure, as for the first search, for full-text screening and disagreements. The screening procedures resulted in a full list of studies potentially relevant for one or more of the six above-mentioned evidence syntheses.

A second title and abstract screening and full-text screening of the full list was then performed independently by two reviewers using screening criteria specific to this systematic review. Reasons for excluding trials were recorded, and disagreements between the reviewers were resolved through discussion. If consensus was not reached, a third reviewer was involved.

Data extraction

We developed and pilot tested a data extraction form to extract data regarding study characteristics and outcomes of interest. Two reviewers independently extracted data, with other reviewers available to resolve disagreements. The following study characteristics were extracted from each of the included studies: bibliographic information, study aim, study design, setting, country, inclusion period, area of interest, results, and conclusion. Further, data for this study’s primary and secondary outcomes were extracted; these included the percentage of original studies using systematic reviews to justify their study and how the systematic reviews of previous research were used (e.g., within the text to justify the study) by the original studies.

Risk-of-bias assessment

No standard tool was identified to assess the risk of bias in empirical meta-research studies. The Editorial Group of the Evidence-Based Research Network prepared a risk-of-bias tool for the planned five systematic reviews with list of items important for evaluating the risk of bias in meta-research studies. For each item, one could classify the study under examination as exhibiting a “low risk of bias”, “unclear risk of bias” or “high risk of bias”. We independently tested the list of items upon a sample of included studies. Following a discussion of the different answers, we adjusted the number and content of the list of items to ten and defined the criteria to evaluate the risk of bias in the included studies (Table 1). Each of the included meta-research studies was appraised independently by two reviewers using the customized checklist to determine the risk of bias. Disagreements regarding the risk of bias were solved through discussion. No study was excluded on the grounds of low quality.

Data synthesis and interpretation

In addition, to narratively summarizing the characteristics of the included meta-research studies and their risk-of-bias assessments, the percentage of original studies using systematic review of previous similar studies to justify a new study (primary outcome) was calculated as the number of studies using at least one systematic review, divided by the total number of original studies within each of the included meta-research studies. A meta-analysis using the random-effects model (DerSimonian and Laird) was used to estimate the overall estimate and perform the forest plot as this model is the default when using the metaprop command. Heterogeneity was evaluated estimating the I2 statistics (the percentage of variance attributable to heterogeneity i.e., inconsistency) and the between study variance tau2. When investigating reasons for heterogeneity, a restricted maximum likelihood (REML) model was used and covariates with the ability to reduce tau2 was deemed relevant. [15].

All analyses were conducted in Stata, version 17.0 (StataCorp. 2019. Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC).

Results

Study selection

In total, 30,592 publications were identified through the searches. Of these, 69 publications were determined eligible for one of the six evidence syntheses. A total of 21 meta-research studies fulfilled the inclusion criteria for this systematic review [10, 11, 1634]; see Fig 1.

Study characteristics

The 21 included meta-research studies were published from 2007 to 2021, representing 3,621 original studies or protocols and one survey with 106 participants; only three of these studies were published before 2013 [10, 18, 26]. The sample of the original study within each of the included meta-research studies varied. One meta-research study surveyed congress delegates [29], one study examined first-submission protocols for randomized controlled trials submitted to four hospital ethics committees [17], and 14 studies examined randomized or quasi-randomized primary studies published during a specific time period in a range of journals [10, 11, 18, 2128, 31, 32, 34] or in specific databases [16, 19, 20, 30]. Finally, one study examined the use of previously published systematic reviews when publishing a new systematic review [33]. Further, the number of original studies within each included meta-research study varied considerably, ranging from 18 [10] to 637 original studies [27]. The characteristics of the included meta-research studies are presented in Table 2.

thumbnail
Table 2. Characteristics of the included meta–research studies (N = 21).

https://doi.org/10.1371/journal.pone.0276955.t002

Risk of bias assessment

Overall, most studies were determined to exhibit a low risk of bias in the majority of items, and all of the included meta-research studies reported an unambiguous aim and a match between aim and methods. However, only a few studies provided argumentation for their choice of data source [17, 20, 24, 30], and only two of the 21 studies referred to an available a-priori protocol [16, 21]. Finally, seven studies provided poor or no discussion of the limitations of their study [10, 19, 22, 2628, 34]. The risk-of-bias assessments are shown in Table 3.

thumbnail
Table 3. Risk of bias of the included meta–research studies N = 21.

https://doi.org/10.1371/journal.pone.0276955.t003

Synthesis of results

Of the included 21 studies, a total of 18 studies were included in the meta-analysis. Two studies included two cohorts each, and both cohorts in each of these studies were included in our meta-analysis [21, 30]. The survey by Clayton and colleagues, with a response rate of 17%, was not included in the meta-analysis as the survey did not provide data to identify the use of systematic reviews to justify specific studies. However, their results showed that 42 of 84 respondents (50%) reported using a systematic review for justification [29]. The study by Chow, which was also not included in the meta-analysis, showed that justification varied largely within and between specialties. However, only relative numbers were provided, and, therefore, no overall percentage could be extracted [11]. The study by Seehra et al. counted the SR citations in RCTs and not the number of RCTs citing SRs and is therefore not included in the meta-analysis either [23].

The percentage of original studies that justified a new study with a systematic review within each meta-research study ranged from 16% to 87%. The pooled percentage of original studies using systematic reviews to justify their research question was 42% (95% CI: 36% to 48%) as shown in Fig 2. Where the confidence interval showed the precision of the pooled estimate in a meta-analysis, the prediction interval showed the distribution of the individual studies. The heterogeneity in the meta-analysis assessed by I2 was 94%. The clinical interpretation of this large heterogeneity is seen in a the very broad prediction interval ranging from 16 to 71%, meaning that based on these studies there is 95% chance that the results of the next study will show a prevalence between 16 to 71%.

thumbnail
Fig 2. Forest plot.

Forest plot prevalence and 95% confidence intervals for the percentage of studies using an SR to justify the study.

https://doi.org/10.1371/journal.pone.0276955.g002

Further, we conducted an explorative subgroup analysis of the study of Helfer et al. and the study of Joseph et al. as these two studies were on meta-analyses and protocols and therefore differ from the other included studies. This analysis did only marginally change the pooled percentage to 39% (95% CI; 33% to 46%) and the between-study variance (tau2) was reduced with 23%.

The 21 included studies varied greatly in their approach and in their description of how systematic reviews were used, i.e., if the original studies referred and whether the used systematic reviews in the original studies were relevant and/or of high-quality. Nine studies assessed, to varying degrees, whether the used systematic reviews were relevant for the justification of the research [1620, 25, 30, 32, 34]. Overall, the information reported by the meta-research studies was not sufficient to report the percentage of primary studies referring to relevant systematic reviews. No details were provided regarding the methodological quality of the systematic reviews used to justify the research question or if they were recently published reviews, except for Hoderlein et al., who reported that the mean number of years from publication of the cited systematic review and the trial report was four years [30].

Discussion

We identified 21 meta-research studies, spanning 15 publication years and 12 medical disciplines. The findings showed substantial variability in the use of systematic reviews when justifying new clinical studies, with the incidence of use ranging from 16% to 87%. However, fewer than half of the 19 meta-analysis-eligible studies used a systematic review to justify their new study. There was wide variability, and a general lack of information, about how systematic reviews were used within many of the original studies. Our systematic review found that the proportion of original studies justifying their new research using evidence syntheses is sub-optimal and, thus, the potential for research redundancy continues to be a challenge. This study corroborates the serious possible consequences regarding research redundancy previously problematized by Chalmers et al. and Glasziou et al. [35, 36].

Systematic reviews are considered crucial when justifying a new study, as is emphasized in reporting guidelines such as the CONSORT statement [37]. However, there are challenges involved in implementing an evidence-based research approach. The authors of the included meta-research study reporting the highest use of systematic reviews to justify a new systematic review study point out that even though the authors of the original studies refer to some of the published systematic reviews, they neglect others on the same topic, which may be problematic and result in a biased approach [33]. Other issues that have been identified are the risk of research waste when a systematic review may not be methodologically sound [12, 38] and that there is also redundancy in the conduct of systematic reviews, with many overlapping systematic reviews existing on the same topic [3941]. In the original studies within the meta-research studies, the use of systematic reviews was not consistent and, further, it was not explicated whether the systematic reviews used were the most recent and/or of high methodological quality. These issues speak to the need for refinement in the area of systematic review development, such as mandatory registration in prospective registries. Only two out of the included 21 studies in this study referred to an available a-priori protocol [16, 21]. General recommendations in the use of systematic reviews as justification for a new study are difficult as these will be topic specific, however researchers should be aware to use the most robust and methodologically sound of recently published reviews, preferably with á priori published protocols.

Efforts must continue in promoting the use of evidence-based research approaches among clinical health researchers and other important stakeholders, such as funders. Collaborations such as the Ensuring Value in Research Funders Forum, and changes in funding review criteria mandating reference to previously published systematic reviews when justifying the research question within funding proposals, are examples of how stakeholders can promote research that is evidence-based [8, 41].

Strengths and limitations

We conducted a comprehensive and systematic search. The lack of standard terminology for meta-research studies resulted in search strategies that retrieved thousands of citations. We also relied on snowballing efforts to identify relevant studies, such as by contacting experts and scanning the reference lists of relevant studies.

There is also a lack of tools to assess risk of bias for meta-research studies, so a specific risk-of bias tool for the five conducted reviews was created. The tool was discussed and revised continuously throughout the research process; however, we acknowledge that the checklist is not yet optimal and a validated risk-of-bias tool for meta-research studies is needed.

Many of the included meta-research studies did not provide details as to whether the systematic reviews used to justify the included studies were relevant, high-quality and/or recently published. This may raise questions as to the validity of our findings, as the majority of the meta-research studies only provide an indication of the citation of systematic reviews to justify new studies, not whether the systematic review cited was relevant, recent and of high-quality, or even how the systematic review was used. We did not assess this further either. Nonetheless, even if we assumed that these elements were provided for every original study included in the included meta-research studies (i.e. taking a conservative approach), fewer than half used systematic reviews to justify their research questions. The conservative approach used in this study therefore does not underestimate, and perhaps rather overestimates, the actual use of relevant systematic reviews to justify studies in clinical health science across disciplines.

Different study designs were included in the meta-analysis, which may have contributed to the high degree of heterogeneity observed. Therefore, the presented results should be interpreted with caution due to the high heterogeneity. Not only were there differences in the methods of the included meta-research studies, but there was also heterogeneity in the medical specialties evaluated [42, 43].

Conclusion

In conclusion, justification of research questions in clinical health research with systematic reviews continues to be inconsistent; fewer than half of the primary studies within the included meta-research studies in this systematic review were found to have used a systematic review to justify their research question. This indicates that the risk of redundant research is still high when new studies across disciplines and professions in clinical health are initiated, thereby indicating that evidence-based research has not yet been successfully implemented in the clinical health sciences. Efforts to raise awareness and to ensure an evidence-based research approach continue to be necessary, and such efforts should involve clinical health researchers themselves as well as important stakeholders such as funders.

Acknowledgments

This work has been prepared as part of the Evidence-Based Research Network (ebrnetwork.org). The Evidence-Based Research Network is an international network that promotes the use of systematic reviews when justifying, designing, and interpreting research. The authors thank the Section for Evidence-Based Practice, Department for Health and Function, Western Norway University of Applied Sciences for their generous support of the EBRNetwork. Further, thanks to COST Association for supporting the COST Action “EVBRES” (CA 17117, evbres.eu) and thereby the preparation of this study. Thanks to Gunhild Austrheim, Head of Unit, Library at Western Norway University of Applied Sciences, Norway, for helping with the second search. Thanks to those helping with the screening: Durita Gunnarsson, Gorm Høj Jensen, Line Sjodsholm, Signe Versterre, Linda Baumbach, Karina Johansen, Rune Martens Andersen, and Thomas Aagaard.

We gratefully acknowledge the contribution from the EVBRES (COST ACTION CA 17117) Core Group, including Anne Gjerland (AG) and her specific contribution to the search and screening process.

References

  1. 1. Lund H, Brunnhuber K, Juhl C, et al. Towards evidence based research. BMJ 2016; 355: 1–5. pmid:27797786
  2. 2. Moher D, Glasziou P, Chalmers I, et al. Increasing value and reducing waste in biomedical research: Who’s listening? Lancet 2016; 387: 1573–1586. pmid:26423180
  3. 3. Association WM. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. Jama 2013; 310: 2191–2194. pmid:24141714
  4. 4. Robinson KA, Brunnhuber K, Ciliska D, et al. Evidence-Based Research Series-Paper 1: What Evidence-Based Research is and why is it important? J Clin Epidemiol 2021; 129: 151–157. pmid:32979491
  5. 5. Lund H, Juhl CB, Nørgaard B, et al. Evidence-Based Research Series-Paper 2: Using an Evidence-Based Research approach before a new study is conducted to ensure value. J Clin Epidemiol 2021; 129: 158–166. pmid:32987159
  6. 6. Lund H, Juhl CB, Nørgaard B, et al. Evidence-Based Research Series-Paper 3: Using an Evidence-Based Research approach to place your results into context after the study is performed to ensure usefulness of the conclusion. J Clin Epidemiol 2021; 129: 167–171. pmid:32979490
  7. 7. Evidence Synthesis International, https://evidencesynthesis.org/.
  8. 8. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Med; 7. Epub ahead of print 2010. pmid:20877712
  9. 9. Ioannidis JPA. Meta-research: Why research on research matters. PLoS Biol 2018; 16: 1–6. pmid:29534060
  10. 10. Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: A status report. J R Soc Med 2007; 100: 187–190. pmid:17404342
  11. 11. Chow JTY, Lam K, Naeem A, et al. The pathway to RCTs: How many roads are there? Examining the homogeneity of RCT justification. Trials 2017; 18: 1–7.
  12. 12. Créquit P, Trinquart L, Yavchitz A, et al. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: The example of lung cancer. BMC Med; 14. Epub ahead of print 2016. pmid:26792360
  13. 13. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ; 372. Epub ahead of print 2021. pmid:33782057
  14. 14. Nørgaard B, Draborg E, Andreasen J, et al. Systematic reviews are rarely used to inform study design—a systematic review and meta-analysis. J Clin Epidemiol 2022; 145: 1–13. pmid:35045317
  15. 15. Cochrane. No Title. Cochrane Handbook, https://training.cochrane.org/handbook/current/chapter-i.
  16. 16. Ker K, Roberts I. Exploring redundant research into the effect of tranexamic acid on surgical bleeding: Further analysis of a systematic review of randomised controlled trials. BMJ Open 2015; 5: 1–7. pmid:26303335
  17. 17. Joseph PD, Caldwell PHY, Barnes EH, et al. Completeness of protocols for clinical trials in children submitted to ethics committees. J Paediatr Child Health 2019; 55: 291–298. pmid:30161286
  18. 18. Goudie AC, Sutton AJ, Jones DR, et al. Empirical assessment suggests that existing evidence could be used more fully in designing randomized controlled trials. J Clin Epidemiol 2010; 63: 983–991. pmid:20573483
  19. 19. Bolland MJ, Grey A, Avenell A. Assessment of research waste part 2: Wrong study populations- an exemplar of baseline vitamin D status of participants in trials of vitamin D supplementation. BMC Med Res Methodol 2018; 18: 1–10.
  20. 20. Chapman SJ, Aldaffaa M, Downey CL, et al. Research waste in surgical randomized controlled trials. Br J Surg 2019; 106: 1464–1471. pmid:31393612
  21. 21. Johnson AL, Walters C, Gray H, et al. The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis. Injury 2020; 51: 212–217. pmid:31711650
  22. 22. Rauh S, Nigro T, Sims M, et al. The use of systematic reviews to justify randomized controlled trials in obstetrics & gynecology publications. Eur J Obstet Gynecol Reprod Biol 2020; 252: 627–628.
  23. 23. Seehra J, Liu C, Pandis N. Citation of prior systematic reviews in reports of randomized controlled trials published in dental speciality journals. J Dent 2021; 109: 103658. pmid:33836249
  24. 24. Shepard S, Wise A, Johnson BS, et al. Are randomized controlled trials in urology being conducted with justification? J Osteopath Med. Epub ahead of print 2021. pmid:34019751
  25. 25. Torgerson T, Evans S, Johnson BS, et al. The use of systematic reviews to justify phase III ophthalmology trials: an analysis. Eye 2020; 34: 2041–2047. pmid:31965083
  26. 26. Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet 2010; 376: 20–21. pmid:20609983
  27. 27. Walters C, Torgerson T, Fladie I, et al. Are randomized controlled trials being conducted with the right justification? J Evid Based Med 2020; 13: 181–182. pmid:32615030
  28. 28. Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc 2013; 24: 145–148.
  29. 29. Clayton GL, Smith IL, Higgins JPT, et al. The INVEST project: Investigating the use of evidence synthesis in the design and analysis of clinical trials. Trials 2017; 18: 1–11.
  30. 30. Hoderlein X, Moseley AM, Elkins MR. Citation of prior research has increased in introduction and discussion sections with time: A survey of clinical trials in physiotherapy. Clin Trials 2017; 14: 372–380. pmid:28747106
  31. 31. Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: A meta-epidemiological study. Eur J Pain (United Kingdom) 2018; 22: 1844–1849. pmid:29978522
  32. 32. De Meulemeester J, Fedyk M, Jurkovic L, et al. Many randomized clinical trials may not be justified: a cross-sectional analysis of the ethics and science of randomized clinical trials. J Clin Epidemiol 2018; 97: 20–25. pmid:29306063
  33. 33. Helfer B, Prosser A, Samara MT, et al. Recent meta-analyses neglect previous systematic reviews and meta-analyses about the same topic: A systematic examination. BMC Med 2015; 13: 1–8.
  34. 34. Rosenthal R, Bucher HC, Dwan K. The use of systematic reviews when designing and reporting surgical trials. Ann Surg 2017; 265: e35–e36. pmid:28266978
  35. 35. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86–89. pmid:19525005
  36. 36. Https://blogs.bmj.com/bmj/2016/01/14/paul-glasziou-and-iain-chalmers-is-85-of-health-research-really-wasted/). No Title.
  37. 37. Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 2010; 63: 834–840. pmid:20346629
  38. 38. Ioannidis JPA. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q 2016; 94: 485–514. pmid:27620683
  39. 39. Siontis KC, Hernandez-Boussard T, Ioannidis JPA. Overlapping meta-analyses on the same topic: Survey of published studies. BMJ 2013; 347: 1–11. pmid:23873947
  40. 40. Sigurdson MK, Khoury MJ, Ioannidis JPA. Redundant meta-analyses are common in genetic epidemiology. J Clin Epidemiol 2020; 127: 40–48. pmid:32540390
  41. 41. Riaz I Bin Khan MS, Riaz H, et al. Disorganized Systematic Reviews and Meta-analyses: Time to Systematize the Conduct and Publication of These Study Overviews? Am J Med 2016; 129: 339.e11–339.e18.
  42. 42. Joyce J, Rabe-Hesketh S, Wessely S. Reviewing the reviews. The example of chronic fatigue syndrome. J Am Med Assoc 1998; 280: 264–266.
  43. 43. Misemer BS, Platts-Mills TF, Jones CW. Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: A cross-sectional analysis. Trials 2016; 17: 1–7.