Figures
Abstract
Background
Previous research looking at published systematic reviews has shown that their search strategies are often suboptimal and that librarian involvement, though recommended, is low. Confidence in the results, however, is limited due to poor reporting of search strategies the published articles.
Objectives
To more accurately measure the use of recommended search methods in systematic reviews, the levels of librarian involvement, and whether librarian involvement predicts the use of recommended methods.
Methods
A survey was sent to all authors of English-language systematic reviews indexed in the Database of Abstracts of Reviews of Effects (DARE) from January 2012 through January 2014. The survey asked about their use of search methods recommended by the Institute of Medicine, Cochrane Collaboration, and the Agency for Healthcare Research and Quality and if and how a librarian was involved in the systematic review. Rates of use of recommended methods and librarian involvement were summarized. The impact of librarian involvement on use of recommended methods was examined using a multivariate logistic regression.
Results
1560 authors completed the survey. Use of recommended search methods ranged widely from 98% for use of keywords to 9% for registration in PROSPERO and were generally higher than in previous studies. 51% of studies involved a librarian, but only 64% acknowledge their assistance. Librarian involvement was significantly associated with the use of 65% of recommended search methods after controlling for other potential predictors. Odds ratios ranged from 1.36 (95% CI 1.06 to 1.75) for including multiple languages to 3.07 (95% CI 2.06 to 4.58) for using controlled vocabulary.
Citation: Koffel JB (2015) Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors. PLoS ONE 10(5): e0125931. https://doi.org/10.1371/journal.pone.0125931
Academic Editor: Brett Thombs, McGill University, CANADA
Received: January 25, 2015; Accepted: March 25, 2015; Published: May 4, 2015
Copyright: © 2015 Jonathan B. Koffel. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The author has no support or funding to report.
Competing interests: The author has declared that no competing interests exist.
Introduction
Systematic reviews are considered to provide the best evidence for health care decisions by gathering, appraising and synthesizing all available literature around a question in a systematic and reproducible manner [1, 2]. The foundation of any systematic review is a comprehensive and systematic search that identifies all eligible studies on the topic and utilizes specialized search tools and methods [3]. Although several prominent organizations have created guides on how to conduct and report effective searches [2, 4–7], many published systematic reviews have been found to contain errors in the design and conduct of the searches that could affect their quality [8–10]. In addition, many reviews have incomplete reporting of the search strategy in the published article [9, 11–15]. Due to the poor search reporting, it is very difficult to know if searches are being conducted poorly or just reported poorly.
One key recommendation for how to improve search quality has been to include a librarian or search specialist (hereafter librarian) when designing and conducting the literature search [2, 4, 5]. Librarians have regularly advocated for their increasing involvement in systematic reviews [10, 16–19] and accumulating evidence supports this advocacy by showing that search quality and reproducibility may be improved when a librarian is involved [9, 20]. Despite these promising findings, existing studies have been limited to examining data available in the published articles and thus may be biased by the poor reporting of search strategies in published articles. In addition, many articles do not provide enough information to determine who conducted the search and their credentials [9, 15, 21], so it is unclear what the true rate of librarian involvement may be. A more accurate measure of librarian involvement is crucial for continued progress in determining librarian impact.
This study will make important contributions to the literature on search strategies and librarian involvement in systematic reviews in two ways. First, it will directly survey recent systematic review authors to determine their use of recommended search strategies and compare these to rates in previous studies. This will allow a larger and more generalizable sample than in previous studies and circumvents the issue of poor reporting, allowing us to determine if searches are being conducted poorly or just reported poorly. Second, it will survey authors on the involvement of a librarian in the review and their role, providing a significantly more accurate and complete estimate than in previous studies. The impact of librarian involvement on the use of recommended search strategies and methods will be examined with the hypothesis that their involvement will be associated with higher use of these strategies.
Methods
Citation Identification
Systematic reviews published in January, 2012 through January, 2014 were identified in the Database of Abstracts of Reviews of Effects (DARE), which conducts highly-sensitive weekly searches of a wide range of sources to identify new systematic reviews [22]. A two-year publication window was chosen to balance the need for a representative and robust sample with authors’ ability to recall the specifics of their search strategy. Systematic reviews from the Cochrane Database of Systematic Reviews were excluded since they follow specific guidelines, represent a subset of all published systematic reviews and are not representative of those in general or specialty journals [12, 13, 23]. DARE was selected since existing hedges and filters for Embase/MEDLINE have sub-optimal precision [24] and thus are inefficient for identifying large numbers of systematic reviews.
The citations from DARE were enriched with additional citation and abstract information from the Scopus/Embase [25] and Web of Science [26] databases by matching digital object identifiers (DOIs) and PubMed ID (PMID) numbers. The ISI Impact Factor for each journal was extracted from the 2013 Journal Citation Reports [27] and converted to its relative quartile rank within its subject category. Since impact factor ranges can vary between disciplines [28], this normalization allowed journals’ impact factors to be more directly compared.
The retrieved citation data was cleaned and elements such the corresponding author’s country and email address were extracted from the address field and placed in unique fields. Only English language articles where an email address for the corresponding author could be identified were included in the study.
Survey
An online survey was designed to gather information on the authors’ background with systematic reviews and searching, search methods in a specific review, and experiences collaborating with librarians (S1 Text). Items on the conduct of the search were gathered from existing lists of best practices for systematic review searches and reporting [2, 4, 5, 12, 29]. Items on the systematic review process and author background were modified and expanded from those used in a previous study [30]. Additional items were developed based on author knowledge and review of previous trends and findings. The survey was pilot-tested and reviewed by local systematic review authors, systematic review experts and search experts for content and validity.
In order to ensure that each corresponding author was only contacted once and was only associated with a single article, a copy of the list of candidate articles was generated and duplicate email addresses were removed. When appropriate, the most recent article from each author was chosen. Each author was sent an e-mail invitation to participate in the study and a unique survey link. The invitation included a description of the current study, the title of the specific article associated with the author in the data set and instructions to answer the questions about the search strategy used in that specific article. The article title was stated again in the survey introduction. Due to the potentially sensitive nature of the survey, all responses were anonymous.
Analyses
In order to determine the proportion of systematic reviews using recommended search methods, the frequency of each method’s use was computed. If a respondent reported that they were “unclear” if a search method was used, it was coded as “no” in order to provide a more conservative estimate and a sensitivity analysis was conducted.
In order to determine the impact of librarian involvement on use of recommended search methods, the frequency of librarian involvement and nature of that involvement was calculated. In addition, information was gathered from authors on variables beyond librarian involvement that have a demonstrated or plausible effect on systematic review quality, and thus potentially on the use of recommended search methods. These include specialized knowledge in or training on systematic review methods [31, 32], publishing in a higher impact journal [32, 33], previous experience writing systematic reviews [34], use of a reporting guidelines [35, 36], years of professional practice, time taken to write the review, year of publication and self-reported confidence in systematic review methodology and the review topic area.
A multivariate logistic regression was conducted to determine the impact of librarian involvement on the use of each recommended search method, controlling for variables reviewed above. These variables were entered in a single step as a covariate block, with librarian involvement entered as the second step. Odds ratios, confidence intervals and significance values for the association between librarian involvement and search method use were calculated.
Differences in demographic variables for included and excluded articles were examined using a χ2test.
All analyses were conducted using SPSS (version 22) and a significance level of α =. 05 was used.
Missing data was not imputed due to the nature of the data and responses were included if the respondent had answered at least two questions. Frequencies were chosen over raw counts due to varying response rates to each question and for ease of interpretation.
Results
Article and Respondent Characteristics
A search was conducted in the DARE database on February 2nd, 2014 and citations for 9,926 articles were retrieved (Fig 1). Expanded citation information could be extracted from the Scopus (which includes Embase) and Web of Science databases for 8,873 of these and a list of 6,726 unique author emails was generated. The survey was successfully sent to 6,322 authors and fully or partially completed by 1,560 of these (25% response rate).
Characteristics of the full list of articles (n = 8873) and those articles where the author opened the survey (n = 1638) are described in Table 1. Due to the anonymity of the responses, for this analysis it was only possible to identify articles where the author had opened the survey, not those where the author had fully or partially completed it. This resulted in a small number (78 articles, 5%) being incorrectly counted as included.
Included articles differed significantly from those not included in terms of year (higher prevalence of 2013 articles), journal quartile (higher prevalence of first quartile articles), number of authors and the corresponding author’s country. While the top five countries of the corresponding authors are identical, a larger proportion of respondents came from English-speaking countries and fewer from China. The strong majority of articles had the words “systematic review” or “meta(-)analysis” in the title or abstract of the article (92%) and most (87%) were indexed as systematic reviews or meta-analyses by the Embase literature database.
Characteristics of the respondents’ background and on pre-defined covariates that will be included in the regression analysis are presented in Table 2. Authors tended towards being earlier in their careers with a median of 8 years of professional experience and 75% having 15 or fewer years of experience. While authors had completed a median of 3 previous reviews, there was wide variation (1–300) with 17% reporting having written 10 or more.
Half (52%) of authors reported having received formal training in the conduct of systematic reviews. In the majority of cases, this training was through a university course rather than a formal workshop from the Cochrane Collaboration or other outside group.
Half (51%) of authors reported that a librarian was involved in the review in some way. Librarian contributions were typically recognized through authorship (26%) or in the acknowledgments (33%). In 36% of cases, including 25% of those where the librarian created or executed the search, their contributions were not recognized in the publication.
Use of Recommended Search Methods
Table 3 reports the percentage of authors who reported the use of each of the recommended search strategies in this study and comparable rates from previous studies.
Use of recommended traditional search methods (items 1–7) was generally high and the majority ranged from 88%-98%. Lower rates were reported for searching in multiple languages (64%) and adjusting the strategy for each literature database (70%). In all cases, rates were higher than in prior studies that examined the published literature only. In some cases, such as use of controlled vocabulary, rates in the current study were markedly higher.
Use of extended search methods (items 8–13) were generally much lower than for traditional methods and the majority ranged from 40%-57%. Nearly all authors (97%), however, reported reviewing the references of included articles. As with traditional search methods, rates were often considerably higher than in comparable studies. Handsearching and use of grey literature were higher in some prior studies, though this may be due to the specific disciplines or topics studied in these articles.
Use of recommended methods for search management and reporting (items 14–17) were mixed. While 60%-80% of authors reported using most of the methods, only 9% reported registering their study in the PROSPERO systematic review register. As with previous methods, the rates in the current study were considerably higher than in previously published studies.
Librarian Impact on Use of Recommended Search Methods
The association between librarian involvement and the use of each search method was investigated using a logistic regression and reported in Table 3, controlling for variables that have a demonstrated or plausible effect on systematic review quality.
Librarian involvement was found to be significantly associated with 12 out of 17 search items (71%) in a univariate (unadjusted) regression and 11 out of 17 items (65%) in a multivariate (adjusted) regression controlling for covariates. Adjusted odds ratios ranged from 1.36 to 3.07. Associations were strongest and most prevalent with traditional searching items (items 1–7), though there were also associations with items related to extended search methods (items 8–13) and the management and reporting of the review (items 14–17).
A sensitivity analysis was performed to determine the effect of coding responses of “unclear” for search elements as “yes” rather than “no” (S2 Text). All existing associations remained significant and, in the majority of cases, became larger. In addition, in the sensitivity analysis librarian involvement was found to significantly predict use of a citation index.
Impact of Guideline Use and Systematic Review Training and Knowledge
As a post-hoc analysis, the association between each of the co-variates in the regression analysis (listed below Table 3) and the use of recommended strategies was investigated. For most of the variables examined, the associations were weak or non-significant. Three variables, however, did show a pattern of significant associations with the use of recommended search methods (S3 Text). Use of a reporting guideline was associated with use of 10 out of 17 methods (59%) with adjusted odds ratios ranging from 2.01–6.69. Formal training in systematic review methodology was associated with use of 6 out of 17 methods (35%) with adjusted odds ratios ranging from 1.32–1.96. Finally, an increase (one point on a five point scale) in self-reported knowledge of systematic review methodology was associated with the use of 8 out of 17 methods (47%) with adjusted odds ratios ranging from 1.18–2.20.
Discussion
This study provides the largest examination to date of search strategies and librarian involvement in systematic reviews and is the first to survey authors across disciplines directly rather than relying on published strategies. By surveying authors directly, it was able to control for the generally poor reporting of systematic reviews and generate a more accurate estimate of the use of recommended search methods and the impact and levels of librarian involvement.
Use of recommended search strategies was generally high (see Table 3), and was equivalent or higher than reported in comparable studies examining article-reported rates in most cases. For some items, such as use of controlled vocabulary, it was considerably higher. In other cases, such as inclusion of a full search strategy, it was difficult to make an accurate comparison since definitions differ widely [14]. Major et al. [30] provide the only known study where author-reported search strategies in dental systematic reviews were compared with article-reported search strategies and they found only fair levels of agreement between the two sources with author-reported rates higher in all cases. By surveying authors directly, the current study may have generated a more accurate or complete measure of the use of recommended search methods. Although the findings regarding the frequency of use of recommended search methods are encouraging, certain recommended methods are still underused. Specifically, authors could improve their searches by adapting the strategy to each source, including multiple languages, searching for unpublished or grey literature, updating and peer-reviewing their searches and ensuring that enough information is provided in the article to allow the search strategy to be appraised and reproduced.
In this study, half of all authors reported involving a librarian or search specialist in the systematic review, often as the designer of the search strategy. This is significantly higher than in other studies that found at most a quarter of reviews involved a librarian [9–11, 20, 21, 51]. This difference is likely due to many authors not reporting librarian contributions in the published article, as was the case in a third of cases in this study. Librarian involvement was strongly associated with use of the majority of recommended search methods, even after controlling for potential confounding variables. This parallels previous studies that found an association between stated librarian involvement and search quality in published reviews. As part of a larger examination of search strategies in systematic reviews of adverse effects, Golder et al. [9] found that searches conducted by information professionals were more likely to be reproducible, include more search terms and search more databases. While the current study did not look at number of search terms used, librarians were significantly associated with searching two or more databases and with including the full search strategy. More recently, Rethlefsen et al. [20] found that librarian involvement predicted higher-quality and more reproducible searches, with a stronger effect for librarian co-authorship, in systematic reviews from high-impact general medicine journals. While this study and the current one found many similar associations, they found a negative association with handsearching in contrast to the positive association in this study. The differences between these studies and the current one are minor and most likely attributable to differences in the journals and topics included. The results show that systematic review authors should ensure that they are following best practices for conducting and reporting searches in their reviews and consider working with librarians as a method to improve the quality of their searches.
While this study was focused on the impact of librarian involvement, the post-hoc analysis found that use of reporting guidelines, formal training in systematic review methodology, and increased self-reported knowledge of systematic review methodology were also associated with the use of many recommended methods. This is exciting since, along with librarian involvement, these are modifiable factors and suggest ways to increase systematic review quality on a larger scale. Additional research, however, is necessary to better understand the associations and mechanisms. For instance, this study only asked about the use of reporting guidelines, which focus on what information needs to be included in the published paper and thus are unlikely to directly affect the methods used. It is probable that authors who follow reporting guidelines are more likely to follow guidelines on the conduct of the review, but this has yet to be established. Similarly, more research is needed on what training methods or self-paced information resources are associated with gains in systematic review and search strategy quality. As the number of systematic reviews grows, identifying interventions such as these that might improve quality will become increasingly important.
This study had several limitations. First and foremost, it is based on self-report, potentially several years after the event, and it is possible authors did not know what each recommended method entailed or had forgotten the details. For example, two-thirds of authors reported including a full search strategy in the published article, but examinations of published articles have found much lower inclusion rates ranging from 9–40%, indicating a disconnect between what authors believe they are doing and what is actually happening [14, 20]. For convenience in obtaining a large sample, systematic reviews were drawn from a pre-identified list of systematic reviews (DARE). While DARE aims to be as comprehensive as possible [22], it is likely that some systematic reviews were missed. In addition, DARE is focused on systematic reviews that address health-related topics, so the results of this study may not generalize to systematic reviews in the social sciences or other disciplines examining non-health questions. Only English-language articles were included to match the English-language survey, which could have resulted in a bias towards methods used in English-speaking countries. Systematic reviews published by the Cochrane Collaboration were excluded from this study since they are required to follow the guidelines of the Cochrane Collaboration [2], which include recommending the use of a Trial Search Coordinator or similar expert, and have been shown to have different quality and reporting of search strategies than non-Cochrane reviews [12, 13, 23]. Cochrane reviews represent a subset of the total number of systematic reviews published annually, estimated at 20% by Moher et al. in 2004 [13] and currently at 13% (869 new or updated Cochrane reviews published in 2014 [52] and 6589 non-Cochrane reviews published in 2014 indexed in DARE [22]). While excluding them means that the results cannot be generalized to Cochrane reviews, the results are generalizable to the majority of systematic reviews and are less likely to be biased by the required methods of any single publisher. Nevertheless, it would be interesting for future research to look at Cochrane reviews and see if their more proscribed methods decrease the association between librarian involvement and use of recommended methods. Finally, the response rate was low (25%) and it is possible that authors were more likely to respond, and thus bias the results, if they had collaborated with a librarian or felt they had conducted a high-quality search. Based on the demographics in Table 1, however, there are few meaningful differences between responders and non-responders. The main exception was a lower response rate from Chinese authors, but this may be expected for an English-language survey and it is unclear that Chinese authors differ from those from other countries on study variables. A more targeted survey with higher levels of participation would be needed to determine if a response bias was present.
A first target for future research would be a multi-methods approach that directly compares author-reported and article-reported search strategies. This would allow us to directly measure reporting deficits and improve on the indirect comparisons performed in this study. Research should also investigate whether authors understand the recommended search methods and reporting requirements, such as what constitutes including a full, reproducible strategy in the article. It is possible that authors believe they are following best practices when they are not. Finally, this and other studies have found that librarian involvement is associated with higher-quality searches, but it is unclear if this is due to correlation or causation. Future research could seek to more directly measure and quantify the specific impact of the librarian on the search process and articles retrieved.
Conclusion
This study has shown that author-reported searching in systematic reviews is better than previously reported, but still sub-optimal. In particular, use of methods such as grey literature searching, adjusting terms and methods for each literature database and reporting of reproducible search strategies need to be improved. Librarian involvement was strongly associated with the use of many recommended search methods and could improve the quality of the review, contributing to the replicability and robustness of meta-analytic findings. It is important to note that librarian involvement uniquely contributed to search strategies after controlling for potential confounders. Despite the demonstrated positive impact of librarian involvement on the use of recommended search strategies and the relative ease with which this variable can be modified, librarian involvement is still low and their contributions need to be better acknowledged in the published article. A comprehensive and reproducible search strategy is core to any systematic review, and librarians represent important collaborators in the endeavor to produce high quality reviews.
Supporting Information
S1 Data. Article Characteristics—Cleaned Data.
Contains information about each of the eligible articles and its corresponding author for this study. This includes both authors that completed the survey and those that did not. Data has been cleaned for consistency and to address data errors. Comma-delimited file. Field definitions and coding information are inserted below the data.
https://doi.org/10.1371/journal.pone.0125931.s001
(CSV)
S2 Data. Survey Responses—Cleaned Data.
Contains the survey responses relevant to this study from each participant. Data has been cleaned and recoded for consistency and to address data errors. Comma-delimited file. Field definitions and coding information are inserted below the data.
https://doi.org/10.1371/journal.pone.0125931.s002
(CSV)
Acknowledgments
Erin Koffel, Minneapolis VA Medical Center, is thanked for her input and suggested revisions to the manuscript.
Author Contributions
Conceived and designed the experiments: JK. Performed the experiments: JK. Analyzed the data: JK. Contributed reagents/materials/analysis tools: JK. Wrote the paper: JK.
References
- 1.
OCEBM Levels of Evidence Working Group. The Oxford Levels of Evidence 2 website. Oxford, UK: Oxford Centre for Evidence-Based Medicine. Available: http://www.cebm.net/index.aspx?o=5653. Accessed 1 April 2015.
- 2.
Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.
- 3. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100. pmid:19621070
- 4.
Institute of Medicine Committee on Standards for Systematic Reviews of Comparative Effectiveness Research. Finding what works in health care: standards for systematic reviews. Eden J, Levit L, Berg A, Morton S, editors. Washington (DC): National Academies Press (US); 2011.
- 5.
Relevo R, Balshern H. Finding evidence for comparing medical interventions. In: Methods guide for effectiveness and comparative effectiveness reviews, AHRQ publication no. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014. Available: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=605. Accessed 1 April 2015.
- 6. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283(15):2008–12. pmid:10789670
- 7. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12. pmid:19631508
- 8. Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006;59(10):1057–63. pmid:16980145
- 9. Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8. pmid:18394536
- 10. Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7. pmid:24841794
- 11. Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049–54. pmid:21694568
- 12. Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54. pmid:18586178
- 13. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. pmid:17388659
- 14. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009;97(1):21–9. pmid:19158999
- 15. Golder S, Loke YK, Zorzela L. Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews. Health Info Libr J. 2014;31(2):92–105. pmid:24754741
- 16. Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA. 2014;312(10):999–1000. pmid:25203078
- 17. Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4. pmid:15088073
- 18. McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80. pmid:15685278
- 19. McKibbon A. Systematic reviews and librarians. Libr Trends. 2006;55(1):202–15.
- 20.
Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiology. 2015;In press.
- 21. Zhang L, Sampson M, McGowan J. Reporting of the role of the expert searcher in Cochrane reviews. Evid Based Libr Inf Pract. 2006;1(4):3–16.
- 22.
Database of Abstracts of Reviews of Effects [website]. York, UK: Centre for Reviews and Dissemination; 2014 [cited October 6, 2014]. Available: http://www.crd.york.ac.uk/CRDWeb/. Accessed 1 April 2015.
- 23. Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30. pmid:19282144
- 24. Lee E, Dobbins M, Decorby K, McRae L, Tirilis D, Husson H. An optimal search filter for retrieving systematic reviews and meta-analyses. BMC Med Res Methodol. 2012;12:51. pmid:22512835
- 25.
Scopus [website]. Elsevier. 2014. Available: http://www.scopus.com/. Accessed 1 April 2015.
- 26.
Web of Science [website]. Thomson Reuters. 2014. Available: http://wokinfo.com/. Accessed 1 April 2015.
- 27.
Journal Citation Reports [website]. Thomson Reuters. 2014. Available: http://thomsonreuters.com/journal-citation-reports/. Accessed 1 April 2015.
- 28. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997;314(7079):498–502. pmid:9056804
- 29. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52. pmid:19230612
- 30. Major MP, Warren S, Flores-Mir C. Survey of systematic review authors in dentistry: challenges in methodology and reporting. J Dent Educ. 2009;73(4):471–82. pmid:19339434
- 31. Klimo P Jr, Thompson CJ, Ragel BT, Boop FA. Methodology and reporting of meta-analyses in the neurosurgical literature. J Neurosurg. 2014;120(4):796–810. pmid:24460488
- 32. Fleming PS, Koletsi D, Seehra J, Pandis N. Systematic reviews published in higher impact clinical journals were of higher quality. J Clin Epidemiol. 2014;67(7):754–9. pmid:24709031
- 33. Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Reporting characteristics of meta-analyses in orthodontics: methodological assessment and statistical recommendations. Eur J Orthod. 2014;36(1):74–85. pmid:23493385
- 34. Aziz T, Compton S, Nassar U, Matthews D, Ansari K, Flores-Mir C. Methodological quality and descriptive characteristics of prosthodontic-related systematic reviews. J Oral Rehabil. 2013;40(4):263–78. pmid:23330989
- 35. Panic N, Leoncini E, de Belvis G, Ricciardi W, Boccia S. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8(12):e83138. pmid:24386151
- 36. Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611. pmid:22110690
- 37. Beller EM, Chen JK-H, Wang UL-H, Glasziou PP. Are systematic reviews up-to-date at the time of publication? Syst Rev. 2013;2:36. pmid:23714302
- 38. Cornelius VR, Perrio MJ, Shakir SA, Smith LA. Systematic reviews of adverse effects of drug interventions: a survey of their conduct and reporting quality. Pharmacoepidemiol Drug Saf. 2009;18(12):1223–31. pmid:19757414
- 39. Faggion CM Jr, Atieh MA, Park S. Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol. 2013;40(9):883–8. pmid:23834263
- 40. Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Res Synth Methods. 2011;2(1):15–32.
- 41. Flores-Mir C, Major MP, Major PW. Search and selection methodology of systematic reviews in orthodontics (2000–2004). Am J Orthod Dentofacial Orthop. 2006;130(2):214–7. pmid:16905066
- 42. Ge L, Wang JC, Li JL, Liang L, An N, Shi XT, et al. The assessment of the quality of reporting of systematic reviews/meta-analyses in diagnostic tests published by authors in China. PLoS One. 2014;9(1):e85908. pmid:24465781
- 43. Gianola S, Gasparini M, Agostini M, Castellini G, Corbetta D, Gozzer P, et al. Survey of the reporting characteristics of systematic reviews in rehabilitation. Phys Ther. 2013;93(11):1456–66. pmid:23744458
- 44. Golder S, Loke YK, Zorzela L. Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011. J Clin Epidemiol. 2013;66(3):253–60. pmid:23347849
- 45. Lawson ML, Pham B, Klassen TP, Moher D. Systematic reviews involving complementary and alternative medicine interventions had higher quality of reporting than conventional medicine reviews. J Clin Epidemiol. 2005;58(8):777–84. pmid:16018912
- 46. Lundh A, Knijnenburg SL, Jorgensen AW, van Dalen EC, Kremer LC. Quality of systematic reviews in pediatric oncology—a systematic review. Cancer Treat Rev. 2009;35(8):645–52. pmid:19836897
- 47. Major MP, Major PW, Flores-Mir C. An evaluation of search and selection methods used in dental systematic reviews published in English. J Am Dent Assoc. 2006;137(9):1252–7. pmid:16946429
- 48. Major MP, Major PW, Flores-Mir C. Benchmarking of reported search and selection methods of systematic reviews by dental speciality. Evid Based Dent. 2007;8(3):66–70. pmid:17891119
- 49. Mallett S, Deeks JJ, Halligan S, Hopewell S, Cornelius V, Altman DG. Systematic reviews of diagnostic tests in cancer: review of methods and reporting. BMJ. 2006;333(7565):413. pmid:16849365
- 50. Turner L, Galipeau J, Garritty C, Manheimer E, Wieland LS, Yazdi F, et al. An evaluation of epidemiological and reporting characteristics of complementary and alternative medicine (CAM) systematic reviews (SRs). PLoS One. 2013;8(1):e53536. pmid:23341949
- 51. Tannery NH, Maggio LA. The role of medical librarians in medical education review articles. J Med Libr Assoc. 2012;100(2):142–4. pmid:22514513
- 52.
About Cochrane Reviews [website]. London, UK: The Cochrane Collaboration; 2015. Available: http://www.cochranelibrary.com/about/about-cochrane-systematic-reviews.html. Accessed 1 April 2015.