Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Citation Classics in Systematic Reviews and Meta-Analyses: Who Wrote the Top 100 Most Cited Articles?

  • Olalekan A. Uthman ,

    olalekan.uthman@warwick.ac.uk

    Affiliations Warwick Centre for Applied Health Research and Delivery (WCAHRD), Division of Health Sciences, Warwick Medical School, the University of Warwick, Coventry, United Kingdom, International Health Group, Liverpool School of Tropical Medicine, Liverpool, United Kingdom, Centre for Evidence-Based Health Care, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa

  • Charles I. Okwundu,

    Affiliations Centre for Evidence-Based Health Care, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa, South African Cochrane Centre, South African Medical Research Council, Tygerberg, South Africa

  • Charles S. Wiysonge,

    Affiliation Centre for Evidence-Based Health Care, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa

  • Taryn Young,

    Affiliations Centre for Evidence-Based Health Care, Faculty of Medicine and Health Sciences, Stellenbosch University, Tygerberg, South Africa, South African Cochrane Centre, South African Medical Research Council, Tygerberg, South Africa

  • Aileen Clarke

    Affiliation Division of Health Sciences, Warwick Medical School, the University of Warwick, Coventry, United Kingdom

Citation Classics in Systematic Reviews and Meta-Analyses: Who Wrote the Top 100 Most Cited Articles?

  • Olalekan A. Uthman, 
  • Charles I. Okwundu, 
  • Charles S. Wiysonge, 
  • Taryn Young, 
  • Aileen Clarke
PLOS
x

Abstract

Background

Systematic reviews of the literature occupy the highest position in currently proposed hierarchies of evidence. The aims of this study were to assess whether citation classics exist in published systematic review and meta-analysis (SRM), examine the characteristics of the most frequently cited SRM articles, and evaluate the contribution of different world regions.

Methods

The 100 most cited SRM were identified in October 2012 using the Science Citation Index database of the Institute for Scientific Information. Data were extracted by one author. Spearman’s correlation was used to assess the association between years since publication, numbers of authors, article length, journal impact factor, and average citations per year.

Results

Among the 100 citation classics, published between 1977 and 2008, the most cited article received 7308 citations and the least-cited 675 citations. The average citations per year ranged from 27.8 to 401.6. First authors from the USA produced the highest number of citation classics (n=46), followed by the UK (n=28) and Canada (n=15). The 100 articles were published in 42 journals led by the Journal of the American Medical Association (n=18), followed by the British Medical Journal (n=14) and The Lancet (n=13). There was a statistically significant positive correlation between number of authors (Spearman’s rho=0.320, p=0.001), journal impact factor (rho=0.240, p=0.016) and average citations per year. There was a statistically significant negative correlation between average citations per year and year since publication (rho = -0.636, p=0.0001). The most cited papers identified seminal contributions and originators of landmark methodological aspects of SRM and reflect major advances in the management of and predisposing factors for chronic diseases.

Conclusions

Since the late 1970s, the USA, UK, and Canada have taken leadership in the production of citation classic papers. No first author from low or middle-income countries (LMIC) led one of the most cited 100 SRM.

Background

Systematic reviews of the literature occupy the highest position in currently proposed hierarchies of evidence [1] and occupy this top position for two fundamental premises. Firstly, a systematic review involves the application of scientific strategies which limit bias by systematic assembly, critical appraisal and synthesis of relevant studies on a particular topic [2,3]. Secondly, reviews that include a meta-analysis provide precise estimates of the association studied[4]. Because of the importance of systematic reviews in summarizing the advances of health care knowledge, their number is growing rapidly [4]. If systematic reviews in fact represent the best level of evidence, they are likely to have great clinical importance[4]. It follows that they may be cited often in the literature. The acknowledgement that one article gives to another is a reference; the acknowledgement that one article receives from another is a citation [5]. The number of citations an article receives after publication reflects its impact on the scientific community. There have been a few recent attempts to identify and analyze “the most cited articles” in various specialties [59]. However, an analysis of the most frequently cited systematic review and meta-analysis (SRM) articles has not yet been reported. Montori and colleagues examined whether systematic reviews receive more citations than narrative reviews [4]. They found that rigorous systematic reviews were cited significantly more often than narrative reviews. In this paper we sought to identify and examine the characteristics of the most cited SRM related articles, such as ranking, year of publication, publishing journal, topic and contribution of different world regions to most cited SRM articles.

In addition, we assessed whether there was an association between year of publication, number of authors, number of pages, journals’ impact factor, and average citations per year.

Methods

The Science Citation Index of the Institute for Scientific Information (ISI) was searched in October 2012 for systematic review and meta-analysis related articles. We searched for articles using validated keywords for identifying SRM [10]. To accredit an article to countries, the method of "absolute country counting" was adopted, in which each country contributing to an article received one paper credit based on the lead author’s correspondence or reprint address [11]. The 100 most-cited articles were selected for further descriptive analyses. Data collected included the year of publication, the topic covered, lead author’s correspondence country of origin, and number of citations.

We used a density-equalizing map to visualize the citation classics by the corresponding address of the author. We used Gastner and Newman's algorithm [12] in order to produce a map of the world in which countries were re-sized according to the number of most cited SRM articles. These calculations employ a diffusion equation in the Fourier domain borrowed from elementary physics, which allows variable resolution by tracking moving boundaries [12].

The impact factors and immediacy factors of journals listed in the 2012 Journal Citation Reports Science Edition were adopted as quantitative tools for evaluating journals in which these articles were published. A journal’s impact factor is a measure of the frequency with which the "average article" in the journal has been cited in a given period of time. The impact factor for a journal is calculated based on a three-year period, and can be considered to be the average number of times published papers are cited up to two years after publication. Non-parametric (Spearman’s) correlation was used to assess the association between years since publication (with reference to the year 2012), numbers of authors, article’s length (number of pages), journal’s impact factor and average citations per year.

Results

The 100 articles are listed in Table 1 in descending order, ranked according to the total number of citations since publication. Among the 100 citation classics, the most cited article received 7308 citations, and the least-cited 675 citations. The average citations per year ranged from 27.8 to 401.6. Figure 1 shows the density-equalizing map illustrating the number of contributions for each country in SRM citation classics. Density equalising mapping demonstrates that a relatively small number of countries were responsible for the majority of the top cited SRM articles (Figure 1). First authors from the USA produced the highest citation classics (n=46), followed by the UK (n=28) and Canada (n=15) (Figure 1). All the 100 most cited articles were published in the English Language.

ArticlesYearTotal no. of citationsAverage citations per year
NoRankNoRank
Dersimonian R, Laird N: Metaanalysis in Clinical-Trials. Controlled Clinical Trials 1986, 7: 177-188.198673081281.17
Egger M, Smith GD, Schneider M, Minder C: Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 1997, 315: 629-634.199751972346.52
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ et al.: Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials 1996, 17: 1-12.199645353283.46
Higgins JPT, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. British Medical Journal 2003, 327: 557-560.200341114456.81
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical-Evidence of Bias - Dimensions of Methodological Quality Associated with Estimates of Treatment Effects in Controlled Trials. Jama-Journal of the American Medical Association 1995, 273: 408-412.199528845169.617
Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS: Evidence based medicine: What it is and what it isn't - It's about integrating individual clinical expertise and the best external evidence. British Medical Journal 1996, 312: 71-72.199628816180.114
Higgins JPT, Thompson SG: Quantifying heterogeneity in a meta-analysis. Statistics in Medicine 2002, 21: 1539-1558.200227557275.58
Baigent C, Keech A, Kearney PM, Blackwell L, Buck G, Pollicino C et al.: Efficacy and safety of cholesterol-lowering treatment: prospective meta-analysis of data from 90,056 participants in 14 randomised trials of statins. Lancet 2005, 366: 1267-1278.2002264082649
Boushey CJ, Beresford SAA, Omenn GS, Motulsky AG: A Quantitative Assessment of Plasma Homocysteine As A Risk Factor for Vascular-Disease - Probable Benefits of Increasing Folic-Acid Intakes. Jama-Journal of the American Medical Association 1995, 274: 1049-1057.199525329148.919
Abe O, Abe R, Enomoto K, Kikuchi K, Koyama H, Masuda H et al.: Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials. Lancet 2005, 365: 1687-1717.2005235610336.63
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M et al.: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998, 352: 609-613.1999233511179.615
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D et al.: Meta-analysis of observational studies in epidemiology - A proposal for reporting. Jama-Journal of the American Medical Association 2000, 283: 2008-2012.2000233512194.612
Lewington S, Clarke R, Qizilbash N, Peto R, Collins R: Age-specific relevance of usual blood pressure to vascular mortality: a meta-analysis of individual data for one million adults in 61 prospective studies. Lancet 2002, 360: 1903-1913.2002232813232.810
Alberti W, Anderson G, Bartolucci A, Bell D, Villalba JB, Brodin O et al.: Chemotherapy in Non-Small-Cell Lung-Cancer - A Metaanalysis Using Updated Data on Individual Patients from 52 Randomized Clinical-Trials. British Medical Journal 1995, 311: 899-909.1995208714122.824
Baigent C, Sudlow C, Collins R, Peto R: Collaborative meta-analysis of randomised trials of antiplatelet therapy for prevention of death, myocardial infarction, and stroke in high risk patients. British Medical Journal 2002, 324: 71-86.2005204015291.45
Block G, Patterson B, Subar A: Fruit, Vegetables, and Cancer Prevention - A Review of the Epidemiologic Evidence. Nutrition and Cancer-an International Journal 1992, 18: 1-29.199218261691.335
Begg CB, Mazumdar M: Operating Characteristics of A Bank Correlation Test for Publication Bias. Biometrics 1994, 50: 1088-1101.199417561797.631
Lazarou J, Pomeranz BH, Corey PN: Incidence of adverse drug reactions in hospitalized patients - A meta-analysis of prospective studies. Jama-Journal of the American Medical Association 1998, 279: 1200-1205.1998170718121.925
Buchwald H, Avidor Y, Braunwald E, Jensen MD, Pories W, Fahrbach K et al.: Bariatric surgery: A systematic review and meta-analysis. Jama-Journal of the American Medical Association 2004, 292: 1724-1737.2004170619213.311
Grady D, Rubin SM, Petitti DB, Fox CS, Black D, Ettinger B et al.: Hormone-Therapy to Prevent Disease and Prolong Life in Postmenopausal Women. Annals of Internal Medicine 1992, 117: 1016-1037.199217022085.141
Marshall D, Johnell O, Wedel H: Meta-analysis of how well measures of bone mineral density predict occurrence of osteoporotic fractures. British Medical Journal 1996, 312: 1254-1259.199615832198.930
Barrick MR, Mount MK: The Big 5 Personality Dimensions and Job-Performance - A Metaanalysis. Personnel Psychology 1991, 44: 1-26.199115822275.351
Davis DA, Thomson MA, Oxman AD, Haynes RB: Changing Physician Performance - A Systematic Review of the Effect of Continuing Medical-Education Strategies. Jama-Journal of the American Medical Association 1995, 274: 700-705.199515772392.834
Appleby P, Baigent C, Collins R, Flather M, Parish S, Peto R et al.: Indications for Fibrinolytic Therapy in Suspected Acute Myocardial-Infarction - Collaborative Overview of Early Mortality and Major Morbidity Results from All Randomized Trials of More Than 1000 Patients. Lancet 1994, 343: 311-322.199415192484.443
Nissen SE, Wolski K: Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. New England Journal of Medicine 2007, 356: 2457-2471.2007150725301.44
Grimshaw JM, Russell IT: Effect of Clinical Guidelines on Medical-Practice - A Systematic Review of Rigorous Evaluations. Lancet 1993, 342: 1317-1322.199313772672.555
Abe O, Abe R, Enomoto K, Kikuchi K, Koyama H, Masuda H et al.: Effects of radiotherapy and of differences in the extent of surgery for early breast cancer on local recurrence and 15-year survival: an overview of the randomised trials. Lancet 2005, 366: 2087-2106.2005132427189.113
Danesh J, Whincup P, Walker M, Lennon L, Thomson A, Appleby P et al.: Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses. British Medical Journal 2000, 321: 199-204.2004127728159.618
Farrer LA, Cupples LA, Haines JL, Hyman B, Kukull WA, Mayeux R et al.: Effects of age, sex, and ethnicity on the association between apolipoprotein E genotype and Alzheimer disease - A meta-analysis. Jama-Journal of the American Medical Association 1997, 278: 1349-1356.199712652984.344
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 1999, 354: 1896-1900.199812623090.137
Armitage CJ, Conner M: Efficacy of the theory of planned behaviour: A meta-analytic review. British Journal of Social Psychology 2001, 40: 471-499.2001123431112.228
Dellinger RP, Carlet JM, Masur H, Gerlach H, Calandra T, Cohen J et al.: Surviving Sepsis Campaign guidelines for management of severe sepsis and septic shock. Critical Care Medicine 2004, 32: 858-873.2004117232146.520
Pignon JP, Bourhis J, Domenge C, Designe L: Chemotherapy added to locoregional treatment for head and neck squamous-cell carcinoma: three meta-analyses of updated individual data. Lancet 2000, 355: 949-955.200011553396.332
Guyatt G: Evidence-Based Medicine - A New Approach to Teaching the Practice of Medicine. Jama-Journal of the American Medical Association 1992, 268: 2420-2425.199211453457.366
Phan KL, Wager T, Taylor SF, Liberzon I: Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 2002, 16: 331-348.2002113635113.627
Kramer MS: Determinants of Low Birth-Weight - Methodological Assessment and Meta-Analysis. Bulletin of the World Health Organization 1987, 65: 663-737.2003113036125.623
Lohmueller KE, Pearce CL, Pike M, Lander ES, Hirschhorn JN: Meta-analysis of genetic association studies supports a contribution of common variants to susceptibility to common disease. Nature Genetics 2003, 33: 177-182.198711303745.281
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication Bias in Clinical Research. Lancet 1991, 337: 867-872.199111193853.373
Anderson JW, Johnstone BM, Cooknewell ME: Metaanalysis of the Effects of Soy Protein-Intake on Serum-Lipids. New England Journal of Medicine 1995, 333: 276-282.199511133965.560
Danesh J, Collins R, Appleby P, Peto R: Association of fibrinogen, C-reactive protein, albumin, or leukocyte count with coronary heart disease - Meta-analyses of prospective studies. Jama-Journal of the American Medical Association 1998, 279: 1477-1482.199811024078.748
Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. Jama-Journal of the American Medical Association 1999, 282: 1054-1060.200110584196.233
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S et al.: Grading quality of evidence and strength of recommendations. British Medical Journal 2004, 328: 1490-1494.2004101242126.522
Smith ML, Glass GV: Meta-Analysis of Psychotherapy Outcome Studies. American Psychologist 1977, 32: 752-760.1977979432899
Brewin CR, Andrews B, Valentine JD: Meta-analysis of risk factors for posttraumatic stress disorder in trauma-exposed adults. Journal of Consulting and Clinical Psychology 2000, 68: 748-766.2000972448146
Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA: Getting research findings into practice - Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. British Medical Journal 1998, 317: 465-468.19989644568.957
Dickersin K, Scherer R, Lefebvre C: Systematic Reviews - Identifying Relevant Studies for Systematic Reviews. British Medical Journal 1994, 309: 1286-1291.19949524652.974
Anderson RJ, Freedland KE, Clouse RE, Lustman PJ: The prevalence of comorbid depression in adults with diabetes - A meta-analysis. Diabetes Care 2001, 24: 1069-1078.20019334784.842
Oxman AD, Thomson MA, Davis DA, Haynes RB: No Magic Bullets - A Systematic Review of 102 Trials of Interventions to Improve Professional Practice. Canadian Medical Association Journal 1995, 153: 1423-1431.19959284854.672
Ohara MW, Swain AM: Rates and risk of postpartum depression - A meta-analysis. International Review of Psychiatry 1996, 8: 37-54.19969134957.167
Clarke R, Collins R, Lewington S, Donald A, Alfthan G, Tuomilehto J et al.: Homocysteine and risk of ischemic heart disease and stroke - A meta-analysis. Jama-Journal of the American Medical Association 2002, 288: 2015-2022.2002910509136
DiMatteo MR, Lepper HS, Croghan TW: Depression is a risk factor for noncompliance with medical treatment - Meta-analysis of the effects of anxiety and depression on patient adherence. Archives of Internal Medicine 2000, 160: 2101-2107.2000900517552
Colditz GA, Brewer TF, Berkey CS, Wilson ME, Burdick E, Fineberg HV et al.: Efficacy of Bcg Vaccine in the Prevention of Tuberculosis - Metaanalysis of the Published Literature. Jama-Journal of the American Medical Association 1994, 271: 698-702.19948905249.479
Wald DS, Law M, Morris JK: Homocysteine and cardiovascular disease: evidence on causality from a meta-analysis. British Medical Journal 2002, 325: 1202-1206K.20028725387.239
Danesh J, Wheeler JG, Hirschfield GM, Eda S, Eiriksdottir G, Rumley A et al.: C-reactive protein and other circulating markers of inflammation in the prediction of coronary heart disease. New England Journal of Medicine 2004, 350: 1387-1397.20008605471.756
Gabriel SE, Jaakkimainen L, Bombardier C: Risk for Serious Gastrointestinal Complications Related to Use of Nonsteroidal Antiinflammatory Drugs - A Metaanalysis. Annals of Internal Medicine 1991, 115: 787-796.19918545540.788
Harris EC, Barraclough B: Suicide as an outcome for mental disorders - A meta-analysis. British Journal of Psychiatry 1997, 170: 205-228.19978505656.769
Linn MC, Petersen AC: Emergence and Characterization of Sex-Differences in Spatial Ability - A Meta-Analysis. Child Development 1985, 56: 1479-1498.19858505731.598
Bongartz T, Sutton AJ, Sweeting MJ, Buchan I, Matteson EL, Montori V: Anti-TNF antibody therapy in rheumatoid arthritis and the risk of serious infections and malignancies - Systematic review and meta-analysis of rare harmful effects in randomized controlled trials. Jama-Journal of the American Medical Association 2006, 295: 2275-2285.20068465814121
Mcgeer PL, Schulzer M, Mcgeer EG: Arthritis and anti-inflammatory agents as possible protective factors for Alzheimer's disease: A review of 17 epidemiologic studies. Neurology 1996, 47: 425-432.19968265951.677
Mensink RP, Katan MB: Effect of Dietary Fatty-Acids on Serum-Lipids and Lipoproteins - A Metaanalysis of 27 Trials. Arteriosclerosis and Thrombosis 1992, 12: 911-919.19928226041.187
Sheppard BH, Hartwick J, Warshaw PR: The Theory of Reasoned Action - A Meta-Analysis of Past Research with Recommendations for Modifications and Future-Research. Journal of Consumer Research 1988, 15: 325-343.19888226134.395
Strong WB, Malina RM, Blimkie CJR, Daniels SR, Dishman RK, Gutin B et al.: Evidence based physical activity for school-age youth. Journal of Pediatrics 2005, 146: 732-737.200581562116.426
Colquitt JA, Conlon DE, Wesson MJ, Porter COLH, Ng KY: Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology 2001, 86: 425-445.20018116373.754
Sacks HS, Berrier J, Reitman D, Anconaberk VA, Chalmers TC: Meta-Analyses of Randomized Controlled Trials. New England Journal of Medicine 1987, 316: 450-455.19878106432.497
Llovet JM, Bruix J: Systematic review of randomized trials for unresectable hepatocellular carcinoma: Chemoembolization improves survival. Hepatology 2003, 37: 429-442.20038046589.338
Peyron R, Laurent B, Garcia-Larrea L: Functional imaging of brain responses to pain. A review and meta-analysis (2000). Neurophysiologie Clinique-Clinical Neurophysiology 2000, 30: 263-288.20008016666.858
Lipsey MW, Wilson DB: The Efficacy of Psychological, Educational, and Behavioral Treatment - Confirmation from Metaanalysis. American Psychologist 1993, 48: 1181-1209.19937976741.985
Dahlof B, Pennert K, Hansson L: Reversal of Left-Ventricular Hypertrophy in Hypertensive Patients - A Metaanalysis of 109 Treatment Studies. American Journal of Hypertension 1992, 5: 95-110.19927966839.889
Ernst E, Resch KL: Fibrinogen As A Cardiovascular Risk Factor - A Metaanalysis and Review of the Literature. Annals of Internal Medicine 1993, 118: 956-963.19937946941.886
Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC: A Comparison of Results of Metaanalyses of Randomized Control Trials and Recommendations of Clinical Experts - Treatments for Myocardial-Infarction. Jama-Journal of the American Medical Association 1992, 268: 240-248.19927827039.191
Patrick DL, Cheadle A, Thompson DC, Diehr P, Koepsell T, Kinne S: The Validity of Self-Reported Smoking - A Review and Metaanalysis. American Journal of Public Health 1994, 84: 1086-1093.19947777143.283
Maron BJ: Hypertrophic cardiomyopathy - A systematic review. Jama-Journal of the American Medical Association 2002, 287: 1308-1320.20027737277.349
Miller ER, Pastor-Barriuso R, Dalal D, Riemersma RA, Appel LJ, Guallar E: Meta-analysis: High-dosage vitamin E supplementation may increase all-cause mortality. Annals of Internal Medicine 2005, 142: 37-46.200576473109.129
Wright IC, Rabe-Hesketh S, Woodruff PWR, David AS, Murray RM, Bullmore ET: Meta-analysis of regional brain volumes in schizophrenia. American Journal of Psychiatry 2000, 157: 16-25.20007637463.662
Horvath AO, Symonds BD: Relation Between Working Alliance and Outcome in Psychotherapy - A Metaanalysis. Journal of Counseling Psychology 1991, 38: 139-149.19917617536.294
Voyer D, Voyer S, Bryden MP: Magnitude of Sex-Differences in Spatial Abilities - A Metaanalysis and Consideration of Critical Variables. Psychological Bulletin 1995, 117: 250-270.19957567644.582
Wong DKH, Cheung AM, Orourke K, Naylor CD, Detsky AS, Heathcote J: Effect of Alpha-Interferon Treatment in Patients with Hepatitis-B E-Antigen-Positive Chronic Hepatitis-B - A Metaanalysis. Annals of Internal Medicine 1993, 119: 312-323.19937527739.690
Capes SE, Hunt D, Malmberg K, Gerstein HC: Stress hyperglycaemia and increased risk of death after myocardial infarction in patients with and without diabetes: a systematic overview. Lancet 2000, 355: 773-778.20007467862.264
Moore FA, Feliciano DV, Andrassy RJ, Mcardle AH, Booth FVM, Morgensteinwagner TB et al.: Early Enteral Feeding, Compared with Parenteral, Reduces Postoperative Septic Complications - the Results of A Metaanalysis. Annals of Surgery 1992, 216: 172-183.1992740793793
Juni P, Altman DG, Egger M: Systematic reviews in health care - Assessing the quality of controlled clinical trials. British Medical Journal 2001, 323: 42-46.19997378056.768
Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JHP et al.: Empirical evidence of design-related bias in studies of diagnostic tests. Jama-Journal of the American Medical Association 1999, 282: 1061-1066.19997318156.270
Ozer EJ, Best SR, Lipsey TL, Weiss DS: Predictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis. Psychological Bulletin 2003, 129: 52-73.20037308281.145
Wellman HM, Cross D, Watson J: Meta-analysis of theory-of-mind development: The truth about false belief. Child Development 2001, 72: 655-684.20017288366.259
Cosgrove SE, Sakoulas G, Perencevich EN, Schwaber MJ, Karchmer AW, Carmeli Y: Comparison of mortality associated with methicillin-resistant and methicillin-susceptible Staphylococcus aureus bacteremia: A meta-analysis. Clinical Infectious Diseases 2003, 36: 53-59.20037258480.647
Hart RG, Benavente O, McBride R, Pearce LA: Antithrombotic therapy to prevent stroke in patients with atrial fibrillation: A meta-analysis. Annals of Internal Medicine 1999, 131: 492-50119997178555.271
Berlin JA, Colditz GA: A Metaanalysis of Physical-Activity in the Prevention of Coronary Heart-Disease. American Journal of Epidemiology 1990, 132: 612-628.19907138632.496
Eaden JA, Abrams KR, Mayberry JF: The risk of colorectal cancer in ulcerative colitis: a meta-analysis. Gut 2001, 48: 526-535.20017118764.661
Neal B, MacMahon S, Chapman N, Cutler J, Fagard R, Neal B et al.: Effects of ACE inhibitors, calcium antagonists, and other blood-pressure-lowering drugs: results of prospectively designed overviews of randomised trials. Lancet 2000, 356: 1955-1964.20007098859.165
Stuck AE, Siu AL, Wieland GD, Adams J, Rubenstein LZ: Comprehensive Geriatric Assessment - A Metaanalysis of Controlled Trials. Lancet 1993, 342: 1032-1036.19937098937.392
Zeggini E, Scott LJ, Saxena R, Voight BF, Marchini JL, Hu T et al.: Meta-analysis of genome-wide association data and large-scale replication identifies additional susceptibility loci for type 2 diabetes. Nature Genetics 2008, 40: 638-645.200870690176.516
Brattstrom L, Landgren F, Israelsson B, Lindgren A, Hultberg B, Andersson A et al.: Lowering blood homocysteine with folic acid based supplements: meta-analysis of randomised trials. British Medical Journal 1998, 316: 894-898.19986979149.878
Labbe KA, Detsky AS, Orourke K: Metaanalysis in Clinical Research. Annals of Internal Medicine 1987, 107: 224-233.19876949227.8100
Hites RA: Polybrominated diphenyl ethers in the environment and in people: A meta-analysis of concentrations. Environmental Science & Technology 2004, 38: 945-956.20046929386.540
Claxton AJ, Cramer J, Pierce C: A systematic review of the associations between dose regimens and medication compliance. Clinical Therapeutics 2001, 23: 1296-1310.20016919462.863
Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical industry sponsorship and research outcome and quality: systematic review. British Medical Journal 2003, 326: 1167-1170B.20036899576.650
Hunt DL, Haynes RB, Hanna SE, Smith K: Effects of computer-based clinical decision support systems on physician performance and patient outcomes - A systematic review. Jama-Journal of the American Medical Association 1998, 280: 1339-1346.19986879649.180
Deci EL, Koestner R, Ryan RM: A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin 1999, 125: 627-668.19996859752.775
Larosa JC, He J, Vupputuri S: Effect of statins on risk of coronary disease - A meta-analysis of randomized controlled trials. Jama-Journal of the American Medical Association 1999, 282: 2340-2346.19996809852.376
Kluger AN, DeNisi A: The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 1996, 119: 254-284.19966779942.384
Lewis CM, Levinson DF, Wise LH, Delisi LE, Straub RE, Hovatta I et al.: Genome scan meta-analysis of schizophrenia and bipolar disorder, part II: Schizophrenia. American Journal of Human Genetics 2003, 73: 34-48.20036751007553

Table 1. The 100 most frequently cited systematic review and meta-analysis articles.

CSV
Download CSV
thumbnail
Figure 1. Spatial distribution of the 100 most frequently cited systematic reviews and meta-analyses related articles using density equalizing mapping.

The area of each country were re-sized in proportion to its total number of 100 most frequently cited systematic reviews and meta-analyses related articles.

https://doi.org/10.1371/journal.pone.0078517.g001

The year of publication with the relevant number of classics identified is shown in Figure 2. The oldest article was published in 1977 and the most recent article in 2008. Figure 3A shows correlation between average citations per year and year since publication (with reference to 2012). There was a statistically significant negative correlation between average citations per year and year since publication (Spearman’s rho = -0.636, 95% CI -0.739 to -0.501, p=0.0001), such that the average citations per year reduces with the number of years since publication.

thumbnail
Figure 2. Graph demonstrating the time period of publication of the 100 most cited systematic reviews and meta-analyses related articles.

https://doi.org/10.1371/journal.pone.0078517.g002

thumbnail
Figure 3. Correlation between since publication, number of authors, journals’ impact factor and average citation per year.

https://doi.org/10.1371/journal.pone.0078517.g003

The number of authors of the most cited articles ranged from one to 22. Four of the articles were authored by a single author and 18 articles by two authors. There was a statistically significant positive correlation between number of authors and average citations per year (rho=0.320, 95% CI 0.132 to 0.486, p=0.001), such that the greater the number of authors, the higher the average citation per year (Figure 3B). The median length of article was 10 pages (range: 2 to 75 pages). There was no statistically significant correlation between length and average citations per year (rho = -0.052, 95% CI -0.246 to 0.146, p=0.608). The most cited articles were published in 42 journals (Table 2), led by Journal of The American Medical Association (n=18) followed by the British Medical Journal (n=14), The Lancet (n=13), and the Annals of Internal Medicine (n=7). Journal impact factors ranged from 1.412 (for Biometrics) to 51.658 (for the New England Journal of Medicine).

Journal titleNumber of articlesImpact factor (2012)
Journal of The American Medical Association1829.978
British Medical Journal1417.215
Lancet1339.06
Annals of Internal Medicine713.976
New England Journal of Medicine451.658
Psychological Bulletin415.575
American Psychologist25.1
Child Development24.915
Controlled Clinical Trials21.597
Nature Genetics235.209
American Journal of Epidemiology14.78
American Journal of Human Genetics111.202
American Journal of Hypertension13.665
American Journal of Psychiatry114.721
American Journal of Public Health13.93
Annals of Surgery16.329
Archives of Internal Medicine111.462
Arteriosclerosis And Thrombosis16.338
Biometrics11.412
British Journal of Psychiatry16.606
British Journal of Social Psychology11.816
Bulletin of The World Health Organization15.25
Canadian Medical Association Journal16.465
Clinical Infectious Diseases19.374
Clinical Therapeutics12.23
Critical Care Medicine16.124
Diabetes Care17.735
Environmental Science Technology15.257
Gut110.732
Hepatology112.003
International Review of Psychiatry11.608
Journal of Applied Psychology14.758
Journal of Consulting And Clinical Psychology15.011
Journal of Consumer Research13.542
Journal of Counseling Psychology12.628
Journal of Pediatrics14.035
Neuroimage16.252
Neurology18.249
Neurophysiologie Clinique Clinical Neurophysiology12.553
Nutrition and Cancer An International Journal12.695
Personnel Psychology13.702
Statistics In Medicine12.044

Table 2. Journals in which the most cited articles were published.

CSV
Download CSV

There was a statistically significant positive correlation between average citations per year and journal impact factor (rho=0. 240, 95% CI 0.045 to 0.416, p=0. 016) (Figure 3C). General and internal medicine were the main topics covered by these highly cited articles (n=59). Considerable attention was also given to Psychology and Psychiatry (n=13).

The top-100 list contained landmark contributions dealing with methodological aspects of conducting systematic reviews and meta-analysis (n=17). At number 1, DerSimonian and Laird’s landmark article which introduced a novel simple random effects model for combining studies. Egger et al. (number-2) examined the prevalence of funnel plot asymmetry among published meta-analyses. Higgins et al. (number-4 and number-7) developed a new measure (I2) for quantifying heterogeneity between studies included in a meta-analysis. Stroup et al. (number-12) reported a proposal for reporting meta-analyses of observational studies. The list of the most cited articles also reflects major advances in the management of non-communicable diseases (n=40) and in the identification of their predisposing factors for such diseases over the last 30 years. Baigent and colleagues (number-8) examined the efficacy and safety of statins on cholesterol lowering. Abe and co-researchers examined effects of chemotherapy and hormonal therapy for early breast cancer recurrence (number-10) and Lewington et al. (number-13) examined age-specific relevance of usual blood pressure and vascular mortality.

Discussion

This study identified and characterised the 100 most cited SRM related articles published in the past three decades, providing an overview of the citation frequency of these most cited articles. The list of the most cited articles identifies first authors and topics which reflect advances in methodological techniques in meta-analysis, major advances in the management of chronic diseases, and identification of predisposing factors over the last 30 years. Some of the most frequently cited articles were methodological papers. As expected the most highly cited papers were more likely to be published in journals high on the impact factor list [13,14]. It is important to note that, at present, no Cochrane review is among the 100 most cited SRM related articles. Cochrane reviews are systematic reviews of primary research in human health care and health policy, and are internationally recognised as the highest standard in evidence-based health care [15,16]. They investigate the effects of interventions for prevention, treatment and rehabilitation [15,16]. They also assess the accuracy of a diagnostic test for a given condition in a specific patient group and setting. They are published online in The Cochrane Library [15,16]. The low citations received by Cochrane reviews may be due to improper citations of Cochrane reviews, and the relatively recent tracking of Cochrane reviews by ISI. In addition, ISI Science Citation Index database covers all new and substantially updated Cochrane reviews from January 2005, and the first impact factor for Cochrane Database of Systematic Review was released in June 2008.

We found that almost half of the most cited SRM related articles originated in the US. This Figure is comparable with the origin of citation classics in other fields [59]. The overwhelming influence the US has on medical research may be due to its large underlying population, enormous financial resources available to the scientific community in the country and its high population of active citing researchers compared to other countries [6,7]. Studies have demonstrated that biomedical research productivity worldwide is largely dependent on each country’s per capita gross national product and the expenditure allotted for research and development [17,18].

Our results support previous findings [59] that first authors from Africa, Asia, and South America had minimal or no contributions in the most cited articles. Scientific publishing activity worldwide over the past decades shows that most countries in these regions have low levels of publication [19]. The above finding is not a surprise because difficulties in research, publication, information access, and language barriers facing the least-developed countries are profound and seem almost intractable. Most information published in journals based in low and middle-income countries (LMICs) never leaves there home borders because these journals are largely excluded from major bibliographic databases. In addition, most of the reviews produced to date address health conditions that are priorities in the developed world[20]. Many major health concerns in LMICs have yet to be made the subject of a citation classic review, although there are signs that this may be changing[21].

In addition, the difficulties of conducting randomised controlled trials and other high quality studies in resource-poor situations result in the exclusion of many LMIC studies from systematic reviews [22]. However, there is a need to challenge the status quo. Scientists from these regions should forge multiple collaborations beyond historical, political, and cultural lines to share knowledge and expertise on SRM. In addition, there is a need to promote research in SRM in less developed regions of the world. This may involve but is not limited to the political will for research capacity development among LIMC health policymakers, the training of LMIC researchers to be competent in systematic review techniques, the development of infrastructure including research and academic institutes, the improvement of current collaborative partnerships with developed nations, increased sponsorship and support from world agencies such as the World Health Organization and the United Nations Organization.

Although we have tried to eliminate potential flaws in our citation analysis, some limitations were inevitable and are linked to the inherent problems of citation analysis [23,24]. The citation of a scientific article usually follows a time course, it is usually not cited until one to two years after publication, reaches a maximum after three to ten years, then declines [6]. Another problem is oriented or biased citing, including various types of conscious or unconscious biases, such as self-citation (bias towards one’s own work), in-house (bias towards friends or colleagues), journal or powerful person (bias towards reviewers, editors, members of grant awarding bodies), negative citation (bias towards potential negative credits), English language (bias towards publishing and referencing English articles), and omission bias (bias towards not referencing competitors or sources contradictory to one’s own results) [25]. Other limitations include the incorrect citation of origin for the authors. By using the author addresses listed in the bylines of research articles, one can only identify countries and organizations where the authors were employed when the research was done or where the article was written [18].

Conclusion

Since the late 1970s, the USA, UK, and Canada have taken leadership in the production of citation classic papers. No author from LMICs led any of the most cited 100 SRM. There is a need to challenge the status quo. Scientists from LMIC should forge multiple collaborations to share knowledge and expertise on SRM. In addition, there is a need to strengthen research capacity in these countries and more support should be provided for the advancement of research efforts.

Author Contributions

Conceived and designed the experiments: OAU. Performed the experiments: OAU. Analyzed the data: OAU CO CW TY AC. Contributed reagents/materials/analysis tools: OAU CO CW TY AC. Wrote the manuscript: OAU CO CW TY AC.

References

  1. 1. Evans D (2003) Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs 12: 77-84. doi:10.1046/j.1365-2702.2003.00662.x. PubMed: 12519253.
  2. 2. Manchikanti L (2008) Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Pain Physician 11: 161-186. PubMed: 18354710.
  3. 3. Wright RW, Brand RA, Dunn W, Spindler KP (2007) How to write a systematic review. Clin Orthop Relat Res 455: 23-29. doi:10.1097/BLO.0b013e31802c9098. PubMed: 17279036.
  4. 4. Montori VM, Wilczynski NL, Morgan D, Haynes RB (2003) Systematic reviews: a cross-sectional study of location and citation counts. BMC Med 1: 2. doi:10.1186/1741-7015-1-2. PubMed: 14633274.
  5. 5. Paladugu R, Schein M, Gardezi S, Wise L (2002) One hundred citation classics in general surgical journals. World J Surg 26: 1099-1105. doi:10.1007/s00268-002-6376-7. PubMed: 12209239.
  6. 6. Baltussen A, Kindler CH (2004) Citation classics in critical care medicine. Intensive Care Med 30: 902-910. doi:10.1007/s00134-004-2195-7. PubMed: 14985952.
  7. 7. Baltussen A, Kindler CH (2004) Citation classics in anesthetic journals. Anesth Analg 98: 443-451, table. PubMed: 14742385.
  8. 8. Fenton JE, Roy D, Hughes JP, Jones AS (2002) A century of citation classics in otolaryngology-head and neck Surgery journals. J Laryngol Otol 116: 494-498. PubMed: 12238666.
  9. 9. Tsai YL, Lee CC, Chen SC, Yen ZS (2006) Top-cited articles in emergency medicine. Am J Emerg Med 24: 647-654. doi:10.1016/j.ajem.2006.01.001. PubMed: 16984831.
  10. 10. Montori VM, Wilczynski NL, Morgan D, Haynes RB (2005) Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 330: 68. doi:10.1136/bmj.38336.804167.47. PubMed: 15619601.
  11. 11. Egghe L, Rousseau R, van Hooydonk G (2000) Methods for accrediting publications to authors or countries: consequences for evaluation studies. J Am Soc Inf Sci 51: 145-157. doi:10.1002/(SICI)1097-4571(2000)51:2.
  12. 12. Gastner MT, Newman ME (2004) From The Cover: Diffusion-based method for producing density-equalizing maps. Proc Natl Acad Sci U S A 101: 7499-7504. 10.1073 PubMed: 15136719.
  13. 13. Schein M, Fingerhut A (2000) Where can surgeons publish? Br J Surg 87: 261-264. doi:10.1046/j.1365-2168.2000.01419.x. PubMed: 10718791.
  14. 14. Schein M, Paladugu R, Sutija VG, Wise L (2000) What American surgeons read: a survey of a thousand Fellows of the American College of Surgeons. Curr Surg 57: 252-258. doi:10.1016/S0149-7944(00)00177-X. PubMed: 10930623.
  15. 15. Higgins JPT, Green S (2011) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 Updated March 2011. The Cochrane Collaboration. Available: www.cochrane-handbook.org.
  16. 16. The Cochrane Collaboration (2013) ochrane Reviews.
  17. 17. Rahman M, Fukui T (2003) Biomedical research productivity: factors across the countries. Int J Technol Assess Health Care 19: 249-252. PubMed: 12701955.
  18. 18. Uthman OA, Uthman MB (2007) Geography of Africa biomedical publications: an analysis of 1996-2005 PubMed papers. Int J Health Geogr 6: 46. doi:10.1186/1476-072X-6-46. PubMed: 17927837.
  19. 19. Perez-Iratxeta C, Andrade MA (2002) Worldwide scientific publishing activity. Science 297: 519. doi:10.1126/science.297.5581.519b. PubMed: 12143877.
  20. 20. Swingler GH, Volmink J, Ioannidis JP (2003) Number of published systematic reviews and global burden of disease: database analysis. BMJ 327: 1083-1084. doi:10.1136/bmj.327.7423.1083. PubMed: 14604930.
  21. 21. Waters E, Doyle J (2004) Systematic reviews of public health in developing countries are in train. BMJ 328: 585. doi:10.1136/bmj.328.7439.585-a. PubMed: 15001523.
  22. 22. Chinnock P, Siegfried N, Clarke M (2005) Is evidence-based medicine relevant to the developing world?: Systematic reviews have yet to achieve their potential as a resource for practitioners in developing countries. Evid Based Complement Alternat Med 2: 321-324. doi:10.1093/ecam/neh114. PubMed: 16136211.
  23. 23. Gisvold SE (1999) Citation analysis and journal impact factors--is the tail wagging the dog? Acta Anaesthesiol Scand 43: 971-973. doi:10.1034/j.1399-6576.1999.431001.x. PubMed: 10593457.
  24. 24. Seglen PO (1998) Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthop Scand 69: 224-229. doi:10.3109/17453679809000920. PubMed: 9703393.
  25. 25. Dumont JE (1989) The bias of citations. Trends Biochem Sci 14: 327-328. doi:10.1016/0968-0004(89)90164-3. PubMed: 2799904.