• Loading metrics

Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap

  • Julian H. Elliott ,

    Affiliations Department of Infectious Diseases, Alfred Hospital and Monash University, Melbourne, Australia, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia

  • Tari Turner,

    Affiliations School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia, World Vision Australia, Melbourne, Australia

  • Ornella Clavisi,

    Affiliation National Trauma Research Institute, Alfred Hospital, Melbourne, Australia

  • James Thomas,

    Affiliation EPPI-Centre, Institute of Education, University of London, London, England

  • Julian P. T. Higgins,

    Affiliations School of Social and Community Medicine, University of Bristol, Bristol, England, Centre for Reviews and Dissemination, University of York, York, England

  • Chris Mavergames,

    Affiliation Informatics and Knowledge Management Department, The Cochrane Collaboration, Freiburg, Germany

  • Russell L. Gruen

    Affiliations National Trauma Research Institute, Alfred Hospital, Melbourne, Australia, Department of Surgery, Monash University, Melbourne, Australia

Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap

  • Julian H. Elliott, 
  • Tari Turner, 
  • Ornella Clavisi, 
  • James Thomas, 
  • Julian P. T. Higgins, 
  • Chris Mavergames, 
  • Russell L. Gruen


  • The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action.
  • Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed.
  • We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence.
  • Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication.
  • Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.

The Bridge from Evidence to Practice

Health research promises societal benefit by making better health possible. However, there has always been a gap between research findings (what is known) and health care practice (what is done), described as the “evidence-practice” or “know-do” gap [1]. The reasons for this gap are complex [2], but it is clear that synthesising the complex, incomplete, and at times conflicting findings of biomedical research into forms that can readily inform health decision making is an essential component of the bridge from “knowing” to “doing.”

Systematic reviews (SRs) and meta-analyses have provided incalculable benefit for human health by contributing to the bridge from knowing to doing, but this benefit is limited by characteristics of the current SR enterprise [3]. The methods of SR and meta-analysis are well developed [3][5], but less progress has been achieved with the other essential component of accuracy—currency. The time from the date of the last search to SR publication is commonly over a year [6], and in an analysis of the time taken for primary study results to be incorporated into an SR, the median time from primary study publication to SR publication ranged from 2.5 to 6.5 years (Figure 1) [7]. Once published, only a minority of reviews are updated within 2 years of publication [8], and this inability to maintain currency leads to significant inaccuracy. By 2 years post-publication 23% of SRs that have not been updated will have failed to incorporate new evidence that would substantively change conclusions about the effectiveness or harms of therapies [9].

Figure 1. Time from primary study publication to incorporation in systematic review.

Analysis of 792 study reports incorporated into 73 systematic reviews across 28 high priority topics in the field of neurotrauma. Study reports were included in the analysis if they were incorporated into a systematic review relevant to one of the high priority topics and published in the period 2001–2009. Systematic reviews were included in the analysis if they were relevant to one of the high priority topics and published in the period 2001–2012. Bars represent medians and interquartile range.

Current approaches to updating SRs focus on detecting SRs most in need of updating [10],[11]. While these methods contribute to the currency of SRs, they cannot adequately reduce the inaccuracy caused by out-of-date SRs. It is often difficult to assemble an authorship team to complete prioritised updates; publication of updates takes many months, during which time the SR remains out of date and therefore potentially inaccurate; and many SRs are not designated as high priority and therefore remain out of date and inaccurate. Despite the availability of these methods and the concerted efforts of many individuals, much of the global corpus of SRs remains out of date.

Incremental advances in traditional SR updating is unlikely to lead to substantial and sustained improvements in the currency of SRs in the context of exponential growth in primary research and SRs [12]. In order to address this considerable source of SR inaccuracy, and produce evidence summaries that are both methodologically rigorous and up to date, a new approach is needed. This innovation has previously been unachievable because rigorous reviews are demanding of time and resources, and “up to date” required rapid processes, which limited the feasibility of rigorous methods. New technologies create the opportunity to resolve this trade-off and enable answers to health questions that are both methodologically rigorous and up to date. We have named this new approach to the updating of SR “living systematic review.” Our aim in proposing this approach is to address the challenges facing contemporary evidence synthesis, while retaining the strengths of SR that have been a critical underpinning of knowledge for health over the last quarter century. This approach to SR adds to rather than replaces existing methods and is particularly relevant for reviews of controlled trials in fast moving topic areas.

Living Systematic Reviews

Living systematic reviews are high quality, up-to-date online summaries of health research that are updated as new research becomes available, consistent with the vision of the pioneering Oxford Database of Perinatal Trials to “include a library of trial overviews, which will be updated when new data become available” [13]. There are four fundamental differences between conventional SR and living systematic review: publication format, work processes, author team management, and statistical methods.

The essential difference between living systematic review and conventional SR is the publication format. Instead of a conventional static SR report or update, living systematic reviews are dynamic, persistent, online-only evidence summaries, which are updated rapidly and frequently. The corollary of a living publication is three changes to SR production. First, work processes must be adapted. A literature search strategy is maintained and outputs fed continuously into an SR workflow, including continuous updating of identified studies, assessment of study quality, data extraction, meta-analysis, and SR report. Instead of the intense, sporadic effort of conventional SRs and SR updates, living systematic review requires moderate, ongoing contribution. Second, author team management must be responsive to a continuous workflow, coordinating effort over long periods of time and allowing evolution in the author team, while maintaining institutional memory.

Third, updating meta-analyses entails a reanalysis of data and, as with repeated analysis of accumulating primary trial data, an inflated rate of false-positive findings is likely if statistical tests are repeated naively [14]. Also, effect size estimates can be unstable, especially in the early stages of evidence generation [15]. These issues are relevant to all updates to meta-analyses, but are particularly important for living systematic reviews given the potential frequency of updates. Statistical monitoring of meta-analyses using formal sequential methods can control the risk of spurious findings, while achieving pre-specified power to identify a pre-specified clinically relevant magnitude of effect [16],[17]. Sequential methods are controversial in meta-analysis since they are based on testing rather than estimation, and an estimate that is widely disseminated on the basis of a significance test result may be biased. As an alternative, a Bayesian approach provides a natural framework for monitoring accumulating evidence in which prior distributions can be used to reduce the probability of falsely concluding superiority of an intervention in the early stages of the review [18], and to stabilize the meta-analysis by exploiting external information about the likely degree of statistical heterogeneity across the studies [19]. Furthermore, a Bayesian framework feeds naturally into decision making.

Production of Living Systematic Reviews

In addition to the three essential changes to review production described above, a key enabler of living systematic review is SR production efficiency. Improvements in efficiency can have profound effects on the application of health technologies. For example, a 4-log decrease in the cost of sequencing a human genome, from US$100 million to US$10,000 over 10 years [20], has catalysed a genomics revolution with profound benefits for health. In contrast, rising methodological expectations have led to an increase in the complexity and cost of SR and production timelines often in excess of 1–2 years [21]. We describe below several recent developments that have the potential to improve dramatically the efficiency of conventional SR and enable the widespread production of living systematic reviews.

Workflow and Collaboration Tools

Despite the availability of some specific tools, the efforts of most SR authors are fragmented across generic word processing, spreadsheet, email, reference management, and statistical analysis tools [22],[23]. This fragmentation hampers the production and updating of SRs, undermines the experience and engagement of SR authors, and limits the availability of process data. Growing innovation in tools and platforms [22] will enable more efficient SR production, but the right incentives and partnerships need to be in place for these innovations to translate into broadly available applications [24].


Text-mining technologies are currently being developed to improve the efficiency of SR production [25]. While experimental work encompasses many stages of the review process the most refined techniques are currently concerned with study identification [26]. Here, machine learning technologies have the potential to reduce the manual “screening” of titles and abstracts by up to 50% in new reviews [27] and more than 90% in review updates [28], greatly enhancing the efficiency of review production. Other initiatives are developing semi-automation technologies to assist with the development of search strategies, assessment of study quality, extraction of data from documents, and production of SR protocols and reports.

Data Repositories and Linked Data

Important health care questions are often the subject of evidence syntheses by multiple independent groups isolated from each other in redundant effort. The value embedded in the process and output of these parallel activities is only partially captured in discrete, static, and unstructured document-based outputs. Efforts to encourage registration of SRs can help minimise unnecessary duplication [29]. Initiatives that enable SR process and output data to be prospectively stored and reused by others are important developments that will reduce redundant effort [30],[31]. When these data are stored in structured formats using shared ontologies [32],[33], following W3C formats for linked data (RDF, OWL), unnecessary duplication can be avoided, but opportunities will also arise to draw from and contribute to the rapidly expanding world of linked open data.

Participation and the Crowd

Despite the fact that SRs are important, resource intensive, and time critical, most SRs are conducted by small academic teams, working part-time over many months. Larger authorship groups increase the efficiency of SR production [34] and the expertise available to each review group [3], but remain underutilised, particularly for high priority questions in which both the demand for evidence and engagement are high. In clinical and laboratory research, high priority questions are often addressed by collaborations of dozens or hundreds of researchers working together, but similar undertakings do not currently exist in evidence synthesis. Increasing the involvement of end users in SR production improves the outputs of SR [35][37], and “citizen science” approaches in which citation screening is crowd-sourced from a network of non-expert contributors have also been tested [38]. Efforts to identify smaller units of scholarly effort for dissemination and attribution [39][41] may be applicable to SR and encourage broader participation. These approaches to participation need to be evaluated in comparison to conventional approaches and to manage the risk of bias associated with contributors' potential competing interests.

Publication of Living Systematic Reviews

The shift to a persistent, dynamic online-only publication format will be enabled by at least two other developments. First is efficient peer and editorial review of a living document. When an ongoing search strategy identifies no new studies for inclusion, the review can be updated with the date of last search without further review. When new studies or data are identified for inclusion, but these make negligible difference to summary estimates and have no effect on review conclusions, these data can be incorporated into the review with a modest form of review (e.g., editorial only). If new studies or data are identified that result in significant changes to summary estimates or the review's conclusions, these should undergo rapid, but nonetheless robust, peer and editorial review. In the latter situations post-publication peer review can contribute to the accuracy of published reviews.

Second, living systematic reviews should be compatible with the norms of scholarly communication. Attribution of contribution to the living publication can follow existing norms, such as ICMJE criteria. Once authors' contributions no longer fulfil criteria for authorship, they can be removed from the author list and acknowledged as former contributors. Citation can also follow existing practices, including version number/date and date accessed. Finally, current practice can also inform listing in bibliographic databases with minor updates appended to an existing entry and major updates listed as a new publication.

Living Systematic Review and a New Evidence Ecosystem

Living systematic review contributes to the translation of knowledge into practice, primarily because of the contribution currency makes to accuracy and utility, but this approach to evidence synthesis also supports, and is enhanced by, associated upstream and downstream innovations.

In the growing deluge of research the noble science of systematic review resembles archeology: academic teams searching for buried artifacts and working tirelessly to reveal their true meaning. The growth in primary research and availability of diverse research outputs—protocols, trial registration, clinical study reports, and individual patient data—will continue to challenge current SR models. New methods are needed to identify datasets relevant to specific health questions and enable synthesis and insight. For example, annotation of research outputs with richer meta-data will increase the efficiency of review-specific search and screening. More importantly, publication and verification of research outputs in structured forms (e.g., using semantic technologies) will transform review-specific quality assessment and data extraction [42]. Living systematic review, together with these upstream innovations, will ensure that the potential rich insights from large datasets such as clinical study reports, individual patient data, and health system “big data” are made available for health decision making in a rigorous, efficient, and timely manner.

Living systematic review also enhances the efficiency and opportunities for knowledge translation. First, living systematic review enables “living knowledge translation,” including living guidelines, standards, policies, and decision support systems. Second, the value inside the “container” of an SR can be unlocked when the associated data are made available, particularly as open access to linked data formats. The availability of these data opens up opportunities for integration with guideline development platforms and clinical decision support systems [43] to create a new evidence “ecosystem” (Figure 2).

Figure 2. Current and emerging health knowledge ecosystems.

The current health knowledge ecosystem (inner circle) is characterized by inefficiencies that hamper the flow of knowledge from health practice through primary research, systematic review and guidelines, and finally back to impacts on health practice. The new health knowledge ecosystem that is emerging (outer circle) is characterized by a continuous flow of knowledge between efficient, living components, including the growing importance of learning health care systems, which together with traditional primary research will populate common data repositories. Living evidence services derived from these repositories, supporting living guidance and decision support systems will close a “living” health knowledge loop.


An essential link between health research and societal benefit is the transformation of millions of published research studies into accurate and usable summaries for health decision making. Many individuals and organisations are committed to this task and their efforts have improved the health of our societies, but increasing demand for accurate evidence—methodologically rigorous and up to date—is not being met. We propose living systematic review as a contribution to the methods of evidence synthesis that addresses these challenges by combining currency with rigour to enhance the accuracy and utility of health evidence. Living systematic review involves modifications to review production and publication, enabled by improved production efficiency and adherence to the norms of scholarly communication. The approach is widely applicable and although challenges remain (Table 1), feasible responses to these challenges exist. Together with emerging innovations in the reporting of primary research and in the creation and use of evidence in health systems, living systematic review contributes to a new evidence ecosystem in which health knowledge and practice are efficiently and rigorously entwined.

Table 1. Living systematic review: modifications to conventional systematic review and key challenges.

Author Contributions

Wrote the first draft of the manuscript: JHE. Contributed to the writing of the manuscript: JHE TT OC JT JH CM RLG. ICMJE criteria for authorship read and met: JHE TT OC JT JH CM RLG. Agree with manuscript results and conclusions: JHE TT OC JT JH CM RLG.


  1. 1. (2005) Bridging the “know-do” gap: meeting on knowledge translation in global health. Geneva: World Health Organization.
  2. 2. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE (2012) Knowledge translation of research findings. Implement Sci 7: 50.
  3. 3. Institute of Medicine (2011) Finding what works in health care: standards for systematic reviews. Washington (D.C.): National Academies Press.
  4. 4. Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097.
  5. 5. Methodological Expectations of Cochrane Intervention Reviews (MECIR) (2013) Available: Accessed 8 November, 2013.
  6. 6. Sampson M, Shojania KG, Garritty C, Horsley T, Ocampo M, et al. (2008) Systematic reviews can be produced and published faster. J Clin Epidemiol 61: 531–536.
  7. 7. Bragge P, Clavisi O, Turner T, Tavender E, Collie A, et al. (2011) The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol 11: 92.
  8. 8. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, et al. (1998) Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 280: 278–280.
  9. 9. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, et al. (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147: 224–233.
  10. 10. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al.. (2008) When and how to update systematic reviews. Cochrane Database Syst Rev: MR000023.
  11. 11. Pattanittum P, Laopaiboon M, Moher D, Lumbiganon P, Ngamjarus C (2012) A comparison of statistical methods for identifying out-of-date systematic reviews. PLoS ONE 7: e48894.
  12. 12. Bastian H, Glasziou P, Chalmers I (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7: e1000326.
  13. 13. Chalmers I (1986) Electronic publications for updating controlled trial reviews. Lancet 328: 287.
  14. 14. Berkey CS, Mosteller F, Lau J, Antman EM (1996) Uncertainty of the time of first significance in random effects cumulative meta-analysis. Control Clin Trials 17: 357–371.
  15. 15. Ioannidis J, Lau J (2001) Evolution of treatment effects over time: empirical insight from recursive cumulative metaanalyses. Proc Natl Acad Sci U S A 98: 831–836.
  16. 16. Wetterslev J, Thorlund K, Brok J, Gluud C (2008) Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol 61: 64–75.
  17. 17. Higgins JP, Whitehead A, Simmonds M (2011) Sequential methods for random-effects meta-analysis. Stat Med 30: 903–921.
  18. 18. Spieglehalter D (2004) Incorporating Bayesian ideas into health-care evaluation. Statist Sci 19: 156–174.
  19. 19. Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JP (2012) Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol 41: 818–827.
  20. 20. Wetterstrand K (2013) DNA Sequencing costs: data from the NHGRI Genome Sequencing Program (GSP). Available: Accessed 11 July 2013.
  21. 21. Tricco AC, Brehaut J, Chen MH, Moher D (2008) Following 411 Cochrane protocols to completion: a retrospective cohort study. PLoS ONE 3: e3684.
  22. 22. Elliott J (2012) Exploiting innovations in technology to improve the efficiency of review production. 20th Cochrane Colloquium. Auckland.
  23. 23. Ciapponi A, Glujovsky D (2012) Survey among Cochrane authors about early stages of systematic reviews. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012 Auckland, New Zealand
  24. 24. Elliott J, Mavergames C, Becker L, Meerpohl J, Thomas J, et al. (2013) The efficient production of high quality evidence reviews is important for the public good. BMJ 346: f846.
  25. 25. Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research Synthesis Methods 2: 1–14.
  26. 26. Thomas J, Brunton J (2010) EPPI-Reviewer 4.0: software for research synthesis. London: Social Science Research Unit, Institute of Education.
  27. 27. Wallace B, Trikalinos T, Lau J, Brodley C, Schmid C (2010) Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 11: 55.
  28. 28. Wallace BC, Small K, Brodley CE, Lau J, Schmid CH, et al. (2012) Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med 14: 663–669.
  29. 29. Booth A, Clarke M, Dooley G, Ghersi D, Moher D, et al. (2013) PROSPERO at one year: an evaluation of its utility. Systematic Reviews 2: 4.
  30. 30. (2013) SRDR: Systematic Review Data Repository. Available: Accessed 8 November 2013.
  31. 31. (2013) Cochrane Register of Studies. Available: Accessed 8 November 2013.
  32. 32. (2013) Cochrane Reviews Ontology. Available: Accessed 8 November 2013.
  33. 33. (2013) OCRe: The Ontology of Clinical Research. Available: Accessed 8 November 2013.
  34. 34. McGee R, Willis N, Jones A, Higgins G, Craig J, et al.. (2012) Factors associated with the timely completion of a Cochrane review. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand
  35. 35. Sung NS, Crowley WF, Genel M, Salber P, Sandy L, et al. (2003) Central challenges facing the national clinical research enterprise. JAMA 289: 1278–1287.
  36. 36. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, et al. (2005) Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy 10 Suppl 135–48.
  37. 37. Colby DC, Quinn BC, Williams CH, Bilheimer LT, Goodell S (2008) Research glut and information famine: making research evidence more useful for policymakers. Health Aff (Millwood) 27: 1177–1182.
  38. 38. Noel-Storr A, Struthers C, Huckvale C, Dooley G, McShane R (2012) ALOIS, ScreenToGo and the Cochrane Register of Studies: using mobile technology to engage and involve consumers in the work of the Cochrane Collaboration. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand.
  39. 39. What is the Human Variome Project? Nat Genet 39: 423.
  40. 40. Giardine B, Borg J, Higgs DR, Peterson KR, Philipsen S, et al. (2011) Systematic documentation and analysis of human genetic variation in hemoglobinopathies using the microattribution approach. Nat Genet 43: 295–301.
  41. 41. Mons B, van Haagen H, Chichester C, Hoen PB, den Dunnen JT, et al. (2011) The value of data. Nat Genet 43: 281–283.
  42. 42. US Food and Drug Administration (2013) Study data standards for regulatory submissions: position statement. Available: Accessed 8 November 2013.
  43. 43. Brandt L, Kristiansen A, Guyatt G, Vandvik P (2012) Providing likeable and understandable guidelines using GRADE in the EMR with direct links to individual patient data. Phase 1. In: Proceedings of the 9th Guidelines International Network Conference; 22–25 August 2012Berlin, Germany.