BM has received research grants from Intel, Tunstall, Chief Scientist Office, and NHS Lothian to explore the use of Telehealthcare in long-term conditions. AS is on the
Aziz Sheikh and colleagues report the findings of their systematic overview that assessed the impact of eHealth solutions on the quality and safety of health care.
There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.
We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.
There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. In addition, there is a lack of robust research on the risks of implementing these technologies and their cost-effectiveness has yet to be demonstrated, despite being frequently promoted by policymakers and “techno-enthusiasts” as if this was a given. In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.
There is considerable international interest in exploiting the potential of digital health care solutions, often referred to as eHealth—the use of information and communication technologies—to enhance the quality and safety of health care. Often accompanied by large costs, any large-scale expenditure on eHealth—such as electronic health records, picture archiving and communication systems, ePrescribing, associated computerized provider order entry systems, and computerized decision support systems—has tended to be justified on the grounds that these are efficient and cost-effective means for improving health care. In 2005, the World Health Assembly passed an eHealth resolution (WHA 58.28) that acknowledged, “eHealth is the cost-effective and secure use of information and communications technologies in support of health and health-related fields, including health-care services, health surveillance, health literature, and health education, knowledge and research,” and urged member states to develop and implement eHealth technologies. Since then, implementing eHealth technologies has become a main priority for many countries. For example, England has invested at least £12.8 billion in a National Programme for Information Technology for the National Health Service, and the Obama administration in the United States has committed to a US$38 billion eHealth investment in health care.
Despite the wide endorsement of and support for eHealth, the scientific basis of its benefits—which are repeatedly made and often uncritically accepted—remains to be firmly established. A robust evidence-based perspective on the advantages on eHealth could help to suggest priority areas that have the greatest potential for benefit to patients and also to inform international eHealth deliberations on costs. Therefore, in order to better inform the international community, the authors systematically reviewed the published systematic review literature on eHealth technologies and evaluated the impact of these technologies on the quality and safety of health care delivery.
The researchers divided eHealth technologies into three main categories: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. Then, implementing methods based on those developed by the Cochrane Collaboration and the NHS Service Delivery and Organisation Programme, the researchers used detailed search strategies and maps of health care quality, safety, and eHealth interventions to identify relevant systematic reviews (and related theoretical, methodological, and technical material) published between 1997 and 2010. Using these techniques, the researchers retrieved a total of 46,349 references from which they identified 108 reviews. The 53 reviews that the researchers finally selected (and critically reviewed) provided the main evidence base for assessing the impact of eHealth technologies in the three categories selected.
In their systematic review of systematic reviews, the researchers included electronic health records and picture archiving communications systems in their evaluation of category 1, computerized provider (or physician) order entry and e-prescribing in category 2, and all clinical information systems that, when used in the context of eHealth technologies, integrate clinical and demographic patient information to support clinician decision making in category 3.
The researchers found that many of the clinical claims made about the most commonly used eHealth technologies were not substantiated by empirical evidence. The evidence base in support of eHealth technologies was weak and inconsistent and importantly, there was insubstantial evidence to support the cost-effectiveness of these technologies. For example, the researchers only found limited evidence that some of the many presumed benefits could be realized; importantly, they also found some evidence that introducing these new technologies may on occasions also generate new risks such as prescribers becoming over-reliant on clinical decision support for e-prescribing, or overestimate its functionality, resulting in decreased practitioner performance.
The researchers found that despite the wide support for eHealth technologies and the frequently made claims by policy makers when constructing business cases to raise funds for large-scale eHealth projects, there is as yet relatively little empirical evidence to substantiate many of the claims made about eHealth technologies. In addition, even for the eHealth technology tools that have proven to be successful, there is little evidence to show that such tools would continue to be successful beyond the contexts in which they were originally developed. Therefore, in light of the lack of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, the authors say that future eHealth technologies should be evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle, and include socio-technical factors to maximize the likelihood of successful implementation and adoption in a given context. Furthermore, it is equally important that eHealth projects that have already been commissioned are subject to rigorous, multidisciplinary, and independent evaluation.
Please access these websites via the online version of this summary at
The authors' broader study is: Car J, Black A, Anandan C, Cresswell K, Pagliari C, McKinstry B, et al. (2008) The Impact of eHealth on the Quality and Safety of Healthcare. Available at:
More information is available on the World Health Assembly
The World Health Organization provides information at
The European Commission provides Information on
More information is provided on
Implementations of potentially transformative eHealth technologies are currently underway internationally, often with significant impact on national expenditure. England has, for example, invested at least £12.8 billion in a National Programme for Information Technology (NPfIT) for the National Health Service, and the Obama administration in the United States (US) has similarly committed to a US$38 billion eHealth investment in health care
Moving this agenda forward thus requires a scientifically informed perspective. However, there remains a disparity between the evidence-based principles that underpin health care generally and the political, pragmatic, and commercial drivers of decision making in the commissioning of eHealth tools and services. Obtaining an evidence-informed perspective on the current situation may serve to ground unrealistic expectations that might hinder longer-term progress within the field, help to suggest priorities by identifying areas with greatest potential for benefit, and also inform ongoing deliberations on eHealth implementations that are being considered internationally.
To inform these global deliberations, we systematically reviewed the preexisting systematic review literature on eHealth technologies and their impact on the quality and safety of health care delivery. We synthesised and contextualised our findings with the broader theoretical and methodological literature with a view to producing a comprehensive and accessible overview of the field. We present here a synopsis and updated version of a much larger recently published report covering the period 1997–2010
Systematic reviews of reviews have been particularly advocated to inform policy, clinical, and research deliberations by providing an evidence-based summary of inter-related technologies
Inherent difficulties associated with systematic reviews of health care organisation and delivery intervention include the considerable effort required at the outset to facilitate their conduct
For quality and safety considerations, we identified existing taxonomies and frameworks to facilitate this conceptual mapping exercise, which helped to delineate the scope of our work. For the field of eHealth, we drew from existing team members' conceptual and empirical work to aid our construction of a conceptual map for eHealth technologies
We drew on established Cochrane-based systematic review principles to search for relevant systematic reviews. An inclusive string of MeSH and free terms (
On the basis of the areas identified for prioritisation, we developed a detailed list of interventions that were to be included/excluded (
All systematic reviews having been identified as potentially suitable were assessed for inclusion by two independent reviewers, with arbitration by a third reviewer if necessary. Data from systematic reviews meeting the above criteria, henceforth referred to as “reviews,” were independently critically reviewed by two reviewers, and relevant data were abstracted. Systematic reviews not primarily concerned with assessing impact on patients, professionals, or the organisation, but nonetheless intervention focused, were drawn on to provide additional contextual information. These supplementary systematic reviews (henceforth referred to as “supplementary reviews”) were not subjected to formal critical appraisal.
Critical appraisal was undertaken using an adapted version of the Critical Appraisal Skills Programme (CASP) tool for systematic reviews
A standard approach was taken for each of the eHealth technologies of interest. Definitions were first clarified and then the individual use and broader scope for deployment conceptualised. Juxtaposing this with the aforementioned conceptual maps of the fields of eHealth, quality and safety provided a literature-based framework for delineating the principal theorised benefits and risks associated with each intervention. We used this framework to guide synthesis of the empirically demonstrated benefits and risks of implementing eHealth technologies.
The body of literature identified was too diverse to allow quantitative synthesis of empirical evidence and we therefore undertook a narrative synthesis. This synthesis involved initially describing the technologies and outcomes studies using the above-described framework for each of the included reviews, which was followed by developing a summary of our assessment of and the key findings from each review (
Our searches retrieved a total of 46,349 references from which we selected a total of 108 reviews for inclusion (
Lead Author and Year | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | Q12 | Q13 | Q14 | Q15 | Totala |
Ammenwerth 2008 | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 24 |
Anderson 1997 | 2 | 2 | 2 | 1 | 2 | 1 | 1 | 2 | 1 | 2 | 1 | 2 | 1 | 1 | 2 | 23 |
Balas 2004 | 1 | 1 | 2 | 1 | 2 | 1 | 2 | 1 | 2 | 2 | 1 | 2 | 1 | 2 | 1 | 22 |
Bennett 2003 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 18 |
Bryan 2008 | 1 | 2 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 19 |
Charvet-Protat 1998 | 1 | 2 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 12 |
Chatellier 1998 | 2 | 2 | 2 | 0 | 1 | 1 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 21 |
Chaudhry 2006 | 1 | 1 | 2 | 2 | 0 | 1 | 1 | 1 | 2 | 1 | 0 | 2 | 0 | 2 | 2 | 18 |
Clamp 2005 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 2 | 1 | 17 |
Delpierre 2004 | 1 | 1 | 0 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 1 | 1 | 1 | 1 | 13 |
Dexheimer 2008 | 2 | 1 | 1 | 1 | 2 | 1 | 2 | 1 | 2 | 2 | 1 | 2 | 1 | 0 | 0 | 19 |
Durieux 2008 | 2 | 2 | 1 | 2 | 2 | 1 | 2 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 23 |
Eslami 2007 | 2 | 2 | 1 | 1 | 0 | 1 | 2 | 2 | 1 | 1 | 0 | 1 | 1 | 2 | 1 | 18 |
Eslami 2008 | 2 | 2 | 1 | 1 | 0 | 1 | 2 | 2 | 1 | 1 | 0 | 1 | 1 | 2 | 1 | 18 |
Eslami 2009 | 2 | 2 | 1 | 0 | 1 | 2 | 2 | 0 | 1 | 0 | 0 | 2 | 1 | 1 | 1 | 16 |
Fitzmaurice 1998 | 2 | 1 | 0 | 2 | 2 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 12 |
Garg 2005 | 1 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 20 |
Georgiou 2007 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 2 | 2 | 1 | 0 | 2 | 1 | 2 | 1 | 21 |
Hayward 2009 | 2 | 0 | 0 | 1 | 0 | 1 | 2 | 1 | 2 | 1 | 0 | 2 | 1 | 1 | 1 | 15 |
Hender 2000 | 1 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 11 |
Heselmans 2009 | 2 | 2 | 2 | 2 | 2 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 22 |
Hider 2002 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 21 |
Irani 2009 | 2 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 18 |
Jamal 2009 | 1 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 2 | 1 | 0 | 1 | 1 | 0 | 0 | 16 |
Jerant 2000 | 1 | 2 | 0 | 0 | 2 | 1 | 1 | 1 | 2 | 1 | 0 | 2 | 1 | 1 | 1 | 16 |
Kaushal 2003 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 2 | 0 | 2 | 1 | 1 | 0 | 0 | 0 | 16 |
Mador 2009 | 2 | 2 | 1 | 2 | 2 | 2 | 2 | 2 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 20 |
Mitchell 2001 | 1 | 1 | 1 | 2 | 2 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 2 | 1 | 17 |
Montgomery 1998 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 14 |
Niazkhani 2009 | 2 | 2 | 2 | 1 | 0 | 1 | 1 | 2 | 2 | 1 | 1 | 2 | 1 | 2 | 2 | 22 |
Oren 2003 | 1 | 2 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 2 | 2 | 14 |
Pearson 2009 | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 23 |
Poissant 2005 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 1 | 25 |
Randell 2007 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 17 |
Reckmann 2009 | 2 | 2 | 1 | 2 | 0 | 2 | 1 | 1 | 1 | 1 | 0 | 2 | 1 | 2 | 2 | 20 |
Rothschild 2004 | 1 | 1 | 1 | 1 | 0 | 1 | 2 | 1 | 2 | 1 | 0 | 2 | 1 | 1 | 1 | 16 |
Schedlbauer 2009 | 1 | 2 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 1 | 22 |
Shachak 2009 | 2 | 1 | 1 | 0 | 0 | 0 | 2 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 15 |
Shamliyan 2008 | 1 | 2 | 1 | 1 | 1 | 2 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 13 |
Shebl 2007 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 17 |
Shekelle 2006 | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 2 | 1 | 2 | 1 | 20 |
Shekelle 2009 | 1 | 1 | 2 | 1 | 0 | 2 | 1 | 1 | 2 | 1 | 1 | 2 | 2 | 2 | 2 | 21 |
Shiffman 1999 | 1 | 1 | 2 | 0 | 0 | 2 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 14 |
Shojania 2009 | 1 | 2 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 1 | 1 | 1 | 0 | 22 |
Sintchenko 2007 | 1 | 1 | 1 | 0 | 1 | 1 | 2 | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 17 |
Smith 2007 | 1 | 2 | 0 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 0 | 2 | 1 | 2 | 1 | 17 |
Tan 2005 | 2 | 2 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 1 | 0 | 1 | 0 | 0 | 0 | 19 |
Thompson 2009 | 2 | 1 | 1 | 1 | 0 | 0 | 1 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 9 |
Uslu 2008 | 2 | 1 | 0 | 0 | 2 | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 16 |
van Rosse 2009 | 2 | 2 | 1 | 1 | 2 | 2 | 2 | 0 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | 24 |
Wolfstadt 2008 | 1 | 1 | 0 | 0 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 13 |
Wong 2010 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 22 |
Yourman 2008 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 2 | 2 | 1 | 1 | 1 | 0 | 0 | 16 |
Maximum total score of 30, each question
The EHR is a complex construct encompassing digitised health care records and the information systems into which these are embedded
The theorised benefits and risks associated with EHRs are largely related to data storage and management functionality. These functions include increased accessibility, legibility, “searchability,” manipulation, transportation, sharing, and preservation of electronic data. Consequently, improved organisational efficiency and secondary uses of data are typically amongst the most commonly expected benefits. However, digitising health records can also introduce new risks. Paper persistence can result in threats to patient safety, unsecured networks can lead to illegitimate access, and increased time needed to document and retrieve patient data can result in organisational inefficiency. Moreover, the dynamic of the patient-provider interaction could become less personal with the intrusion by the computer as a “third person” in the consultation. If anticipated benefits are not realised, this may therefore mean that ultimately the EHR may be rendered cost-ineffective.
Although a number of reviews purporting to assess the impact of EHRs were found, many of these in fact investigated auxiliary systems such as CDSS, CPOE, and ePrescribing. As a result, most of the impacts assessed were more relevant to these other systems. We found only anecdotal evidence of the fundamental expected benefits and risks relating to the organisational efficiency resulting from the storage and management facilities within the EHR and thus the potential for secondary uses (
Benefits | |||||||
Review ID | Data Security | Legibility | Accessibility | Completeness | Comprehensiveness | Efficiency | Secondary Uses |
Clamp 2005 | N/A | + | + | + | N/A | +/- | N/A |
Irani 2009 | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
Jamal 2009 | N/A | N/A | N/A | +/++ | N/A | + | N/A |
Mador 2009 | N/A | N/A | ++ | N/A | N/A | +/- | +/- |
Mitchell 2001 | +/- | + | N/A | + | + | +/+ + | N/A |
Poissant 2005 | N/A | N/A | N/A | N/A | N/A | +/+ + | N/A |
Shachak 2009 | N/A | N/A | N/A | + | +/++ | N/A | N/A |
Shekelle 2006 | N/A | + | N/A | + | + | +/+ + | N/A |
Shekelle 2009 | N/A | N/A | N/A | N/A | N/A | +/- | + |
Thompson 2009 | N/A | N/A | N/A | N/A | N/A | + | N/A |
Uslu 2008 | N/A | + | + | N/A | N/A | ++ | N/A |
Evidence of benefits: N/A, not assessed; +/-, none; +, weak; +/++, weak to moderate; ++, moderate.
Risks | |||||
Review ID | Paper Persistence | Patient Disengagement | Insecure Data | Increased Time | Increased Costs |
Clamp 2005 | N/A | - | N/A | - | - |
Irani 2009 | N/A | +/- | N/A | N/A | N/A |
Jamal 2009 | N/A | N/A | N/A | +/- | +/- |
Mador 2009 | N/A | N/A | N/A | - | N/A |
Mitchell 2001 | N/A | - | - | - | - |
Poissant 2005 | N/A | N/A | N/A | - - | N/A |
Shachak 2009 | - | -/- - | N/A | N/A | N/A |
Shekelle 2006 | N/A | N/A | N/A | - | - |
Shekelle 2009 | N/A | N/A | N/A | - | - |
Thompson 2009 | N/A | N/A | N/A | +/- | N/A |
Uslu 2008 | N/A | N/A | N/A | N/A | +/- |
Evidence of risks: N/A, not assessed; +/-, none; -, weak; -/- -, weak to moderate; --, moderate.
PACS are clinical information systems used for the acquisition, archival, and post-processing distribution of digital images. An image must either be directly acquired using digital radiography or be digitised from a paper-based format. It can be stored using an electronic, magnetic, or optical storage device. PACS can be integrated or interface with EHRs and CDSSs, or be stand-alone systems.
Much like the digitisation of health records, certain benefits – i.e., accessibility, image (rather than data) quality, searchability, transportation, sharing, and preservation – can be expected from the digitisation of medical images, which were previously film based. Again, certain improvements to organisational efficiency should in theory follow on from this digitisation, including time-savings, continuity of care, and ability to remotely view images. Conversely, digitising medical images can lead to decreased organisational efficiency if increased time is needed for retrieval owing to the difficulties associated with navigating a new or cumbersome system or in the event of system downtime. If the potential benefits of a PACS implementation are not realised, high expenditure might render the application cost-inefficient.
Although only three reviews on PACS were located, in contrast to the reviews on EHRs the impacts assessed in reviews of PACS were more congruent with the theoretically derived benefits (
Benefits | ||||||
Review ID | Data Integrity | Image Resolution | Image Access | Cost Savings | Time Savings | Diagnostic Accuracy |
Anderson 1997 | + | +/- | + | +/- | + | +/- |
Charvet-Protat 1998 | + | N/A | + | + | + | N/A |
Clamp 2005 | + | N/A | + | +/- | + | + |
Evidence of benefits: N/A, not assessed; +/-, none; +, weak; +/++, weak to moderate; ++, moderate.
Risks | ||||
Review ID | Film Persistence | Record Loss | Increased Time | Increased Costs |
Anderson 1997 | +/- | +/- | +/- | - |
Charvet-Protat 1998 | N/A | N/A | +/- | - |
Clamp 2005 | N/A | N/A | +/- | - |
Evidence of risks: N/A, not assessed; +/-, none; -, weak; -/- -, weak to moderate; --, moderate.
CPOE systems are typically used by clinicians to enter, modify, review, and communicate orders; and return results for laboratory tests, radiological images, and referrals (for pharmacy see ePrescribing)
We found relatively few reviews on CPOE that were not focused primarily on the ordering of medications, rather than the ordering of laboratory tests and medical images. Within the reviews, we found that what had been empirically evaluated generally mirrored the theorised impacts (
Benefits | |||||
Review ID | Resource Utilisation | Indicated Care | Patient Outcomes | Cost Savings | Time Savings |
Chaudry 2006 | +/+ + | +/++ | +/- | +/- | +/- |
Garg 2005 | +/+ + | +/++ | +/- | + | N/A |
Georgiou 2007 | +/- | + | +/- | + | +/- |
Jamal 2009 | + | + | +/- | N/A | N/A |
Niyazkhani 2009 | N/A | + | N/A | N/A | +/+ + |
Poissant 2005 | N/A | N/A | N/A | N/A | +/+ + |
Rothschild 2004 | +/+ + | + | +/- | + | + |
Shekelle 2006 | +/+ + | +/++ | +/- | + | + |
Evidence of benefits: N/A, not assessed; +/-, none; +, weak; +/++, weak to moderate; ++, moderate.
Risks | ||||
Review ID | Increased Time | Interruptions | Increased Costs | Workarounds |
Chaudry 2006 | +/- | N/A | +/- | N/A |
Garg 2005 | N/A | - | +/- | N/A |
Georgiou 2007 | - | N/A | +/- | N/A |
Jamal 2009 | N/A | N/A | N/A | N/A |
Niyazkhani 2009 | - | -- | +/- | - |
Poissant 2005 | - - | - | N/A | N/A |
Rothschild 2004 | +/- | N/A | +/- | N/A |
Shekelle 2006 | +/- | N/A | +/- | N/A |
Evidence of risks: N/A, not assessed; +/-, none; -, weak; -/- -, weak to moderate; --, moderate.
ePrescribing refers to clinical information systems that are used by clinicians to enter, modify, review, and output or communicate medication prescriptions. This term thus includes stand-alone CDSSs for prescribing purposes
ePrescribing was the most commonly studied intervention amongst the included reviews. Consequently, we found multiple papers covering most of the theorised impacts (
Benefits | |||||||
Reference ID | Surrogate Outcomes | Guideline Adherence | Safer Prescribing | Communication | Patient Outcomes | Resource/Cost Savings | Time Savings |
Ammenwerth 2008 | Pot. ADEs +/+ + | N/A | MEs ++ | N/A | ADEs + | N/A | N/A |
Bryan 2008 | + | + | N/A | N/A | +/- | + | N/A |
Chatellier 1998 | +/++ | N/A | ++/+ | N/A | Death +/-Haemorrhage +/-Thromboembolic events +/- | N/A | N/A |
Clamp 2005 | ++ | + | MEs ++ | + | ADEs + | + | + |
Delpierre 2004 | +/- | + | MEs+ | N/A | +/- | N/A | N/A |
Duriex 2008 | +/++ | + | +/++ | N/A | Death +/- | +/++ | N/A |
Eslami 2007 | + | +/+ + | +/- | N/A | +/- | + | + |
Eslami 2008 | + | +/+ + | + | + | +/- | + | + |
Eslami 2009 | + | + | N/A | N/A | +/- | N/A | N/A |
Fitzmaurice 1998 | + | N/A | + | N/A | + | N/A | N/A |
Garg 2005 | +/++ | + | +/- | N/A | +/- | +/- | N/A |
Hider 2002 | +/++ | +/+ + | + | +/+ + | + | + | N/A |
Jamal 2009 | + | +/++ | + | N/A | +/- | + | N/A |
Mitchell 2001 | + | N/A | N/A | N/A | +/- | + | + |
Mollon 2009 | +/++ | N/A | N/A | N/A | +/- | +/- | N/A |
Niyazkhani 2009 | + | N/A | N/A | +/++ | N/A | N/A | +/++ |
Poissant 2005 | N/A | N/A | N/A | N/A | N/A | N/A | +/- |
Rothschild 2004 | +/++ | + | MEs ++ | N/A | ADEs + | + | + |
Schedlbauer 2009 | +/++ | + | MEs ++ | N/A | Renal ADEs+Falls+ | + | N/A |
Shamliyan 2008 | + | N/A | MEs ++ | N/A | ADEs + | N/A | N/A |
Shekelle 2006 | +/+ + | N/A | MEs + | N/A | ADEs + | +/- | N/A |
Shiffman 1999 | N/A | +/+ + | N/A | N/A | +/- | N/A | N/A |
Shojania 2009 | + | + | N/A | N/A | + | N/A | N/A |
Sintchenko 2007 | ++ | + | N/A | N/A | Death +/-ADEs + | N/A | N/A |
Tan 2005 | + | N/A | MEs + | N/A | +/- | +/- | + |
Van Rosse 2009 | +/+ + | N/A | ++ | + | Death +/-ADEs +/- | N/A | + |
Wolfstadt 2008 | N/A | N/A | N/A | N/A | ADEs +/- | N/A | N/A |
Yourman 2008 | + | + | N/A | N/A | +/- | +/- | N/A |
Evidence of benefits: N/A, not assessed; +/-, none; +, weak; +/++, weak to moderate; ++, moderate.
Risks | |||
Reference ID | Patient Harm | Increased Time | Increased Costs |
Ammenwerth 2008 | +/- | N/A | N/A |
Bryan 2008 | +/- | N/A | N/A |
Chatellier 1998 | +/- | N/A | N/A |
Clamp 2005 | +/- | - | +/- |
Delpierre 2004 | +/- | N/A | N/A |
Durieux 2008 | +/- | N/A | +/- |
Eslami 2007 | +/- | -/- - | - |
Eslami 2008 | +/- | -/- - | - |
Eslami 2009 | +/- | N/A | N/A |
Fitzmaurice 1998 | N/A | N/A | N/A |
Garg 2005 | +/- | N/A | N/A |
Hider 2002 | +/- | N/A | N/A |
Jamal 2009 | +/- | N/A | N/A |
Mitchell 2001 | +/- | +/- | +/- |
Mollon 2009 | +/- | N/A | +/- |
Niyazkhani 2009 | N/A | -/-- | N/A |
Poissant 2005 | N/A | -/- - | N/A |
Rothschild 2004 | N/A | +/- | +/- |
Schedlbauer 2009 | +/- | N/A | +/- |
Shamliyan 2008 | +/- | N/A | N/A |
Shekelle 2006 | +/- | N/A | +/- |
Shiffman 1999 | +/- | N/A | N/A |
Shojania 2009 | +/- | N/A | N/A |
Sintchenko 2007 | +/- | N/A | N/A |
Tan 2005 | N/A | +/- | +/- |
Van Rosse 2009 | +/- | +/- | N/A |
Wolfstadt 2008 | N/A | N/A | N/A |
Yourman 2008 | N/A | N/A | +/- |
Evidence of risks: N/A, not assessed; +/-, none; -, weak; -/- -, weak to moderate; --, moderate.
CDSSs are, when used in the context of eHealth technologies, clinical information systems that integrate clinical and demographic patient information to provide support for decision making by clinicians
In principle, the fundamental impact of CDSSs should be improved clinical decision making. This improvement should, in turn, lead to improved practitioner performance in a variety of care activities (e.g., provision of preventive care, diagnosis, disease management) and ways in which these care activities are delivered (e.g., more evidence-based or guideline adherent decisions). These systems should also be able to help address disparities in care by facilitating standardisation, especially when part of an EHR, PACS, CPOE, or ePrescribing system. Improved practitioner performance should result in a variety of beneficial impacts depending on the care activity targeted (e.g., increased immunisation rates, reduced resource utilisation, more timely diagnosis) or better disease control. In addition, if practitioner's performance is directly related to patient outcomes, then these too should improve. The main theorised risks relating to the use of CDSSs include a potential decline in practitioner performance due to deskilling or flawed system design, and related threats to patient safety.
Actual improved practitioner performance rather than just behaviour change in general was supported by only weak evidence (
Benefits | ||||
Reference ID | Indicated Care | Guideline Adherence | Surrogate Outcomes | Patient Outcomes |
Balas 2004 | + | ++ | +/++ | +/- |
Bryan 2008 | + | + | + | +/- |
Chaudhry 2006 | + | ++ | + | +/- |
Delpierre 2004 | ++ | +/- | + | +/- |
Dexheimer, 2008 | ++ | + | + | +/- |
Garg 2005 | +/++ | ++ | +/++ | +/- |
Hayward 2009 | +/- | N/A | +/- | +/- |
Heselmans 2009 | N/A | +/- | +/- | +/- |
Jamal 2009 | +/++ | ++ | + | +/- |
Jerant, 2000 | ++ | + | + | +/- |
Montgomery 1998 | + | N/A | + | +/- |
Randell 2007 | +/- | +/- | +/- | +/- |
Shekelle 2006 | ++ | ++ | +/++ | +/- |
Shiffman 1999 | + | +/++ | + | +/- |
Shojania 2009 | +/++ | ++ | +/- | |
Sintchenko 2007 | + | + | +/++ | +/- |
Smith 2007 | N/A | N/A | +/- | +/- |
Tan 2009 | +/- | N/A | +/- | +/- |
Evidence of benefits: N/A, not assessed; +/-, none; +, weak; +/++, weak to moderate; ++, moderate.
Risks | ||
Reference ID | Practitioner performance | Patient outcomes |
Balas 2004 | N/A | +/- |
Bryan 2008 | N/A | +/- |
Chaudhry 2006 | N/A | +/- |
Delpierre 2004 | +/- | +/- |
Dexheimer, 2008 | N/A | +/- |
Garg 2005 | N/A | +/- |
Hayward 2009 | N/A | +/- |
Heselmans 2009 | N/A | N/A |
Jamal 2009 | +/- | +/- |
Jerant, 2000 | +/- | +/- |
Montgomery 1998 | +/- | - |
Randell 2007 | - | +/- |
Shekelle 2006 | N/A | +/- |
Shiffman 1999 | +/- | +/- |
Shojania 2009 | N/A | +/- |
Sintchenko 2007 | +/- | +/- |
Smith 2007 | N/A | +/- |
Tan 2009 | +/- | +/- |
Evidence of risks: N/A, not assessed; +/-, none; -, weak; -/- -, weak to moderate; --, moderate.
Our systematic review of systematic reviews on the impact of eHealth has demonstrated that many of the clinical claims made about the most commonly deployed eHealth technologies cannot be substantiated by the empirical evidence. Overall, the evidence base in support of these technologies is weak and inconsistent, which highlights the need for more considered claims, particularly in relation to the patient-level benefits, associated with these technologies. Also of note is that we found virtually no evidence in support of the cost-effectiveness claims (
This work is characterised by a number of strengths and limitations, which need to be considered when interpreting this work. Strengths include the multifaceted approach to the identification of systematic reviews and the synthesis of this body of evidence. Juxtaposing the conceptual maps of the fields of quality, safety, and eHealth permitted us to produce a comprehensive framework for assessing the impact of these technologies in an otherwise poorly ordered discipline. In addition, reflecting on methodological considerations and socio-technical factors enabled us to produce an overview that is sensitive to the intricacies of the discipline.
Given the poor indexing of this literature and the fact that our searches were centred on English-language databases, there is the possibility that we may have missed some systematic reviews. Our use of a novel, multimethod approach may be criticised as being less rigorous than a conventional systematic review in that we were not in a position to appraise individual primary studies. These more novel methods of synthesis are less well developed and employed, and therefore less evaluated
At the most elementary level, the literature that constitutes the evidence base is poorly referenced within bibliographic databases reflecting the nonstandard usage of terminology and lack of consensus on a taxonomy relating to eHealth technologies
Our greatest cause for concern was the weakness of the evidence base itself. A strong evidence base is characterised by quantity, quality, and consistency. Unfortunately, we found that the eHealth evidence base falls short in all of these respects. In addition, relative to the number of eHealth implementations that have taken place, the number of evaluations is comparatively small. Apart from several barriers and challenges that impede the evaluation of eHealth interventions per se
Another commonly criticised element of the existing evidence base is its utility
A handful of high-profile primary studies demonstrating the greatest evidence of benefit often serve as exemplars of the transformative power of clinical information systems
Keeping in mind the above, the maturation of evaluation is vital to the success of eHealth
We found an important literature pertaining to the design and deployment aspects of eHealth technologies. This literature is central to understanding why some interventions succeed and others fail (or being judged as such). At the individual level, “human factors” play an important role in the design of an intervention, determining usability and ultimately adoption
It is clear that there is now a large volume of work studying the impact of eHealth on the quality and safety of health care. This might be seen as setting a firm foundation for realising the potential benefits of eHealth. However, although seminal reports on quality and safety of health care invariably point to eHealth as one of the main vehicles for driving forwards sweeping improvements
Our major finding from reviewing the literature is that empirical evidence for the beneficial impact of most eHealth technologies is often absent or, at best, only modest. While absence of evidence does not equate with evidence of ineffectiveness, reports of negative consequences indicate that evaluation of risks – anticipated or otherwise – is essential. Clinical informatics should be no less concerned with safety and efficacy than the pharmaceutical industry. Given this, there is a pressing need for further evaluations before substantial sums of money are committed to large-scale national deployments under the auspices of improving health care quality and/or safety.
Promising technologies, unless properly evaluated with results fed back into development, might not “mature” to the extent that is needed to realise their potential when deployed in everyday clinical settings. The paradox is that while the number of eHealth technologies in health care is growing, we still have insufficient understanding of how and why such interventions do or do not work
Finally, it is equally important that deployments already commissioned are subject to rigorous, multidisciplinary, and independent evaluations. In particular, we should take every opportunity to learn from the largest eHealth commissioning and deployment project in health care in the world – the £12.8 billion NPfIT and the at least equally ambitious national programme that has recently begun in the US
Critical appraisal form.
(0.05 MB DOC)
Characteristics and main findings of “reviews.”
(0.42 MB DOC)
Search strategy (databases, string, and filters).
(0.05 MB DOC)
Intervention inclusion and exclusion criteria.
(0.03 MB DOC)
We are grateful to the Independent Project Steering Committee comprising Denis Protti (chair), David Bates, Richard Lilford, Maureen Baker, Antony Chuter, and Jo Foster for their valuable guidance and support. Our many thanks to Ulugbek Nurmatov for his work in quality assessment as well as to Ann Hansen for her work in running the searches. This work draws on a report published by the NHS Connecting for Health Evaluation programme, the full text of which is available from:
computerised decision support system
computerised provider (or physician) order entry
electronic health record
electronic prescribing
National Health Service
picture archiving and communication system