Introduction

Patients presenting to the emergency department (ED) have varying needs for urgent medical attention and limited hospital resources often necessitate prioritization of some patients over others [1]. Overcrowding of the ED is a common and increasing problem and can lead to adverse patient outcomes [2]. EDs therefore need to quickly determine the urgency and level of care required for each patient in order to optimize the allocation of scarce hospital resources [3]. To achieve this, most modern EDs have a triage process to assess patients’ severity of illness or injury upon arrival, assign priorities and then provide appropriate treatment [3,4]. Currently, ED triage is most commonly guided by semi-subjective scale-based systems, with some notable examples being the Emergency Severity Index (ESI) [5] and the Canadian Triage and Acuity Scale (CTAS) [6]. Using a mixture of qualitative and quantitative metrics, these scale-based protocols guide the healthcare practitioner in assigning the patient to a label that reflects his/her required level of care. Although scale-based systems have been widely implemented and have shown their usefulness, their accuracy is highly dependent on the triaging doctors’ or nurses’ experience [3]. In recent years, various prediction models have been developed for ED patients that could complement subjective scale-based triage processes and further optimize management of patients in the ED [7,8]. These models are typically derived from real-world data and utilize a range of statistical and machine learning tools, ranging from traditional regression models to cutting-edge neural networks [9]. Some examples include models predicting in-hospital mortality [10], intensive care unit (ICU) admission or readmission [11–13]. However, among the many models developed, few were externally validated and even fewer had their impact on clinical practice analysed [7].

Nevertheless, some of these prediction models may have been implemented into routine clinical practice with the increasing emphasis on harnessing big data and building learning healthcare systems [14]. While the area under curve (AUC) and other quantitative summary statistics are used in model development and validation, they do not entirely capture the actual consequences of model implementation [15]. Predictive analytics promise to improve patient outcomes but several intervening steps leading to providers responding appropriately to model outputs are necessary to result in actual patient benefit [16]. Studying the implementation process and its impact on outcomes can identify potential barriers and facilitators to implementation and strategies that may promote implementation [17].

There have been systematic reviews providing informative overviews of prediction models in the ED primarily in terms of model structure, development, and performance [7,12,18]. One review on clinical decision support systems for triage in ED found that less than half of the included studies had an implementation phase even though the majority of them showed promising potential in the validation phase [7]. These findings suggest that there exists a host of barriers which are unrelated to performance and that despite the paucity in implementation, these barriers can be overcome with proper knowledge and execution. Furthermore, to the best of our knowledge, there has been no review of the process and outcomes of implementing these models into routine clinical practice. Therefore, a gap still exists in our current understanding of the logistical and administrative challenges involved in prediction model implementation in the ED. This gulf in the current body of knowledge also extends to an understanding of how outcomes of such implemented models are assessed and perceived by the healthcare providers. This points to a need for a scoping review on this topic, a type of systematic evidence synthesis where the intent is to assess and understand the extent of knowledge or map the concepts in a particular field, rather than to answer a specific clinical question to aid decision making as is the case for a systematic review [19].

To understand both the contextual determinants as well as outcomes affecting implementation success into routine clinical practice, we will be using the revised, enhanced Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM)/Practical, Robust, Implementation, and Sustainability Model (PRISM) 2019 framework to guide this review (Fig 1) [20]. The revised RE-AIM/PRISM framework retains the original 5 RE-AIM domains, which can be used as evaluation outcomes for an implementation effort and includes multilevel contextual determinants from PRISM that can explain these implementation outcomes [20]. This would be useful in guiding future implementation studies, but also in providing valuable insight into the factors that influence to the success of implementing prediction models in the ED [21,22].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The revised RE-AIM/PRISM framework [20].

https://doi.org/10.1371/journal.pone.0267965.g001

This review is also focused on implementation of prediction models into routine ED practice. Implementation is a process, where implementation studies can occur from pre-implementation to the maintenance phase. Studies focused on implementation outcomes at any stage are therefore relevant for our review. Prediction models typically progress from development, internal validation, external validation to impact assessment before being implemented [23,24]. Model performance prior to routine implementation is therefore theoretical, even if performed on actual patient data. For effectiveness, we therefore chose to only include results after implementation, as this represents ‘real-world effectiveness’.

The aim of this scoping review is to summarize the current landscape and understanding of implementation of prediction models in the ED from an implementation science angle. Specifically, apart from a descriptive summary of characteristics of prediction models implemented in clinical practice, we will summarize the implementation outcomes and contextual determinants affecting implementation success according to the revised RE-AIM/PRISM framework [20].

Methods

This scoping reviews follows the Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist [25]. At the time of writing, we have completed the searching, title and abstract screening, and full-text screening.

Inclusion and exclusion criteria

We expected that reports on prediction model implemented in the ED would have varying degrees of focus on implementation outcomes and reporting of implementation strategies. We also recognized that not all implementation studies may be comparative. For clarity, we have combined the PICOT (Population, Intervention, Comparison, Outcome and Time) framework and the Standards for Reporting Implementation Studies (StaRI) guidelines [26] to differentiate the intervention and implementation strategy of interest. The intervention describes the type of prediction models of focus in this review, and implementation strategy describes the studies. We therefore included studies that satisfied the PICOT elements for both the intervention and implementation strategy (Table 1). For the implementation strategy, the intervention and comparison elements were seen as optional to accommodate non-comparative studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Inclusion and exclusion criteria.

https://doi.org/10.1371/journal.pone.0267965.t001

All types of primary research studies were included (e.g., randomized, quasi-experimental, observational, qualitative, etc). For reviews that met our inclusion criteria, we included the relevant primary studies within instead of the review itself, allowing more flexibility with data extraction and synthesis.

We excluded conference abstracts and papers, theses and dissertations, letters to editors and commentaries as these were unlikely to contain sufficient information to contribute meaningfully to the review. We also excluded non-research documents in the grey literature and studies with no full-text in English due to lack of resources for searching and translation, respectively.

A wide variety of predictive tools may be used in the ED. For the sake of clarity, the following types of interventions were not included:

• Triage models that rely on subjective judgement and/or do not involve quantitative variables, including simple criteria-based rules.
• Models used in the ED but not applied on patients (e.g., for operations planning, staff roster planning, analysis of scans or reports, etc).
• Models for prediction or prognostication based on a single predictor or diagnostic procedure (e.g., troponin, CT scan, etc).
• Treatment protocols, guidelines, or pathways with multiple decision points/actions. Although some of these may include triage or prediction, they are typically only one of many components of a complex intervention. It is therefore difficult to assess the implementation aspects of the prediction component alone. We will focus on studies where the prediction model is the primary focus for implementation.
• Models or tools for diagnosis or measuring a single construct (e.g., pain, alcohol intake, etc).
• Models focused on improving operational efficiency (e.g., quality improvement studies).

Search strategy

A list of keywords and index terms for informative PICOT elements for the intervention and implementation strategy was generated. The index terms for each database were searched and curated according to the controlled vocabulary of the database. For example, for PubMed, MeSH terms were searched using the keywords and the most relevant ones chosen. The keywords and index terms within each concept were then combined using the BOOLEAN operator ‘OR’ and searched in all databases. The results from the 3 concepts were then combined using the BOOLEAN operator ‘AND’ to narrow the search. The team then reviewed a sample of the initial search results and updated the search terms with additional keywords found in relevant articles. The following filters were applied: English, “Full text” and “Journal Article” to remove conference abstracts and other non-research articles. The final search terms are shown in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Literature search terms.

https://doi.org/10.1371/journal.pone.0267965.t002

Discussion

This scoping review is the first to focus on the implementation process and real-world outcomes of prediction models implemented in the ED. The body of existing reviews on prediction models in the ED have principally focused on model development and validation [7,12,18]. This review aims to fill a gap in the current literature and complement existing reviews by providing an overview of how validated prediction models in routine clinical use are implemented and evaluated. It will therefore complement existing reviews that focus only on the performance of prediction models. Additionally, considering the burgeoning prevalence of high-performing Artificial Intelligence (AI) based models [31] and the increasing adoption of computerized clinical decision support systems [32], an inquiry into the implementation of prediction models carries tremendous practical implications. This is especially so in the ED, where a range of undesirable outcomes such as overcrowding [33], patient readmission [34] and septic shock [35] continue to persist even after patients leave the ED and are not directly addressed by triage itself.

Despite advancements in the development and validation of ED-based prediction models, there has been comparatively little progress in the implementation and integration of such models into clinical practice. One major barrier to adoption of prediction models is lack of evidence of clinical utility, which requires the model to provide information over and above what is already known, thereby prompting actions that lead to improved outcomes compared to without the model [36]. A host of other challenges such as data barriers, lack of transparency, regulation and certification, ethics, need for education and training, exists especially for models designed to harness electronic medical records to produce real-time prediction [37]. This review will summarize the collective experience of implementing various types of prediction models in EDs around the world using an implementation science framework and provide a sense of what factors promote or hinder implementation, what strategies might work in promoting implementation, how models perform in the real world and which area are lacking in implementation research. For EDs considering implementing certain prediction models in their setting, this review can provide valuable information on their model’s potential clinical impact, key strategies to maximize implementation success and what implementation studies to perform while doing so. This review can also reveal pitfalls and gaps in the implementation of certain prediction models. Taken together, a collection of both positive and negative real-world experiences can provide a holistic perspective of ED prediction model implementation, potentially aiding and streamlining the implementation process for future prediction models.

There are some strengths and limitations of our study. The key strength is that this review focuses on implementation, which is a gap in the review literature currently. Another strength is that we intend to include a broad range of models, which increases the applicability of the findings. The limitations are firstly, the exclusion of protocols, guidelines or pathways that include a prediction component. Prediction models are often not used in isolation but part of a care plan. However, the focus of this review is to inform how to implement new prediction models that are likely to be an addition to current clinical workflows rather than creation of a whole new workflow, although that might be necessary in some cases. Secondly, the summary of implementation outcomes may require our interpretation and categorization of study findings. This is inevitable and necessary as terminology within the implementation science is not standardized [38]. Moreover, many studies may not even have explicitly utilized implementation science methods or tools.

In conclusion, this scoping review will be a valuable resource for informing future implementation studies of prediction models in the ED.