Introduction

Machine learning (ML) can be applied to emergency department (ED) data to develop clinical decision support systems (CDSS) for paediatric patients, providing health professionals with objective criteria. With the abundance of medical data being collected in electronic healthcare record systems, each patient has become a ‘big data’ challenge. Problems that would have involved collective thinking from clinicians to solve have now become too complex, reaching beyond the ability of the human mind [1], directing us towards ML as a decision support tool, helping to provide patients with the best possible care [2].

Artificial Intelligence (AI) is an umbrella term that includes a wide range of sub fields such as robotics, natural language processing, expert systems, knowledge representation, computer vision, intelligent agents, navigation, planning and predictive analytics [3]. ML is one of these sub fields and the focus of this research. ML is a technique used to train computer systems to automatically learn from data and past experiences to identify patterns and make predictions without being given explicit instructions, mimicking human intelligence [4]. A CDSS is an application used to enhance healthcare delivery by incorporating specific clinical knowledge, patient data, and other health information into medical decisions. CDSSs can be categorised as knowledge based and non-knowledge based. Programmable rules (example IF-THEN statements) are used in knowledge-based systems, retrieving data to evaluate the rule and providing an output or action. Non-knowledge based systems apply AI or ML to the data source as opposed to being programmed [5]. Similar to how imaging technology enhances medical decision making, ML as a CDSS tool has the potential to assist with complex medical decisions [6] and operational management within an environment that needs fast and accurate decision making such as the ED. Research has shown that CDSSs have proven to support clinicians in a range of decisions and patient care activities, and today they actively and widely enhance the provision of high quality care [5]. However, it is believed that a computer cannot fully replace the essential and intimate relationship between a clinician and a patient, nor can an algorithm fully replace the capacity of human reasoning [7], therefore caution should be exercised to avoid overdependence and to use these tools to enhance decision-making in an appropriate manner.

The objective of the scoping review is to perform a search of the literature for ML models used as CDSS tools in the ED. The data relating to challenges with paediatric datasets, areas of application, composition and validation will be sought in the literature for paediatric ML CDSS tools for use in the ED. Models developed using adult or mixed patient data will also be included as there may be potential to apply these models to paediatric patients in the ED. It will also enable us to compare the amount of research devoted to the development of ML CDSS tools for adults compared to paediatrics in the ED. To our knowledge no scoping reviews exist that focuses solely on the application of ML in the ED for clinical decision support, which also gathers information specific to the development of ML CDSS tools for paediatric patients in the ED.

Materials and methods

This protocol will outline the process for conducting the scoping review. The methodology followed will be informed by the scoping study framework of Arksey and O’Malley [8] consisting of six stages.

1. Identifying the research question
2. Identifying relevant studies
3. Study selection
4. Charting the data
5. Collating, summarising and reporting the results
6. Consultation

The Joanna Briggs Institute (JBI) guidelines for scoping reviews [9] will be used to supplement the approach providing additional guidance, enabling transparency and replication of the study by predefining objectives, methodology and how the scoping review will be reported. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Scoping Review Extension (PRISMA-ScR) [10] will be used in conjunction with these approaches (S1 Appendix). This scoping review protocol was registered at Open Science Framework (https://osf.io/75gvs).

3. Study selection

Each database’s search results will be imported into Rayyan (https://www.rayyan.ai) [16] which will be used to eliminate any duplicates and assist with the selection and review process. At the start of the screening process Zotero will be used to identify and exclude retracted articles. The labels used to assist in the categorisation of study types and the reasons for exclusion of articles will be decided by the review team (based on the inclusion criteria) and will be set up in Rayyan. Following similar methodology of previous researchers that distributed the workload using teams of reviewers [17–19], teams consisting of two reviewers will independently screen each article for inclusion against the eligibility criteria using the title and abstract. Discussion between the reviewers will resolve any disagreements. The independent groups of researchers will then review the full text of the remaining articles using the same inclusion criteria and will also examine the reference lists of included articles for additional sources.

Several practice runs were performed by the review team, using between 25 and 100 randomly selected articles imported into Rayyan to ensure a robust screening process, inclusion criteria and to discuss how disagreements will be resolved. It was evident from these practice runs that the greatest source of disagreement originated from the reviewer’s background knowledge. The team consists of reviewers with strong clinical or computer science knowledge, therefore it was decided to leverage this knowledge by dividing the teams and the screening process into two stages. In the first stage, the reviewers with clinical knowledge will determine if the setting and the intended use of the CDSS tool is the ED. In the second stage, the reviewers with computer science knowledge will assess whether ML was used in the study to create a CDSS tool. For both stages, individual reviewers will be used to uphold the independent screening process. Following JBI guidelines [9], the full scoping review will not proceed until a 75% agreement or greater is achieved in the practice runs for both stage one and stage two. A checklist based on the PCC inclusion criteria (divided into the two stages) was created to assist the reviewers further in defining what should and should not be included (Fig 1). Using PRISMA-ScR guidelines a flow diagram representing the selection of sources of evidence will be included.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Article checklist used for title and abstract screening.

https://doi.org/10.1371/journal.pone.0294231.g001

4. Charting the data

A preliminary charting table has been included which will contain key information relevant to the review question (Table 1). The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines [20] were also used to inform the data items in the charting table, along with the approach taken by Andaur Navarro et al. [21]. As per the study selection stage the reviewers conducted a trial run using 100 articles in Rayyan to identify useful information to be included in the charting table. Additional items may be added to this table as the study progresses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Preliminary charting table.

https://doi.org/10.1371/journal.pone.0294231.t001

Discussion

Previous literature reviews have broadened the scope beyond the ED to include emergency medicine, which covered pre-hospital settings (e.g. paramedics, emergency medical services) [4, 22], others narrowed it by focusing on particular areas such as triage [23, 24], predicting in-hospital admissions [25] and others that excluded some application areas of planning and operational management [4]. Several reviews only included studies that had evidence that the model outperformed accepted practices [14] or in the study by Kareemi et al., manuscripts were selected based on evaluating the performance of the models against standard care for prognosis and diagnosis prediction [26]. When looking at the intended users of these CDSS tools, Boonstra and Laven restricted their research to use by ED physicians [14]. Several researchers excluded paediatric studies from their reviews [23, 26]. Medical databases were predominately used in the search, with some authors including technology databases [22]. Liu et al. opted to include one clinical trials database when searching for registered trials on AI in the ED [27]. Many researchers included all AI applications [4, 14, 22] as opposed to ML, with some excluding research that used classical methods such as logistic regression if not a validated model [26] and others excluding studies that did not self-identify as AI [22]. Four reviews were identified as encompassing some of the aspects that this scoping review will use in the search strategy with some potential gaps (Table 2). Liu at al. also carried out a review of AI in emergency medicine [28], but there was no evidence that a scoping or systematic review was carried out.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Pre-existing scoping and systematic literature reviews.

https://doi.org/10.1371/journal.pone.0294231.t002

There are several strengths to consider. This scoping review will be the first to focus on ML CDSS tools for use in the ED, that will also seek to understand where the development of paediatric specific ML CDSS tools lies within that research landscape. As this scoping review will include reviewers from diverse backgrounds and expertise, a different approach will be taken in the screening process to leverage this knowledge appropriately. Splitting the review teams in the early stage of the screening process into those with clinical knowledge (stage 1) and computer science knowledge (stage 2) will decrease the time spent on resolving disagreements that may arise due to lack of either clinical or ML knowledge. During a trial run involving the screening of 100 articles, this difference in backgrounds and area of expertise proved to be one of the top reasons for disagreements on literature to include in the review.

Limitations of this study include the exclusion of literature whose abstract or full text is not in English. Studies using entirely synthesised data or simulated models have also been excluded as this scoping review seeks to capture data on the evaluation and validation methods of models using real world data. The databases to be included in the search are predominately medical based, therefore some useful research may be missed from other types of databases such as computer science.

To conclude, this scoping review will be an important source of information for future researchers looking to develop ML CDSS tools for paediatric patients in the ED.

Supporting information