Objectives

This review aims to evaluate the applications of NLP at ED triage by answering the following questions:

1. How has NLP been applied to data acquired at ED triage?
2. Do NLP based models outperform humans or current risk stratification techniques when predicting outcomes?
3. Does incorporating free-text improve predictive performance of ML models when compared to ML models that use only structured data?

Study selection

This process is summarised in a PRISMA Flow Diagram (Fig 1). There were 5329 records identified following database searching and a further 11 records identified through other sources. Following removal of duplicates, 3730 records remained and underwent title and abstract screening. 3597 records were excluded. The remaining 173 full-text articles were assessed for eligibility. In total, 153 articles were excluded, and 20 studies remained for inclusion (Fig 1). There were no unresolved disagreements as to study inclusion or results of data extraction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Prisma flow diagram.

https://doi.org/10.1371/journal.pone.0279953.g001

Characteristics of included studies

A summary of the included studies is shown in Table 1. There were 19 retrospective studies [17, 18, 30, 34–49]. One study reported their ML model was developed using retrospective data then validated using prospective data [50]. All used observational cohort designs. Two studies were international multi-centre studies (USA and Portugal); 12 were conducted in the USA; 2 were from South Korea; one each from Australia, Brazil, China, and France. The most common outcomes assessed were prediction of triage score (six studies), prediction of admission (five studies), and prediction of critical illness (three studies). Two studies predicted need for imaging within the ED, two studies looked at the assignment of provider assigned chief complaint label, one study predicted diagnosis of infection in the ED, and one study aimed to identify and classify temporal expressions used in triage notes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Details of included studies.

https://doi.org/10.1371/journal.pone.0279953.t001

The population size varied greatly between studies ranging from 598 triage notes to 1.8 million patients. Five studies used a population of under 100 000, four studies had a population of between 100 000 and 200 000, six studies had a population of between 200 001 and 300 000, and six studies had a population of over 300 000. Twelve studies used data from a single site and eight studies used data from multiple sites. The largest number of sites used was 642 by Zhang et al.

Fifteen studies applied NLP to free-text history of presenting complaint, seven studies applied NLP to a free-text chief complaint, two studies applied NLP to a structured chief complaint label, and one study applied NLP to simulated triage dialogues that had been transcribed by either a human or an ML model. The other most frequently used input variables were patient demographics (13 studies), patient vital signs (heart rate, respiratory rate, oxygen saturation, blood pressure, and temperature) (15 studies), pain score (12 studies), triage score (10 studies), mode of arrival (10 studies), time of arrival (9 studies) and past medical history (7 studies). Other input variables included mental status (5 studies), and blood glucose level (5 studies).

Prediction of admission

Overall, NLP models and multimodal models achieved high Area Under the Receiver Operating Characteristic Curve (AUC) in predicting admission at time of triage for adult and paediatric patients (Table 2) [18, 30, 35, 41, 46]. Of the five studies focusing on predicting admission to hospital, Roquette et al. achieved the highest (AUC) using a gradient boosting model (AUC 0.89). Tahayori et al. achieved a similar AUC (0.88) using only free-text history of presenting complaint. Tahayori et al. were the only authors that compared their model to emergency physician performance. Their model achieved a higher accuracy than five emergency consultants (0.83 vs 0.78) and higher specificity (0.86 vs 0.77), but lower sensitivity (0.72 vs 0.9). Roquette et al. and Zhang et al. both compared ML models trained using structured data only with ML models that incorporated both structured data and text data. They found that the addition of text data results in a small improvement (Zhang AUC 0.823 to AUC 0.844, Roquette AUC 0.872 to AUC 0.891) when compared to the use of structured data alone. This improvement was not assessed for statistical significance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results of included studies, grouped by outcome.

https://doi.org/10.1371/journal.pone.0279953.t002

Prediction of critical illness

Of the three studies that predicted critical illness at triage (defined as ICU admission, cardiopulmonary arrest within 24 hours, or death within 24 hours of triage), Fernandes et al. achieved the highest AUC (0.96) in predicting in-hospital death or cardiopulmonary arrest within 24 hours of triage using an extreme gradient boosting model (Table 2) [43–45]. They found no difference in AUC when using clinical variables only or clinical variables and structured chief complaint processed by NLP. Joseph et al. found their NLP model (AUC 0.857) significantly outperformed an abnormal vital sign trigger (AUC 0.521) and ESI ≤ 2 (AUC 0.672) in predicting critical illness. The addition of free-text data improved the performance of their neural network model (from AUC 0.820 to AUC 0.857).

Prediction of triage score

NLP has been retrospectively applied to data acquired from multiple different triage systems [17, 36–38, 47, 48]. NLP models and multimodal models have achieved high AUCs in assigning triage categories using structured and free-text data (Table 2) [17, 36–38, 47, 48]. Wang et al. achieved the highest performance in predicting ESI using their "DeepTriager" model (AUC 0.96). Kim et al. achieved an AUC of 0.89 in assigning a KTAS category to auto-transcribed simulated triage dialogue. This was only slightly lower than the performance achieved using human-transcribed simulated triage dialogue (AUC 0.90).

Three studies compared the accuracy of triage scores assigned by multimodal models incorporating NLP to triage scores assigned by nurses [17, 36, 47]. Such models were reported to be more accurate than nurses in two out of three papers [17, 36, 47]. Ivanov et al. (2021) used a random sample test set of 729 records to assess their model’s ability to predict ESI. In this test set, ESI had been assigned to each record with unanimous agreement by three expert clinicians. They also compared ED site nurses’ original ESI against the expert consensus assigned ESI. When applied to the test set, their clinical-NLP model achieved an AUC of 0.85 in predicting ESI and original ED nurse triage achieved an AUC of 0.75. Three members of the study team (two emergency clinicians and one emergency nurse) also assigned ESI to this test set, achieving similar AUCs to the NLP model (clinician one AUC 0.86, clinician two AUC 0.85, clinician three AUC 0.82). Sterling et al. (2020) retrospectively calculated number of resources used (ESI), and then used a test set of 1000 randomly selected records to compare their model performance against two experienced ED nurses. In this test set, their model achieved a similar F1 score (0.589 vs 0.659) but lower accuracy (0.589 vs 0.659) than the two ED nurses in predicting number of resources used. Gligorijevic et al. (2018) approximated nurses’ performance by analysing predicted (ESI) versus actual number of resources used. They then compared approximated nurses’ performance against their model’s capability at predicting actual resources used. Overall, their NLP model achieved a higher accuracy than approximated nurses’ performance in assigning number of resources used category (43.6% vs 59.6%). The addition of text data compared to structured data alone improved performance in assigning triage score [36, 37].

Prediction of provider-assigned chief complaint

NLP models and multimodal models incorporating NLP were able to accurately map free-text history of presenting complaint to structured chief complaints (Table 2) [42, 50]. Chang et al. (2020) used BERT to predict provider-assigned chief complaint labels (Top-5 structured label AUC 0.92). Greenbaum et al. (2019) applied NLP to free text triage notes to rank structured chief complaint labels by their predicted probability. This improved structured data capture from 26.2% to 97.2%.

Prediction of investigations

Multimodal models incorporating NLP were used to predict diagnostic imaging performed in the ED (Table 2) [39, 40]. Zhang et al. developed a model to predict need for advanced diagnostic imaging (computed tomography, ultrasound, magnetic resonance imaging) in the ED, and obtained an AUC 0.78. Zhang et al. also achieved an AUC 0.824 in predicting the need for any diagnostic imaging in a paediatric population. In both cases, the inclusion of structured variables improved performance slightly when compared to unstructured variables alone (AUC from 0.74 to 0.78, and AUC from 0.81 to 0.82).

Identifying infection

Horng et al. (2017) found that the incorporation of free-text data improves the discriminatory ability (increase in AUC from 0.67 to 0.86) for identifying sepsis (defined by ICD-9-CM code) in the ED at triage (Table 2).

Extracting temporal information from triage notes

Irving et al. (2008) applied an NLP system to extract and classify temporal information contained in free-text triage notes. Such information included the relative time of event compared to triage time, and duration of event. They report better performance was obtained using a Decision Tree compared to Naive Bayes.

Multimodal models

Eleven papers compared ML models that used only structured data to multimodal models that incorporated both structured data and free-text data (Table 2) [34–40, 43–46]. The best performing model in each of these papers incorporated free-text. The largest improvement in model performance from incorporating free-text was found by Horng et al. (increase in AUC from 0.67 to 0.86 for identifying infection). The addition of free-text did not improve model AUC in one case, however, did improve model average precision [44]. There were no cases where the incorporation of free-text into the model resulted in worse performance. Six papers assessed models that used only free-text, with no structured data [30, 36, 37, 39, 40, 42]. Tahayori et al. were able to use only free-text data to predict admission with high accuracy (83%). Zhang et al. used free-text to predict performance of diagnostic imaging. Gligorijevic’s “Deep Attention” models using only unstructured data outperformed those using only structured data. Incorporating both structured data and free-text data improved results when compared to models that used only free-text data, though often only a small improvement was found.

Modern NLP compared to traditional NLP

Three papers directly compared modern NLP based on DL to more traditional ML techniques such as bag-of-words and topic modelling (Table 2) [30, 38, 48]. Modern DL based NLP outperformed traditional ML based NLP in two cases [30, 38]. In contrast, Kim et al. found that a BERT based DL model did not perform better than ML based models, though their population was relatively small. Chang et al. compared the performance of multiple modern DL based models, finding BERT slightly outperformed Embeddings from Language Models (ELMo) and Long Short-Term Memory (LSTM) networks in mapping free-text chief complaints to structured fields.

Integration into practice

Greenbaum et al. was the only study that reported the deployment of an NLP based model into clinical practice. Greenbaum et al. aimed to increase the ease of high-quality structured data collection of patients’ chief complaint at triage through the use of an NLP based model. Their model used both free-text triage notes and structured data to provide contextual autocomplete of chief complaint label, and also show the user a list of the top five most likely chief complaints. Prior to implementation of their model, chief complaint was more commonly entered as unstructured free text, with only 26.2% of patient encounters resulted in structured data capture. Following implementation, this increased to 97.2%. The authors aggregated multiple incidents of unscheduled downtime that occurred throughout the study to opportunistically assess the impact of their model. When ML based autocomplete was not operational (and instead alphabetised autocomplete was shown), the percent of encounters that resulted in structured data capture decreased from 97.2% to 89.2%. The number of keystrokes typed for each presenting problem decreased from 11.6 pre-implementation to 0.6 post-implementation. Contextual autocomplete was associated with qualitatively more complete and higher quality (as assessed by three independent reviewers on a four-point Likert scale) structured documentation of chief complaints.

Study quality—Risk of bias within and across studies

A summary of the PROBAST assessment is provided in Table 3. Overall, 16 out of 20 studies were considered to have a high risk of bias. A common reason for overall high risk of bias in the PROBAST assessment was lack of external validation. Four studies were assessed as having a low risk of bias. One study had high applicability concerns and 19 studies had low applicability concerns. The four studies assessed as having low risk of bias also had low applicability concerns. No studies referred to a previously published or publicly registered protocol.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. PROBAST assessment of the included studies.

https://doi.org/10.1371/journal.pone.0279953.t003

Availability of datasets and code

Availability of study datasets and code is shown in Table 4. Data was publicly available for three studies (all by Zhang et al.) and was available on request from study authors for a further four studies [30, 34, 35, 39, 40, 43, 44]. One study reported plans to release a modified de-identified dataset, however at the time of this review this is still pending approvals [45]. The model code was publicly available for two studies [42, 45]. Notably, the code repository from Chang et al. was well organised and contained clear instructions for researchers on how to download their pretrained model and apply it to their own dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Availability of dataset and code for included studies.

https://doi.org/10.1371/journal.pone.0279953.t004

NLP at triage

This review finds that NLP has been applied to data available at the time of ED triage to predict a range of outcomes, with a focus on predicting need for admission and assigned triage score. This review also the combining free-text nursing triage notes with structured data appears to result in the best model performance, however free-text nursing triage notes alone have been used by NLP algorithms to predict need for admission and need for diagnostic imaging [18, 30, 39, 40]. A potential benefit of developing models that require only free-text as an input is that it may allow for easier portability of predictive models between different triage systems [30].

Structured data capture

Accurate and consistent structured capture of patients’ presenting complaints is important for research, service improvement, and public health initiatives [50]. Common medical ontologies also improve system interoperability [51]. However collection of structured data is often difficult, especially when contrasted with the ease and expressiveness of free-text entry [50]. In a rare singular example of NLP being deployed into routine clinical practice at ED triage, Greenbaum et al. developed, implemented, and prospectively evaluated an NLP driven user interface in an attempt to improve structured data capture [50]. Promisingly, they report that their NLP based contextual auto-predict did not add additional burden to users, made structured data collection easier than unstructured data collection, and significantly increased structured data collection.

NLP compared to humans

Human performance may be a reasonable baseline for ML models to meet to be considered accurate enough for implementation into clinical practice. Few studies have compared NLP models at triage to human performance. Such comparisons will be crucial in future work. Tahayori et al. was the only study that compared results from NLP models to emergency physicians [30]. Ivanov et al., Sterling et al, and Gligorijevic et al. compared NLP based models to nurses in assigning triage scores and found model accuracy was similar to nurses [17, 36, 47].

Modern NLP

While it is difficult to compare studies due to their heterogeneity, advanced DL based NLP appears to outperform traditional NLP. This is certainly the case when compared internally within studies and is consistent with previous NLP research [52]. BERT appears to be the most popular advanced NLP that has been used. BERT was released in October 2018 and at the time of release, BERT outperformed other NLP models [27]. However, of the 16 papers published since the release of BERT, only three have used it. Other large models have subsequently been released. For example, GPT-3 is a 175 billion parameter language model that was released in 2020 and is reported to outperform BERT in various circumstances [28]. Chowdhery et al. have recently published Pathways Language Model (PaLM), a 540-billion parameter model that achieves further increases in performance [29].

Incorporating clinical gestalt

Sterling et al. 2020 noted the difficulty in capturing the general clinical impression of the triage nurse [17]. Ivanov et al. also noted that important contextual aspects at triage were not available for consideration by ML models [47]. Future work could assess the impact of incorporating triage nurses’ gestalt into predictive models. Other contextual data available at the time of triage includes the number of patients currently waiting to be seen, the number of patients currently in the ED, and number of admitted patients in the hospital could also be incorporated into ML models. However, it is unknown if the inclusion of such data would improve model performance.

Integration with other AI systems

Kim et al. provides an interesting example of how various AI based technologies can be combined [48]. Future work could assess if it is feasible to integrate triage NLP models with other novel AI based interventions, such as automated monitoring of patients’ vital signs while they are in the waiting room, or with data entered by patients themselves in self-triage applications. However, combining multiple AI based predictive models may not result in improved performance.

Pre-trained models for ED triage

Publicly available large DL based language models have often been trained on corpuses containing text from newspapers, books, and websites [27, 28]. Triage notes are often quite short and contain a number of unique and idiosyncratic abbreviations and acronyms not common in everyday English language [17, 30]. NLP models that have been applied to triage notes were often based on models that were not developed specifically for this purpose. DL based NLP models that have been fine-tuned on large corpuses of medical text have been released, however they have not been applied to ED triage. Large publicly available clinical databases such as MIMIC-IV that contain ED triage notes with linked outcomes may be helpful in further model development and may facilitate direct comparisons between models developed by different research groups [53, 54]. Triage focused NLP research could potentially benefit from groups sharing large language models that have been pre-trained on triage data, though it is unknown whether such models’ performance would generalise across different healthcare settings and triage systems.

Interpretability

NLP models have become increasingly complex. Inputs to DL based NLP models may be processed through billions of interconnected processing units prior to the model generating an output [26]. This has increased their predictive capability, however the explanation of why a model gives a certain output when presented with a particular input has become less interpretable to humans. Some have suggested that it is unethical to implement such models into clinical practice if the reasoning behind their predictions cannot be interpreted by humans [55]. It has also been suggested that improved interpretability may help end users detect model biases and improve patient safety [56]. Other however, argue that model interpretability is unnecessary and should not be pursued at the expense of model accuracy [57]. It is currently unknown if clinicians will accept a model into routine clinical practice if the model cannot explain why an output was given in a way that is interpretable to humans. Developing “explainable AI” models is an active area of research [58, 59]. Few papers attempted to address human interpretability of their NLP model’s output. Wang et al. show how models could be somewhat more interpretable [38]. Their triage model is able to highlight free-text triage notes, with a darker colour corresponding to the sections of text that was more heavily weighted by the model. This provides an initial "sense check" that humans can then combine with their own experience and knowledge.

Prospective and external validation is needed

The majority of research so far has been retrospective with a high risk of bias, and completed in the USA. There is a significant need for prospective evaluation and external validation, especially in other countries and triage systems. Further research is also required to assess the impact of integrating NLP at triage on patient orientated outcomes, as there is currently little evidence that NLP at triage improves outcomes compared to usual clinical practice.

Clinical impact and risk

NLP models have rarely been deployed at ED triage. As such, it is unknown what impact these tools could have on clinical practice. The introduction of a new tool into a complex system is likely to have unintended consequences, and use of the tool may itself change practice. Triage notes may be written in a different way if it is known that they are being used for predictive purposes. It is also unknown if the length of triage notes impacts model performance. There may also be unintended harms. For example, telling a patient at triage that they are likely to be admitted or to have a long wait time, could potentially influence their behaviour and affect the number of patients who leave without being seen. It may be useful to establish the performance benchmarks predictive models must meet prior to implementation into clinical practice. This could be achieved through further studies comparing NLP model performance to emergency physicians and nurses.

Predictive models are trained on data that reflects current practice. This engrains the assumption that current practice is appropriate, which may not be the case. If NLP models are retrained and updated as new data becomes available, then model performance may change over time. It will be important to ensure that there is appropriate algorithm stewardship in place prior to clinical use [60].

Acceptability

It is also unknown if the use of NLP at triage is acceptable to patients and staff. It will be important to involve clinicians, patients, and healthcare consumer groups in the development and governance of any future implementation projects. It will also be important to ensure that these systems do not place further burden on users. Ease of use and perceived clinical impact will likely be important factors for adoption by clinicians.

Ethical issues

Racial, age, and gender biases at ED triage have been previously reported [61–63]. Concerns over bias in ML models have been well described [64, 65]. The assessment and reduction of bias in ML is an ongoing and active area of research [66]. At its best, NLP at triage could help reduce bias through standardising triage decisions and providing a more objective triage score. However, at its worst NLP at triage could further ingrain existing biases into practice, under the guise of objectivity and hidden in the opacity of abstract algorithms. Patient apprehensions and concerns about the use of AI will also need to be considered. An emerging body of literature suggests that in while in general, patients view AI positively, they do have some concerns with its use in healthcare [67]. These include perceptions that AI will be less accurate than clinicians, that there is a lack of transparency in predictions, and that there are risks to the privacy of their personal healthcare data [68–73]. Further research investigating the impact of NLP based tools on vulnerable and minority populations is warranted.

Study level

Only one study contained prospectively validated results, and no studies contained results that were externally validated at a separate site. Results reported may not be generalisable to other settings. There was inconsistent reporting of methods and results among studies. The majority of studies (79%) were assessed to have a high risk of bias.

Review level

Heterogeneity of the included studies precluded meta-analysis which limits the level of evidence this review provides. All studies reported positive results for NLP at triage, which may reflect publication bias. While we took significant care to ensure our search strategy was broad enough to capture all relevant literature, the variety of NLP and ML terminology means that some studies may have been missed. Non-English articles, and articles published prior to 2012 were also excluded from our search. We used the PROBAST tool to assess for risk of bias and applicability concerns for included studies. However, other tools to assess risk of bias do exist and their use may have resulted in different risk of bias assessments.