Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Advanced diagnostic imaging utilization during emergency department visits in the United States: A predictive modeling study for emergency department triage

  • Xingyu Zhang ,

    Contributed equally to this work with: Xingyu Zhang, Joyce Kim

    Roles Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing

    Affiliations University of Michigan School of Nursing, Applied Biostatics Laboratory, Ann Arbor, MI, United States of America, Department of Surgery, Emory University School of Medicine, Atlanta, GA, United States of America

  • Joyce Kim ,

    Contributed equally to this work with: Xingyu Zhang, Joyce Kim

    Roles Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Internal Medicine, Emory University School of Medicine, Atlanta, GA, United States of America

  • Rachel E. Patzer,

    Roles Conceptualization, Methodology, Resources, Supervision

    Affiliations Department of Surgery, Emory University School of Medicine, Atlanta, GA, United States of America, Department of Epidemiology, Rollins School of Public Health, Atlanta, GA, United States of America, Health Services Research Center, Emory University School of Medicine, Atlanta, GA, United States of America

  • Stephen R. Pitts,

    Roles Conceptualization, Methodology, Resources, Supervision, Writing – review & editing

    Affiliations Health Services Research Center, Emory University School of Medicine, Atlanta, GA, United States of America, Department of Emergency Medicine, Emory University School of Medicine, Atlanta, GA, United States of America

  • Falgun H. Chokshi,

    Roles Conceptualization, Methodology, Resources, Supervision, Writing – original draft, Writing – review & editing

    Affiliations Department of Radiology & Imaging Sciences, Emory University School of Medicine, Atlanta, GA, United States of America, Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States of America

  • Justin D. Schrager

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

    Affiliation Department of Epidemiology, Rollins School of Public Health, Atlanta, GA, United States of America



Emergency department (ED) crowding is associated with negative health outcomes, patient dissatisfaction, and longer length of stay (LOS). The addition of advanced diagnostic imaging (ADI), namely CT, ultrasound (U/S), and MRI to ED encounter work up is a predictor of longer length of stay. Earlier and improved prediction of patients’ need for advanced imaging may improve overall ED efficiency. The aim of the study was to detect the association between ADI utilization and the structured and unstructured information immediately available during ED triage, and to develop and validate models to predict utilization of ADI during an ED encounter.


We used the United States National Hospital Ambulatory Medical Care Survey data from 2009 to 2014 to examine which sociodemographic and clinical factors immediately available at ED triage were associated with the utilization of CT, U/S, MRI, and multiple ADI during a patient’s ED stay. We used natural language processing (NLP) topic modeling to incorporate free-text reason for visit data available at time of ED triage in addition to other structured patient data to predict the use of ADI using multivariable logistic regression models.


Among the 139,150 adult ED visits from a national probability sample of hospitals across the U.S, 21.9% resulted in ADI use, including 16.8% who had a CT, 3.6% who had an ultrasound, 0.4% who had an MRI, and 1.2% of the population who had multiple types of ADI. The c-statistic of the predictive models was greater than or equal to 0.78 for all imaging outcomes, and the addition of text-based reason for visit information improved the accuracy of all predictive models.


Patient information immediately available during ED triage can accurately predict the eventual use of advanced diagnostic imaging during an ED visit. Such models have the potential to be incorporated into the ED triage workflow in order to more rapidly identify patients who may require advanced imaging during their ED stay and assist with medical decision-making.


Emergency department (ED) crowding is a well-recognized problem in the United States [13]. Problems associated with ED crowding have been extensively documented: longer wait time and length of stay (LOS) during ED visit; staff and patient dissatisfaction; higher hospital costs; and negative patient outcomes [48]. As a result, many emergency departments are moving toward physician triage models in which physicians perform rapid evaluations to expedite work-ups and dispositions while patients are still in the waiting area. This method has shown promise—with studies demonstrating decreased LOS and decreased number of patients who leave without being seen [912]. Algorithmic clinical decision support, specifically predictive analytics, may be of benefit in this clinical setting[13]; however its use has not been sufficiently described or tested.

The decision by an ED provider to pursue advanced diagnostic imaging (ADI) studies during an ED visit is a major contributor to increased ED LOS [14, 15], and ADI use in the ED has been increasing for more than a decade [16]. The median LOS for ED patients with ADI is 114 minutes longer than those without ADI [17]. This increased LOS can be attributed to clinical factors—such as the amount of time it takes to obtain and interpret a CT scan—and diagnostic factors—such as the time it takes to clinically evaluate a patient and decide if they will need ADI. Early prediction of eventual ADI use has the potential to shorten diagnostic time. To date, research has not examined the role of a predictive model that can use information immediately available to a triage provider upon patient arrival (e.g. patient demographics, vitals, medical history, and the patient’s own descriptions of their reason for visit) to estimate the probability that the patient will undergo ADI during their ED visit. Such a predictive model—if implemented and tested in the clinical setting perhaps as an adjunct to the Electronic Health Record (EHR) or as a standalone program—could support clinicians in making rapid, informed decisions regarding ADI. Furthermore, few studies have utilized the important information that exists within the free-text reason for visit that patient’s give on arrival to the ED to make predictions regarding processes and outcomes in the ED [18, 19]. This free-text reason for visit information can be utilized via natural language processing (NLP), a method through which text data can be extracted and processed for analysis and has been shown to improve models related to health outcomes [2023].

In a nationally representative sample, we examined patient information that would readily be available during the ED triage process, including free-text reason for visit, to develop predictive models for ADI use including computed tomography (CT), ultrasound (US), and magnetic resonance imaging (MRI) during the ED encounter.

Materials and methods

Study population

This study is a secondary analysis of data collected from the 2009–2014 National Hospital Ambulatory Medical Care Survey ED Subfile (NHAMCS-ED)[2426], a multistage, stratified probability sample of ED visits in the United States administered by the National Center for Health Statistics, a branch of the Centers for Disease Control and Prevention. The NHAMCS-ED sample is collected during a random 4-week period each year. Study staff visit approximately 300 hospital-based EDs, which are randomly selected from approximately 1,900 geographically defined areas covering all 50 States and the District of Columbia. A standardized form and protocol are utilized to abstract data from approximately 100 patient charts per ED. Details of the survey methodology are available from the National Center for Health Statistics [25, 26]. A total of 179,036 patient visits were included in the survey datasets from 2009 to 2014. After excluding pediatric visits (n = 39,886), 139,150 (77.7%) adult patients (≥18 years old) visits remained for analysis.

Study variables

Outcome variables.

The primary outcome variables for this study were the eventual use of advanced diagnostic imaging during an ED visit. ADI categories were analyzed independently and include any ADI, CT only, US only, MRI only, or multiple ADI use.

Structured variables.

Structured covariates included only those that would be immediately available at the time of ED triage: sex, age category, race/ethnicity, type of residence (private residence, nursing home, homeless, or other), source of payment (using the NHAMCS algorithm to classify multiple payers), whether the patient arrived via ambulance, arrival day and time, initial vital signs (body temperature, heart rate, respiratory rate, blood pressure, pulse oximetry), pain scale, whether the patient arrived on oxygen, whether the patient had used the ED within the past 72 hours, the episode of care (initial vs. follow-up visit to the ED for the presenting problem), past medical history (cancer, cerebrovascular disease, chronic obstructive pulmonary disease, condition requiring dialysis, congestive heart failure, dementia, diabetes, myocardial infarction, pulmonary embolism, and HIV), whether the visit was related to an injury, poisoning, or adverse effect of medical treatment (and if related to injury/poisoning, whether it was self-inflicted, related to assault, unintentional, or unknown), and visit acuity at triage (triage level which by convention is changed to a 5 point system by NHAMCS researchers when EDs use a 3-point or 4-point acuity scale). Five past medical history diagnoses (cancer, chronic obstructive pulmonary disease, dementia, myocardial infarction, and pulmonary embolism) were collected starting from 2012; therefore, these were missing in the 2009–2011 survey years. Information on whether the patient arrived on oxygen was not collected in the 2014 survey year.

Reason for visit.

Reason for visit information, extracted using NLP, included up to three reasons for visit or cause of injury recorded by the providers for each patient in the ED triage notes. The survey methodology used in classifying patient reasons for visit has been described previously and is designed to approximate the patient’s own words [27]. The reason for visit classification system derived by the National Center for Health Statistics is a modular framework into which the reason for visit is broadly categorized as a type of complaint (e.g., symptoms, diseases, injury) and a methodology for systematically recording these complaints within a specific organ system or body area. The system then records the complaint in a pre-specified fashion according to an alphabetical index of complaints (for example, “eye pain” is changed to “pain, eye”) while maintaining the emphasis on the patient’s lay terminology rather than a clinician’s translation of the patient’s reason for visit.

Missing data.

Missing values for age, sex, race, and ethnicity, approximately 0.1%, 0.9%, 16.8%, and 30.3% respectively, were imputed by the NHAMCS investigators in the dataset prior to public release. According to the NHAMCS, the investigators imputed age and sex using a hot deck based on 3-digit ICD-9-CM code for primary diagnosis, triage level, ED volume, and geographic region, while they imputed patient ethnicity using a model-based single, sequential regression method [26]. We imputed missing values for all other variables with the median of the corresponding variable before establishing the statistical models for this study; these variables include vital signs, mode of arrival, patient’s residence type, source of payment, episode of care, whether the visit was related to injury/poisoning, triage level, and pain scale (Table 1).

Table 1. Baseline characteristics of U.S. patients presenting to the ED, stratified by advanced imaging techniques, NHAMCS 2009–2014.

Statistical analyses

Topic modeling.

NLP is a branch of computational linguistic techniques that extract and analyze information from unstructured and semi-structured text or speech data. Topic modeling is a commonly used technique for NLP, which can identify patterns hidden in the free text to evaluate an underlying theme or topic of the text [28]. The model based on the Latent Dirichlet Allocation (LDA) algorithm[29, 30] was used to break all the free text into different themes after preprocessing [31, 32].

The mathematical principles and algorithm of LDA have been described in prior research [28, 29]. Briefly, the free-text reasons for visit from each patient is a mixture of several topics composed of a set of words. For example, in a two-topic model, reasons for visit from patient 1 may contain 20% topic A (gastrointestinal problem) and 80% topic B (respiratory problems), while patient 2’s reason for visit could be 90% topic A and 10% topic B. The most common terms in the gastrointestinal topic might be “hematemesis” and “vomit”, while the respiratory problems may be composed of words including “breath”, “asthma”, and “shortness”. The LDA method can identify the mixture of topics, which describes each free-text reason for visit, while determining the mixture of words that associated with each topic. The correlation coefficient between each patient and each topic can be estimated. In this way, the free text reason for visit were transformed into a structured matrix of correlation coefficients between the patients and topics, which can be used for predicting the outcome. We employed the ldatatuning package in R for this analysis as previously described [32].

Regression modelling.

Logistic regression models were used to measure the association between the outcome and the structured and unstructured predictors, and to predict the outcomes. To determine the predictive performance in identifying patients with advanced imaging use, we analyzed three models: (1) models with structured variables; (2) models with free-text data using NLP; (3) models with both structured and free-text variables. This was done for any ADI use, any CT use (including multiple cases), any U/S scan (including multiple cases), any MRI (including multiple cases), and multiple types of ADI use.

Ten-fold cross-validation was used to validate the performance of each model. The dataset was randomly divided into 10 sets; 9 of the 10 sets were used to train the models while the one remaining was used as the testing set. The area under the receiver-operating curve (ROC) was recorded for the testing set. The average ROC curve was derived by comparing the prediction values from all 10 cross-validation testing set. The probabilities of ADI use for each patient were calculated with this model. The best cutoff of the probabilities was determined by using the point on the ROC curve with the shortest distance to the upper left corner (where sensitivity = 1 and specificity = 1). The best cutoff of the probabilities for prediction and the corresponding sensitivity, specificity, and overall accuracy were recorded [33].

To evaluate the effect of the missing values on the models, specifically for the five comorbidities variables that were not part of the dataset prior to 2012 as well as one variable that was not collected in one survey year, we performed a sensitivity analysis using cases without missing values in any of the variables considered. Among a total of 139,150 cases, there were 14,009 (10.1%) cases without any missing values. Basic data organization was done in SAS 9.4. The text analyses were performed in R 3.3.2. The modeling of logistic regression was performed in Matlab R2016b.


Characteristics of ED patients

Among 139,150 ED patient visits from December 2008 to December 2014, 21.9% of visits resulted in ADI use, including 16.8% who had CTs, 3.6% with U/S, 0.4% with MRIs, and 1.2% who had multiple types of ADI. The ADI use proportion increased in the older age groups. Females presented higher ADI use (22.8%) than males (20.7%). Hispanic patients had higher ADI use (22.5%) than non-Hispanics patients (21.8%) White had higher ADI use proportion (23%) than African American patients (18.1%). Patients from nursing home (32.9%) and private residence (21.8%) had higher proportion of ADI use than homeless patients (11.9%). Medicare patients (27.5%) and private insurance patients (23.6%) have higher ADI use proportions than Medicaid (18.6%) and of uninsured patients (17.9%). Patients who arrived by ambulance (33.1%) presented higher proportion of ADI use than patients those who did not (19.4%) (Table 1).

Factors associated with ADI use

The adjusted odds ratio of ED visits resulting in different types of ADI use (vs. no ADI use) for each variable using multinomial logistic regression analyses are presented in Table 2 and S1 Table. Adjusted analyses showed male patients were 8% less likely to receive any ADI compared to female patients (OR: 0.92 95% CI 0.90–0.95). African American patients were 21% less likely to have ADI (OR: 0.79, 95% CI 0.76–0.82). Compared to those with private insurance, patients with Medicare were 24% less likely have ADI (OR: 0.76, 95% CI 0.73–0.79), while patients with Medicaid were 23% less likely (OR: 0.77, 95% CI 0.74–0.80) and uninsured patients were 22% less likely to have ADI (OR: 0.78, 95% CI 0.74–0.81). Unadjusted analyses are presented in S2 Table.

Table 2. Adjusted odds ratio of selected characteristics associated with the use of advanced diagnostic imaging studies during the emergency department visit (vs. no advanced imaging use), NHAMCS 2009–2014.

Age, triage level, arrival mode, place of residence, and certain comorbidities were also predictive of the eventual use of ADI. For example, the odds of ADI use increased progressively with increasing age; compared to patients in the age 18–29 group, the adjusted odds of ADI use was 1.97 times higher for patients ≥ 75 years old (95% CI 1.86–2.08), 1.69 times higher for patients in the 65–74 age group (95% CI 1.59–1.79), 1.25 times higher for patients in the 45–64 age group (95% CI 1.20–1.30), and 1.11 times higher for patients in the 30–44 age group (95% CI 1.06–1.15) (Table 2). Those who arrived via ambulance were 1.87 times more likely to receive ADI than those who did not (95% CI 1.80–1.93). Compared to those who lived in a private residence, nursing home patients were 10% less likely (OR: 0.90, 95% CI 0.83–0.98), while those who were homeless were 46% less likely to have any ADI use (OR: 0.54 95% CI 0.46–0.64). Comorbidities had varying likelihood of any ADI use; patients with history of cerebrovascular disease were 1.74 times more likely (95% CI 1.63–1.85) and those with dementia were 1.56 times more likely (95% CI 1.35–1.80) than those who did not have respective comorbidities to have any ADI use (Table 2). Increasing pain scale was associated progressively with an increased likelihood of any ADI use, as well as of CT use (S1 Fig). The trends in the likelihood of CT use only and multiple types of ADI use (vs. no ADI use) closely mirrored that of any ADI use with race, age, triage level, mode arrival, place of residence, source of payment, and certain comorbidities (cerebrovascular disease and dementia) being most predictive based on the odds ratios (Table 2).

Text variables extracted using topic modelling to predict ADI use

The top 10 terms in each topic for the first 20 topics are presented in S3 Table. Although these topics cannot all be generalized into terms that are clinically meaningful, words that have been grouped into topics may indicate a theme. For example, the first topic shows a theme related to the extremities, the second gastrointestinal problems, the third respiratory problems, and the fourth trauma. The first 20 topics all show significant odds in all types of ADI use; for example, “topic 12” has an odds ratio of 0.07 (95% CI 0.04–0.11) for predicting any ADI use.

Predictive performance of multivariable logistic regression models

Applying the three logistic regression models (model 1: structured variables only, model 2: text-based reason for visit variables only, and model 3: both text-based and structured variables), we found that the predictive accuracy for ADI use was higher for models with text-based reason for visit variables only compared to models with structured variables only. The predictive accuracy was the highest when both text-based reason for visit and structured variables were included (Table 3 and Fig 1). For models that included both unstructured and unstructured variables, the AUC was 0.78 (0.77–0.78) for any ADI use, 0.79 (0.79–0.79) for CT use, 0.83 (0.82–0.84) for U/S use, 0.80 (0.79–0.80) for MRI use, and 0.78 (0.77–0.79) for multiple ADI use.

Fig 1. Receiver operating curves for the logistic regression models for different types of advanced diagnostic imaging use (Any ADI, CT, MRI, US, Multiple ADI).

The red point on the curve minimized the Euclidian distance between the ROC curve and the upper left coordinate which defined the best cutoff for this study.

Table 3. Predictive performance of logistic regression models with 10-fold classification in identifying patients with various advanced imaging use during emergency department triage, NHAMCS 2009–2014.

Estimated coefficients and standardized coefficients of the structured variables from logistic regression between the outcome of ADI use and the predictors were presented as a modeling example (S4 Table), which can be used for perspective study. Standardized coefficients can be compared to present which variable have a greater effect on the ADI use prediction. The item “whether the injury/poisoning intentional” and the immediate triage level presented highest standardized coefficients among the structured variables.

We performed a sensitivity analysis for missing values. Among the 139,150 records, there were 14,009 records without any missing values of which 3,409 (24.4%) resulted in ADI use, including 2,560 (18.3%) CTs, 613 (4.38%) USs, 65 (0.46%) MRIs, and 171 (1.22%) multiple AMIs. With 100 topic models from the unstructured data included, the AUC was 0.76 (95% CI 0.75–0.77). With the structured variables included only, the AUC was 0.70 (95% CI 0.68–0.72). With both the structured variables and 80 topics from unstructured data included, the AUC reached 0.79 (95% CI 0.78–0.81).


Improving ED efficiency may help address the continued problem and negative consequences of ED crowding in the U.S. [2, 6, 34]. One previously unexplored solution to address this problem may be to identify patients more likely to eventually obtain ADI earlier in their ED encounter. Our study applied predictive analytics and natural language processing modeling techniques with six years of nationally representative survey data to create a model to predict ADI use during the ED triage process.

One of the novel aspects of this study was the use of not only structured variables (examples: age, race, residence type), but also text-based information (reason for visit and cause of injury) via natural language processing. Specifically, we chose LDA topic modelling, which balances predictive performance and ease of information interpretation by grouping words into topics [28]. With the inclusion of reason for visit information in the model, the AUC ranged from 0.78 to 0.83 for all outcomes. When choosing the best probability cut-off given in the study (p = 0.05) as the threshold, the best overall accuracy of this model for ultrasound use, for example, reached 78% (with sensitivity of 0.73 and specificity of 0.78), which means that with the model given in the study, physicians can predict with an accuracy of 73% whether a patient will eventually receive ultrasound during their ED stay, and offers a 78% discriminatory accuracy for those who will not receive ADI.

During our exploration and model development, we observed surprising and substantial racial and socioeconomic disparities in the use of ADI in this sample. Similar to previous studies, African Americans were less likely to have ADI compared to white patients [35, 36]. There are several potential explanations for these differences. For example, some evidence suggests that injury severity varies by race, thus warranting differential use of ADI [35, 37]. In addition, the extent of overcrowding in an ED has been shown to affect the thoroughness of patients’ evaluation, which disproportionately affects hospitals that serve higher number of African Americans [37, 38]. Other potential explanations for racial differences in ADI use include provider implicit bias [35], and/or potential overuse of ADI in white patients rather than underuse by African American patients [39].

Patients with Medicaid and uninsured patients were also less likely to receive ADI compared to patients with private insurance [40]. Compared to patients that live in private residences, patients from nursing homes and patients who were homeless had decreased likelihood of any ADI use. Reasons for these disparities should be further explored in future research to determine the appropriateness of including or excluding these variables in prediction models [37] based on the clinical context, such as the one proposed in this study. This will be important to determine whether such prediction models can serve as a more objective tool to predict whether a patient will need ADI by excluding factors that may be influenced by clinician bias, for example. It may also be of value to explore the relationship between measurements of disparities, such as the role that insurance type plays in the racial differences we observed in ADI, or the influence of the ED specific characteristics such as urbanicity, teaching hospital designation, or safety net designation on ADI utilization. Further studies are needed to determine the effect of predictive clinical decision support algorithms such as the one constructed for this study, on the clinical use of ADI in settings where it can potentially be deployed to reduce racial and socioeconomic disparities in ADI use.

We also found that age, triage level, arrival mode, place of residence, and certain comorbidities were predictive of the eventual use of ADI during ED visit. As expected, patients with emergent and immediate triage levels had the highest likelihood of ADI use. These patients are often immediately placed in an ED room for workup shortly after arrival and early identification of their need for ADI would likely have less of an impact on ED LOS, and the decision tree that can lead to ADI in these patients often bypasses the traditional triage processes. However, patients who were triaged as urgent (typically triage level 3) or semi-urgent (typical triage level 4) also had increased odds of ADI use. These patients typically spend a longer portion of their LOS in the waiting area prior to being placed in a room—after which a provider typically makes the decision to pursue ADI. Urgent and semi-urgent patients comprise the majority of all ED patients (80.2% in this sample) and stand to benefit the most from this form of predictive modelling as they utilize the majority of ED ADI (79.8% of ADI in this sample).

When the use of each type of ADI (CT, MRI, U/S) was analyzed, we found that the general trends closely mirrored that of any ADI use except for U/S. One explanation for this difference is the increasing number of ultrasounds performed unofficially as a point of care test at bedside or in a fashion that was not captured in the dataset. For example, the Focused Assessment with Sonography in Trauma (FAST) ultrasound [41] is often not captured, which would result in under-reporting in the current dataset. Despite having been shown in prior research to be a poor predictor of health outcomes[42, 43], we found that patients indicating higher levels of pain on the traditional ten-point pain scale had increased odds of receiving ADI (Table 2, S1 Fig). This may reflect the fact that physicians tend to do more for patients who complain of severe pain[44].

The triage process in the ED represents the earliest in-person point of contact between a patient and a medical provider, often a nurse, after arriving in the ED. This is an extremely important encounter, but one that is often quite brief. A decision support system built on models such as the ones proposed in this study may be valuable to triage personnel, charge nurses, hospital leaders, hospital flow coordinators, and ED physicians. Further research will be needed to test the effect of using such a system in the clinical setting. Because these models were derived from nationally representative survey data, the clinical use of this type of modelling strategy may benefit from location-specific data and additional calibration of models for specific regions of the county or patient populations.

The present study has several limitations. First, missing values in the datasets affect the predictive performance of the models; sensitivity analyses were performed to limit this potential source of bias. Second, the interpretation of the topic models used for natural language processing in extracting data from unstructured information is not always straightforward as these topics are computer generated and take into account multiple layers of variable interactions. Third, the specific test ordered (example: CT of head) was not available in the dataset for analysis; however, this may be important to explore in future studies. Fourth, the dataset does not provide the results of ADI studies, the medical indication for those studies, who ordered the study, or the time of ordering the study. Therefore, we lack the temporal information to know if an ADI study was ordered for a patient immediately upon arrival by triage personnel or if was ordered later by a different provider who was privy to additional clinical information such as lab results or changes in clinical status. Additionally, we used the ED physicians’ decision to pursue ADI as the gold standard to establish the outcome for the predictive model—not whether the imaging yielded or ruled out a diagnosis. As a result, the appropriateness of these decisions cannot be assessed. To determine the utility of these models in supporting physicians’ ED triage decisions, these predictive models should be developed and prospectively validated in the clinical setting. Because these predictive models were designed on national data and focused on a specific time point (triage) during an ED encounter, it is not possible to account for the impact of serial interactions in the ED that lead toward or away from ADI. Further studies are needed to assess the impact of using this type of predictive model on the triage behaviors, ordering patterns, clinical pathways, overall imaging utilization, and ED flow.


This investigation used six years of nationally representative ED data to construct statistical models to predict the eventual use of advanced diagnostic imaging—using only the information that would be available at the time of ED triage: vital signs, general medical information, and the patient’s stated reason for visit. The overall discriminatory accuracy of these models supports prospective testing for use as an adjunct clinical decision support tool.

Supporting information

S1 Fig.

Adjusted Odds of receiving CT (left) and Ultrasound (right) by pain scale score.


S1 Table. Adjusted odds ratio of selected characteristics associated with the use of advanced diagnostic imaging studies during the emergency department visit (vs. no advanced imaging use), NHAMCS 2009–2014.


S2 Table. Crude odds ratio of characteristics associated with the use of various types of advanced imaging studies during an emergency department visit (vs. no advanced imaging use), NHAMCS 2009–2014.


S3 Table. Most frequent 10 words in each topic model for Topics 1–20.


S4 Table. Parameter estimation between the outcome of ADI use and structured imaging during the emergency department visit, NHAMCS 2009–2014.



  1. 1. Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo CA. A conceptual model of emergency department crowding. Annals of emergency medicine. 2003;42(2):173–80. pmid:12883504
  2. 2. Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects, and solutions. Annals of emergency medicine. 2008;52(2):126–36. e1. pmid:18433933
  3. 3. Eastman AB. The Future of Emergency Care in the United States Health System: an Institute of Medicine report. Clin Neurosurg. 2007;54:192–4. pmid:18504917.
  4. 4. Pines JM, Hollander JE. Emergency department crowding is associated with poor care for patients with severe pain. Annals of emergency medicine. 2008;51(1):1–5. pmid:17913299
  5. 5. Moskop JC, Sklar DP, Geiderman JM, Schears RM, Bookman KJ. Emergency department crowding, part 1—concept, causes, and moral consequences. Annals of emergency medicine. 2009;53(5):605–11. pmid:19027193
  6. 6. Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605–11 e6. pmid:23218508; PubMed Central PMCID: PMCPMC3690784.
  7. 7. Chalfin DB, Trzeciak S, Likourezos A, Baumann BM, Dellinger RP, group D-Es. Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Critical care medicine. 2007;35(6):1477–83. pmid:17440421
  8. 8. Forero R, McCarthy S, Hillman K. Access block and emergency department overcrowding. Critical Care. 2011;15(2):1.
  9. 9. Rogg JG, White BA, Biddinger PD, Chang Y, Brown DF. A Long‐term Analysis of Physician Triage Screening in the Emergency Department. Academic Emergency Medicine. 2013;20(4):374–80. pmid:23701345
  10. 10. Partovi SN, Nelson BK, Bryan ED, Walsh MJ. Faculty triage shortens emergency department length of stay. Academic Emergency Medicine. 2001;8(10):990–5. pmid:11581086
  11. 11. Soremekun OA, Capp R, Biddinger PD, White BA, Chang Y, Carignan SB, et al. Impact of physician screening in the emergency department on patient flow. The Journal of emergency medicine. 2012;43(3):509–15. pmid:22445677
  12. 12. Burström L, Nordberg M, Örnung G, Castrén M, Wiklund T, Engström M-L, et al. Physician-led team triage based on lean principles may be superior for efficiency and quality? A comparison of three emergency departments with different triage models. Scandinavian journal of trauma, resuscitation and emergency medicine. 2012;20(1):57.
  13. 13. Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Ann Emerg Med. 2017. Epub 2017/09/11. pmid:28888332.
  14. 14. Burns K, Chernyak V, Scheinfeld MH. Emergency department imaging: are weather and calendar factors associated with imaging volume? Clinical Radiology. 2016;71(12):1312.e1–.e6.
  15. 15. Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003;5(3):155–61. pmid:17472779.
  16. 16. Chokshi FH, Hughes DR, Wang JM, Mullins ME, Hawkins CM, Duszak R Jr. Diagnostic Radiology Resident and Fellow Workloads: A 12-Year Longitudinal Trend Analysis Using National Medicare Aggregate Claims Data. J Am Coll Radiol. 2015;12(7):664–9. pmid:25972250.
  17. 17. Kanzaria HK, Probst MA, Ponce NA, Hsia RY. The association between advanced diagnostic imaging and ED length of stay. The American journal of emergency medicine. 2014;32(10):1253–8. pmid:25176565
  18. 18. McCallum A. Information extraction: Distilling structured data from unstructured text. Queue. 2005;3(9):48–57.
  19. 19. Horng S, Sontag DA, Halpern Y, Jernite Y, Shapiro NI, Nathanson LA. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS One. 2017;12(4):e0174708. pmid:28384212; PubMed Central PMCID: PMCPMC5383046.
  20. 20. Claster W, Shanmuganathan S, Ghotbi N. Text Mining of Medical Records for Radiodiagnostic Decision-Making. 2008.
  21. 21. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? Journal of biomedical informatics. 2009;42(5):760–72. pmid:19683066
  22. 22. Huhdanpaa HT, Tan WK, Rundell SD, Suri P, Chokshi FH, Comstock BA, et al. Using Natural Language Processing of Free-Text Radiology Reports to Identify Type 1 Modic Endplate Changes. Journal of digital imaging. 2017. Epub 2017/08/16. pmid:28808792.
  23. 23. Shin B, Chokshi F, Lee T, Choi J. Classification of Radiology Reports Using Neural Attention Models2017.
  24. 24. McCaig LF, Burt CW. Understanding and Interpreting the National Hospital Ambulatory Medical Care Survey: Key Questions and Answers. Annals of Emergency Medicine. 2012;60(6):716–21. pmid:23083968
  25. 25. McAdams-Demarco MA, Grams ME, Hall EC, Coresh J, Segev DL. Early hospital readmission after kidney transplantation: patient and center-level associations. American journal of transplantation: official journal of the American Society of Transplantation and the American Society of Transplant Surgeons. 2012;12(12):3283–8. Epub 2012/09/29. pmid:23016838.
  26. 26. Harhay M, Lin E, Pai A, Harhay MO, Huverserian A, Mussell A, et al. Early rehospitalization after kidney transplantation: assessing preventability and prognosis. American journal of transplantation: official journal of the American Society of Transplantation and the American Society of Transplant Surgeons. 2013;13(12):3164–72. Epub 2013/10/30. pmid:24165498; PubMed Central PMCID: PMCPMC4108077.
  27. 27. Schneider D, Appleton L, McLemore T. A reason for visit classification for ambulatory care. Vital and health statistics Series 2, Data evaluation and methods research. 1979;(78):i–vi, 1–63. Epub 1979/02/01. pmid:433160.
  28. 28. Wallach HM, editor Topic modeling: beyond bag-of-words. Proceedings of the 23rd international conference on Machine learning; 2006: ACM.
  29. 29. Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. Journal of machine Learning research. 2003;3(Jan):993–1022.
  30. 30. Bíró I, Szabó J, Benczúr AA, editors. Latent dirichlet allocation in web spam filtering. Proceedings of the 4th international workshop on Adversarial information retrieval on the web; 2008: ACM.
  31. 31. Asghar MZ, Khan A, Ahmad S, Kundi FM. Preprocessing in natural language processing. Editorial board. 2013;152.
  32. 32. Vijayarani S, Ilamathi MJ, Nithya M. Preprocessing Techniques for Text Mining-An Overview.
  33. 33. Froud R, Abel G. Using ROC Curves to Choose Minimally Important Change Thresholds when Sensitivity and Specificity Are Valued Equally: The Forgotten Lesson of Pythagoras. Theoretical Considerations and an Example Application of Change in Health Status. PLoS ONE. 2014;9(12):e114468. PMC4256421. pmid:25474472
  34. 34. Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo CA Jr. A conceptual model of emergency department crowding. Annals of emergency medicine. 2003;42(2):173–80. Epub 2003/07/29. pmid:12883504.
  35. 35. Natale JE, Joseph JG, Rogers AJ, Tunik M, Monroe D, Kerrey B, et al. Relationship of Physician-identified Patient Race and Ethnicity to Use of Computed Tomography in Pediatric Blunt Torso Trauma. Acad Emerg Med. 2016;23(5):584–90. pmid:26914184.
  36. 36. Hryhorczuk AL, Mannix RC, Taylor GA. Pediatric abdominal pain: use of imaging in the emergency department in the United States from 1999 to 2007. Radiology. 2012;263(3):778–85. Epub 2012/04/27. pmid:22535565.
  37. 37. Payne NR, Puumala SE. Racial disparities in ordering laboratory and radiology tests for pediatric patients in the emergency department. Pediatric emergency care. 2013;29(5):598–606. Epub 2013/04/23. pmid:23603649.
  38. 38. Hwang U, Weber EJ, Richardson LD, Sweet V, Todd K, Abraham G, et al. A research agenda to assure equity during periods of emergency department crowding. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2011;18(12):1318–23. Epub 2011/12/16. pmid:22168197; PubMed Central PMCID: PMCPMC3368012.
  39. 39. Natale JE, Joseph JG, Rogers AJ, Mahajan P, Cooper A, Wisner DH, et al. Cranial computed tomography use among children with minor blunt head trauma: association with race/ethnicity. Archives of pediatrics & adolescent medicine. 2012;166(8):732–7. Epub 2012/08/08. pmid:22869404.
  40. 40. Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK. National trends in use of computed tomography in the emergency department. Annals of emergency medicine. 2011;58(5):452–62. e3. pmid:21835499
  41. 41. Reardon R. Ultrasound in trauma–the FAST exam. Focused assessment with sonography in trauma. Ultrasound Guide to Emergency Physicians 2013.
  42. 42. Liu N, Koh ZX, Goh J, Lin Z, Haaland B, Ting BP, et al. Prediction of adverse cardiac events in emergency department patients with chest pain using machine learning for variable selection. BMC Medical Informatics and Decision Making. 2014;14:75–. PMC4150554. pmid:25150702
  43. 43. Seupaul RA. Visual analog pain scores and the need for analgesia. Academic Emergency Medicine. 2003;10(9):1012–.
  44. 44. Zisk RY, Grey M, Medoff-Cooper B, MacLaren JE, Kain ZN. The squeaky wheel gets the grease: parental pain management of children treated for bone fractures. Pediatric emergency care. 2008;24(2):89–96. pmid:18277844