https://orcid.org/0000-0002-5172-2500
Figures
Abstract
Introduction
In the Netherlands, most emergency department (ED) patients are referred by a general practitioner (GP) or a hospital specialist. Early risk stratification during telephone referral could allow the physician to assess the severity of the patients’ illness in the prehospital setting. We aim to assess the discriminatory value of the acute internal medicine (AIM) physicians’ clinical intuition based on telephone referral of ED patients to predict short-term adverse outcomes, and to investigate on which information their predictions are based.
Methods
In this prospective study, we included adult ED patients who were referred for internal medicine by a GP or a hospital specialist. Primary outcomes were hospital admission and triage category according to the Manchester Triage System (MTS). Secondary outcome was 31-day mortality. The discriminatory performance of the clinical intuition was assessed using an area under the receiver operating characteristics curve (AUC). To identify which information is important to predict adverse outcomes, we performed univariate regression analysis. Agreement between predicted and observed MTS triage category was assessed using intraclass and Spearman’s correlation.
Results
We included 333 patients, of whom 172 (51.7%) were referred by a GP, 146 (43.8%) by a hospital specialist, and 12 (3.6%) by another health professional. The AIM physician’s clinical intuition showed good discriminatory performance regarding hospital admission (AUC 0.72, 95% CI: 0.66–0.78) and 31-day mortality (AUC 0.73, 95% CI: 0.64–0.81). Univariate regression analysis showed that age ≥65 years and a sense of alarm were significant predictors. The predicted and observed triage category were similar in 45.2%, but in 92.5% the prediction did not deviate by more than one category. Intraclass and Spearman’s correlation showed fair agreement between predicted and observed triage category (ICC 0.48, Spearman’s 0.29).
Citation: van Dam PMEL, Lasso Peña RE, Mommertz JA, Borggreve HF, van Loon NPH, Zelis N, et al. (2024) Acute internal medicine physicians’ clinical intuition based on acute care telephone referral: A prospective study. PLoS ONE 19(6): e0305566. https://doi.org/10.1371/journal.pone.0305566
Editor: Filip Haegdorens, Universiteit Antwerpen, BELGIUM
Received: January 31, 2024; Accepted: May 31, 2024; Published: June 14, 2024
Copyright: © 2024 van Dam et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Assessing the severity of the patient’s illness is an important challenge for physicians in the emergency department (ED). This risk stratification influences the urgency of diagnostic testing, treatment, and clinical decision-making. In the Netherlands, nearly all ED patients are referred after initial triage by a health professional (i.e. general practitioner (GP) or hospital specialist). The GP provides primary care to low-risk patients, and high-risk patients are referred to secondary care [1, 2]. High-risk patients (i.e. who are in need of (acute) hospital care) have to receive treatment in a timely fashion, as delays lead to overcrowding in the ED and higher mortality rates [3–6]. Rapid and accurate discrimination between high and low-risk patients is therefore an important aspect of emergency care [7].
Clinical intuition is a non-analytic, subconscious and instinctive process, through which physicians assess the severity of the patient’s illness [8–11]. The ED physician’s clinical intuition has demonstrated commendable discriminatory performance in previous studies, predicting adverse outcomes in ED patients (i.e. hospital admission or short-term mortality) with an area under the receiver operating characteristics curve (AUC) ranging from 0.74 to 0.85 [12–15]. Therefore, clinical intuition is an important tool for discrimination between high and low-risk patients in the ED.
Clinical intuition based on information shared between the referring physician and the ED physician could offer a rapid and early risk stratification tool before the patient arrives at the ED. During the referral of a patient by telephone, the ED physician can potentially estimate the severity of the patient’s illness and the need for hospital admission in the prehospital setting. This early risk stratification would allow the ED physician to manage the patient flow in the ED more effectively. However, as far as we know, no studies have investigated the discriminatory performance of clinical intuition based on telephone referral information to predict adverse outcomes.
Therefore, in this prospective study, we aimed to investigate the discriminatory performance of the ED physician’s clinical intuition based on telephone referral information to predict short-term outcomes. In addition, we aimed to investigate on which information the ED physicians based their predictions.
Methods
Study design and setting
This prospective cohort study was performed at the ED of the Maastricht University Medical Center + (MUMC+). This is a combined secondary/tertiary care center in the Netherlands, with 22,000 ED visits every year. The medical ethics committee of the MUMC+ approved this study (METC 2018–0838) and waived the requirement of informed consent. This study was conducted and reported in accordance with the STROBE guidelines (Strengthening the Reporting of Observation studies in Epidemiology) (S1 Table) [16].
In contrast to many other countries where patients can visit the ED without referral by a health professional (open access ED), most non-trauma ED patients in the Netherlands are referred by a GP or hospital specialist. After this initial evaluation, the referring physicians call the ED physicians to inform them about the reasons for referral. Self-referral is possible, however higher costs have been introduced to discourage self-referral [2].
Study sample
Adult patients (18 years or older) who were referred by telephone to the ED of the MUMC+ for assessment and treatment by an internist, were eligible for inclusion. In our ED, internists and their residents attend all patients with problems related to internal medicine and gastroenterology, in addition to non-differentiated non-trauma patients. Patients were included in the period between March 18th and June 30th 2022.
Patients were included by the on-call acute internal medicine (AIM) physician. AIM physicians are internists who are specialized in stabilization, diagnosis and treatment of patients with acute presentations of internal diseases in the ED, the medical ward and the outpatients clinic. Emergency physicians (EPs) are specialized in emergency medicine (medical patients as well as trauma patients) in the ED. In many Dutch hospitals, AIM physicians and EPs work together in the ED, making use of each other’s specific expertise [2]. The AIM physicians who participated in our study are specialists or residents in the last stage of their specialist training. Since the AIM physicians were predominantly present on weekdays during day shifts (and to a lesser extent in the evenings and weekends), the study sample was derived on a convenience sample.
Questionnaire
Patients were included in the study using a questionnaire filled out by the AIM physician during or immediately after the telephone referral. The questions were answered before the patient arrived at the ED and the responses could not be changed afterwards. The questionnaire was developed based on questionnaires from previous studies [12, 14, 17–25], and adapted to be feasible in clinical practice in the ED. The questionnaire is shown in the supporting information (S1 Fig).
The questionnaire consisted of nine questions (Q): Q1) Does the referring physician know the patient? (Yes/No); Q2) Which information was given during the telephone referral? (Previous medical history, Alarm symptoms, Physical examination, Diagnostic testing, Previous treatment, Diagnosis); Q3) What is your gut feeling? (Everything fits/Sense of alarm) [26]; Q4) How severely ill is this patient? (0–100%); Q5) Which Manchester Triage System (MTS) urgency do you expect this patient to receive? (Red/Orange/Yellow/Green/Blue) [27]; Q6) What is the chance that this patient will be admitted to the hospital? (0–100%); Q7) To what department will this patient be admitted? (“General ward” or “Intensive care unit/Medium care unit (ICU/MCU)”; Q8) What is the chance that this patient will be admitted for longer than 7 days? (0–100%); Q9) What is the chance that this patient dies within 31 days? (0–100%).
Data collection
From the ED charts and the electronic medical records, data were collected on age, sex, comorbidity based on the Charlson Comorbidity Index (CCI), mode of transportation to the ED, and triage category based on the MTS [27, 28]. The triage was performed by the triage nurse, who was unaware of the triage category predicted by the AIM physician. It was also recorded whether the referring physician knew the patient (i.e. was the referral based on a continuous clinical relationship or on the first clinical encounter?), and whether the AIM physician who received the telephone referral was the same as the physician who later made the treatment plan in the ED. The main reason for referral to the ED was recorded according to the International Classification of Diseases (ICD)-10 system [29]. The patients’ vital signs and the results of laboratory tests during the ED visit were also retrieved. Finally, data were collected on hospital admission, length of hospital stay, admission to ICU/MCU, and 31-day mortality. Data on mortality were verified using the medical records. In the Netherlands, all deaths are registered by the municipal administration office, and these data are linked to the medical records. Data collection was performed by medical students and resident doctors, who were not blinded for the results of the questionnaires. The quality of the data was checked by sample by two researchers, and discrepancies were resolved through discussion with another investigator.
Outcomes
The primary outcomes for assessing the discriminatory performance of the AIM physician’s clinical intuition during telephone referral were hospital admission and MTS triage category in the ED. The secondary outcome was 31-day mortality.
Statistical analysis
Regarding sample size, in order to investigate the discriminatory performance of clinical intuition in a prehospital setting to predict hospital admission, we aimed to comply with the rule of thumb to include approximately 100 patients who met the primary outcome, similar to other studies [30]. Assuming an admission rate of 65%, we calculated a required sample size of approximately 154 patients (65% of 154 patients equals 100 admitted patients). In order to address any selection bias in our convenience sample, we used the Chi square test to compare the age, hospital admission rate and MTS triage categories of our study sample to all ED patients with problems related to internal medicine or gastroenterology during the entire year of 2022.
Baseline characteristics were analyzed using descriptive statistics. Continuous variables were reported as medians with interquartile ranges (IQR), remaining variables were reported as totals with percentages. In case of missing estimates on clinical intuition or outcome, the patient could not be included in further analysis. To identify which information during the telephone referral is most important to predict the patients’ outcome, we performed univariate logistic regression analyses, and odds ratios (OR) with 95% confidence intervals (95% CI) were reported. For the purpose of the univariate logistic regression analysis, we divided the observed MTS triage category into urgent (i.e. yellow, orange and red) and non-urgent (i.e. blue and green). A p-value < 0.05 was considered to be statistically significant. The agreement between the predicted and observed MTS triage category was analyzed using the Spearman’s rank correlation and the intraclass correlation coefficient (ICC). The discriminatory performance of clinical intuition was assessed by calculating an AUC with 95% CI. An AUC of 0.5 corresponds with very poor discriminatory performance, whereas an AUC of 1.0 means perfect accuracy. Calibration was assessed by visually inspecting the calibration plot.
All data were analyzed using IBM SPSS Statistics for Windows, IBM Corporation, Armonk NY, version 25.0. DeLong tests were performed in R, version 4.0.0.
Results
Study sample
During the study period, 333 ED patients were included after telephone referral (Table 1). The median age was 70 years (IQR 56–80), and 191 patients (57.4%) were male. The median time between referral and arrival to the ED was 54 minutes (IQR: 31–97 minutes).
In total, 202 patients (60.7%) were admitted to the hospital, 188 patients (56.4%) were triaged as urgent (yellow, orange or red, according to MTS), and 30 patients (9.0%) died within 31 days after the ED visit. The 31-day follow-up period was complete for all patients.
During the study period, a total of 1664 patients visited our ED. We found no significant differences in MTS triage category and hospital admission rate of our study sample when compared to all ED patients with problems related to internal medicine or gastroenterology in 2022 (S2 Table).
Questionnaires
In total, 10 AIM physicians with a median experience level of 17 years (IQR 13–22) participated in the inclusion of patients. The answers to the questions in the questionnaire are summarized in Table 2. In total, 172 patients (51.7%) were referred by a GP, 146 patients (43.8%) by a hospital specialist and 12 patients (3.6%) by another health professional (e.g. nursing home physician or ambulance). In most cases (63.0%), the referral was based on a continuous clinical relationship (i.e. the referring physician knew the patient). In 130 patients (39.0%), the AIM physician who received the telephone referral was the same as the physician who later made the treatment plan in the ED (S3 Table).
The information given during telephone referral was recorded (Table 2). Out of the 6 items that the AIM physician could fill out on the questionnaire, a median of 3 items (IQR 2–4) was scored. This number of items was comparable in those who were referred by a GP and those who were referred by a specialist. The medical history was discussed most frequently (82.6%), followed by the presence of alarm symptoms (61.6%) and the results of physical examination (48.0%).
Prediction of hospital admission
In our sample, 202 patients (60.7%) were admitted to the hospital and 9 patients (2.7%) were admitted to ICU (Table 1). The median length of hospital stay was 5 days (IQR 2–9) and 64 patients (19.2%) were admitted longer than 7 days. The AIM physicians predicted a median chance of hospital admission of 80% (IQR: 60–100) and a median chance of prolonged hospital admission (>7 days) of 30% (IQR: 5–50). The predicted number of ICU/MCU admissions was 4 (1.2%), and 3 out of the 9 ICU/MCU admissions were predicted correctly.
The clinical intuition of the AIM physicians showed good discriminatory performance regarding hospital admission with an AUC of 0.72 (95% CI: 0.66–0.78). The calibration plot showed average overestimation of the chance of admission by the AIM physician and a slope of <1 (Fig 1). In a subgroup analysis of patients where the AIM physician who received the telephone referral was the same as the physician who later made the treatment plan in the ED, we found no significant differences in observed outcomes or discriminatory performance regarding hospital admission (S3 Table).
Calibration plot of the predicted chance of hospital admission (x axis) using clinical intuition. The calibration plot shows average overestimation of the chance of admission and a slope of < 1 compared to the dotted line (which would indicate perfect calibration).
Of the patients who were referred by a GP, 107 patients (62.2%) were admitted to the hospital. The clinical intuition yielded an AUC of 0.74 (95% CI: 0.65–0.82) regarding admission. Of the patients who were referred by a hospital specialist, 86 patients (58.9%) were admitted and the clinical intuition yielded an AUC of 0.67 (95% CI: 0.58–0.77).
The univariate logistic regression analysis showed that age 65 years or older and the presence of a sense of alarm in the AIM physician were the significant predictors of hospital admission during telephone referral (Table 3).
Prediction of MTS triage category
The majority of patients (56.4%) were triaged at the ED by the triage nurse as urgent (yellow, orange or red, according to MTS) and most patients were triaged as green or yellow (Table 4). According to the AIM physicians, green (120 patients, 36.0%) and yellow (155 patients, 46.5%) were the most frequently predicted triage category as well.
In total, the predicted triage category by the AIM physician was similar to the triage category observed in the ED in 45.2%, and did not deviate by more than one urgency level in 92.5% of the patients (Table 4). The ICC was 0.48 (95% CI: 0.35–0.58, p <0.001) and Spearman’s correlation coefficient was 0.29 (95% CI: 0.19–0.39, p <0.001), indicating moderate to fair agreement between the predicted and observed MTS triage category. The accuracy of the predicted triage category did not depend on whether the referring physician knew the patient as in both the correct and incorrect predictions, the physician knew the patient in approximately 70% of cases (70.1% and 68.2%). The predicted MTS triage category by the AIM physician showed poor discriminatory performance regarding hospital admission with an AUC of 0.61 (95% CI: 0.55–0.67).
We found that in both patients who were referred by a GP and those who were referred by a hospital specialist, there was at most a moderate correlation between the predicted and observed triage category. In patients who were referred by a GP, the ICC was 0.52 (CI 95%: 0.35–0.65, p <0.001) and Spearman’s correlation coefficient was 0.33 (95% CI: 0.19–0.47, p<0.001), indicating fair agreement (S4 Table). In the patients who were referred by a hospital specialist, the ICC was 0.36 (CI 95%: 0.11–0.54, p = 0.004) and the Spearman’s correlation coefficient was 0.20 (95% CI: 0.04–0.36, p = 0.014), indicating poor agreement (S5 Table).
The univariate logistic regression analysis showed that the presence of a sense of alarm in the AIM physician was the only significant predictor of both the predicted and observed MTS triage category during telephone referral (Table 5).
Prediction of mortality
In our sample, 30 patients (9.0%) died within 31 days after the ED visit (Table 1). The AIM physicians predicted a median chance of 31-day mortality of 10% (Table 2). The clinical intuition of the AIM physicians showed good discriminatory performance to predict 31-day mortality with an AUC of 0.73 (95% CI: 0.64–0.81). The calibration plot showed average overestimation of the 31-day mortality risk by AIM physicians and a slope of <1 (Fig 2).
Calibration plot of the predicted chance of 31-day mortality (x axis) using clinical intuition. The calibration plot shows average overestimation of the chance of mortality and a slope of < 1 compared to the dotted line (which would indicate perfect calibration).
The univariate logistic regression analysis showed that age 65 years or older and the presence of a sense of alarm in the AIM physician were the significant predictors of 31-day mortality (Table 3). However, the ORs were not statistically significant.
Discussion
In this prospective study, we investigated the discriminatory performance of the clinical intuition of AIM physicians based on telephone referral of ED patients to predict short-term outcomes. In our cohort of 333 patients, the AIM physicians filled out questionnaires during or immediately after telephone referral by a GP, specialist or other health professional. The AIM physicians’ clinical intuition to predict hospital admission showed good discriminatory performance with an AUC of 0.72, and the calibration plot showed average overestimation of the chance of admission. The discriminatory performance to predict hospital admission was slightly higher in patients who were referred by a GP than in those who were referred by a hospital specialist (AUC 0.74 versus 0.67). We found that age 65 years or older and the presence of a sense of alarm in the AIM physician were significant predictors of hospital admission during telephone referral. The AIM physicians predicted the same MTS triage category in the ED in just under half of the cases (45.2%), but in the majority of the cases (92.5%) the prediction did not deviate by more than one urgency level. The accuracy of this prediction did not depend on who referred the patient (GP or hospital specialist), nor on whether the referring physician knew the patient. We found that the presence of a sense of alarm in the AIM physician was the only significant predictor of the MTS triage category. The AIM physicians’ clinical intuition to predict 31-day mortality showed good discriminatory performance with an AUC of 0.73. However, given the low number of deaths, our study was underpowered to adequately analyze this secondary outcome.
Clinical intuition in the prehospital setting
Our findings show that overall the discriminatory performance of the clinical intuition of AIM physicians during telephone referral of ED patients is good. Therefore, AIM physicians have the opportunity to assess the severity of the patients’ illness in the prehospital setting. In our cohort, the patients took a median of 54 minutes to arrive at the ED after telephone referral. Therefore, the prediction in the prehospital setting has a longitudinal character. This may affect the accuracy of the prediction, since the patients’ clinical condition can improve or worsen between referral and arrival. The accuracy of the prediction could also be influenced by who the referring physician is. GPs usually refer after clinically assessing the patient themselves. When a hospital specialist refers a patient, this assessment usually takes place by telephone. However, in our cohort we found no significant differences in discriminatory performance between these groups. Furthermore, whether the referring physician knew the patient did not play a major role in the accuracy of the prediction.
Our findings regarding the discriminatory performance of clinical intuition are in line with those of other studies that report discriminatory performance of clinical intuition regarding short-term mortality or hospital admission with an AUC ranging from 0.71 to 0.77 [12–14, 31]. However, these studies were all performed in an hospital setting (i.e. ED or acute medical unit). To our knowledge, this is the first prospective study to investigate the discriminatory performance of clinical intuition based on telephone referral in a prehospital phase. One study in the UK assessed the accuracy of telephone triage in identifying the need for emergency care in more than 40,000 patients with suspected COVID-19 infection [32]. The authors found that 60% of the patients received a non-urgent assessment and were indeed at low risk of adverse outcome. Another study assessed the accuracy to identify life-threatening conditions during telephone calls concerning patients with loss of consciousness in a Danish police-operated emergency call-center [33]. They found a sensitivity of 82% and a positive predictive value of 39%.
In our study, the AIM physician was able to make a good estimate of the risk of adverse outcome based on a telephone referral of a patient to the ED. The calibration plot showed average overestimation of the chance of admission and mortality, indicating that the physician uses a safe margin when estimating the severity of the patients’ illness. In our study, the accuracy to predict the MTS triage category was lower and there was only fair agreement between the predicted and observed triage category. Previous studies showed inferior performance of the MTS triage category in older medical patients [34]. In our study, the MTS showed poor discriminatory performance to predict hospital admission and with an AUC of only 0.61.
Our group of AIM physicians consisted of ten physicians with a median of 17 years (IQR: 13–22) of experience. The results of our study show that our group of experienced physicians can play an important coordinating role within the acute care chain, creating a possibility for early risk stratification, to assist in the logistics of a crowded ED and to guide allocating healthcare resources. Our results also emphasize the importance of providing information by the referring GP or hospital specialist.
Study limitations
Our study has several limitations. First, our study was performed in a single medical centre, limiting the generalizability of the results. However, our sample of ED patients and the number of participating AIM physicians was relatively large, and the follow-up of all patients was complete. Second, there is a risk of bias since we used a convenience sample. To address selection bias, we compared the age, MTS triage categories and hospital admission rate of our study sample with all medical (non-trauma) ED patients during an entire year and in a previous study in our ED, and found no differences (S2 Table) [35]. Therefore, the study sample appears to be a representative sample of the ED population at the MUMC+. Third, there is a risk of bias since in 130 patients (39.0%) the AIM physician who received the telephone referral was the same as the physician who later made the treatment plan in the ED. However, in a subgroup analysis in these patients we found no significant differences in observed patient outcomes or discriminatory performance (S3 Table). Last, in studies assessing clinical intuition there is a theoretical risk of bias caused by the Hawthorne effect, because completing questions regarding clinical intuition may influence the outcome of the patients’ treatment in the ED (e.g. decisions regarding hospital admission). In our study, the questionnaires were completed in the prehospital phase and were not modified afterwards. In addition, the physician who treated the patient in the ED was often not the one who received the telephone referral, as the median time between the referral and the patients’ arrival in the ED was 54 minutes.
Conclusion
In conclusion, the clinical intuition of an AIM physician based on a telephone referral can be used to accurately predict hospital admission and the triage category before the patients enter the ED. The results of our study suggest that when experienced physicians fulfil a coordinating role within the acute care chain, this creates a possibility for early risk stratification, to assist in the logistics in a crowded ED and to guide allocating healthcare resources.
Supporting information
S1 Table. STROBE statement, checklist of items that should be included in reports of observational studies.
https://doi.org/10.1371/journal.pone.0305566.s002
(DOCX)
S2 Table. Patient characteristics of the study sample.
https://doi.org/10.1371/journal.pone.0305566.s003
(DOCX)
S3 Table. Subgroup analysis in patients where the AIM physician who received the telephone referral was the same as the physician who made the treatment plan in the ED.
https://doi.org/10.1371/journal.pone.0305566.s004
(DOCX)
S4 Table. Predicted versus observed MTS triage category in patients referred by the GP.
https://doi.org/10.1371/journal.pone.0305566.s005
(DOCX)
S5 Table. Predicted versus observed MTS triage category in patients referred by hospital specialists.
https://doi.org/10.1371/journal.pone.0305566.s006
(DOCX)
Acknowledgments
PD, RLP, JM and PS collected clinical data for this study. PD and RP interpreted data. PD, RLP and PS performed the statistical analysis. PD drafted the first version of the manuscript. RLP, HB, NL, NZ, DW, RH, DP, JC and PS critically reviewed the manuscript.
References
- 1. Thijssen WA, Giesen PH, Wensing M. Emergency departments in The Netherlands. Emerg Med J. 2012;29(1):6–9. pmid:22036937
- 2. Kremers MNT, Wachelder JJH, Nanayakkara PWB, Haak HR. Organisation of internal medicine in acute care in the Netherlands: a detailed overview. Neth J Med. 2020;78(5):251–60. pmid:33093250
- 3. van der Linden N, van der Linden MC, Richards JR, Derlet RW, Grootendorst DC, van den Brand CL. Effects of emergency department crowding on the delivery of timely care in an inner-city hospital in the Netherlands. Eur J Emerg Med. 2016;23(5):337–43. pmid:25831039
- 4. Lu TC, Tsai CL, Lee CC, Ko PC, Yen ZS, Yuan A, et al. Preventable deaths in patients admitted from emergency department. Emerg Med J. 2006;23(6):452–5. pmid:16714507
- 5. Ter Avest E, Onnes BT, van der Vaart T, Land MJ. Hurry up, it’s quiet in the emergency department. Neth J Med. 2018;76(1):32–5. pmid:29380730
- 6. Liew D, Liew D, Kennedy MP. Emergency department length of stay independently predicts excess inpatient length of stay. Med J Aust. 2003;179(10):524–6. pmid:14609414
- 7. Gill TM. The central role of prognosis in clinical decision making. JAMA. 2012;307(2):199–200. pmid:22235093
- 8. Stolper E, van Bokhoven M, Houben P, Van Royen P, van de Wiel M, van der Weijden T, et al. The diagnostic role of gut feelings in general practice. A focus group study of the concept and its determinants. BMC Fam Pract. 2009;10:17. pmid:19226455
- 9. Vogel L. Gut feelings a strong influence on physician decisions. CMAJ. 2018;190(33):E998–E9. pmid:30127045
- 10. Van den Brink N, Holbrechts B, Brand PLP, Stolper ECF, Van Royen P. Role of intuitive knowledge in the diagnostic reasoning of hospital specialists: a focus group study. BMJ Open. 2019;9(1):e022724. pmid:30696671
- 11. Latten GH, Claassen L, Coumans L, Goedemondt V, Brouwer C, Muris JW, et al. Vital signs, clinical rules, and gut feeling: an observational study among patients with fever. BJGP Open. 2021;5(6). pmid:34475019
- 12. Zelis N, Mauritz AN, Kuijpers LIJ, Buijs J, de Leeuw PW, Stassen PM. Short-term mortality in older medical emergency patients can be predicted using clinical intuition: A prospective study. PLoS One. 2019;14(1):e0208741. pmid:30601815
- 13. Rohacek M, Nickel CH, Dietrich M, Bingisser R. Clinical intuition ratings are associated with morbidity and hospitalisation. Int J Clin Pract. 2015;69(6):710–7. pmid:25689155
- 14. Brabrand M, Hallas J, Knudsen T. Nurses and physicians in a medical admission unit can accurately predict mortality of acutely admitted patients: a prospective cohort study. PLoS One. 2014;9(7):e101739. pmid:25019354
- 15. Sinuff T, Adhikari NK, Cook DJ, Schunemann HJ, Griffith LE, Rocker G, et al. Mortality predictions in the intensive care unit: comparing physicians with scoring systems. Crit Care Med. 2006;34(3):878–85. pmid:16505667
- 16. Cuschieri S. The STROBE guidelines. Saudi J Anaesth. 2019;13(Suppl 1):S31–S4. pmid:30930717
- 17. Beglinger B, Rohacek M, Ackermann S, Hertwig R, Karakoumis-Ilsemann J, Boutellier S, et al. Physician’s first clinical impression of emergency department patients with nonspecific complaints is associated with morbidity and mortality. Medicine (Baltimore). 2015;94(7):e374. pmid:25700307
- 18. Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NK. The "surprise question" for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ. 2017;189(13):E484–E93. pmid:28385893
- 19. Henning DJ, Bhatraju PK, Johnson NJ, Kosamo S, Shapiro NI, Zelnick LR, et al. Physician Judgment and Circulating Biomarkers Predict 28-Day Mortality in Emergency Department Patients. Crit Care Med. 2019;47(11):1513–21. pmid:31369425
- 20. Moretti D, Buncuga MG, Scolari Pasinato CM, Rossi FE, Quinones ND, Laudanno CD. [Value of the PROFUND index in comparison with clinical intuition for the prognosis of multipathological hospitalized patients.]. Rev Fac Cien Med Univ Nac Cordoba. 2021;78(4):376–83.
- 21. Oliva-Fanlo B, March S, Gadea-Ruiz C, Stolper E, Esteva M, group CO. Prospective Observational Study on the Prevalence and Diagnostic Value of General Practitioners’ Gut Feelings for Cancer and Serious Diseases. J Gen Intern Med. 2022;37(15):3823–31. pmid:35088202
- 22. Radtke A, Pfister R, Kuhr K, Kochanek M, Michels G. Is ’gut feeling’ by medical staff better than validated scores in estimation of mortality in a medical intensive care unit?—The prospective FEELING-ON-ICU study. J Crit Care. 2017;41:204–8. pmid:28577477
- 23. Turnbull S, Lucas PJ, Redmond NM, Christensen H, Thornton H, Cabral C, et al. What gives rise to clinician gut feeling, its influence on management decisions and its prognostic value for children with RTI in primary care: a prospective cohort study. BMC Fam Pract. 2018;19(1):25. pmid:29402235
- 24. van der Vegt AE, Holman M, ter Maaten JC. The value of the clinical impression in recognizing and treating sepsis patients in the emergency department. Eur J Emerg Med. 2012;19(6):373–8. pmid:23108381
- 25. White N, Kupeli N, Vickerstaff V, Stone P. How accurate is the ’Surprise Question’ at identifying patients at the end of life? A systematic review and meta-analysis. BMC Med. 2017;15(1):139. pmid:28764757
- 26. Stolper E, Van Royen P, Van de Wiel M, Van Bokhoven M, Houben P, Van der Weijden T, et al. Consensus on gut feelings in general practice. BMC Fam Pract. 2009;10:66. pmid:19761589
- 27. Cooke MW, Jinks S. Does the Manchester triage system detect the critically ill? J Accid Emerg Med. 1999;16(3):179–81. pmid:10353042
- 28. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83. pmid:3558716
- 29.
WHO. The ICD-10 classification of diseases and related health problems. World Health Organization (WHO); 2004.
- 30. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–26. pmid:26553135
- 31. Wiswell J, Tsao K, Bellolio MF, Hess EP, Cabrera D. "Sick" or "not-sick": accuracy of System 1 diagnostic reasoning for the prediction of disposition and acuity in patients presenting to an academic ED. Am J Emerg Med. 2013;31(10):1448–52. pmid:23972480
- 32. Marincowitz C, Stone T, Bath P, Campbell R, Turner JK, Hasan M, et al. Accuracy of telephone triage for predicting adverse outcomes in suspected COVID-19: an observational cohort study. BMJ Qual Saf. 2022.
- 33. Bach A, Christensen EF. Accuracy in identifying patients with loss of consciousness in a police-operated emergency call centre—first step in the chain of survival. Acta Anaesthesiol Scand. 2007;51(6):742–6. pmid:17465976
- 34. Brouns SHA, Mignot-Evers L, Derkx F, Lambooij SL, Dieleman JP, Haak HR. Performance of the Manchester triage system in older emergency department patients: a retrospective cohort study. BMC Emerg Med. 2019;19(1):3. pmid:30612552
- 35. van Dam PMEL, Lievens S, Zelis N, van Doorn WPTM, Meex SJR, Cals JWL, et al. Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study. Ann Med. 2023;55(2):2290211. pmid:38065678