To assess the impact of a clinical decision model for febrile children at risk for serious bacterial infections (SBI) attending the emergency department (ED).
Randomized controlled trial with 439 febrile children, aged 1 month-16 years, attending the pediatric ED of a Dutch university hospital during 2010-2012. Febrile children were randomly assigned to the intervention (clinical decision model; n=219) or the control group (usual care; n=220). The clinical decision model included clinical symptoms, vital signs, and C-reactive protein and provided high/low-risks for “pneumonia” and “other SBI”. Nurses were guided by the intervention to initiate additional tests for high-risk children. The clinical decision model was evaluated by 1) area-under-the-receiver-operating-characteristic-curve (AUC) to indicate discriminative ability and 2) feasibility, to measure nurses’ compliance to model recommendations. Primary patient outcome was defined as correct SBI diagnoses. Secondary process outcomes were defined as length of stay; diagnostic tests; antibiotic treatment; hospital admission; revisits and medical costs.
The decision model had good discriminative ability for both pneumonia (n=33; AUC 0.83 (95% CI 0.75-0.90)) and other SBI (n=22; AUC 0.81 (95% CI 0.72-0.90)). Compliance to model recommendations was high (86%). No differences in correct SBI determination were observed. Application of the clinical decision model resulted in less full-blood-counts (14% vs. 22%, p-value<0.05) and more urine-dipstick testing (71% vs. 61%, p-value<0.05).
In contrast to our expectations no substantial impact on patient outcome was perceived. The clinical decision model preserved, however, good discriminatory ability to detect SBI, achieved good compliance among nurses and resulted in a more standardized diagnostic approach towards febrile children, with less full blood-counts and more rightfully urine-dipstick testing.
Nederlands Trial Register NTR2381
Citation: de Vos-Kerkhof E, Nijman RG, Vergouwe Y, Polinder S, Steyerberg EW, van der Lei J, et al. (2015) Impact of a Clinical Decision Model for Febrile Children at Risk for Serious Bacterial Infections at the Emergency Department: A Randomized Controlled Trial. PLoS ONE 10(5): e0127620. https://doi.org/10.1371/journal.pone.0127620
Academic Editor: Susanna Esposito, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Università degli Studi di Milano, ITALY
Received: December 19, 2014; Accepted: April 4, 2015; Published: May 29, 2015
Copyright: © 2015 de Vos-Kerkhof et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Authors EK and RN are supported by ZonMW, the Dutch organization for Health Research and Development, and Erasmus University Medical Centre Rotterdam. Author RO is supported by a fellowship of the European Society of Pediatric Infectious Diseases. No funding bodies had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Fever is one of the most common symptom among children presenting to the emergency department (ED) [1–3] and accountable for 10–20% of all acute admissions. [4–6] Fever may have various causes, ranging from self-limiting viral infections to serious bacterial infections (SBI) (e.g. septicemia, pneumonia, urinary tract infections). Febrile children at the ED pose a diagnostic challenge, as physicians need to identify that relatively small proportion of SBI with a potential fatal course, in this large group of children with self-limiting diseases.[7–9] To support physicians several guidelines and decision models have been developed focussing on improving diagnosis, limiting diagnostic tests and improved cost-effectiveness. ([4, 7, 10–17] However, the true impact of these diagnostic tools in clinical practice, taking the translation of diagnostic risk predictions to clinical management recommendations and the subsequent compliance of the clinicians with these recommendations into account, is hardly subject of research. [18–20]
In this study we aimed to evaluate the impact of a previously developed clinical decision model for febrile children in the daily practice of the ED. This model includes clinical signs, symptoms and the biomarker C- reactive protein (CRP).  Results of this study will bridge the often remaining gap in translating decision models into clinical practice.[19, 21, 22]
Study design and setting
We conducted a randomized controlled trial of a clinical decision model at the emergency department (ED) of the Erasmus MC-Sophia Children’s Hospital in Rotterdam, The Netherlands. The protocol and CONSORT statement are available as supporting information; see S2 and S3 Files. This large inner-city university hospital is visited annually by nearly 9000 children with a mixed ethnic population of which 90% involved basic pediatric care. During the study period there was no national guideline available for handling febrile children at the ED, although most physicians were familiar with the NICE febrile child guideline.
We prospectively enrolled all consecutive pediatric patients (≥1 month—<16 years) presenting with fever at our ED from the first of September 2010 until June 30, 2012. Febrile children were eligible if fever had been noted at home in the 24 hours prior to presentation, when body temperature measured at the ED was ≥38.5°C or fever was used as a positive discriminator of the Manchester Triage System (MTS). From the major principle of reducing diagnostic uncertainty by using a diagnostic decision model  we excluded well appearing febrile children (no amber/red alarming signs) with a clear focus of uncomplicated rhinitis/otitis and severely ill children (emergent triage category).[23, 24] Children with chronic co-morbidity were excluded because of their increased risk of having serious infections and developing a complicated course. Finally, children who reattended the ED within one week of their first presentation were only included at their initial visit.
The intervention consisted of the implementation of the clinical decision model, based on a polytomous logistic regression model of 2,717 febrile children who presented to the ED and included eleven predictors of pneumonia and other SBI versus no SBI (Table 1). Instructions on the use of the clinical decision model and the study process included individual briefings, practice cases for emergency staff (nurses/physicians), information by email, and posters mounted at the ED. During the trial period, feedback and periodic teaching sessions were organized. The clinical decision model presented patient-specific risk estimates (percentages) for both pneumonia and other SBI, which were categorized in high- or low risk groups and consequently were accompanied by recommendations for further diagnostic testing (Fig 1). These recommendations and risk estimates were only presented for children allocated to the intervention group; others were designated ‘usual care’ after completion of entering the model predictors (control group). Cut-off points for additional diagnostic testing were based on the diagnostic performance of the original prediction model at different risk thresholds, and were agreed upon by expert panel. For the chosen cut-off points, the expected gain was maximized based on the ratio of false positive/negative errors and expected adverse consequences of a delayed diagnosis SBI. For high-risk children, nurses were instructed to initiate additional testing before physician’s assessment. This recommendation implied chest-radiography in children with estimated risk for pneumonia ≥15% and urine-dipstick and culture if estimated risk for other SBI was ≥30%. Following the local protocol  and international guidelines  urine-dipstick testing was also recommended in low-risk children without a clear focus for their febrile illness. Nurses and physicians were blinded for these cut-off values and for the contribution of predictors on risk scores. Physicians were able to overrule the recommendations by adding to or refraining from additional diagnostic testing. In the control group the physician first examined the patient and ordered diagnostic procedures according to their own judgment, but with the knowledge of a CRP-value which was required to enable randomization.
Clinical decision model with its predictors (above) and recommendations (below).
All children who attended the ED were routinely triaged with the Manchester Triage System (MTS).[23, 24, 28, 29] After patient triage, model predictors were entered by the nurse for each individual patient. We collected patient characteristics (e.g. gender, age, reason of ED visit), referral profile, duration of the febrile episode, clinical signs/symptoms, observations and measures from physical examination (e.g., vital signs, clinical appearance). Data on all performed laboratory tests (e.g. full blood-count, CRP), additional diagnostic tests (e.g. chest-radiography, urine-dipstick, blood/urine-culture) and treatment/follow-up were registered prospectively in the computer-based hospital information system. The clinical decision model was implemented as a stand-alone device, accessible from each computer at the ED. A central logbook recorded data from all entered participants (Fig 1). Completing all variables of the clinical decision model was mandatory before randomization could be initiated. Once the risk estimate was shown to the nurse, no changes could be made to the entered model predictors. The randomization mechanism, based on even/odd seconds indicated by the digital computer clock, was unknown to nurses and physicians.
Ethics was obtained by the institutional review board (IRB) of the ErasmusMC. According to IRB-review the intervention contained no additional risks as patients were not subjected to additional operations and no rules of conduct were imposed. Written informed consent was required by IRB and it was obtained from all caretakers, or guardians on behalf of the children enrolled (MEC-2008-071).
Evaluation of the clinical decision model included diagnostic performance and feasibility of the model. Primary outcome measures were correct diagnoses (SBI) and their related false positive/negative diagnoses by using the clinical decision model in routine practice. Secondly, we measured process outcomes including length of stay (LOS) at the ED and items on diagnostics, treatment and follow-up as defined below.
Previous research in the same setting showed that approximately 50% of febrile children at the ED were submitted to diagnostic procedures (blood tests, urine-cultures and chest-radiography). To detect a reduction of unnecessary diagnostic tests from 50% to 35% (false positives) to improve our primary outcome of correct diagnosis, the intervention and control group should include 180 children each (80% power; type-I error of 5%; 2-sided test). This number of patients also allows detecting a difference of 10 minutes patients’ ED length of stay (30 minutes standard deviation (SD)). Adjusted for 10–15% dilution effect by nurses managing both intervention and control patients during the same shift  and adjusted for 10% not evaluable cases, the trial planned to include 500 children with fever.
Evaluation clinical decision model.
We first assessed the performance of the clinical prediction model for our trial population by evaluating discriminative ability according to the area-under-the-receiver-operating-characteristic-curve (AUC) and calibration. Predicted risks of pneumonia and other SBI were compared with the observed proportions of pneumonia and other SBI to assess calibration.[10, 33] Feasibility was measured by compliance of the nurses to the recommendations of the clinical decision model.
SBI were defined according to a reference standard and included abnormal radiographic findings and positive cultures from otherwise sterile body sites (urine, blood, spinal fluid) or excluded by uneventful follow-up by telephone three days after ED discharge. A consensus diagnosis was made if the reference standard was inconclusive (by investigators EK, RGN, RO).  All final diagnoses were classified as either pneumonia, other SBI or no SBI. Outcome measures were coded blinded for the allocated randomization arm.
The impact of the decision model was analysed by intention to treat. We evaluated the impact of the clinical decision model by measuring correct diagnoses (SBI) and their related false positive/negative diagnoses,as our primary patient outcome, indicating effects of using the intervention strategy irrespective of overruling or non-compliance by the physician versus the usual care strategy. Primary patient outcomes of the decision model in intervention and usual care were compared using Chi-square analysis. We defined false positives as children without SBI incorrectly exposed to diagnostic tests and false negatives as children with SBI incorrectly refrained from diagnostic tests. We presented sensitivity (children with pneumonia/UTI and performance of chest-radiography/urine-culture), specificity (children without pneumonia/UTI and no performance of chest-radiography/urine-culture) and their related positive/negative likelihood ratios of the complete model. In the control group exposure to testing was based on judgment of the physician only; in the intervention group this was based on the recommendation of the clinical decision model and physicians’ judgment. LOS at the ED was based on the triage starting time and ED departure time as registered in the nursing record; Diagnostic procedures included chest-radiography, urine-dipstick, full blood-count and cultures (blood, urine and others). Treatment/follow-up included antibiotic prescriptions, hospitalization and revisits. Comparisons of process outcomes were tested with Chi-square or Student’s t test analysis, P-values were two-tailed. All analyses were performed with SPSS PASW statistics software (version 20.0; Chicago, Illinois) and R statistical packages (version 2.14, Vienna, Austria).
Out of 1,769 eligible febrile children 836 children were excluded because of chronic co-morbidity, emergent triage category, uncomplicated rhinitis/otitis or revisits within 7 days, resulting in 933 eligible children. 439 evaluable children were included for analysis (Fig 2) with a median age of 1.8 years (IQR 0.9–4.1), 57% were boys (n = 249), and the SBI prevalence was 13% (n = 55) including 33 children with pneumonia and 22 children with other SBI (Table 1).
Flowchart patient selection and Randomization.
Evaluation clinical decision model
The discriminative ability of the clinical decision model was 0.83 (95% CI 0.75–0.90) for pneumonia and 0.81 (95% CI 0.72–0.90) for other SBI. The clinical prediction model had good calibration in the study population, except for the upper quintiles due to the limited number of cases with pneumonia or other SBI (Fig 3). We also observed limited variability of the predicted risks (SD 0.06 for pneumonia and 0.08 for other SBI) with only 10% of the children (n = 44) being categorized as high risk. Consequently, model recommendations were limited and the majority of children rightly did not undergo diagnostics. 60% (18/30) of the children assigned to high-risk for pneumonia did not have final diagnosis of pneumonia, for other SBI this was 71% (10/14). Five percent (21/409) of children assigned to low-risk for pneumonia had a final diagnosis of pneumonia, for other SBI this was 4% (18/425). All children assigned to high-risk categories with a final diagnosis of SBI received antibiotics (20/20) compared to 57% of high-risk children without final diagnoses of SBI (13/23). Compliance to the recommendations of the clinical decision model in the intervention arm by nurses was high; only three high-risk patients did not receive the recommended diagnostics for either pneumonia (n = 2) or other SBI (n = 1) because of overruling by the physician. In 28 low-risk patients without clear other focus, no urine-dipstick testing was done, leading to a compliance of 86% (188/219).
Calibration plot of the predicted risks of pneumonia and other SBI (x-axis) and the observed frequencies of pneumonia and other SBI (with 95% CI, y-axis). The triangles represent the mean (predicted vs. observed) risk estimates of pneumonia and other SBI by quintiles of predicted risk. The dashed diagonal line represents ideal calibration. The distribution of the predicted risks of patients with pneumonia (n = 33) and other SBI (n = 22) and other patients (n = 406 and n = 417) is shown in the bottom of the graph, parallel to the x-axis.
Primary patient outcome
In 60% (25/42) of all children allocated to the intervention group chest-radiography was done but no pneumonia was diagnosed (false positives, Table 2). This percentage was not significantly different from the 57% (16/28) false positives of the control group (p = 0.84). In 67% (12/18) of all children in the intervention group urine-culture was collected but no UTI was diagnosed (false positives), but did not differ from the control group (53%, 9/17) (p = 0.41). The clinical decision model performed similarly between both study groups with regard to false negatives (Table 2).
Secondary process outcomes
Median LOS did not differ for children in both the intervention as the control group (1h57min vs. 1h54min, respectively) (Table 2). In the intervention group less full blood-count tests (14% vs. 22%, p-value <0.05) and more urine-dipsticks were done correctly according to current guidelines  (71% vs. 61%, p-value <0.05). The number of chest-radiographies and urine-cultures were somewhat higher in the intervention group, although not significant. Overall treatment with antibiotics did not substantially differ between both study groups, (23% in intervention group vs. 27% in control group) (p = 0.30) (Table 2). Finally, no differences in hospitalization or revisits were noted between both study groups.
This impact analysis showed good compliance to the decision model recommendations and its good discriminative ability for detecting SBI was confirmed in febrile children presenting to the ED. Unfortunately, we could not demonstrate improved assessment of correct diagnoses by the intervention. Application of the clinical decision model resulted in a more standardized diagnostic approach towards the febrile child, with significantly less full blood-counts and more rightfully performed urine-dipstick testing in the intervention group.
Comparison with other studies
This randomized controlled impact trial provides the methodological step after external validation of a developed decision model with good discriminative ability for predicting the presence of pneumonia and other SBI. As an essential step before proven clinical applicability, we performed this impact analysis to test whether or not the decision model actually improved clinical decisions, benefitted patient care or reduced costs.[10, 19–21]
From a literature review it is known that only a minority of new developed clinical prediction models underwent broad validation and no model had undergone impact analysis. The results of our trial were comparable with a recent literature review or individual studies on the effect of these models which showed effects on process outcomes, but only sparse effects on patient outcomes, costs or efficiency.[37, 38] However, this review included only two prediction rules applicable to pediatric emergency care.[31, 39]
Clinical and research implications
Although our study reports good compliance to and high accuracy of the clinical decision model, positive impact on routine care was lacking. A number of reasons have been described that explain why an accurate clinical prediction model may not result in improved patient care and processes.[40–42] One important reason is that physicians’ intuitive estimation of probabilities may be as good as, if not better than, the prediction model. This is reflected by high positive and negative likelihood ratios  in the detection of pneumonia/ UTI for both the intervention as the usual care group (Table 2). Clinicians approach in febrile child management is already on the safe side, as reflected by the absence of worse primary outcomes which was part of our primary outcome. As in practice one would never accept increased false negative diagnoses, it can be considered that trials as ours can only have effects on secondary (process) outcomes like diagnostic tests and treatment. Next, it can be argued that studying improved patient outcomes in febrile children is difficult anyway, with decreasing prevalence of the most severe diagnoses as sepsis and meningitis. Furthermore, this implementation trial was conducted at a highly specialized and experienced university pediatric ED, with residents and experienced supervising pediatricians. We expect that future use of the clinical decision model in a more general setting with less specialized medical staff may lead to improved process outcomes.
One major influence explaining the lack of impact on LOS was probably the availability of CRP bedside-testing in both study groups. Previous research showed that CRP bedside-guided decision making reduced the LOS at the pediatric ED by 19% (median LOS 148 minutes). The rationale for testing CRP in all children was already discussed before in the original paper. For this trial we wanted to test the effect of the decision model as a whole, rather than testing the individual effect of CRP. Finally, we completed our impact trial with a cost-minimisation study (S1 File). Costs of our process outcomes (e.g. blood counts, urine analysis) were low and no difference were found in e.g. hospitalization rate or costs of adverse events. As a consequence, neither this impact analysis, nor a sensitivity analysis could demonstrate substantial cost savings of using the clinical decision model compared with usual care management.
Compliance to the recommendations of the clinical decision model was good. However, we did not include 379 children for unknown reasons, which may suggest some feasibility problems of the decision model itself, but also reflects the difficulties of performing a randomized controlled trial (with the need of informed consent) in emergency care. As we did not observe significant differences on median age (1.9 (IQR 2.0–3.0) vs. 1.8 (IQR 2.0–3.0)), gender (55.7% males vs. 56.7% males) and prevalence of SBI (14.2% vs. 12.5%), when we compared non-included with included patients, we feel that this most likely did not affect generalizability of our results. Another measure of feasibility could be whether the model variables were completed directly after the patient’s triage. This was not the case for all patients, as some were seen by a physician directly during the triage process of the nurse.[5, 23] In more densely crowded ED settings that require triage systems to truly prioritize physicians’ tasks, higher efficiency due to the implementation of the decision model can be expected.
Strengths and limitations
The main strength of this study is its randomized design to evaluate the last step required for translating prediction research into clinical practice. Second, as we modelled children’s age in a linear piecewise manner, we accounted for the differentiated risks and the uniqueness of signs and symptoms of children in the broad included age group. A subgroup analysis on children aged ≤36 months (n = 287; 65%), who are considered the primary focus in many diagnostic fever studies , had similar conclusions. Next, we created optimal conditions for the implementation before we started our impact trial. First, we ensured the compliance of the nurses to the computerized patient triage system at our ED was already high (90% compliance and 97% adherence to the MTS advice). Second, there was an electronic patient record for nurse’s evaluation and, the clinical decision model was implemented in the routine workflow at the pediatric ED. Thirdly, involvement of the nurses during the process of tailoring the format of decision models to the local circumstances resulted in willingness to adopt the system in routine care. Finally, we incorporated a structured follow-up by telephone for ruling-out the possibility of missed and clinically relevant SBI diagnoses in children not fulfilling reference standard criteria. This helps us to reduce verification bias, and is recommended as a valid proxy for missing reference tests .
Our study has some limitations. First, reference tests were ordered based on the recommendations of the clinical decision model or at the physician’s discretion. However, to reduce verification bias we used a standardized follow-up period to ensure no clinically relevant SBI were missed in children who did not have chest-radiography or specimens bacterial culture.[48, 49] In addition, as antibiotic prescription is relatively low in our study (around 30%), limiting the number of children with true SBI who had no reference standards done by pre-emptive antibiotics use, we can assume that associated false negative diagnoses were low. Second, some interobserver variability might have influenced the validity of the clinical decision model. As the discriminative ability of the model was good in this study, as in previous settings  and the model included mainly vital (relative objective) signs, this limitation has probably not influenced our results negatively. Third, although the clinical decision model was integrated in the nurses’ regular work-flow as stand-alone software, optimal implementation would have been a fully integrated system in the computer-based hospital information system including automatic pop-up screens and reminders. Fourth, from the perspective to focus on febrile children with the largest diagnostic uncertainty, selection criteria differed slightly from the original population in which the model was derived. As the validity was preserved in this new (randomized) population, selection of slightly different patients won’t have affected results, but may have some consequences for generalization of results to other settings.
Finally, our clinical decision model advised on performing chest-radiographies which is in contrast with recent advice of the British Thoracic Society  on limited use of chest-radiographies in suspected childhood pneumonia. As the model fairly classifies high/low risk of SBI, we expect it can validly guide decisions on appropriate treatment as well. This might be supported by the lower unnecessary antibiotic prescriptions in the intervention group. Therefore, future recommendations of the clinical decision model may preferably focus on impact on therapeutic management like antibiotic prescription.
This impact analysis translates a validated clinical prediction model for febrile children into clinical practice. Although we observed well implementation, no substantial impact on patient outcome was perceived. Further evaluation may focus on impact in other settings with greater variability in experience in the assessment of the febrile child, in patient numbers, and in prevalence of outcome measures, or may include guidance on therapeutic management.
We gratefully acknowledge the emergency staff and medical students of our emergency department for their participation and careful collection of the required data. We want to thank Marcel de Wilde for the collaboration and the development of the clinical decision model for practical use. We acknowledge Yolanda de Rijke, Barry Koelewijn and other members of the POCT-team for their participation and careful collection of the C-reactive protein samples. Finally, we thank Dorien Geurts for monitoring patient inclusion on the emergency department.
Conceived and designed the experiments: EVK RGN YV EWS JL HAM RO. Performed the experiments: EVK RGN RO. Analyzed the data: EVK RGN YV SP HAM RO. Contributed reagents/materials/analysis tools: EVK RGN YV SP EWS JL HAM RO. Wrote the paper: EVK RGN YV SP EWS JL HAM RO. Responsible for protocol development and obtaining funding: HAM RO. Responsible for the implementation of the study protocol, monitoring patient inclusion, collection of the data and data analysis: EVK RGN. Supervised the implementation of the study protocol, patient inclusion, data analysis and writing of the manuscript: RO HAM. Supervised protocol development and analysis and interpretation of the data: YV EWS. Responsible for the economic analysis: SP. Contributed protocol development, technical support, and data acquisition: JL.
- 1. Merrill CT, Owens PL, Stocks C. Pediatric Emergency Department Visits in Community Hospitals from Selected States, 2005: Statistical Brief #52. 2006. Epub 2011/07/08. NBK56039 [bookaccession]. pmid:21735571.
- 2. Bouwhuis CB, Kromhout MM, Twijnstra MJ, Buller HA, Moll HA. [Few ethnic differences in acute pediatric problems: 10 years of acute care in the Sophia Children's Hospital in Rotterdam]]. Nederlands tijdschrift voor geneeskunde. 2001;145(38):1847–51. pmid:11593789.
- 3. Armon K, Stephenson T, Gabriel V, MacFaul R, Eccleston P, Werneke U, et al. Determining the common medical presenting problems to an accident and emergency department. Arch Dis Child. 2001;84(5):390–2. Epub 2001/04/24. pmid:11316679; PubMed Central PMCID: PMC1718762.
- 4. National Institute for Health and Care Excellence. Feverish illness in children—Assessment and initial management in children younger than 5 years 2013 [updated May 2013]. Available: http://guidance.nice.org.uk/CG160/NICEGuidance/pdf/English.
- 5. Oostenbrink R, Thompson M, Steyerberg EW, members E. Barriers to translating diagnostic research in febrile children to clinical practice: a systematic review. Arch Dis Child. 2012;97(7):667–72. Epub 2012/01/06. archdischild-2011-300667 [pii] pmid:22219168.
- 6. Slater M, Krug SE. Evaluation of the infant with fever without source: an evidence based approach. Emerg Med Clin North Am. 1999;17(1):97–126, viii–ix. Epub 1999/04/02. pmid:10101343.
- 7. Bleeker SE, Derksen-Lubsen G, Grobbee DE, Donders AR, Moons KG, Moll HA. Validating and updating a prediction rule for serious bacterial infection in patients with fever without source. Acta Paediatr. 2007;96(1):100–4. pmid:17187613.
- 8. Van den Bruel A, Haj-Hassan T, Thompson M, Buntinx F, Mant D, European Research Network on Recognising Serious Infection i. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet. 2010;375(9717):834–45. Epub 2010/02/06. S0140-6736(09)62000-6 [pii] pmid:20132979.
- 9. Craig JC, Williams GJ, Jones M, Codarini M, Macaskill P, Hayen A, et al. The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ (Clinical research ed. 2010;340:c1594. Epub 2010/04/22. pmid:20406860; PubMed Central PMCID: PMC2857748.
- 10. Nijman R, Vergouwe Y, Thompson M, Veen van M, Meurs van A, Lei van der J, et al. Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study. BMJ (Clinical research ed. 2013;346: f1706. pmid:23550046
- 11. Galetto-Lacour A, Zamora SA, Andreola B, Bressan S, Lacroix L, Da Dalt L, et al. Validation of a laboratory risk index score for the identification of severe bacterial infection in children with fever without source. Arch Dis Child. 2010;95(12):968–73. pmid:20515973.
- 12. Van den Bruel A, Aertgeerts B, Bruyninckx R, Aerts M, Buntinx F. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract. 2007;57(540):538–46. Epub 2007/08/31. pmid:17727746; PubMed Central PMCID: PMC2099636.
- 13. Thompson M, Coad N, Harnden A, Mayon-White R, Perera R, Mant D. How well do vital signs identify children with serious infections in paediatric emergency care? Arch Dis Child. 2009;94(11):888–93. Epub 2009/07/18. adc.2009.159095 [pii] pmid:19608555.
- 14. Pantell RH, Newman TB, Bernzweig J, Bergman DA, Takayama JI, Segal M, et al. Management and outcomes of care of fever in early infancy. Jama. 2004;291(10):1203–12. Epub 2004/03/11. 291/10/1203 [pii]. pmid:15010441.
- 15. Berger RM, Berger MY, van Steensel-Moll HA, Dzoljic-Danilovic G, Derksen-Lubsen G. A predictive model to estimate the risk of serious bacterial infections in febrile infants. Eur J Pediatr. 1996;155(6):468–73. Epub 1996/06/01. pmid:8789763.
- 16. Young Infants Clinical Signs Study G. Clinical signs that predict severe illness in children under age 2 months: a multicentre study. Lancet. 2008;371(9607):135–42. Epub 2008/01/15. S0140-6736(08)60106-3 [pii] pmid:18191685.
- 17. Brent AJ, Lakhanpaul M, Ninis N, Levin M, MacFaul R, Thompson M. Evaluation of temperature-pulse centile charts in identifying serious bacterial illness: observational cohort study. Arch Dis Child. 2011;96(4):368–73. Epub 2011/01/15. adc.2010.183129 [pii] pmid:21233079; PubMed Central PMCID: PMC3158667.
- 18. Maguire JL, Kulik DM, Laupacis A, Kuppermann N, Uleryk EM, Parkin PC. Clinical prediction rules for children: a systematic review. Pediatrics. 2011;128(3):e666–77. Epub 2011/08/24. peds.2011-0043 [pii] pmid:21859912.
- 19. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Annals of internal medicine. 2006;144(3):201–9. pmid:16461965.
- 20. Wallace E, Smith SM, Perera-Salazar R, Vaucher P, McCowan C, Collins G, et al. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC Med Inform Decis Mak. 2011;11:62. Epub 2011/10/18. 1472-6947-11-62 [pii] pmid:21999201; PubMed Central PMCID: PMC3216240.
- 21. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381. Epub 2013/02/09. PMEDICINE-D-12-02102 [pii]. pmid:23393430; PubMed Central PMCID: PMC3564751.
- 22. Stiell IG, Bennett C. Implementation of clinical decision rules in the emergency department. Acad Emerg Med. 2007;14(11):955–9. Epub 2007/10/10. j.aem.2007.06.039 [pii] pmid:17923717.
- 23. Mackway-Jones K MJ, Windle J. Emergency triage. Group MT, editor: Oxford: Blackwell Publishing Ltd; 2006.
- 24. M-J K.. Emergency triage. Group MT, editor. London, UK: BMJ Publishing Group; 1997.
- 25. Nijman R, Moll H, Vergouwe Y, Rijke de Y, Oostenbrink R. C-Reactive Protein bedside testing in febrile children lowers length of stay at the emergency department. Accepted: PCARE-D-14-00014R1. 2014.
- 26. Geurts DH, Vos W, Moll HA, Oostenbrink R. Impact analysis of an evidence-based guideline on diagnosis of urinary tract infection in infants and young children with unexplained fever. Eur J Pediatr. 2013. Epub 2013/11/14. pmid:24221603.
- 27. Mori R, Lakhanpaul M, Verrier-Jones K. Diagnosis and management of urinary tract infection in children: summary of NICE guidance. BMJ (Clinical research ed. 2007;335(7616):395–7. Epub 2007/08/25. 335/7616/395 [pii] pmid:17717369; PubMed Central PMCID: PMC1952472.
- 28. Maconochie I, Dawood M. Manchester triage system in paediatric emergency care. BMJ (Clinical research ed. 2008;337:a1507. Epub 2008/09/24. pmid:18809588.
- 29. van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, van der Lei J, et al. Manchester triage system in paediatric emergency care: prospective observational study. BMJ (Clinical research ed. 2008;337:a1501. Epub 2008/09/24. pmid:18809587; PubMed Central PMCID: PMC2548283.
- 30. Decision rule for children with fever at the emergency room. [updated 2014]. Available: www.erasmusmc.nl/feverkidstool.
- 31. Roukema J, Steyerberg EW, van der Lei J, Moll HA. Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source. J Am Med Inform Assoc. 2008;15(1):107–13. Epub 2007/10/20. M2164 [pii] pmid:17947627; PubMed Central PMCID: PMC2273109.
- 32. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ (Clinical research ed. 2001;322(7282):355–7. Epub 2001/02/13. pmid:11159665; PubMed Central PMCID: PMC1119583.
- 33. Van Calster B, Vergouwe Y, Looman CW, Van Belle V, Timmerman D, Steyerberg EW. Assessing the discriminative ability of risk models for more than two outcome categories. Eur J Epidemiol. 2012;27(10):761–70. Epub 2012/10/12. pmid:23054032.
- 34. Bleeker SE, Moons KG, Derksen-Lubsen G, Grobbee DE, Moll HA. Predicting serious bacterial infection in young children with fever without apparent source. Acta Paediatr. 2001;90(11):1226–32. pmid:11808890.
- 35. Oostenbrink R, Moons KG, Theunissen CC, Derksen-Lubsen G, Grobbee DE, Moll HA. Signs of meningeal irritation at the emergency department: how often bacterial meningitis? Pediatric emergency care. 2001;17(3):161–4. pmid:11437138.
- 36. R: a language and environment for statistical computing ln: Team RdC eRFfSC. 2006.
- 37. Stiell IG, Wells GA, Vandemheen K, Clement C, Lesiuk H, Laupacis A, et al. The Canadian CT Head Rule for patients with minor head injury. Lancet. 2001;357(9266):1391–6. pmid:11356436.
- 38. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Annals of internal medicine. 2012;157(1):29–43. Epub 2012/07/04. 1206700 [pii] pmid:22751758.
- 39. Bourgeois FC, Linder J, Johnson SA, Co JP, Fiskio J, Ferris TG. Impact of a computerized template on antibiotic prescribing for acute respiratory infections in children and adolescents. Clin Pediatr (Phila). 2010;49(10):976–83. Epub 2010/08/21. 0009922810373649 [pii] pmid:20724348.
- 40. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. Jama. 2000;284(1):79–84. pmid:10872017.
- 41. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6(2):104–14. Epub 1999/03/27. pmid:10094063; PubMed Central PMCID: PMC61349.
- 42. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8(6):527–34. Epub 2001/11/01. pmid:11687560; PubMed Central PMCID: PMC130063.
- 43. Martin NG, Sadarangani M, Pollard AJ, Goldacre MJ. Hospital admission rates for meningitis and septicaemia caused by Haemophilus influenzae, Neisseria meningitidis, and Streptococcus pneumoniae in children in England over five decades: a population-based observational study. The Lancet Infectious diseases. 2014;14(5):397–405. pmid:24631222.
- 44. Baraff LJ, Lee SI. Fever without source: management of children 3 to 36 months of age. Pediatr Infect Dis J. 1992;11(2):146–51. Epub 1992/02/01. pmid:1741193.
- 45. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ (Clinical research ed. 2005;330(7494):765. Epub 2005/03/16. bmj.38398.500764.8F [pii] pmid:15767266; PubMed Central PMCID: PMC555881.
- 46. Roukema J, Steyerberg EW, van Meurs A, Ruige M, van der Lei J, Moll HA. Validity of the Manchester Triage System in paediatric emergency care. Emerg Med J. 2006;23(12):906–10. Epub 2006/11/30. 23/12/906 [pii] pmid:17130595; PubMed Central PMCID: PMC2564249.
- 47. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ (Clinical research ed. 1998;317(7162):858–61. Epub 1998/09/25. pmid:9748183; PubMed Central PMCID: PMC31096.
- 48. Biesheuvel C, Irwig L, Bossuyt P. Observed differences in diagnostic test accuracy between patient subgroups: is it real or due to reference standard misclassification? Clin Chem. 2007;53(10):1725–9. Epub 2007/09/22. 53/10/1725 [pii] pmid:17885138.
- 49. Reitsma JB, Rutjes AW, Khan KS, Coomarasamy A, Bossuyt PM. A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol. 2009;62(8):797–806. pmid:19447581.
- 50. Edmonds ZV, Mower WR, Lovato LM, Lomeli R. The reliability of vital sign measurements. Annals of emergency medicine. 2002;39(3):233–7. Epub 2002/02/28. S0196064402423244 [pii]. pmid:11867974.
- 51. Harris M, Clark J, Coote N, Fletcher P, Harnden A, McKean M, et al. British Thoracic Society guidelines for the management of community acquired pneumonia in children: update 2011. Thorax. 2011;66 Suppl 2:ii1–23. Epub 2011/10/19. thoraxjnl-2011-200598 [pii] pmid:21903691.