Study design and setting

We conducted a randomized controlled trial of a clinical decision model at the emergency department (ED) of the Erasmus MC-Sophia Children’s Hospital in Rotterdam, The Netherlands. The protocol and CONSORT statement are available as supporting information; see S2 and S3 Files. This large inner-city university hospital is visited annually by nearly 9000 children with a mixed ethnic population of which 90% involved basic pediatric care.[2] During the study period there was no national guideline available for handling febrile children at the ED, although most physicians were familiar with the NICE febrile child guideline.[4]

Study population

We prospectively enrolled all consecutive pediatric patients (≥1 month—<16 years) presenting with fever at our ED from the first of September 2010 until June 30, 2012. Febrile children were eligible if fever had been noted at home in the 24 hours prior to presentation, when body temperature measured at the ED was ≥38.5°C or fever was used as a positive discriminator of the Manchester Triage System (MTS).[23] From the major principle of reducing diagnostic uncertainty by using a diagnostic decision model [5] we excluded well appearing febrile children (no amber/red alarming signs)[4] with a clear focus of uncomplicated rhinitis/otitis and severely ill children (emergent triage category).[23, 24] Children with chronic co-morbidity were excluded because of their increased risk of having serious infections and developing a complicated course.[25] Finally, children who reattended the ED within one week of their first presentation were only included at their initial visit.

Study intervention

The intervention consisted of the implementation of the clinical decision model, based on a polytomous logistic regression model of 2,717 febrile children who presented to the ED and included eleven predictors of pneumonia and other SBI versus no SBI (Table 1)[10]. Instructions on the use of the clinical decision model and the study process included individual briefings, practice cases for emergency staff (nurses/physicians), information by email, and posters mounted at the ED. During the trial period, feedback and periodic teaching sessions were organized. The clinical decision model presented patient-specific risk estimates (percentages) for both pneumonia and other SBI, which were categorized in high- or low risk groups and consequently were accompanied by recommendations for further diagnostic testing (Fig 1). These recommendations and risk estimates were only presented for children allocated to the intervention group; others were designated ‘usual care’ after completion of entering the model predictors (control group). Cut-off points for additional diagnostic testing were based on the diagnostic performance of the original prediction model at different risk thresholds, and were agreed upon by expert panel.[10] For the chosen cut-off points, the expected gain was maximized based on the ratio of false positive/negative errors and expected adverse consequences of a delayed diagnosis SBI. For high-risk children, nurses were instructed to initiate additional testing before physician’s assessment. This recommendation implied chest-radiography in children with estimated risk for pneumonia ≥15% and urine-dipstick and culture if estimated risk for other SBI was ≥30%. Following the local protocol [26] and international guidelines [27] urine-dipstick testing was also recommended in low-risk children without a clear focus for their febrile illness. Nurses and physicians were blinded for these cut-off values and for the contribution of predictors on risk scores. Physicians were able to overrule the recommendations by adding to or refraining from additional diagnostic testing. In the control group the physician first examined the patient and ordered diagnostic procedures according to their own judgment, but with the knowledge of a CRP-value which was required to enable randomization.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Clinical decision model.

Clinical decision model with its predictors (above) and recommendations (below).

https://doi.org/10.1371/journal.pone.0127620.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient characteristics.

https://doi.org/10.1371/journal.pone.0127620.t001

Data collection

All children who attended the ED were routinely triaged with the Manchester Triage System (MTS).[23, 24, 28, 29] After patient triage, model predictors were entered by the nurse for each individual patient. We collected patient characteristics (e.g. gender, age, reason of ED visit), referral profile, duration of the febrile episode, clinical signs/symptoms, observations and measures from physical examination (e.g., vital signs, clinical appearance). Data on all performed laboratory tests (e.g. full blood-count, CRP), additional diagnostic tests (e.g. chest-radiography, urine-dipstick, blood/urine-culture) and treatment/follow-up were registered prospectively in the computer-based hospital information system. The clinical decision model was implemented as a stand-alone device, accessible from each computer at the ED. A central logbook recorded data from all entered participants (Fig 1). Completing all variables of the clinical decision model was mandatory before randomization could be initiated. Once the risk estimate was shown to the nurse, no changes could be made to the entered model predictors.[30] The randomization mechanism, based on even/odd seconds indicated by the digital computer clock, was unknown to nurses and physicians.

Ethics was obtained by the institutional review board (IRB) of the ErasmusMC. According to IRB-review the intervention contained no additional risks as patients were not subjected to additional operations and no rules of conduct were imposed. Written informed consent was required by IRB and it was obtained from all caretakers, or guardians on behalf of the children enrolled (MEC-2008-071).

Outcome measures

Evaluation of the clinical decision model included diagnostic performance and feasibility of the model. Primary outcome measures were correct diagnoses (SBI) and their related false positive/negative diagnoses by using the clinical decision model in routine practice. Secondly, we measured process outcomes including length of stay (LOS) at the ED and items on diagnostics, treatment and follow-up as defined below.

Statistical analysis

Evaluation clinical decision model

The discriminative ability of the clinical decision model was 0.83 (95% CI 0.75–0.90) for pneumonia and 0.81 (95% CI 0.72–0.90) for other SBI. The clinical prediction model had good calibration in the study population, except for the upper quintiles due to the limited number of cases with pneumonia or other SBI (Fig 3). We also observed limited variability of the predicted risks (SD 0.06 for pneumonia and 0.08 for other SBI) with only 10% of the children (n = 44) being categorized as high risk. Consequently, model recommendations were limited and the majority of children rightly did not undergo diagnostics. 60% (18/30) of the children assigned to high-risk for pneumonia did not have final diagnosis of pneumonia, for other SBI this was 71% (10/14). Five percent (21/409) of children assigned to low-risk for pneumonia had a final diagnosis of pneumonia, for other SBI this was 4% (18/425). All children assigned to high-risk categories with a final diagnosis of SBI received antibiotics (20/20) compared to 57% of high-risk children without final diagnoses of SBI (13/23). Compliance to the recommendations of the clinical decision model in the intervention arm by nurses was high; only three high-risk patients did not receive the recommended diagnostics for either pneumonia (n = 2) or other SBI (n = 1) because of overruling by the physician. In 28 low-risk patients without clear other focus, no urine-dipstick testing was done, leading to a compliance of 86% (188/219).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Calibration plot for the risk of pneumonia (left) and other SBI (right).

Calibration plot of the predicted risks of pneumonia and other SBI (x-axis) and the observed frequencies of pneumonia and other SBI (with 95% CI, y-axis). The triangles represent the mean (predicted vs. observed) risk estimates of pneumonia and other SBI by quintiles of predicted risk. The dashed diagonal line represents ideal calibration. The distribution of the predicted risks of patients with pneumonia (n = 33) and other SBI (n = 22) and other patients (n = 406 and n = 417) is shown in the bottom of the graph, parallel to the x-axis.

https://doi.org/10.1371/journal.pone.0127620.g003

Primary patient outcome

In 60% (25/42) of all children allocated to the intervention group chest-radiography was done but no pneumonia was diagnosed (false positives, Table 2). This percentage was not significantly different from the 57% (16/28) false positives of the control group (p = 0.84). In 67% (12/18) of all children in the intervention group urine-culture was collected but no UTI was diagnosed (false positives), but did not differ from the control group (53%, 9/17) (p = 0.41). The clinical decision model performed similarly between both study groups with regard to false negatives (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Patient and process outcome.

https://doi.org/10.1371/journal.pone.0127620.t002

Secondary process outcomes

Median LOS did not differ for children in both the intervention as the control group (1h57min vs. 1h54min, respectively) (Table 2). In the intervention group less full blood-count tests (14% vs. 22%, p-value <0.05) and more urine-dipsticks were done correctly according to current guidelines [26] (71% vs. 61%, p-value <0.05). The number of chest-radiographies and urine-cultures were somewhat higher in the intervention group, although not significant. Overall treatment with antibiotics did not substantially differ between both study groups, (23% in intervention group vs. 27% in control group) (p = 0.30) (Table 2). Finally, no differences in hospitalization or revisits were noted between both study groups.

Main findings

This impact analysis showed good compliance to the decision model recommendations and its good discriminative ability for detecting SBI was confirmed in febrile children presenting to the ED. Unfortunately, we could not demonstrate improved assessment of correct diagnoses by the intervention. Application of the clinical decision model resulted in a more standardized diagnostic approach towards the febrile child, with significantly less full blood-counts and more rightfully performed urine-dipstick testing in the intervention group.

Comparison with other studies

This randomized controlled impact trial provides the methodological step after external validation of a developed decision model with good discriminative ability for predicting the presence of pneumonia and other SBI.[10] As an essential step before proven clinical applicability, we performed this impact analysis to test whether or not the decision model actually improved clinical decisions, benefitted patient care or reduced costs.[10, 19–21]

From a literature review it is known that only a minority of new developed clinical prediction models underwent broad validation and no model had undergone impact analysis.[18] The results of our trial were comparable with a recent literature review or individual studies on the effect of these models which showed effects on process outcomes, but only sparse effects on patient outcomes, costs or efficiency.[37, 38] However, this review included only two prediction rules applicable to pediatric emergency care.[31, 39]

Clinical and research implications

Although our study reports good compliance to and high accuracy of the clinical decision model, positive impact on routine care was lacking. A number of reasons have been described that explain why an accurate clinical prediction model may not result in improved patient care and processes.[40–42] One important reason is that physicians’ intuitive estimation of probabilities may be as good as, if not better than, the prediction model. This is reflected by high positive and negative likelihood ratios [8] in the detection of pneumonia/ UTI for both the intervention as the usual care group (Table 2). Clinicians approach in febrile child management is already on the safe side, as reflected by the absence of worse primary outcomes which was part of our primary outcome. As in practice one would never accept increased false negative diagnoses, it can be considered that trials as ours can only have effects on secondary (process) outcomes like diagnostic tests and treatment. Next, it can be argued that studying improved patient outcomes in febrile children is difficult anyway, with decreasing prevalence of the most severe diagnoses as sepsis and meningitis.[43] Furthermore, this implementation trial was conducted at a highly specialized and experienced university pediatric ED, with residents and experienced supervising pediatricians. We expect that future use of the clinical decision model in a more general setting with less specialized medical staff may lead to improved process outcomes.

One major influence explaining the lack of impact on LOS was probably the availability of CRP bedside-testing in both study groups. Previous research showed that CRP bedside-guided decision making reduced the LOS at the pediatric ED by 19% (median LOS 148 minutes).[25] The rationale for testing CRP in all children was already discussed before in the original paper.[10] For this trial we wanted to test the effect of the decision model as a whole, rather than testing the individual effect of CRP. Finally, we completed our impact trial with a cost-minimisation study (S1 File). Costs of our process outcomes (e.g. blood counts, urine analysis) were low and no difference were found in e.g. hospitalization rate or costs of adverse events. As a consequence, neither this impact analysis, nor a sensitivity analysis could demonstrate substantial cost savings of using the clinical decision model compared with usual care management.

Compliance to the recommendations of the clinical decision model was good. However, we did not include 379 children for unknown reasons, which may suggest some feasibility problems of the decision model itself, but also reflects the difficulties of performing a randomized controlled trial (with the need of informed consent) in emergency care. As we did not observe significant differences on median age (1.9 (IQR 2.0–3.0) vs. 1.8 (IQR 2.0–3.0)), gender (55.7% males vs. 56.7% males) and prevalence of SBI (14.2% vs. 12.5%), when we compared non-included with included patients, we feel that this most likely did not affect generalizability of our results. Another measure of feasibility could be whether the model variables were completed directly after the patient’s triage. This was not the case for all patients, as some were seen by a physician directly during the triage process of the nurse.[5, 23] In more densely crowded ED settings that require triage systems to truly prioritize physicians’ tasks, higher efficiency due to the implementation of the decision model can be expected.

Strengths and limitations

The main strength of this study is its randomized design to evaluate the last step required for translating prediction research into clinical practice. Second, as we modelled children’s age in a linear piecewise manner,[10] we accounted for the differentiated risks and the uniqueness of signs and symptoms of children in the broad included age group. A subgroup analysis on children aged ≤36 months (n = 287; 65%), who are considered the primary focus in many diagnostic fever studies [44], had similar conclusions. Next, we created optimal conditions for the implementation before we started our impact trial.[45] First, we ensured the compliance of the nurses to the computerized patient triage system at our ED was already high (90% compliance and 97% adherence to the MTS advice).[46] Second, there was an electronic patient record for nurse’s evaluation and, the clinical decision model was implemented in the routine workflow at the pediatric ED. Thirdly, involvement of the nurses during the process of tailoring the format of decision models to the local circumstances resulted in willingness to adopt the system in routine care.[47] Finally, we incorporated a structured follow-up by telephone for ruling-out the possibility of missed and clinically relevant SBI diagnoses in children not fulfilling reference standard criteria. This helps us to reduce verification bias, and is recommended as a valid proxy for missing reference tests [48].

Our study has some limitations. First, reference tests were ordered based on the recommendations of the clinical decision model or at the physician’s discretion. However, to reduce verification bias we used a standardized follow-up period to ensure no clinically relevant SBI were missed in children who did not have chest-radiography or specimens bacterial culture.[48, 49] In addition, as antibiotic prescription is relatively low in our study (around 30%), limiting the number of children with true SBI who had no reference standards done by pre-emptive antibiotics use, we can assume that associated false negative diagnoses were low. Second, some interobserver variability might have influenced the validity of the clinical decision model.[50] As the discriminative ability of the model was good in this study, as in previous settings [10] and the model included mainly vital (relative objective) signs, this limitation has probably not influenced our results negatively. Third, although the clinical decision model was integrated in the nurses’ regular work-flow as stand-alone software, optimal implementation would have been a fully integrated system in the computer-based hospital information system including automatic pop-up screens and reminders. Fourth, from the perspective to focus on febrile children with the largest diagnostic uncertainty, selection criteria differed slightly from the original population in which the model was derived. As the validity was preserved in this new (randomized) population, selection of slightly different patients won’t have affected results, but may have some consequences for generalization of results to other settings.

Finally, our clinical decision model advised on performing chest-radiographies which is in contrast with recent advice of the British Thoracic Society [51] on limited use of chest-radiographies in suspected childhood pneumonia. As the model fairly classifies high/low risk of SBI, we expect it can validly guide decisions on appropriate treatment as well. This might be supported by the lower unnecessary antibiotic prescriptions in the intervention group. Therefore, future recommendations of the clinical decision model may preferably focus on impact on therapeutic management like antibiotic prescription.