Design and setting

This implementation research is designed to be a single-centre, pragmatic, controlled before-after study. The study received ethics waivers from the SingHealth centralised institutional review board (CIRB) (CIRB Ref: 2020/2941). Consent was waived by the ethics committee.

The study will take place in the largest tertiary hospital in Singapore, which receives an ED attendance rate of approximately 120,000 annually. The ED provides comprehensive 24-hour emergency medicine (EM) services for a wide range of emergencies, including major trauma. It offers direct access to 24/7 laboratory and diagnostic imaging services. The ED care team is comprised of EM specialists (associate consultants and above) and non-EM specialists (residents, medical officers, resident physicians, etc.).

Sample size

Based on previous data [22,38], we assume an estimated 20% of exams are inappropriate across all imaging procedures (based on previous studies conducted locally). In this study, we seek to demonstrate a reduction in inappropriate exams using a before and after design. Data from independent before and after groups will be compared using t-test. To achieve a power of 0.9, with an alpha error of 0.05 and an effect size of 0.3, we will need 468, therefore, 235 per group, before and after the intervention.

The primary analysis is conducted at the physician (cluster) level. For each physician, we compute the mean appropriateness rate in the pre-implementation and post-implementation phases and analyse the within-physician change. Power calculations were therefore based on a paired t-test of physician-level means, which is asymptotically equivalent to tests for differences in proportions when binary outcomes are summarised at the cluster level.

Sources of previous studies conducted locally:

1. 1). Tay YX, Chan LL, Than SR, et al. Appropriateness of lumbar spine radiography and factors influencing imaging ordering patterns: paving the path toward value-driven health care. Int J Qual Health Care. 2023;35(2):mzad021. doi:10.1093/intqhc/mzad021
2. 2). Tay YX, Foley SJ, Killeen R, et al. Positivity rates and subsequent patient dispositions after utilisation of cervical spine imaging referral guidelines in Singapore. Insights Imaging. 2025;16(1):170. Published 2025 Aug 8. doi:10.1186/s13244-025-02048-9
3. 3). Tay YX, Foley SJ, Ong ME, et al. Using evidence-based imaging referral guidelines to facilitate appropriate imaging: Are they all the same?. Eur J Radiol. 2025;183:111933. doi:10.1016/j.ejrad.2025.111933

Characteristics of participants

A purposive sample of junior ED physicians will be approached via the ED Trauma lead. Inclusion criteria will include non-specialist EM physicians, and exclusion criteria will include non-specialist ED physicians who are unable to commit to the study period of 4 months (i.e., rotating/posting out of the ED within the study timeframe). Only non-specialists ED physicians will be recruited as they are the primary care providers for most of the ED patients and also account for the majority of the X-ray imaging referrals from the ED. Additionally, they are mostly assigned for clinical duty in the P3 and P4 sections of the ED (patient acuity category status P3: ambulant patients with minor emergencies and P4: non-emergency cases), where a substantial volume of X-ray cervical and lumbar spine are requested [38]. Notably, a previously conducted study in the same ED revealed that non-specialists accounted for a greater proportion of inappropriate orders as compared to specialists [22]. Consequently, this navigated us from the pitfall of recruiting physicians with a low baseline percentage of inappropriate imaging requests, which limits the potential for reduction [39].

This study will not employ randomisation or blinding due to its pragmatic design. The research team will take informed consent verbally prior to the study’s commencement, following an introduction and briefing.

Comparator and intervention

Recruited ED physicians will be allocated into the control and intervention groups based on their starting posting date in the ED (e.g., 2^nd quarter, 3^rd quarter etc). Fig 1 illustrates the flowchart and timeline for the respective groups involved in the study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow chart and timeline of the study.

https://doi.org/10.1371/journal.pone.0336801.g001

In the first phase for both groups, ED physicians will follow the usual clinical practice (“Do nothing”) of making imaging referrals without consulting CDSS. Both groups will be informed during their onboarding on the availability of imaging referral guidelines in the training handbook and also on the intranet. Specifically, the physicians in the control group will not receive any CDSS introduction/orientation and no additional imaging referral guidelines education beyond the standard onboarding process. Simultaneously, a retrospective analysis of the electronic medical records will be performed.

The study included consecutive patients referred to the ED’s radiology department for cervical and lumbar spine radiography by the cohort of recruited ED physicians. Patients who had imaging referrals from general practitioners will be omitted due to our lack of access to the clinical notes for this particular cohort. For each case, the initial clinical diagnosis (reasons for the referral), the ordered procedure, patient demographics (age and gender), and ED clinical notes will be collected. To enhance the accuracy of analysing each imaging referral, the initial clinical diagnosis will be integrated with the corresponding ED clinical notes, including signs, symptoms, clinical examination, and red flags. This integration will result in a more comprehensive “indication(s)” for the imaging procedure. The appropriateness of each imaging referral will be determined with reference to localised and institutional approved imaging referral guidelines built on imaging referral guidelines published by ACR, ESR and RCR (Refer to Table 1). The records will be archived to REDCap (Research Electronic Data Capture) hosted by SingHealth for subsequent analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Imaging referral guidelines* for X-ray Cervical and Lumbar Spine.

https://doi.org/10.1371/journal.pone.0336801.t001

During the second phase of the study, the ED physicians in the control group will continue their usual clinical practice of ordering imaging without consulting CDSS (“Do nothing”). We will study the referral practices of the ED physicians in the control group over a consecutive period of two months, using a pragmatic approach. We will perform a retrospective analysis of the electronic medical records, following the same approach as in Phase 1.

Conversely, the ED physicians in the intervention group will receive individual access to a CDSS after a two-month period (Phase 1 – No CDSS intervention), and they will continue clinical practice using the standalone CDSS, which will incorporate localised and approved institutional imaging referral guidelines. The CDSS adopted in this study will be xWave CDS (xWave Technologies Ltd., Dublin, Ireland), which is a cloud-based Software as a Service (SaaS) platform with ISO27001 certification and developed in partnership with both the ESR iGuide and RCR iRefer. xWave CDS is fully hosted in Amazon Web Services (AWS) (Seattle, USA), and access is through the public internet. There is no hardware required onsite other than devices to access a web browser. Approval will be sought from the local hospital information technology team to whitelist the xWave CDS uniform resource locator (URL) for use by physicians on the hospital computer system. As the CDSS is not integrated into the Computerised Provider Order Entry (CPOE) ordering process, physicians will be required to toggle between the CPOE platform and xWave CDS. Modifications will be made to the xWave CDS user interface to incorporate elements that were deemed user-friendly as highlighted by local ED physicians during a previous focus group study [40]. This includes consolidating imaging referral guidelines onto a single page and replacing drop-down boxes with checked boxes.

Briefing slides and an introduction/orientation to the CDSS will be provided to ED physicians utilising the CDSS prior to the start of this second phase (Phase 2). The briefing slides will be curated by the vendor project manager and the principal investigator (PI) and subsequently be reviewed by the senior ED physicians for comprehensibility to ensure that the materials are well-defined and well-understood. Similarly, both the project manager and the PI will blueprint a 15-minute content plan for the introduction/orientation. The introduction/orientation will be conducted virtually by the research team members and the project manager assigned to the implementation. Topics covered include: (1) introduction to CDSS; (2) user workflow; (3) orientation of the CDSS input screen; (4) orientation of the CDSS guidelines screen; and (5) orientation of the CDSS recommendation screen. There will be no evaluation at the end of the introduction/orientation, but time will be dedicated for questions and answers. While no criteria for physician learning are available, it was deemed that the most effective learning for the time-constrained physicians was self-directed through the curated briefing slides, which will be disseminated through email and WhatsApp group.

The physicians will select pre-defined clinical history from the CDSS for imaging recommendations, and they will have the option to either follow or disregard them. We will study the referral practices of the recruited ED physicians in the intervention group over a consecutive period of two months. While ED physicians are not constricted by guidelines, during the study’s second phase, there will be special strategies to improve compliance and to motivate the physicians to adhere to the guidelines. This includes email reminders and the emphasis on the audit process.

The evaluation process in Phase 2 will mimic the pre-implementation evaluation process (Phase 1) to ensure comparable results and data. Essentially, we want to conduct a careful review of the difference between the baseline and post-implementation results. The analysis will include all participating physicians in an intention-to-treat fashion.

Data management plan

No coded/anonymous research data will be sent outside of the study’s institution. We will code the collected data to prevent participant identification. We will store the data in the encrypted/password-protected healthcare cloud (H-cloud), which is the private cloud of Singapore public healthcare. Once the study concludes, we will retain the data for a maximum of seven years before erasing the soft copy. The research data will only be accessible to the PI, who will treat all personal information and results with complete confidentiality. Only authorised personnel will review the data to ensure correct study execution. The PI will also perform data and safety monitoring. The PI will review the recruited data on a monthly basis, and since this study poses only minimal risk, data integrity monitoring is not required. Similarly, this research study does not have any stopping criteria based on efficacy, futility, or safety considerations.

Safety considerations

There is no expected physical, psychological, social, or economic harms associated with this study. The participants will receive complete information regarding the study design and the scientific rationale underlying the proposed study.

The research team is multidisciplinary and comprises radiographers, EM physicians, and radiologists with sufficient expertise and experience to conduct the research. Data are collected from routine (cervical and lumbar spine X-rays) procedures to avoid unnecessary risk to patients. The PI, who is neither from the ED nor has any relationship with the participants, will perform the data analysis, minimising privacy and confidentiality risks. Additionally, the information will not be used for any purpose except for this research, and that participants will never be identified personally; only group data will be reported.

Data and statistical analysis

We will describe participant and patient demographics, clinical and imaging characteristics, and study outcomes using standard methods, such as frequencies and percentages for categorical variables and mean, standard deviation, median, and range for continuous variables.

We will use a t-test to examine the outcomes before and after the intervention (“Do nothing” and CDSS). The primary outcome (appropriateness) is binary at the patient level but analysed at the physician (cluster) level. Each physician’s mean appropriateness in the pre- and post-phases will be compared using a paired t-test on within-physician differences; if assumptions of normality are materially violated, analyses will be repeated with a variance-stabilising transformation.

To complement this, patient-level multilevel logistic regression models will be performed with a random intercept for physician and fixed effects for phase, intervention arm, and their interaction. Patient-level (age, sex, indication type) and physician-level (specialty) covariates will be included to adjust for potential confounding. Both unadjusted cluster-level and adjusted multilevel estimates will be reported side by side. Where possible, we will present all effects with confidence intervals.

Status and timeline

Initial data collection will start on 01 May 2025. We aim to complete data collection by 30 November 2025 and perform all analysis by 31 December 2025.

Strengths

One of the strengths of this study lies in our approach to implementation, in which we will apply implementation science principles to guide the EBI’s deployment. We will utilise JBI’s 7-phase process model to facilitate and direct the integration of evidence into practice, specifically the integration of CDSS into the practice of ED physicians [47]. Notably, we have identified problems with stakeholders, engaged change agents and assessed the context and readiness of ED physicians for change [48]. Through these phases, we have identified barriers and opportunities in the ED/healthcare systems as vital for reducing low-value imaging, developed a better understanding of how physicians make sense of the CDSS intervention, engaged key leadership for endorsement of technological interventions, and ensured shared agreement and engagement where ED physicians, radiologists, and radiographers are involved in the implementation project: developing guidelines, modifying CDSS, user acceptance testing, and developing implementation strategies.

Additionally, we will use a controlled before-after study design to measure changes in the outcomes linked to the intervention. By having a control group, it provides more confidence in the study’s results through the elucidation of changes over time (i.e., attributing causal inference to the intervention).

Although CDSS are designed to improve quality of care and reduce costs, a review has shown no change in the total volume of advanced imaging orders and small changes in appropriateness [49]. This can be a result of suboptimal implementation, which negatively impacts the uptake of evidence in real-world practice [50]. Literature indicates that demonstrating the effectiveness of an EBI does not ensure its integration into routine practice, as uptake is significantly influenced by contextual factors (e.g., environment, specific setting, individual characteristics) [37]. This study will build on previous findings and will provide comprehensive data on the impact of this EBI in real-world setting on healthcare systems and patients while accounting for barriers and facilitators in the implementation process. Drawing from unpublished results from a locally conducted study that assessed the context and readiness of ED physicians to promote evidence-based imaging referral guidelines, targeted implementation strategies will be in place to bridge the evidence-to-practice gap.

We have intentionally adopted a pragmatic methodology for this study to gather empirical insights regarding the use and effects of CDSS in healthcare. In our study, our recruitment approach provides the best opportunity for rapid recruitment of ED physicians during the transition from another institution to the study’s ED, which occurs during the study period. This study will be informative regardless of the outcome.

Limitations

The study design is however also subject to several limitations, which are minimised where possible.

An often-discussed potential drawback of controlled before-after studies is methodological weakness, which results in a higher risk of bias, as there may be unidentified differences between the intervention and control groups that may affect changes in the outcome measure. For instance, the performance at baseline for both groups may differ, and subsequent analyses may be inappropriate, as the baseline imbalance suggests that the control group is not comparable [50]. This could render any apparent effect of the intervention spurious. Indeed, it is often difficult to identify a comparable control group [50]. However, we will endeavour to minimise these differences by ensuring that the participating ED physicians shared similar clinical experiences when recruited, i.e., previous ED postings prior to this study. Nonetheless, it is prudent to acknowledge that there is no direct comparison between the control and intervention groups, therefore the estimate of effect attributable to the intervention should be interpreted with caution. However, we will ensure that a table of baseline comparisons is available for readers, which will facilitate clinical judgement in determining whether any of the differences are substantial enough to have influenced the outcome [51].

Furthermore, in this study, the EBI may be influenced by the Hawthorne effect, which could result in an overestimation of the intervention’s efficacy [52]. However, we contend that the impact will be negligible, as prior quality assurance audits on X-ray lumbar spine, conducted after the implementation of institutional imaging guidelines, revealed minimal variations in both volume and appropriateness, even with ED physicians informed of the ongoing audits.

A possible constraint in the efficacy of CDSS is the emergence of alert fatigue among physicians, leading them to disregard feedback from the CDSS or perceive it as an onerous regulation rather than a tool to enhance the appropriateness of imaging referrals [53]. Moreover, to bypass the standalone CDSS, ED physicians may attempt to “game the system” by selecting indications that will result in their imaging referral being deemed Appropriate. We acknowledge these constraints and therefore seek to adhere to the “Ten Commandments for Effective Clinical Decision Support” to maximise the success of the implementation [54].

Simultaneously, effective doses are estimates and may vary depending on local equipment, protocols, patient body habitus, and radiographer practices. Therefore, the estimates for the cumulative effective doses may differ from other authors. Similarly, the estimates for energy consumption and carbon emissions may vary due to hospital equipment efficiency, energy sources, and operational practices. However, with the complexity involved in estimating these values, we believe the impact will be minimal in terms of magnitude, as we adopted estimates from published works in journals such as Radiology and The Lancet, while also including estimates from an international organisation – ICRP [55].

Our study may also be susceptible to inherent bias. This encompasses biases such as selection and history. To minimise selection bias, the research team will recruit newly posted junior ED physicians whose residency matches are decided by a neutral party (MOH Holdings Pte Ltd., Singapore). Similarly, to mitigate history threat, all changes that could impact the outcomes would be accounted for, and the time span between the before and after is minimised to reduce the effects of trends, which in our case is each two months in close succession [56].

Additionally, the relatively small sample size and single-centredness will limit the external validity of this study. Using implementation science principles in our study, we are able to create a “culturally adapted” intervention that takes into account feedback from ED physicians. This resulted in a refined and more acceptable CDSS for end users, particularly in Singapore. This could serve as a blueprint for other countries to adopt and adapt our approach in implementation of CDSS.

Lastly, it has been recognised that the impact of standalone CDSS is often lower than that of integrated systems due to the extra work required by users to obtain an imaging recommendation amid a busy clinical environment. However, there is a limitation surrounding the local information technology infrastructure, whereby a change in the electronic medical record system is expected shortly, rendering integration work unfeasible.

Despite the limitations, the study is maximising the opportunity to draw meaningful conclusions that will aid future studies and our own research in the implementation of CDSS. Specifically, this study will serve as baseline data and evidence for implementing and adopting integrative CDSS within the local health system/healthcare cluster. Plans are currently in progress for an exploratory study on a pilot integrative CDSS to be implemented at the point of care in academic hospitals and primary care settings, addressing the challenges associated with standalone CDSS, such as workflow burdens and “alert fatigue”.