Introduction

Improving the quality of care that patients receive during outbreaks of infectious disease is paramount to both improving patient outcomes and engendering trust and healthcare seeking. Quality improvement (QI) can be defined as “the use of deliberate and defined methods in continuous efforts to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes” [1]. Whilst QI is well established to drive improvement in health systems and routine care, there are fewer reports of its use during public health emergencies [2] and there “is no reason why the mindset, concepts and tools of improvement and safety science cannot be applied to acute or emergency situations” [3]. During the 2014–2016 West Africa Ebola Epidemic, at times, quality of patient care was neglected [4] and the facilities where patients were cared for were inadequate [5], likely leading to excess mortality and extension of the epidemic. In this article we describe the use of QI and the creation of a standardized assessment tool to assess and improve patient care and patient care facilities during the COVID-19 pandemic in Sierra Leone.

Most reports of the use of QI during the COVID-19 pandemic, are in it’s more traditional sphere, to maintain and improve essential health services [6] and reported from high-income countries [7]. A recent scoping review of QI during COVID-19 found 26 articles, with none conducted in Africa, seven articles focused on QI methods to assess the COVID-19 public health response [7]. Curtis et al applied the Centres for Disease Control US guidelines to assess COVID-19 surveillance for hospitalised patients in Victoria, Australia [8]. A qualitative interview study evaluated the response to Zika virus in Florida, mapped results to the WHO Health Systems Framework and extrapolated the learning to COVID-19 response [9]. Another study assessed preparedness across several countries using the WHO Health System Framework [10]. Another study used annual reporting data from the International Health Regulations to assess preparedness across 182 countries [11]. Whilst Boyce et al describe the creation of a Rapid Urban Health Security Assessment Tool, to assess preparedness for COVID-19 response at the local government level [12]. None of the articles retrieved focused on quality of care at the facility or patient level. Indeed, some authors argue that quality of care was initially overlooked during COVID-19 and that quality of care should be integrated into emergency preparedness and response plans [13].

In Sierra Leone, informed by the previous public health response to Ebola [14], a National COVID-19 Emergency Response Centre (NaCOVERC) including a Case Management (CM) pillar was established at the outset of the Pandemic. The CM Pillar was primarily responsible for the clinical management of suspected and confirmed cases of COVID-19 and published the first Case Management Strategy and Standard Operating Procedures in April 2020 [15]. These guidelines were updated periodically, as new evidence emerged [16]. A QI working group was established within the case management pillar in May 2020, tasked to improve the quality of care for patients with suspected or confirmed COVID-19. In this article, we describe the use of QI to improve quality of care for COVID-19 in Sierra Leone. We describe the creation of a standardised assessment tool for the quality of COVID-19 care and report serial nationwide assessments of quality.

Methods

The context: COVID-19 in Sierra Leone

The first COVID-19 case in Sierra Leone was recorded on the 30 March 2020; cases increased to a peak in June 2020 before steadily declining [17]. As of October 31, 2021, a total of 6,398 cases and 121 related deaths had been confirmed in Sierra Leone and a total of 249,534 COVID-19 reverse transcription polymerase chain reaction (RT-PCR) tests had been conducted and the average positivity rate was 2.56%. Three waves of COVID-19 were recorded, occurring during weeks 15–46 in 2020 (2,369 cases), week 47 in 2020 to week 16 in 2021 (1,665 cases), and weeks 17–43 in 2021 (2,364 cases), respectively (Fig 1) [18].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Timeline of COVID-19 case and deaths in Sierra Leone, with the four quality assessment periods indicated in green, adapted from Liu et al [18].

https://doi.org/10.1371/journal.pgph.0002670.g001

The care of patients with suspected or confirmed COVID-19 was the responsibility of the CM pillar. To provide effective care and reduce the transmission of confirmed or suspected COVID-19 cases, the CM pillar used existing infrastructure to establish COVID-19 treatment centres (CTCs), COVID-19 community care centres (CCCs) and hospital isolation units (HIUs) to manage severe or critically ill patients, asymptomatic or mild COVID-19 cases and suspected COVID-19 cases, respectively. The case management response has previously been described in more detail [17]. A schema demonstrating patient flow between the facilities is provided in S1 Fig.

This manuscript describes the use of QI to assess and improve patient care in Sierra Leone in three phases. The first phase was the development of a quality assessment tool, the second phase was the assessment of quality at COVID-19 care facilities and the third phase was statistical analysis of the quality assessment data.

Results

The first Delphi round to decide the domains to be assessed by the assessment tool was responded to by 12/14 participants. The final domains included were; coordination; infrastructure; staffing; infection prevention and control; patient care processes; diagnostics and COVID-19 testing; drugs and consumables; oxygen; nutrition; vulnerable populations and safeguarding. 11/14 participants responded to the second round to prioritise questions within the ten domains. The final 50 questions included 13 outcome questions, 17 process questions and 20 structure or input questions²². The final tool included ten domains, with five questions for each domain, each question was scored from 0–2, providing a maximum score of 100. The Delphi process was initiated on the 22/06/20 and the tool was finalised on the 06/07/20 and is displayed here S1 Tool.

The pilot cumulative scores of the tool, to assess inter-rater variability were 74 vs 72 for CTCs, 66 vs 63 for HIUs. Mean time to complete the tool was 2 hours and 51 minutes (range: 02:10–03:35).

Quality results

A total of 94 assessments were included in the analysis over four assessment periods; round 1(n = 19) 16/07/20-20/07/20; round 2 (n = 25) 24/09/20-28/09/20; round 3 (n = 25) 19/01/2021–25/01/2021 and round 4 (n = 25) 14/07/2021–22/07/2021. Assessments were conducted at 27 facilities, 10 CTCs and 17 HIUs. Missing item data was low, 0.49% (23/4700) and median value replacement was conducted.

Mean scores per domain for each round of assessment for all 27 facilities are shown in Table 1. The difference in mean total scores increased significantly by 9.3 from round one to round two (P<0.01), increased non-significantly by 1.4 from round two to round three (p<0.64), and then significantly reduced by 7.9 from round three to round four (p<0.03). The highest mean scores were seen in the domains of infrastructure 8.3 (SD:2.1) and IPC 8.3 (SD:1.9) and the drugs domain had the lowest mean score of 6.2 (SD:2.9).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary mean scores by domain and by assessment round for all facilities, paired t-tests compare round of assessment to the previous round.

https://doi.org/10.1371/journal.pgph.0002670.t001

A mixed effects model of quality scores compared to baseline is reported in Table 2. An increase of 8.75 (p = <0.01) from baseline was seen in round 2, 10.67 (p = <0.01) in round 3 compared to baseline and 2.17 (p = 0.43) in round 4 compared to baseline. In round two significant improvements were seen in coordination, diagnostics, IPC, nutrition and vulnerable population domains. In round three, significant improvements compared to baseline were seen in diagnostics, staffing, IPC, nutrition and vulnerable populations. At round four compared to baseline, significant improvements were seen in diagnostics, IPC and nutrition whilst a significant decrease was seen in infrastructure. Disaggregated results, showing CTCs and HIUs separately are presented in the S1 and S2 Tables. Mean cumulative scores for CTCs were higher +5.8 than HIUs (p<0.02) overall and at all four timepoints, S1 and S2 Tables. Sensitivity analysis excluding facilities which were not included in round 1 (baseline), reported similar results, S3 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Mixed effect model compares quality of care score at assessment round 2, 3 and 4 to baseline quality of care assessment score in all facilities.

https://doi.org/10.1371/journal.pgph.0002670.t002

Discussion

Our study describes the creation of a standardised assessment tool for the quality of COVID-19 care and reports serial nationwide assessments of quality. We demonstrate that assessing quality of care during health emergencies is feasible and can be completed in a timely manner. From initiating the creation of the tool to reporting nationwide results took less than a month. There is likely a balance between the time taken to create a tool and the time pressure to establish a baseline and plan improvement of care. In future, pre-prepared tools alongside wider dissemination of tool development methods could further reduce the time taken to establish a baseline of quality of care during infectious disease outbreaks. The low levels of missing item data, suggest the mixed methods used to ascertain data are feasible to undertake at the facility level. The development of assessment tools on electronic tablets and app-based software, may further reduce missing data, reduce transcription errors and reduce data inputting time. Our assessment process achieved good nationwide coverage with all CTCs and HIUs in Sierra Leone being assessed at least twice.

The literature and current tools for COVID_19 largely focus on preparedness rather than quality of care [10, 25, 28, 29]. The Pan American Health Organization Hospital Readiness Checklist for COVID-19 [25] was adapted in Sierra Leone, and the authors report a cross-sectional survey of 9 hospitals. That assessment noted significant gaps in “COVID-19 leadership, coordination, health information, rapid identification, diagnosis, isolation and clinical procedures” [29]. Ogoina et al reports using a modified WHO preparedness tool [25], with 13 domains and 124 indicators, in Nigeria. Twenty of the 68 CTCs in Nigeria responded to the one-off self-assessment [30], with 3 (15%) hospitals reporting adequate readiness. A similar self-assessment in Gondar, Ethiopia using the WHO checklist found only 1 out of 8 assessed hospitals had adequate readiness [31]. In Malawi an existing tool, the Malawi Emergency and Critical Care Survey, developed using Nominal Group Technique process [32] was adapted to rapidly assess baseline readiness at 13 hospitals. The study interviewed 101 clinical staff, triangulating responses from multiple respondents to reduce reporting bias, and found gaps in oxygen access, personal protective equipment and isolation rooms [33]. In our study, we took the next logical step, to move from assessing preparedness to sequentially assessing quality of care. Quality improved significantly from round 1 through to round 3, demonstrating that QI methods can drive improvements in quality of care during an infectious disease outbreak. Our QI intervention was guided by principles for measuring quality of care from the High Quality Health Systems Commission [34]: Accountability and action were key, the Quality working group engaged key accountable people; including the facility in charge; the district medical officer; the regional COVID-19 case management leads and the central case management pillar. Assessment results were disseminated to key accountable individuals and a list of priority actions from each assessment was disaggregated into facility level actions, district level actions, and national level actions, with tasks assigned accordingly. The assessment process was designed to provide timely information to decision makers, with assessment results compiled and presented at the central level and district level within a week of assessment. Results were presented in dashboards, displaying colour coding and cumulative scores, allowing gaps to be identified and good practice to be recognised. Previous outbreak experience [35] and early experience in the pandemic revealed that achieving high quality of care would require a broad set of inputs and processes. This led to the adoption of a broad, holistic view of quality, encompassing staff, space, systems and supplies [36] required to achieve high quality care. This is particularly relevant to the health system in Sierra Leone where it was recognised that a narrow clinical focus would be insufficient to achieve high quality care in a fragmented health system [37] with low numbers of health care workers [38].

The reduction in quality observed from round three to round four may reflect a de-prioritisation of resources to COVID-19 CTCs by the health system [39] due to the reduction in inpatient case load and reported COVID-19 related deaths during that period [18]. This may reflect a healthy re-prioritisation of limited health system resources back towards essential health services [40], however, in the final assessment care was not significantly better than at baseline, and the reduction in the infrastructure domain neither bodes well for future preparedness. The reduction in the infrastructure domain was likely multifactorial; an absence of financing for the maintenance of facilities; a repurposing of facilities back to their original essential health service function as patient admissions declined and a shift of financing towards COVID-19 vaccination efforts instead of patient care.

Mean quality scores were significantly higher for CTCs than HIUs at all timepoints. In the authors collective experience, outbreak response resources and attention are largely concentrated on confirmed cases and treatment centres, to the detriment of suspected cases and HIUs [35], and as a result HIUs are comparatively underinvested in compared to CTCs. HIUs receive patients with suspected disease, with a broader range of underlying disease conditions, requiring a broader range of interventions, equipment and medications. In the capital Freetown, due to periods of significant delay in receiving test results, more COVID-19 deaths occurred in HIUs than in CTCs [41]. Whilst HIUs had lower mean scores at all timepoints, the decrease in mean cumulative scores in the final fourth round of assessment was not statistically significant for HIUs, suggesting some longer term strengthening of HIUs (mainly based in hospitals) may have been achieved in Sierra Leone. This is important as HIUs tend to be the permanent infrastructure of the clinical response compared to CTCs which may be established and then decommissioned as the outbreak wanes. A critical decision in Sierra Leone, in contrast to the Ebola response model, was that NGOs were not allowed to set up stand-alone treatment centres but were asked to support the government to establish HIUs and CTCs within existing government facilities.

The domains of drugs and care processes were resistant to improvement and analysis of individual items highlights some areas where improvement did not occur. Whilst NaCOVERC did obtain and distribute medications for COVID-19, these did not routinely include drugs for common co-morbidities, measured by the assessment tool, such as hypertension and diabetes. Drug distribution may not always have reached the facility level, and the lack of access to medications for common co-morbidities remains concordant with reports from facilities [41]. Patient care processes saw little improvement, and indicators with persistent low scores were risk stratification and acuity-based triage of patients. Oxygen scores only showed a modest improvement, this reflects the lack of regular access to high flow oxygen outside of Freetown and inappropriate oxygen prescription and use. Oxygen access in Sierra Leone was severely limited, throughout the outbreak, whilst oxygen cylinders from a private supplier were distributed by NaCOVERC, these did not meet demand and the scale-up of oxygen producing capacity at government hospitals [42] only began in 2022.

Our study has limitations and strengths. Our first assessment took place in June 2020, two months after the first case of COVID-19 was reported in Sierra Leone. It is not therefore, a pre-COVID baseline and it does not capture the initial interventions in the health system, including initial infrastructure investments. Our process used expert opinion and is therefore limited by the experts who were included in the process. Delphi process participants were from a diverse range of clinical and non-clinical backgrounds; however, we could have included a larger group of experts and patients in the process. Due to the urgency of the task, we did not anonymise participant feedback, and this may have affected participant responses. A further limitation of our study, is that there was insufficient time to comprehensively assess the inter-observer variability of our tool, which was only measured on two occasions, but reflected the need for rapid deployment and use of the tool. Finally, our study was not designed to assess the relationship between quality scores and patient outcomes which are the end goal of the process. Our study benefited from using a clear methodology to design the assessment tool, grounded by a conceptual framework [22]. We used multiple overlapping methods of data collection; direct observation; interview with healthcare workers and review of clinical documentation. We report good coverage of the assessment, assessing all CTCs and HIUs in the country at least twice, and are able to report serial assessments over one year. Further work should include greater piloting of assessment tools before their implementation and the inclusion of patients in the Delphi process if feasible.

Based off our experience, we advocate for the standardized assessment of quality of care and use of QI methods during health emergencies. Assessment tools should be designed using appropriate methodology, provide standardized guidance on assessment methods and how to modify tools. Assessments of preparedness, should be designed with an increased focus on quality of care, to enable serial assessments over time, and be grounded in a conceptual theory of change or QI methodology. Standardized assessment of quality of care; allows standards to be agreed and set; encourages assessment of respectful care of an agreed quality; strengthens the rights of the patient; enhances protection of healthcare workers; enables performance to be compared locally, internationally or over time [43, 44]; identifies critical needs; and targets rapid interventions and investments [45]. Finally, standardized assessment of quality of care of patients during health emergencies could be a vital tool to promote learning health systems [46], encourage the sharing of good practice and thereby strengthen global health security.

Conclusion

This study demonstrates that a quality of care assessment tool can be rapidly created using a modified Delphi process, and that QI methods can drive improvements in care during a health emergency. We recommend that standardized assessments of quality of care and use of QI methods should be incorporated into future health emergencies.

Supporting information

Acknowledgments

We would like to acknowledge all the participants of the modified Delphi process for their expert input and advice. We would like to thank all the assessors for their diligent work in travelling the country and assessing quality of care. We would like to acknowledge all staff in the healthcare system and NaCOVERC who helped drive improved care for our patients. We hope that this study can in a small way ensure that in future all patients receive safe, dignified and quality care during health emergencies.