Study design

A four-dimensional incomplete cross-sectional stepped wedge cluster randomised trial [17] was conducted between 1^st July 2018 and 7^th March 2019. This design was selected because a clinical audit involves both a control (step (ii)) and intervention (step (v)) phase of data collection. Clinical audit is conducted using samples of patients within a healthcare facility; thus, to evaluate the impact of conducting standards-based audits, after introduction of the process at a healthcare facility, clustering by facility was necessary. The dimension was determined by the maximum number of standards to be audited within each healthcare facility. Data for each standard assessed was collected for a cluster of individuals within a subset of steps at participating healthcare facilities (clusters), the design was therefore ‘incomplete’.

Trial intervention

The intervention consisted of the introduction of a process of standards-based audit consisting of 3-month audit cycles, each of which comprised: i) a month in which data were collected to assess compliance with the standard(s) chosen for audit (measurement 1); ii) an ‘action month’ in which the required identified actions were implemented to improve compliance with the standard; and, iii) a third month in which data were collected to re-assess compliance with the standard(s) (measurement 2). For each standard that was audited at a facility the ‘action month’ of the audit cycle was the step in which the classification of the status transitioned from control to intervention.

Trial outcomes

The primary outcome measured for each woman was compliance with defined standard of care without regard to which standard was audited. Each secondary outcome measure was the same for a specified standard.

Study setting

The Ministry of Health in Malawi identified five of its 28 districts for the implementation of a quality improvement programme (opportunistic selection). In each district all healthcare facilities providing Comprehensive or Basic EmONC were selected. A total of 44 facilities were included with 34 providing Basic (BEmONC, health centres, primary level) and 10 providing Comprehensive Emergency Obstetric Care (CEmONC, hospitals, secondary level) in the five implementation districts (S2 Table).

Each participating healthcare facility was to implement two consecutive periods of standards-based audit with each period anticipated to take three months. To avoid confounding in the design the steps in which audit cycle periods commenced were staggered over 3 months (steps); the design thus involved six steps. Randomisation was balanced for each standard, district, and facility type. Fifteen facilities were randomised to start in each of months 1 and 3 and the other 14 were to start in month 2.

CEmONC facilities conducted audit on two standards per quarter (four standards each in total) and BEmONC facilities audited one standard in each quarter (two standards each in total).

Development of standards

A total of 25 standards of care, including standards for respectful care and for clinical content specific to EmONC, were developed and agreed (S3 Table). Based on existing WHO guidelines and the internationally agreed components of EmONC, these standards were developed through discussion and consensus agreement at five multidisciplinary consensus building national and international workshops (including managers, midwives, general doctors, obstetrician-gynaecologists, and researchers).

Training in standards-based audit

Healthcare providers from each facility participated in a four-day participatory training workshop on how to conduct standards-based audit, how and which data to collect, including the design of simple audit data collection tables and before and after comparison of percentage compliance. The healthcare providers from each facility worked together to identify relevant priority standards for their healthcare facility from the list of 25 standards provided, with the understanding that any standard chosen was expected to be applicable to at least 25 clients per month at the relevant healthcare facility. Participants from CEmONC facilities identified six and participants from BEmONC identified four standards in total.

Assigning standards

The trial statistician (SW) subsequently assigned the standards to be audited by each facility to ensure coverage of as many standards of interest as possible, drawn from the standards ranked for each healthcare facility (where possible facilities that ranked a standard first were selected for the standard). The sequence in which standards were to be audited within each facility was assigned by the trial statistician using the runiform() function in a Stata program. For each standard the design of the study was an incomplete stepped wedge design. Thus, for each of the facilities assigned to audit a given standard, the timing of assessments (measurement 1 and measurement 2) for a standard was assigned to follow one of the six sequences listed for a typical standard ‘a’ (S1 Fig). For CEmONC facilities that audited four standards the design was four-dimensional. The dimensions do not correspond with standards. Allocation concealment was not possible for this trial’s design.

Data collection

Each standard was assessed during the first and third month of an audit cycle (S1 Fig). In addition, compliance with all standards selected for that facility was measured in the first and final (sixth) months of data collection at each healthcare facility. For example, S1 Fig indicates all assessments that were to be performed for standards audited within BEmONC facilities auditing standard ‘a’. For each standard a standard-specific data collection tool was developed to capture the data required to determine whether the standard had been achieved in the care provided for each patient assessed. For some standards an exit interview was conducted to collect data (eg respectful care); for other standards data were collected from registers or case notes. At each healthcare facility data for a standard was collected by the trained healthcare providers at the facility for 25 consecutive women attending for care within the relevant month. No patient identification details were captured for any standard. For each audit cycle the healthcare providers kept a log of details of the problems identified and actions undertaken to improve compliance with the standard.

Sample size considerations

Sample size calculations were performed using an R Shiny app to determine differences in compliance detectable for individual standards [18]. It was assumed that data for each standard would be collected according to the incomplete stepped wedge design shown in S1 Fig, with three clusters following each sequence in the design. For assessment of compliance with each standard, samples of 25 clients per standard, month and facility were planned. The calculations performed indicated that there would be at least 80% power to detect an absolute improvement in compliance of 22% (or an odds ratio of 2.6) for a standard audited at 18 healthcare facilities, provided the initial compliance was about 50%, the intra-cluster correlation did not exceed 0.2 and the cluster autocorrelation did not fall below 0.5.

Statistical analysis

As specified in the statistical analysis plan data aggregated across standards were analysed, using Stata version 14, to provide an estimate of the impact of performing standards-based audit on compliance with standard. Analysis used all data collected for each standard and was on an as implemented basis. Analysis used the binary responses (compliant / not) of respondents within facilities in multilevel mixed effects logistic regression models, with fixed effects for Intervention (measurement 2 vs measurement 1), Standard (12 standards), Facility Type (Basic and Comprehensive) and Month (calendar month) (the statistical analysis plan incorrectly stated that Month would be a random effect), and random effects for Facility (cluster), and Month by Facility interaction (model extension B described by Hemming and colleagues) [19]. District was also considered for inclusion as a random effect. The effect of performing standards-based audit on compliance was estimated with a 95% confidence interval, after adjusting for Standard, Facility type, Month and Facility by Month interaction. Each standard considered was also to be analysed separately, to provide an estimate of the effectiveness of audit on compliance with the standard. These analyses involved the same terms as for aggregated data, except for terms involving Standard.

Several sensitivity analyses were performed to explore the impact of assumptions made when planning the trial analysis:

1. i). To examine whether there was any statistical evidence that the baseline compliance rate influences the odds of improvement following the action of the audit cycle the models used were expanded to include baseline compliance (on the percentage scale) as a covariate and interaction of this with the exposure variable. This analysis excluded the five facility-standard combinations for which no baseline data were collected.
2. ii). To examine evidence of any impact of overlap in the content of Standards an indicator factor was added to indicate data collected in period 2 for a standard which overlapped with a standard audited in period 1 at the facility.
3. iii). The planned analysis used only one two-way interaction term in the analysis. Five two-way interaction terms (Intervention by Facility Type, Standard by Facility Type, Standard by Month, Facility Type by Month and healthcare facility by Standard) were each considered for inclusion along with baseline compliance, in separate analyses to explore possible model extensions. These were applied to all eligible data and to the subset of data for the four most audited standards.
4. iv). To assess the impact on the estimates of the inclusion of additional assessments in month 6 (or 1) when audited in period 1 (or 2) two approaches were used: i) the mean (SD) of the change in compliance within the same intervention state (months 3 and 6 when audited in period 1 and months 1 and 4 for audits in period 2) was considered; ii) the main analysis was repeated using only data from audit cycles, i.e. with the additional assessments excluded.

Ethical considerations

Ethical approval was obtained from the Research and Ethics Committee, Liverpool School of Tropical Medicine (Research Protocol 18–028, approved 21^st June 2018). The Ministry of Health and Population, Malawi authorised conduct of the study waiving ethical approval in country (letter dated 20^th June 2018, ref QMD/10). As consent is not normally part of a clinical audit process consent was not sought.

Primary outcome: Aggregated standards

For the primary outcome, using data aggregated for all standards, the improvement in compliance attributed to the use of standards-based audit was statistically significant; the adjusted (for standard, facility type, month, and healthcare facility by month clustering) OR (95% CI) was 2.80 (1.65,4.76) (Table 2). The ICC (95% CI) for healthcare facilities was estimated to be 0.14 (0.04,0.25) and for the interaction of healthcare facility and month the ICC was estimated to be 0.42 (0.34,0.49). Consequently, the CAC was estimated to be 0.34.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Estimates of compliance OR between study arms and correlations.

https://doi.org/10.1371/journal.pone.0310896.t002

A sensitivity analysis, excluding the three standards used in no more than two healthcare facilities, yielded similar results, with an adjusted OR (95% CI) of 2.92 (1.72,4.96). Further analyses considered omission of the random healthcare-facility by month effect and confirmed that the fit of the model was significantly improved when the term was included. Conversely inclusion of a random term for district did not significantly improve the fit of the model.

Secondary outcomes: Compliance with individual standards

Analysis of data for the seven standards with sufficient data for a separate analysis yielded ORs which range from 1.10 (0.25,4.90) (for CS timing standard) to 19.6 (3.10,124) (for respectful care standard) (S6 Table). The 95% confidence intervals for each of these covered the value of 2.80 estimated in the aggregate analysis. However, for four standards (ANC pre-eclampsia screening, partograph, CS timing and MVA exam) the 95% confidence interval included 1.0 and thus these analyses did not provide evidence that the observed increase in compliance was statistically significant. As the partograph standard was the standard most frequently audited, data for it were explored further.

The logistic regression model for the partograph standard estimated that a substantial secular increase in compliance occurred, with the OR for compliance compared with month one increasing to month eight. The distribution of month in which healthcare facilities took action for this standard was not well balanced, with just two facilities per step for steps 4, 5 and 6 but at least five facilities in each of the other three steps (2, 3 and 7). Compliance improved at all four healthcare facilities which took action in steps 4 and 5, between the baseline assessment and the initial assessment of the audit cycle (control state), these improvements would have contributed to the estimated upward secular trend. Additionally, compliance did not improve for the assessment after action in either of the healthcare facilities in step 5 (S7 Table and S2 Fig).

Sensitivity analyses to explore robustness of model fitted

Problems identified and actions taken at healthcare facility as part of standards-based audit

A summary of the problems identified, and actions taken (steps 2 and 3 of the audit cycle) is presented in Table 3. The problems identified were classified into four categories:

1. A—Conduct and practice of healthcare providers—not carrying out duties as required, not aware of need for certain clinical practices and poor documentation of practice. This was noted as a problem by 29 facilities for 55 audit cycles, for all standards audited except the oxytocic standard. Staff shortage and/or non-availability of staff on site was a problem for 3 of 7 audit cycles for CS timing and MVA timing standards.
2. B—Lack of availability of equipment, consumables and tools for documentation was a problem reported at 18 (41%) facilities for 20 of the 69 audit cycles conducted for five standards (oxytocics, sepsis detection, malaria detection, ANC pre-eclampsia screening and partograph). This mainly required additional simple equipment and tools to be made available sometimes from existing stock (thermometers, urine dipsticks, partographs, patient charts).
3. C–General Infrastructure of the healthcare facility was a problem for 6 of 21 facilities auditing two standards (respectful care and CS timing). Examples were: the construction of a labour room not allowing for patient privacy e.g., open room with no separation between beds; no accommodation on site for emergency staff.
4. D–Patient- related factors were a recognised problem for 2 of 3 facilities meeting two standards (Hb after APH / PPH and identification of pre-eclampsia) and in each case simple diagnostic tests (measuring haemoglobin, checking protein in urine using a dipstick) could not be provided to women unable to pay i.e. these services were not free.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of problems identified, and actions taken to improve compliance with standards.

https://doi.org/10.1371/journal.pone.0310896.t003

At healthcare facility level teams worked hard to solve these problems with practical and feasible solutions arrived at and implemented as illustrated for each standard in Table 3.

Main findings

This study shows that for a pre-agreed set of standards reflecting the quality of care in maternal and newborn health, there was a statistically significant improvement in compliance for a standard audited. For a standard selected for audit the estimated OR (95% CI) of compliance with standard, following feedback and action, in the intervention phase compared with the control phase was 2.80 (1.65,4.76).

For individual standards estimates of change in compliance (using ORs) were variable and prone to bias, likely due to the inherent confounding of time and intervention, and imbalance in the data available, which arose from deviations from the randomisation.

As part of the audit cycle healthcare providers identified problems that resulted in lack of compliance with the agreed standards of care. These were most frequently recognised as requiring a change in staff conduct and performance–with reorientation and improved supervision of staff implemented to improve clinical practice and documentation. Lack of consumables and equipment, e.g. thermometers, simple point-of-care diagnostic tests, partographs and patient charts, for five of the standards audited, was noted at 41% of facilities. In several healthcare facilities the overall infrastructure of the labour rooms required more attention as patient privacy could not be provided because of open labour rooms with no structural (or screens) division between beds. Non availability of staff for operative procedures (manual vacuum aspiration and caesarean section) was in part due to the non-availability of accommodation on site for staff required for emergencies after daytime working hours.

Strengths and limitations

To the best of our knowledge this is the first study to assess the effectiveness of standards-based audit as a quality improvement method for Emergency Obstetric and Newborn Care using a stepped-wedge randomised trial design. The estimated impact of the intervention on the odds of compliance was consistent across the sensitivity analyses performed, suggesting that the findings for aggregated data are robust.

For some Standards the deviations from randomisation of both the allocated standards and their timings, created considerable imbalance in data obtained per step for the standard. The small number of observations for each step reduced the precision in estimation of secular trend. This reduced the power of the analysis for some individual standards to detect a benefit from the intervention and produced some imprecise estimates.

One standard (malaria detection) was seasonal since the incidence of malaria is usually higher during the later months of the study. It was audited at 15 healthcare facilities–six in period 1, nine in period 2. During the malaria season there may be a higher underlying chance of malaria being considered as the possible cause for fever and hence an increased chance of being tested for malaria as the study period progressed. If this is the case this could have caused the impact of the intervention to be over-estimated for some of the 15/104 audit cycles completed for this standard.

There was good overall balance in the design of data capture for the aggregated data analysed. Although an underlying upward trend in compliance was estimated, a statistically significant improvement in compliance was detected. In each study month at least 14 facilities contributed data to the aggregate analysis.

Inclusion of the baseline compliance rate as a covariate in the analysis improved the fit of models and is a useful strategy for accounting for part of the variation in compliance with the various standards. Although each healthcare facility contributed data for multiple standards there is little justification for assuming that compliance with all standards will be similar within any given healthcare facility. Indeed, it is anticipated that compliance for standards varies between healthcare facilities which were encouraged to choose standards for audit for which they perceived compliance to be sub-standard (rather than standards for which compliance was considered as already high).

Implications for future trial design

The findings of the sensitivity analyses suggest some additional issues for consideration in planning the design and analysis for any future study using this class of trial design. These issues are itemised below:

1. Analysis including Baseline as a covariate suggests that the magnitude of the impact on the OR is greater when the initial compliance level is lower and therefore consideration should be given to inclusion of Baseline as a covariate.
2. The overlapping content of standards audited had minimal impact on the estimated impact of the intervention. However, because of the potential bias arising from such overlaps it would be prudent to avoid such overlaps in the selection of standards for auditing.
3. As is evident in the estimates derived from analysis of individual standards the sparseness of the design limits the capacity of the analysis to derive valid estimates for multiple effects. Although several interactions were statistically significant when included in an analysis the validity of such estimates is difficult to establish. Inclusion of interaction terms for standard by Month and standard by Facility Type separately indicated that each was statistically significant, but when included together the standard by Month interaction did not improve on the fit without the term in the model. This suggests that the inherent imbalance in the study design, with the distribution of months in which data are measured for each standard being heterogeneous, may be the cause of the apparent interaction when included without standard by Month rather than a real interaction between standards and Facility Type. Alternatively there may be a spurious association between the two interaction terms.
4. Exclusion of the additional baseline and endline assessments resulted in a slightly larger estimated OR for the impact of the intervention, with a decrease in precision. At some healthcare facilities the inclusion of the additional assessments was associated with an improvement in compliance between assessments without the intervention. The additional measurements provide opportunity to improve the precision and hence power of the study to detect an effect of the intervention.

Implications for clinical practice

Improving the quality of healthcare services and making quality an integral component of scaling-up of interventions that are known to be effective is crucial if health outcomes for mothers and babies are to improve [4,5].

Skilled birth attendants need to be trained to have the required competencies and should be provided with an “enabling environment” which includes drugs, supplies, appropriate policies, and a functional referral system [20]. All women require skilled attendance at the time of birth [6,7]. For an estimated 10–15% of women, a potentially life-threatening complication develops during pregnancy, birth, or the post-partum period. In most cases, this complication will be unexpected and unpredictable. Therefore, it is crucial that all women and babies have access to good-quality Emergency Obstetric and Newborn Care [7,8]. The most frequent and main complications are well understood and can be readily prevented and/or treated. They include haemorrhage, sepsis, eclampsia, and complications of obstructed labour and abortion. There are existing effective medical and surgical interventions that are relatively inexpensive which can be put in place to manage these complications [21]. It is important that healthcare providers are confident and competent in the provision of this care [22–24]. Training of health care providers can improve implementation of evidence-based practice but for optimal effect the organisation and quality of health services require additional focus [25].

Multi-country surveys demonstrate that quality improvement activities are in place in many settings but that these tend to focus mainly on audit of adverse events through conduct of maternal and perinatal death audit [25,26]. This is generally effective especially in identifying specific areas of care that require improvement because they are assessed as sub-standard. Data on cause of and factors contributing to maternal and perinatal death are ‘local’ and can help identify specific gaps as well as solutions that can be put in place at the healthcare facility level [27,28]. This requires that a culture of improvement and quality is developed rather than a culture of blame. Specific standards can be agreed that reflect the identified areas of care that need to be improved. Systematic use of standards-based audit with healthcare providers themselves evaluating the care they are giving and making improvements where needed can lead to demonstrable improved motivation, ownership, and sense of responsibility for delivering good quality care [29,30].

Poor quality care is often a function of weak health systems and processes or problems in their implementation generally rather than the fault of individuals. Systematic reviews suggest that effectiveness is likely to be higher when baseline compliance is low, when feedback or audit cycles are conducted more than once, with local expertise in conduct of audit, team effort rather than individual actions with the agreement of explicit actions and development of action plans [16,29,30]. Support is also needed at local and central government levels to enable further scale-up audit.