Study design

The study was designed as a multi-institution, retrospective record review with hospital visits as the main source of program evaluation. A total of ten hospitals, split evenly between the Amhara and Sidama regions, were visited by the research team from July-August 2022. During hospital visits, surgical data practices were observed to assess the flow of information between hospital registries, hospital monthly key performance indicator (KPI) forms, and the District Health Information Software 2 (DHIS2) reports reviewed at the regional and national level. Data owners and instruments were identified at each step. The dates of record review ranged from 1/1/2022 through 6/31/2022. This time period was chosen to evaluate the most recent six-months of hospital data from the time of inquiry. All data accessed was deidentified and the study authors were unable to identify specific individuals from the data. The research team undertook a complete surgical data audit at each hospital visited. Registries within the OR, surgical ward, ICU, and maternity ward were reviewed by each reporting period and data points were aggregated to calculate monthly indicators. Researchers met with hospital quality improvement officers to audit monthly KPI reporting forms and final DHIS2 electronic database reports. At the Regional Health Bureaus, the final data reports from the queried hospitals visited were reviewed. At the national level, the DHIS2 conglomerate data from each Regional Health Bureau was evaluated. Data triangulation compared hospital to regional to national data.

Subjects

Primary and general public hospitals who were previously involved in Safe Surgery 2020 interventions, with approval from the Amhara and Sidama Regional Health Bureau and the Ministry of Health, were selected for this research [19, 21]. Amhara hospitals were further chosen based on their participation in the 2018 surgical data intervention carried out by the research team. Sidama hospitals were selected according to their involvement in Safe Surgery 2020 programming. Records of patients who underwent major surgery, defined as any procedure conducted in an operating room under general, spinal, or regional anesthesia [22], during their hospital stay within the study period were reviewed in this study.

Methods of measurements

Surgical Key Performance Indicators (KPIs) of interest were selected based on a prior surgical data intervention conducted at Amhara hospitals and on feedback from the Federal Ministry of Health (FMOH). The research team selected five indicators: Surgical Volume, Perioperative Mortality Rate (POMR), Rate of Safe Surgery Checklist Utilization (SSC), Surgical Site Infection Rate (SSI), and Anesthetic Adverse Outcome Rate (AAO) for data analysis. Formal definitions for each indicator were defined by the Saving Lives Through Safe Surgery (SALTS) program [22] and are included in Table 1. For collection of data from the registries, KPI reporting forms, and DHIS2 data, the investigators utilized Microsoft Excel forms at each hospital during on-site visits.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Five surgical key performance indicators of focus and definitions [19].

https://doi.org/10.1371/journal.pgph.0002600.t001

Data analysis

Enumerated data from hospital registries were compared with the data in the national database. Analysis consisted of comparing data between sources (registry, KPI reporting forms, and DHIS2) and assessing data quality according to dimension 1 (data completeness) and dimension 2 (internal consistency of reported data) of the Data Quality Review (DQR) framework [22]. Completeness of data was evaluated by reviewing the monthly KPI reporting forms for the study period (Jan-June 2022) and documenting the number of months KPI data was absent for each indicator. For example, if the value for surgical volume was missing for one month at a single hospital, this was denoted as missing 1/6 of surgical volume data for the study period. Inconsistent data was defined as the number of months within the six-month study period where the registry source data did not match the DHIS2 electronic database. Following FMOH guidelines, surgical indicator data was deemed inconsistent if DHIS2 data differed by more than 10% of the registry data. This established verification factor for routine data quality assessment is calculated by the number of recorded events from the source document (registry) divided by the number of reported events in the DHIS2 report, with verification factor greater than 1.1 (under-reporting) or less than 0.9 (over-reporting) deemed inconsistent [23]. The KPI Safe Surgery Checklist utilization was an exception where KPI reporting forms were used as the source data and matched to DHIS2 data, given this indicator is not recorded in the registry.

Ethical approval

Institutional review board approval was obtained from Medical College of Wisconsin (MCW) and the Ethiopian Public Health Institute (EPHI) for all study activities. EPHI approved the following protocol number: EPHI-IRB-454-2022, titled “Assessment of Current Data Practices in Ethiopia.” The MCW Institutional Review Board approved the study under PRO ID: PRO00043474. The study investigators also received a letter of support from the Federal Ministry of Health.

Inclusivity in global research

Additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the S1 Checklist.

Hospital characteristics

Table 2 includes hospital specific data on facility resources and characteristics for the ten study hospitals. Median catchment population was 375,000 patients (70,000–1,800,000) whereas median total hospital beds was 45 (25–145). Integrated Emergency Surgical Officers (IESOs), Master’s-level health professionals intended to provide emergency and essential surgery in Ethiopia, were the predominant surgical provider at most institutions [24].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Hospital-specific characteristics.

https://doi.org/10.1371/journal.pgph.0002600.t002

Data collection processes

At all ten hospitals, data flow mechanisms were observed and compiled into a generalized process illustrated by Fig 1. The data collection process involved recording data elements in registries by hospital staff, then calculation of KPIs by quality improvement (QI) officers, completing monthly KPI reporting forms, and submitting data into the electronic national data system (DHIS2). For the five KPIs evaluated, healthcare providers entered surgical data into registries specific to their department, i.e. wards, ICU, or operating room. At the end of the reporting period, the QI officers reviewed the registries and utilized the raw data to calculate key performance indicators. If they noted any discrepancies or missing data, they would query the health provider responsible for that element. KPI data collection forms for this purpose varied between hospitals. The KPI data is then entered into the electronic DHIS2 and uploaded to the Regional Health Bureau (RHB) for approval. KPI data at the RHB is aggregated from multiple hospitals together to describe the surgical practice in the region and submitted to the Federal Ministry of Health.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Surgical indicator data flow.

Responsible data owner and reporting instrument at each step of the data collection and reporting process. KPI = Key Performance Indicators, DHIS2 = District Health Information Software 2.

https://doi.org/10.1371/journal.pgph.0002600.g001

Surgical key performance indicators

From registry data collected independently, values for the five KPIs of interest were calculated for each month from January to June 2022 (Tir to Sene 2014 in the Ethiopian Calendar). For all ten hospitals over six-months, average monthly surgical volume was 57 cases and SSC utilization was 93% (Table 3). Over one month, the mean surgical volumes were 54 and 59 cases per hospital in Amhara and Sidama, respectively. In Amhara, the majority of hospitals reflected the regional average. In Sidama, three primary hospitals performed less than 15 cases each on average per month, while the two general hospitals each performed between 55 and 214 operations on average per month. This extrapolates to a predicted rate of 50 to 712 operations per 100,000 population annually at primary hospitals in Amhara, 30 to 67 per 100,000 in Sidama primary hospitals, and 80 to 147 per 100,000 in Sidama general hospitals according to their determined catchment populations. When calculated using a standardized catchment population of 1 million for primary and general hospitals, there is a predictive rate of 35 to 121 operations per 100,000 population annually at primary hospitals in Amhara, 4.8 to 12 per 100,000 in Sidama primary hospitals, and 64 to 264 per 100,000 in Sidama general hospitals. Across all ten hospitals over the 6-month period there were thirteen post-operative mortality events, for a collective POMR of 0.38% (13/3399). For the same period, twenty-seven surgical site infections were reported for an overall inpatient SSI rate of 0.79% (27/3399). Five anesthetic adverse outcomes were documented for a collective AAO rate of 0.15% (5/3399).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of registries to DHIS2 data for surgical KPIs.

https://doi.org/10.1371/journal.pgph.0002600.t003

Data quality

Data quality was assessed based on two metrics: completeness and consistency. Our study found data completeness to be an average of 72% for the 10 hospitals, and 70% of hospitals had consistent data for the five surgical KPIs of interest. Verification factor calculations between registry and DHIS2 data, as seen in Fig 2, revealed inconsistencies (>10% difference) at nine total hospitals, commonly SSI (6/10) and surgical volume (5/10). Five hospitals demonstrated incomplete registry data, with the most commonly incomplete indicator data for AAO, SSC, and SSI. Across all five KPIs of interest, surgical volume was the most likely to be recorded within the registry and have inconsistencies between values recorded in the OR registry and DHIS2 (Table 3). Of note, Sidama hospitals frequently had more inconsistent surgical volume data than Amhara hospitals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data quality assessment of five surgical indicators.

Number of hospitals out of 10 with inconsistent (blue) and incomplete (orange) data for each KPI of focus for the 6-month period. SSC = Surgical Safety Checklist utilization, SSI = Surgical Site Infection, POMR = Perioperative Mortality Rate, AAO = Anesthesia Adverse Outcome.

https://doi.org/10.1371/journal.pgph.0002600.g002

After further investigation of inconsistencies between registry data and DHIS2 data, national data is revealing underreported rates (verification factor >1.1) for major surgical complications: SSI, AAO, and POMR. For POMR, 11 of the 13 events documented for all ten hospitals across 6-months were recorded at Hospital 3. However, this value does not correspond to the DHIS2 data which reported zero POMR events for Hospital 3 across the 6-months (Table 3). Furthermore, 1 POMR event observed at Hospital 9 for the reporting period did not transfer to the DHIS2 database but was present in the KPI reporting form. All of the hospitals who recorded any post-operative deaths (3 hospitals in total), had at least one month with inconsistent POMR data between registries and DHIS2. SSI rates were similarly underreported, with over half of the registry-documented surgical site infections at all ten hospitals over six months missing from the overall SSIs within DHIS2 (27/3399 vs 13/3399). SSI rates were inconsistent for seven out of ten hospitals for the evaluated period. For AAO, only Hospital 7 recorded 5 AAO events within their registry for the six-month period, but zero AAO events were present within DHIS2 for all ten hospitals over the study period. SSC data was mostly consistent between KPI forms and DHIS2 data, with the exception of Hospital 9 where median SSC differed by 12.5% and Hospital 3 where the range of SSC values differed by 36% but the median remained the same.

Surgical site infection

For SSI, our study estimated a value of 0.79% (27/3399) across 10 hospitals, lower than the 2% value recorded by the Network for Perioperative and Critical Care (N4PCC) 2021 registry [25] and the 10.2% described in the African Surgical Outcomes Study [11]. Specifically for SSI, patients are often discharged before they will develop signs and symptoms of infection. There is currently no uniform mechanism to enumerate outpatient SSIs.

Anesthesia adverse outcome

Similarly for AAO, our dataset of 3399 surgeries yielded only 5 AAO events for a rate of 0.15%. This is considerably lower than the N4PCC 2021 registry value of 3.1% [25] and closer to the AAO reported value of 0.94% in Zimbabwe [26]. The use of a multicenter cloud-based registry, in-hospital coordinators and data trainers, and utilization of an offline mobile application for data collection most likely improved reporting compliance and identification of adverse events, leading to increased AAO capture rates in the N4PCC study [25]. Variation in AAO rates reported can also be attributed to variations in KPI definitions, as in Zimbabwe where an AAO is defined as a critical incident and includes nausea or post-operative pain [26]. Furthermore for AAO, it may be difficult to detect and diagnose complications such as high spinal anesthesia. This was also the indicator with the greatest variability in recording mechanisms and data ownership, likely accounting for the very low AAO, with only one hospital reporting these events. Additional training for knowledge and skills may be needed in this case.

Peri-operative mortality rate

Our study found a lower POMR at 0.38% (13/3399) than the 0.9% (9/1000) found in the N4PCC 2021 registry surveyed at 4 Ethiopian hospitals [25]. This disparity may be due to the indicators being limited to only inpatient events, thus POMR may be lower than expected as deaths for KPIs are registered only before discharge. There were also some deaths recorded in registries, which were not reported in the national data as mentioned previously. Further external mortality audits for these complications including chart reviews and periodical follow-up surveys, as well as a robust death registry, are recommended to understand the true state of surgical outcomes in Ethiopian hospitals.

Surgical volume

Calculated annual surgical volume for Sidama general hospitals is 80 to 147 operations per 100,000, 50 to 712 operations per 100,000 population at primary hospitals in Amhara, and 30 to 67 per 100,000 in Sidama primary hospitals. In our dataset, surgical volume at all ten hospitals has not yet reached the LCoGS 2030 target of 5000 surgeries per 100,000 population annually [2]. Study findings of surgical volume for Sidama general hospitals (Jan- June 2022) are consistent with Meshesha et al.’s surgical volume of 289 per 100,000 over a 90-day interval (Sept 2020—May 2021) for general hospitals [27]. Our study calculates higher annual surgical volumes for Amhara and Sidama primary hospitals than the 37 per 100,000 found for primary health care units over a 90-day interval. The Amhara primary hospitals showed an overall increase in their surgical volume compared to the previous study during the data intervention in 2018 [28], which may be due to both growth in surgical capacity and population needs.

Surgical safety checklist

Estimated SSC values were 93% for 3399 surgeries, consistent with the 92.1% found in the N4PCC registry for 1595 surgical cases [25] and higher than the 67.6% rate reported by Sibhatu et. al. 2022 [29]. Our study determined 60% of audited hospitals had complete SSC data, consistent with the 60.8% of 659 checklists the latter study found to be filled completely and correctly. The barriers to data collection identified at ten hospitals within the Amhara and Sidama region are consistent with the findings of previous investigations into data quality within the Ethiopian healthcare system. The overreporting of certain surgical KPIs, namely SSC and underreporting metrics of illness or mortality was concurrent with findings from Mekebo et. al. and other studies [28, 30, 31]. Our observation of data flow processes demonstrated that SSC values are calculated by QI officers by selecting a sample of ten patient charts for review. At certain institutions, stop-gap mechanisms were implemented to prevent the progression of patient charts to the Inpatient Admission office (IPAD) until all forms including the SSC were completed. Such processes might explain the disparity Sibhatu et. al found between the DHIS2 SSC value of 81% and the externally audited 60.8% retrospectively calculated from patient charts [27]. High rates of surgical data inconsistency were a major result of this research and are likely due to lack of consistent KPI reporting forms, and the considerable burden of aggregating substantive data elements into complex indicators as discussed previously by Adane et al 2021 [32]. Our observations were in alignment with other challenges with data collection in an Ethiopian setting previously discussed including turnover of hospital staff, poor understanding of data processes, modification, or manipulation of data to compensate for the lack of data, and technological issues with DHIS2 [28, 30].

Limitations

National implications of this study are limited by the small sample size (ten hospitals) and limited study period (six months). Hospital selection was also limited to facilities with prior involvement in Safe Surgery 2020. This limitation could potentially impact the generalizability of these findings to the current surgical landscape in Ethiopia, as system changes are ongoing and interregional differences are likely. The investigators included two regions within Ethiopia as well as both primary and general hospitals as countermeasures to the limited sample size and period. Selecting hospitals involved in the prior surgical data strengthening program (in Amhara) and those not (in Sidama) was also employed to expose potential differences in data practices. Follow-up investigation using qualitative methods, specifically surveying data collectors and data managers at participating hospitals and regions, will provide further context around the findings. Future studies are also planned to expand this initial investigation of key performance indicator data in Ethiopia to a larger number of hospitals and regions.

Additionally, the use of registry data as primary source data generates the risk that some surgical indicators not recorded in the registry were left out of our analysis. Using registry data rather than performing a retrospective review of surgical patient charts could underestimate issues with data quality. Mortality audits, or complication audits, are another potential retrospective mechanism to capture more complete data for these outcomes, requiring access to patient charts [33, 34] Future studies incorporating alternative mechanisms, such as those described, will contribute important information on the accuracy and reliability of registry data recorded during the patient’s hospital course.

Another source of data quality inconsistency could be attributed to reporting period variability between hospitals. This study utilized DHIS2 reporting periods for data auditing. The implications of variable reporting periods are issues with accuracy and comparison interregionally and between hospitals. An under or over-estimation of the KPI data from month to month could occur as well as the omission or double counting of a recordable event during registry review by the KPI data officers. Our study recommends stricter enforcement of DHIS2 reporting periods within hospital data collection systems including but not limited to emphasizing correct reporting periods during KPI training of hospital personnel and providing standardized registries and KPI reporting forms organized by the correct reporting period with clear cut off dates to enhance the data system.

Other limitations include the inability to quantitatively assess the level of data correction and manipulation by hospital staff following recording in the registry and prior to DHIS2 input. Concern over the consequences of reporting values too high or low for a given indicator likely compound this issue, generating issues with data reliability in turn Careful consideration of the procedures used to review registry data, the consequences associated with reporting poor metrics, and the use of this data to allocate government resources to hospitals would likely improve overall data quality.

The study team noted blank entries were often used to denote “0” findings within data forms, challenging the ability to assess data completeness versus a true lack of recordable events for the month. From the perspective of the KPI focal person, whose role it is to review and assimilate the primary source registry data into KPI monthly reporting forms, this limitation affects follow-up with data collectors regarding potential missed events. Our study team recommends universal guidelines for all hospital staff recording and reviewing KPI data, including indicating no recordable events with “0” rather than a blank entry to create a clearer report. Encouraging data ownership and responsibility by primary recorders i.e. Head OR nurse, Liaison officer, etc. would encourage frequent review of recorded data for clarity and serve as a countermeasure for this limitation.

Finally, this study was unable to include hospitals involved in previous data quality interventions in Tigray because of ongoing conflict and travel restrictions. This limits the comparability of this study to prior works by the study team. However, the inclusion of Amhara region hospitals within the current work and prior works improves the comprehensiveness of the findings. Ultimately, addressing data gaps in conflict-affected regions remains a significant challenge. While the DHIS2 database would allow for remote viewing of KPI aggregated data, the collection of this data is highly dependent on local stability, including the existence of permanent infrastructure and electronic access.

Recommendations

Our study recommends a regular, multi-level supervision mechanism focused on identifying and addressing data inconsistencies to ensure standardized data processes across all hospitals. Routine indicator quality inspections at each level of the system are essential to address the currently described issues.

At the national level, an emphasis on the importance of data quality while simultaneously ensuring no penalty for suboptimal outcomes is imperative. Incentives for consistent data quality (hospital recognition at the national level) as opposed to low complication rates would aid in the cultural transformation of this ideal. Additional proposed solutions include the integration of an electronic health system, which is currently in its early stages in the country at select hospitals, to have real-time data entry and minimize the opportunity for error. Reinvigorating the monitoring and evaluation arm of the SaLTS team is another potential avenue for improvement. While there are systems for evaluating national health system data quality, ultimately the emphasis on accuracy needs to extend to the regional and hospital level.

The study team similarly recommends at the regional level, the implementation of permanent survey teams to decrease the risk of errors and improve data quality and utilization with regular supervision and external audits. This system is currently in place in some regions, including Sidama, and initial feedback from this study has been used for this purpose. Having identified areas of data inconsistency, the Sidama regional office has worked with local hospital teams to improve the way in which these complications (specifically POMR) are recorded and reported. While the current system emphasizes data completeness on a monthly basis, quarterly audits of consistency are an additional requirement to enhance indicator capture. While these audits can be resource-intensive, the upfront investment is likely to lead to more sustainable, accurate health data which can ultimately improve the health system. Frequent training and incentives for quality officer retention would improve the consistency of this oversight.

At the hospital level, more consistent quality improvement programs to emphasize the relevance of these indicators can provide an opportunity to focus on surgical complications as areas for better teamwork, while increasing data capture as a byproduct. At one hospital, a department-wide emphasis on reducing surgical site infections, with large publicly visible posters denoting monthly progress helped to improve more accurate collection of this variable. Since the surgical team was motivated to improve this indicator, they were also keen to precisely measure monthly progress. Ultimately, hospital data quality officers will be the ones performing the most in-depth quality assessments of each department’s data on at least a monthly basis as required regionally and nationally. An additional opportunity is for more regular training sessions involving all participants in the data flow process at the hospital, not limited to data quality officers.