Skip to main content
Advertisement
  • Loading metrics

Quality and sustainability of Ethiopia’s national surgical indicators

  • Kayleigh R. Cook ,

    Contributed equally to this work with: Kayleigh R. Cook, Zebenay B. Zeleke

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Surgery, Medical College of Wisconsin, Milwaukee, Wisconsin, United States of America

  • Zebenay B. Zeleke ,

    Contributed equally to this work with: Kayleigh R. Cook, Zebenay B. Zeleke

    Roles Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing

    Affiliation Department of Surgery, Bahir Dar University, Bahir Dar, Amhara Region, Ethiopia

  • Ephrem Gebrehana,

    Roles Conceptualization, Data curation, Investigation, Methodology, Supervision, Writing – review & editing

    Affiliation Department of Surgery, Hawassa University, Hawassa, Sidama Region, Ethiopia

  • Daniel Burssa,

    Roles Conceptualization, Resources, Writing – review & editing

    Affiliation Ethiopian Ministry of Health, Addis Ababa, Ethiopia

  • Bantalem Yeshanew,

    Roles Data curation, Writing – review & editing

    Affiliation Ethiopian Ministry of Health, Addis Ababa, Ethiopia

  • Atkilt Michael,

    Roles Conceptualization, Data curation, Resources, Software, Writing – review & editing

    Affiliation Sidama Regional Health Bureau, Hawassa, Sidama Region, Ethiopia

  • Yoseph Tediso,

    Roles Data curation, Methodology, Resources, Writing – review & editing

    Affiliation Sidama Regional Health Bureau, Hawassa, Sidama Region, Ethiopia

  • Taylor Jaraczewski,

    Roles Writing – review & editing

    Affiliation Department of Surgery, Medical College of Wisconsin, Milwaukee, Wisconsin, United States of America

  • Chris Dodgion,

    Roles Resources, Supervision, Writing – review & editing

    Affiliation Department of Surgery, Medical College of Wisconsin, Milwaukee, Wisconsin, United States of America

  • Andualem Beyene ,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing

    ‡ AB and KRI also contributed equally to this work.

    Affiliation Department of Surgery, Addis Ababa University, Addis Ababa, Ethiopia

  • Katherine R. Iverson

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    katie.r.iverson@gmail.com

    ‡ AB and KRI also contributed equally to this work.

    Affiliation Department of Surgery, Medical College of Wisconsin, Milwaukee, Wisconsin, United States of America

Abstract

In 2015, the Ethiopian Federal Ministry of Health (FMOH) developed the Saving Lives through Safe Surgery (SaLTS) initiative to improve national surgical care. Previous work led to development and implementation of 15 surgical key performance indicators (KPIs) to standardize surgical data practices. The objective of this project is to investigate current practices of KPI data collection and assess quality to improve data management and strengthen surgical systems. The first portion of the study documented the surgical data collection process including methods, instruments, and effectiveness at 10 hospitals across 2 regions in Ethiopia. Secondly, data for KPIs of focus [1. Surgical Volume, 2. Perioperative Mortality Rate (POMR), 3. Adverse Anesthetic Outcome (AAO), 4. Surgical Site Infection (SSI), and 5. Safe Surgery Checklist (SSC) Utilization] were compared between registries, KPI reporting forms, and the DHIS2 (district health information system) electronic database for a 6-month period (January—June 2022). Quality was assessed based on data completeness and consistency. The data collection process involved hospital staff recording data elements in registries, quality officers calculating KPIs, completing monthly KPI reporting forms, and submitting data into DHIS2 for the national and regional health bureaus. Data quality verifications revealed discrepancies in consistency at all hospitals, ranging from 1–3 indicators. For all hospitals, average monthly surgical volume was 57 cases, POMR was 0.38% (13/3399), inpatient SSI rate was 0.79% (27/3399), AAO rate was 0.15% (5/3399), and mean SSC utilization monthly was 93% (100% median). Half of the hospitals had incomplete data within the registries, ranging from 2–5 indicators. AAO, SSC, and SSI were commonly missing data in registries. Non-standardized KPI reporting forms contributed significantly to the findings. Facilitators to quality data collection included continued use of registries from previous interventions and use of a separate logbook to document specific KPIs. Delayed rollout of these indicators in each region contributed to issues in data quality. Barriers involved variable indicator recording from different personnel, data collection tools that generate false positives (i.e. completeness of SSC defined as paper form filled out prior to patient discharge) or missing data because of reporting time period (i.e. monthly SSI may miss infections outside of one month), inadequate data elements in registries, and lack of standardized monthly KPI reporting forms. As the FMOH introduces new indicators and changes, we recommend continuous and consistent quality checks and data capacity building, including the use of routinely generated health information for quality improvement projects at the department level.

Introduction

Creating effective global surgery programs requires a clear understanding of the baseline state of surgical care. While the global surgery movement has gained momentum in the past several years, there remains a lack of quality data regarding the status of surgery in low-resource settings [1]. Our understanding of surgical capacity, resources, and outcomes has been limited by our methods of inquiry and the system in place for measuring results.

The quest for global surgery data was instigated by the Lancet Commission on Global Surgery [2], who proposed six core indicators designed to measure access to safe and affordable surgical care. A number of these were integrated into the World Bank’s World Development Indicators [3] and the World Health Organization’s Global Health Indicators [4]. Based on expert opinion in the Utstein consensus report, the global surgery indicators have since been uniformly defined and narrowed to five (surgical volume, perioperative mortality rate (POMR), surgical workforce, financial risk protection, and geospatial access) [5]. Information on these metrics have come from a variety of methodological approaches in the global community. Literature reviews, modeling studies, facility-based surveys, and regional and international surgical outcomes collaboratives are the diverse sources for these data [614]. While these studies have provided the foundation for global surgery indicator benchmarking, there is a growing need for sustainable and timely country-based systems with surgical metrics integrated into national health information systems [15, 16].

A monitoring and evaluation pillar was included in Ethiopia’s national surgical strategy (SaLTS or Saving Lives Through Safe Surgery) to gauge the current surgical practice in the country and create a sustainable method for tracking progress [17, 18]. The surgical key performance indicators (KPIs) are 15 metrics prospectively collected and reported regularly from each health facility in the country. In 2018, the indicators were piloted at 10 hospitals across two regions of Ethiopia as part of Safe Surgery 2020, an initiative to improve surgical care in the country [19]. Since 2018, the electronic platform DHIS2 has been used to report and aggregate these indicators. Informed by the evaluation of the first five-year surgical strategic plan (SaLTS) in 2020, the Ethiopian Federal Ministry of Health has now initiated the second stage of this strategy, SaLTS II, in 2021. The SaLTS II plan [20] outlines the approach to meet surgical metric goals by the year 2025, that include increased surgical volume and providers in line with international standards, POMR < 2%, and improved access to care within two-hours. To evaluate the sustainability of the SaLTS data system and ensure its utility in monitoring progress towards these goals, this study was created to appraise these key surgical indicators.

The aim of this study is to investigate the data management practices and assess the quality of surgical data in Ethiopia by exploring how surgical indicators are collected and reported at the hospital level since their national implementation.

Material and methods

Study design

The study was designed as a multi-institution, retrospective record review with hospital visits as the main source of program evaluation. A total of ten hospitals, split evenly between the Amhara and Sidama regions, were visited by the research team from July-August 2022. During hospital visits, surgical data practices were observed to assess the flow of information between hospital registries, hospital monthly key performance indicator (KPI) forms, and the District Health Information Software 2 (DHIS2) reports reviewed at the regional and national level. Data owners and instruments were identified at each step. The dates of record review ranged from 1/1/2022 through 6/31/2022. This time period was chosen to evaluate the most recent six-months of hospital data from the time of inquiry. All data accessed was deidentified and the study authors were unable to identify specific individuals from the data. The research team undertook a complete surgical data audit at each hospital visited. Registries within the OR, surgical ward, ICU, and maternity ward were reviewed by each reporting period and data points were aggregated to calculate monthly indicators. Researchers met with hospital quality improvement officers to audit monthly KPI reporting forms and final DHIS2 electronic database reports. At the Regional Health Bureaus, the final data reports from the queried hospitals visited were reviewed. At the national level, the DHIS2 conglomerate data from each Regional Health Bureau was evaluated. Data triangulation compared hospital to regional to national data.

Subjects

Primary and general public hospitals who were previously involved in Safe Surgery 2020 interventions, with approval from the Amhara and Sidama Regional Health Bureau and the Ministry of Health, were selected for this research [19, 21]. Amhara hospitals were further chosen based on their participation in the 2018 surgical data intervention carried out by the research team. Sidama hospitals were selected according to their involvement in Safe Surgery 2020 programming. Records of patients who underwent major surgery, defined as any procedure conducted in an operating room under general, spinal, or regional anesthesia [22], during their hospital stay within the study period were reviewed in this study.

Methods of measurements

Surgical Key Performance Indicators (KPIs) of interest were selected based on a prior surgical data intervention conducted at Amhara hospitals and on feedback from the Federal Ministry of Health (FMOH). The research team selected five indicators: Surgical Volume, Perioperative Mortality Rate (POMR), Rate of Safe Surgery Checklist Utilization (SSC), Surgical Site Infection Rate (SSI), and Anesthetic Adverse Outcome Rate (AAO) for data analysis. Formal definitions for each indicator were defined by the Saving Lives Through Safe Surgery (SALTS) program [22] and are included in Table 1. For collection of data from the registries, KPI reporting forms, and DHIS2 data, the investigators utilized Microsoft Excel forms at each hospital during on-site visits.

thumbnail
Table 1. Five surgical key performance indicators of focus and definitions [19].

https://doi.org/10.1371/journal.pgph.0002600.t001

Data analysis

Enumerated data from hospital registries were compared with the data in the national database. Analysis consisted of comparing data between sources (registry, KPI reporting forms, and DHIS2) and assessing data quality according to dimension 1 (data completeness) and dimension 2 (internal consistency of reported data) of the Data Quality Review (DQR) framework [22]. Completeness of data was evaluated by reviewing the monthly KPI reporting forms for the study period (Jan-June 2022) and documenting the number of months KPI data was absent for each indicator. For example, if the value for surgical volume was missing for one month at a single hospital, this was denoted as missing 1/6 of surgical volume data for the study period. Inconsistent data was defined as the number of months within the six-month study period where the registry source data did not match the DHIS2 electronic database. Following FMOH guidelines, surgical indicator data was deemed inconsistent if DHIS2 data differed by more than 10% of the registry data. This established verification factor for routine data quality assessment is calculated by the number of recorded events from the source document (registry) divided by the number of reported events in the DHIS2 report, with verification factor greater than 1.1 (under-reporting) or less than 0.9 (over-reporting) deemed inconsistent [23]. The KPI Safe Surgery Checklist utilization was an exception where KPI reporting forms were used as the source data and matched to DHIS2 data, given this indicator is not recorded in the registry.

Ethical approval

Institutional review board approval was obtained from Medical College of Wisconsin (MCW) and the Ethiopian Public Health Institute (EPHI) for all study activities. EPHI approved the following protocol number: EPHI-IRB-454-2022, titled “Assessment of Current Data Practices in Ethiopia.” The MCW Institutional Review Board approved the study under PRO ID: PRO00043474. The study investigators also received a letter of support from the Federal Ministry of Health.

Inclusivity in global research

Additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the S1 Checklist.

Results

Hospital characteristics

Table 2 includes hospital specific data on facility resources and characteristics for the ten study hospitals. Median catchment population was 375,000 patients (70,000–1,800,000) whereas median total hospital beds was 45 (25–145). Integrated Emergency Surgical Officers (IESOs), Master’s-level health professionals intended to provide emergency and essential surgery in Ethiopia, were the predominant surgical provider at most institutions [24].

Data collection processes

At all ten hospitals, data flow mechanisms were observed and compiled into a generalized process illustrated by Fig 1. The data collection process involved recording data elements in registries by hospital staff, then calculation of KPIs by quality improvement (QI) officers, completing monthly KPI reporting forms, and submitting data into the electronic national data system (DHIS2). For the five KPIs evaluated, healthcare providers entered surgical data into registries specific to their department, i.e. wards, ICU, or operating room. At the end of the reporting period, the QI officers reviewed the registries and utilized the raw data to calculate key performance indicators. If they noted any discrepancies or missing data, they would query the health provider responsible for that element. KPI data collection forms for this purpose varied between hospitals. The KPI data is then entered into the electronic DHIS2 and uploaded to the Regional Health Bureau (RHB) for approval. KPI data at the RHB is aggregated from multiple hospitals together to describe the surgical practice in the region and submitted to the Federal Ministry of Health.

thumbnail
Fig 1. Surgical indicator data flow.

Responsible data owner and reporting instrument at each step of the data collection and reporting process. KPI = Key Performance Indicators, DHIS2 = District Health Information Software 2.

https://doi.org/10.1371/journal.pgph.0002600.g001

Surgical key performance indicators

From registry data collected independently, values for the five KPIs of interest were calculated for each month from January to June 2022 (Tir to Sene 2014 in the Ethiopian Calendar). For all ten hospitals over six-months, average monthly surgical volume was 57 cases and SSC utilization was 93% (Table 3). Over one month, the mean surgical volumes were 54 and 59 cases per hospital in Amhara and Sidama, respectively. In Amhara, the majority of hospitals reflected the regional average. In Sidama, three primary hospitals performed less than 15 cases each on average per month, while the two general hospitals each performed between 55 and 214 operations on average per month. This extrapolates to a predicted rate of 50 to 712 operations per 100,000 population annually at primary hospitals in Amhara, 30 to 67 per 100,000 in Sidama primary hospitals, and 80 to 147 per 100,000 in Sidama general hospitals according to their determined catchment populations. When calculated using a standardized catchment population of 1 million for primary and general hospitals, there is a predictive rate of 35 to 121 operations per 100,000 population annually at primary hospitals in Amhara, 4.8 to 12 per 100,000 in Sidama primary hospitals, and 64 to 264 per 100,000 in Sidama general hospitals. Across all ten hospitals over the 6-month period there were thirteen post-operative mortality events, for a collective POMR of 0.38% (13/3399). For the same period, twenty-seven surgical site infections were reported for an overall inpatient SSI rate of 0.79% (27/3399). Five anesthetic adverse outcomes were documented for a collective AAO rate of 0.15% (5/3399).

thumbnail
Table 3. Comparison of registries to DHIS2 data for surgical KPIs.

https://doi.org/10.1371/journal.pgph.0002600.t003

Data quality

Data quality was assessed based on two metrics: completeness and consistency. Our study found data completeness to be an average of 72% for the 10 hospitals, and 70% of hospitals had consistent data for the five surgical KPIs of interest. Verification factor calculations between registry and DHIS2 data, as seen in Fig 2, revealed inconsistencies (>10% difference) at nine total hospitals, commonly SSI (6/10) and surgical volume (5/10). Five hospitals demonstrated incomplete registry data, with the most commonly incomplete indicator data for AAO, SSC, and SSI. Across all five KPIs of interest, surgical volume was the most likely to be recorded within the registry and have inconsistencies between values recorded in the OR registry and DHIS2 (Table 3). Of note, Sidama hospitals frequently had more inconsistent surgical volume data than Amhara hospitals.

thumbnail
Fig 2. Data quality assessment of five surgical indicators.

Number of hospitals out of 10 with inconsistent (blue) and incomplete (orange) data for each KPI of focus for the 6-month period. SSC = Surgical Safety Checklist utilization, SSI = Surgical Site Infection, POMR = Perioperative Mortality Rate, AAO = Anesthesia Adverse Outcome.

https://doi.org/10.1371/journal.pgph.0002600.g002

After further investigation of inconsistencies between registry data and DHIS2 data, national data is revealing underreported rates (verification factor >1.1) for major surgical complications: SSI, AAO, and POMR. For POMR, 11 of the 13 events documented for all ten hospitals across 6-months were recorded at Hospital 3. However, this value does not correspond to the DHIS2 data which reported zero POMR events for Hospital 3 across the 6-months (Table 3). Furthermore, 1 POMR event observed at Hospital 9 for the reporting period did not transfer to the DHIS2 database but was present in the KPI reporting form. All of the hospitals who recorded any post-operative deaths (3 hospitals in total), had at least one month with inconsistent POMR data between registries and DHIS2. SSI rates were similarly underreported, with over half of the registry-documented surgical site infections at all ten hospitals over six months missing from the overall SSIs within DHIS2 (27/3399 vs 13/3399). SSI rates were inconsistent for seven out of ten hospitals for the evaluated period. For AAO, only Hospital 7 recorded 5 AAO events within their registry for the six-month period, but zero AAO events were present within DHIS2 for all ten hospitals over the study period. SSC data was mostly consistent between KPI forms and DHIS2 data, with the exception of Hospital 9 where median SSC differed by 12.5% and Hospital 3 where the range of SSC values differed by 36% but the median remained the same.

Discussion

This study evaluated the process and quality of current surgical data practices in Ethiopia through five key performance indicators (KPIs) at ten hospitals in two regions over a 6-month period. The evaluation occurs five years after the creation and implementation of the surgical KPI data system in conjunction with Ethiopia’s national strategic plan (SaLTS) to improve surgical and anesthesia care.

Principle findings include a largely uniform mechanism for data collection and reporting between facilities from hospital registries, to data collection forms, to the DHIS2 electronic database. However, the lack of standardization of data collection forms and staff practices between each hospital may be a source of discrepancies in comparing outcomes from registries to DHIS2 data. While standardized registries have improved the collection of data in the setting of their origination, the development and implementation of standardized monthly KPI reporting forms across hospitals would improve consistency in recording and reporting. This will require a collaborative effort between policymakers, healthcare stakeholders, hospital employees, and non-governmental organizations.

Data quality was assessed based on two metrics: completeness and consistency which were found to be above 70% for the five KPIs at all 10 hospitals. Our study found data completeness was 72% on average for the 10 hospitals, and 70% of hospitals had consistent data for the five surgical KPIs of interest. Data quality varied based on the specific indicator. Surgical volume was largely complete throughout the data process, however at least 3 out of 10 hospitals had incomplete data for SSI, SSC, and AAO. Difficulties defining and tracking SSI and AAO perioperative complications are likely due to observed variability in recording forms and registries, reporting mechanisms across data flow, andwhich hospital provider is responsible for data collection.

Most inconsistencies are found for SSIs and surgical volume, with at least half of hospitals having inconsistent data for these two indicators. Some hospitals were not using standard registries, which could lead to missed data elements. For example, SSIs were highest in hospital 6 (n = 14), but only a little over half of these events made it into DHIS2 (n = 8). The initial data intervention included a SSI registry as part of the suite of tools for collecting and recording the indicators. Some hospitals had this SSI logbook to better capture this data, but there was variability in use of this system as opposed to using the hospital ward registries for recording infections.

The lack of or non-standardized data collection tools to transfer indicator components from the registries to DHIS2 also created a major gap in the link between these metrics. For example, Hospital 3 had the highest POMR (11 deaths) according to the registry data, but this was not accounted for in DHIS2 (0 deaths). We observed difficulty in accounting for surgical deaths from multiple different departments (ICU, multiple wards, and OR) which could explain this disparity, as deaths from the ICU were left out of the data process for POMR at this hospital. It also demonstrates the need for specifically trained personnel for data collection and entry, as well as routine data quality checks. Apart from the different areas of data management shortcomings due to lack of appropriate tools, there could be a high turnover of the already trained quality improvement officers. There should also be a uniform understanding of the primary data owners and the QI officers.

There were overall relatively low rates of complications such as SSI (0.79%), AAO (0.15%), and POMR (0.38%). It is possible that the data does not capture the full reality as outcomes are limited to inpatient events. Additionally, many of the hospitals, especially primary hospitals in Sidama, had lower surgical volume and cases were limited to cesarean-sections, which likely contributes to the lack of significant complications in these settings. An accurate assessment of the surgical system is dependent on these indicators as metrics of service delivery and quality of care. As the surgical system in Ethiopia continues to grow, it is imperative the quality of data is a true reflection of hospital operations. Given these rates of surgical complications may be lower than expected, a more accurate system for these indicators will allow for tracking improvements in access to and quality of surgical care.

Surgical site infection

For SSI, our study estimated a value of 0.79% (27/3399) across 10 hospitals, lower than the 2% value recorded by the Network for Perioperative and Critical Care (N4PCC) 2021 registry [25] and the 10.2% described in the African Surgical Outcomes Study [11]. Specifically for SSI, patients are often discharged before they will develop signs and symptoms of infection. There is currently no uniform mechanism to enumerate outpatient SSIs.

Anesthesia adverse outcome

Similarly for AAO, our dataset of 3399 surgeries yielded only 5 AAO events for a rate of 0.15%. This is considerably lower than the N4PCC 2021 registry value of 3.1% [25] and closer to the AAO reported value of 0.94% in Zimbabwe [26]. The use of a multicenter cloud-based registry, in-hospital coordinators and data trainers, and utilization of an offline mobile application for data collection most likely improved reporting compliance and identification of adverse events, leading to increased AAO capture rates in the N4PCC study [25]. Variation in AAO rates reported can also be attributed to variations in KPI definitions, as in Zimbabwe where an AAO is defined as a critical incident and includes nausea or post-operative pain [26]. Furthermore for AAO, it may be difficult to detect and diagnose complications such as high spinal anesthesia. This was also the indicator with the greatest variability in recording mechanisms and data ownership, likely accounting for the very low AAO, with only one hospital reporting these events. Additional training for knowledge and skills may be needed in this case.

Peri-operative mortality rate

Our study found a lower POMR at 0.38% (13/3399) than the 0.9% (9/1000) found in the N4PCC 2021 registry surveyed at 4 Ethiopian hospitals [25]. This disparity may be due to the indicators being limited to only inpatient events, thus POMR may be lower than expected as deaths for KPIs are registered only before discharge. There were also some deaths recorded in registries, which were not reported in the national data as mentioned previously. Further external mortality audits for these complications including chart reviews and periodical follow-up surveys, as well as a robust death registry, are recommended to understand the true state of surgical outcomes in Ethiopian hospitals.

Surgical volume

Calculated annual surgical volume for Sidama general hospitals is 80 to 147 operations per 100,000, 50 to 712 operations per 100,000 population at primary hospitals in Amhara, and 30 to 67 per 100,000 in Sidama primary hospitals. In our dataset, surgical volume at all ten hospitals has not yet reached the LCoGS 2030 target of 5000 surgeries per 100,000 population annually [2]. Study findings of surgical volume for Sidama general hospitals (Jan- June 2022) are consistent with Meshesha et al.’s surgical volume of 289 per 100,000 over a 90-day interval (Sept 2020—May 2021) for general hospitals [27]. Our study calculates higher annual surgical volumes for Amhara and Sidama primary hospitals than the 37 per 100,000 found for primary health care units over a 90-day interval. The Amhara primary hospitals showed an overall increase in their surgical volume compared to the previous study during the data intervention in 2018 [28], which may be due to both growth in surgical capacity and population needs.

Surgical safety checklist

Estimated SSC values were 93% for 3399 surgeries, consistent with the 92.1% found in the N4PCC registry for 1595 surgical cases [25] and higher than the 67.6% rate reported by Sibhatu et. al. 2022 [29]. Our study determined 60% of audited hospitals had complete SSC data, consistent with the 60.8% of 659 checklists the latter study found to be filled completely and correctly. The barriers to data collection identified at ten hospitals within the Amhara and Sidama region are consistent with the findings of previous investigations into data quality within the Ethiopian healthcare system. The overreporting of certain surgical KPIs, namely SSC and underreporting metrics of illness or mortality was concurrent with findings from Mekebo et. al. and other studies [28, 30, 31]. Our observation of data flow processes demonstrated that SSC values are calculated by QI officers by selecting a sample of ten patient charts for review. At certain institutions, stop-gap mechanisms were implemented to prevent the progression of patient charts to the Inpatient Admission office (IPAD) until all forms including the SSC were completed. Such processes might explain the disparity Sibhatu et. al found between the DHIS2 SSC value of 81% and the externally audited 60.8% retrospectively calculated from patient charts [27]. High rates of surgical data inconsistency were a major result of this research and are likely due to lack of consistent KPI reporting forms, and the considerable burden of aggregating substantive data elements into complex indicators as discussed previously by Adane et al 2021 [32]. Our observations were in alignment with other challenges with data collection in an Ethiopian setting previously discussed including turnover of hospital staff, poor understanding of data processes, modification, or manipulation of data to compensate for the lack of data, and technological issues with DHIS2 [28, 30].

Limitations

National implications of this study are limited by the small sample size (ten hospitals) and limited study period (six months). Hospital selection was also limited to facilities with prior involvement in Safe Surgery 2020. This limitation could potentially impact the generalizability of these findings to the current surgical landscape in Ethiopia, as system changes are ongoing and interregional differences are likely. The investigators included two regions within Ethiopia as well as both primary and general hospitals as countermeasures to the limited sample size and period. Selecting hospitals involved in the prior surgical data strengthening program (in Amhara) and those not (in Sidama) was also employed to expose potential differences in data practices. Follow-up investigation using qualitative methods, specifically surveying data collectors and data managers at participating hospitals and regions, will provide further context around the findings. Future studies are also planned to expand this initial investigation of key performance indicator data in Ethiopia to a larger number of hospitals and regions.

Additionally, the use of registry data as primary source data generates the risk that some surgical indicators not recorded in the registry were left out of our analysis. Using registry data rather than performing a retrospective review of surgical patient charts could underestimate issues with data quality. Mortality audits, or complication audits, are another potential retrospective mechanism to capture more complete data for these outcomes, requiring access to patient charts [33, 34] Future studies incorporating alternative mechanisms, such as those described, will contribute important information on the accuracy and reliability of registry data recorded during the patient’s hospital course.

Another source of data quality inconsistency could be attributed to reporting period variability between hospitals. This study utilized DHIS2 reporting periods for data auditing. The implications of variable reporting periods are issues with accuracy and comparison interregionally and between hospitals. An under or over-estimation of the KPI data from month to month could occur as well as the omission or double counting of a recordable event during registry review by the KPI data officers. Our study recommends stricter enforcement of DHIS2 reporting periods within hospital data collection systems including but not limited to emphasizing correct reporting periods during KPI training of hospital personnel and providing standardized registries and KPI reporting forms organized by the correct reporting period with clear cut off dates to enhance the data system.

Other limitations include the inability to quantitatively assess the level of data correction and manipulation by hospital staff following recording in the registry and prior to DHIS2 input. Concern over the consequences of reporting values too high or low for a given indicator likely compound this issue, generating issues with data reliability in turn Careful consideration of the procedures used to review registry data, the consequences associated with reporting poor metrics, and the use of this data to allocate government resources to hospitals would likely improve overall data quality.

The study team noted blank entries were often used to denote “0” findings within data forms, challenging the ability to assess data completeness versus a true lack of recordable events for the month. From the perspective of the KPI focal person, whose role it is to review and assimilate the primary source registry data into KPI monthly reporting forms, this limitation affects follow-up with data collectors regarding potential missed events. Our study team recommends universal guidelines for all hospital staff recording and reviewing KPI data, including indicating no recordable events with “0” rather than a blank entry to create a clearer report. Encouraging data ownership and responsibility by primary recorders i.e. Head OR nurse, Liaison officer, etc. would encourage frequent review of recorded data for clarity and serve as a countermeasure for this limitation.

Finally, this study was unable to include hospitals involved in previous data quality interventions in Tigray because of ongoing conflict and travel restrictions. This limits the comparability of this study to prior works by the study team. However, the inclusion of Amhara region hospitals within the current work and prior works improves the comprehensiveness of the findings. Ultimately, addressing data gaps in conflict-affected regions remains a significant challenge. While the DHIS2 database would allow for remote viewing of KPI aggregated data, the collection of this data is highly dependent on local stability, including the existence of permanent infrastructure and electronic access.

Recommendations

Our study recommends a regular, multi-level supervision mechanism focused on identifying and addressing data inconsistencies to ensure standardized data processes across all hospitals. Routine indicator quality inspections at each level of the system are essential to address the currently described issues.

At the national level, an emphasis on the importance of data quality while simultaneously ensuring no penalty for suboptimal outcomes is imperative. Incentives for consistent data quality (hospital recognition at the national level) as opposed to low complication rates would aid in the cultural transformation of this ideal. Additional proposed solutions include the integration of an electronic health system, which is currently in its early stages in the country at select hospitals, to have real-time data entry and minimize the opportunity for error. Reinvigorating the monitoring and evaluation arm of the SaLTS team is another potential avenue for improvement. While there are systems for evaluating national health system data quality, ultimately the emphasis on accuracy needs to extend to the regional and hospital level.

The study team similarly recommends at the regional level, the implementation of permanent survey teams to decrease the risk of errors and improve data quality and utilization with regular supervision and external audits. This system is currently in place in some regions, including Sidama, and initial feedback from this study has been used for this purpose. Having identified areas of data inconsistency, the Sidama regional office has worked with local hospital teams to improve the way in which these complications (specifically POMR) are recorded and reported. While the current system emphasizes data completeness on a monthly basis, quarterly audits of consistency are an additional requirement to enhance indicator capture. While these audits can be resource-intensive, the upfront investment is likely to lead to more sustainable, accurate health data which can ultimately improve the health system. Frequent training and incentives for quality officer retention would improve the consistency of this oversight.

At the hospital level, more consistent quality improvement programs to emphasize the relevance of these indicators can provide an opportunity to focus on surgical complications as areas for better teamwork, while increasing data capture as a byproduct. At one hospital, a department-wide emphasis on reducing surgical site infections, with large publicly visible posters denoting monthly progress helped to improve more accurate collection of this variable. Since the surgical team was motivated to improve this indicator, they were also keen to precisely measure monthly progress. Ultimately, hospital data quality officers will be the ones performing the most in-depth quality assessments of each department’s data on at least a monthly basis as required regionally and nationally. An additional opportunity is for more regular training sessions involving all participants in the data flow process at the hospital, not limited to data quality officers.

Conclusion

This study demonstrates the ongoing challenges to accurate surgical data collection in a resource-limited setting. Scarce human resources, limited time for data collection, reporting, and auditing, interrupted internet connectivity, and a high volume of patient care needs are all threats to surgical data quality. Five years after the implementation of Ethiopia’s surgical indicators as part of their national surgical and anesthesia strategic plan, there remains difficulty with capturing the true value of surgical complications such as infections and perioperative mortality. This national system for data collection and reporting was a momentous step forward in understanding the current state of surgery in Ethiopia and tracking progress towards measurable goals in improving surgical capacity and quality. However, more unified mechanisms for data transfer, ongoing training for quality officers and health professionals, as well as regular audits for data consistency are needed to improve the current structure. Capturing outpatient surgical outcomes as well as the development of an electronic registry, which is currently in development, will lead to improvements in monitoring and improving surgical capacity and quality in Ethiopia while addressing several of the aforementioned challenges in this setting [21, 25].

Supporting information

Acknowledgments

We thank the Ethiopian Federal Ministry of Health, Sidama Regional Health Bureau, Amhara Regional Health Bureau, and MCW Office of Global Health for their support and collaboration.

References

  1. 1. Global Surgery and Anaesthesia Statistics: The Importance of Data Collection. 2018 [cited 29 September 2021]. In: German Global Surgery Association, Harvard Medical School Program in Global Surgery and Social Change [Internet]. The G4 Alliance -. Available from: https://gallery.mailchimp.com/dd9db680c07f4e88485196260/files/01801d8f-f374-4b14-9404-fcd532ca06a9/Surgical_AnaesthesiaDataReport.pdf/.
  2. 2. Meara JG, Leather AJM, Hagander L, Alkire BC, Alonso N, Ameh EA, et al. Global Surgery 2030: evidence and solutions for achieving health, welfare, and economic development. International Journal of Obstetric Anesthesia. 2016 Feb; 25:75–8.
  3. 3. The World Bank; 2019 [cited 29 September 2021]. Database: World Development Indicators [Internet]. Available from: http://data.worldbank.org
  4. 4. World Health Organization (WHO). Global Reference List of 100 Core Health Indicators (plus health-related SDGs). 2018 [cited April 5 2023]. In: WHO Reportd [Internet]. Geneva: WHO. Available from: https://apps.who.int/iris/handle/10665/25995/.
  5. 5. Davies J, Gelb AW, Gore-Booth J, Martin J, Jannicke Mellin-Olsen Åkerman C, et al. Global surgery, obstetric, and anaesthesia indicator definitions and reporting: An Utstein consensus report. PLOS Medicine. 2021 Aug 20;18(8): e1003749–9. pmid:34415914
  6. 6. Juran S, Gruendl M, Marks IH, Broer PN, Guzmán JM, Davies J, et al. The need to collect, aggregate, and analyze global anesthesia and surgery data. Can J Anesth. 2019 Feb 1;66(2):218–29.
  7. 7. Holmer H, Bekele A, Hagander L, Harrison EM, Kamali P, Ng-Kamstra JS, et al. Evaluating the collection, comparability and findings of six global surgery indicators. British Journal of Surgery. 2018 Dec 20;106(2): e138–50. pmid:30570764
  8. 8. Marks IH, Kamali P, Khan MA, Tahboub H, Hill JC, Zaim C, et al. Data for the sustainable development of surgical systems: a global collaboration. WDI surgical indicators data collection. Program in Global Surgery and Social Change. Harvard Medical School. 2016 [Cited 2022 January 5]. Available from: https://www.pgssc.org/_files/ugd/d9a674_967bde5252664b709f19e691bf6732fd.pdf
  9. 9. Fitzgerald JEF, Khatri C, Glasbey JC, Mohan M, Lilford R, Harrison EM, et al. GlobalSurg Collaborative. Mortality of emergency abdominal surgery in high-, middle- and low-income countries [published correction appears in Br J Surg. 2017 Apr;104(5):632]. Br J Surg. 2016;103(8):971–988. pmid:27145169
  10. 10. Bhangu A, Ademuyiwa AO, Aguilera ML, Alexander P, Al-Saqqa SW, Borda-Luque G, et al. GlobalSurg Collective. Surgical site infection after gastrointestinal surgery in high-income, middle-income, and low-income countries: a prospective, international, multicentre cohort study. The Lancet Infectious Diseases. 2018 May;18(5):516–25.
  11. 11. Biccard BM, Madiba TE, Kluyts HL, Munlemvo DM, Madzimbamuto FD, Basenero A, et al. Perioperative patient outcomes in the African Surgical Outcomes Study: a 7-day prospective observational cohort study. The Lancet. 2018 Apr;391(10130):1589–98. pmid:29306587
  12. 12. Guest GD, McLeod E, Perry WRG, Tangi V, Pedro J, Ponifasio P, et al. Collecting data for global surgical indicators: a collaborative approach in the Pacific Region. BMJ Global Health. 2017 Nov;2(4): e000376. pmid:29225948
  13. 13. Peck G, Roa L, Barthlemy EJ, South S, Foianini JE, Ferreira RV, et al. Improving global emergency and essential surgical care in Latin America and the Caribbean: A collaborative approach. Bull Am Coll Surg. 2019;104(3):24–39.
  14. 14. Zadey S, Iyer H, Nayan A, Shetty R, Sonal S, Smith ER, et al. Evaluating the status of the Lancet Commission on Global Surgery indicators for India. The Lancet Regional Health—Southeast Asia [Internet]. 2023 Jun 1. 13:100178. pmid:37383563
  15. 15. National Surgical, Obstetric and Anaesthesia Planning Manual. United Nations Institute for Training and Research (UNITAR). 2020 [Cited 2023 May 15]
  16. 16. Sonderman KA, Citron I, Mukhopadhyay S, Albutt K, Taylor K, Jumbam D, et al. Framework for developing a national surgical, obstetric and anaesthesia plan. BJS Open. 2019 Jul 24;3(5):722–32. pmid:31592517
  17. 17. Burssa D, Teshome A, Iverson K, Ahearn O, Ashengo T, Barash D, et al. Safe Surgery for All: Early Lessons from Implementing a National Government-Driven Surgical Plan in Ethiopia. World Journal of Surgery. 2017 Oct 13;41(12):3038–45. pmid:29030677
  18. 18. Iverson KR, Ahearn O, Citron I, Garringer K, Mukhodpadhyay S, Teshome A, et al. Development of a surgical assessment tool for national policy monitoring & evaluation in Ethiopia: A quality improvement study. International Journal of Surgery. 2020 Aug; 80:231–40.
  19. 19. Iverson KR, Drown L, Bari S, Abate M, Burssa D, Garringer K, et al. Surgical key performance indicators in Ethiopia’s national health information system: Answering the call for global surgery data. East Cent Afr J Surg. 2021;26(3):92–103. Table 1, SaLTS key performance indicators; p. 94.
  20. 20. National Surgical Care Strategic Plan: Saving Lives Through Safe Surgery II (SaLTS II, 2021–2025). Ministry of Health, Health Service Quality Directorate (HSQD)(Ethiopia). 2021 [Cited 2022 January 5]. Available from: http://repository.iifphc.org/bitstream/handle/123456789/1685/National-Surgical-Care-Strategic-Plan.pdf?sequence=1&isAllowed=y
  21. 21. Belachew FK, Kifleyohanes T, Moore J, Teshome A, Biccard B. Indications, Challenges, and Characteristics of Successful Implementation of Perioperative Registries in Low Resource Settings: A Systematic Review. World Journal of Surgery. 2023 Jan 19;47(6):1387–96. pmid:36656359
  22. 22. Data quality review: module 1: framework and metrics. World Health Organization. 2017 [cited 2022 Aug 11]. Available from: https://apps.who.int/iris/handle/10665/259224
  23. 23. Routine Data Quality Assessment Tool: User Manual. Measure Evaluation. 2017 [cited 2023 Sep 27]. Available from: https://www.measureevaluation.org/resources/publications/ms-17-117/at_download/document
  24. 24. Integrated Emergency Surgical Officers | MINISTRY OF HEALTH—Ethiopia [Internet]. www.moh.gov.et. [cited 2023 Jan 19]. Available from: https://www.moh.gov.et/site/initiatives-4-col/Integrated-Emergency-Surgical-Officers
  25. 25. Network for Peri-operative Critical care (N4PCc). Addressing priorities for surgical research in Africa: implementation of a multicentre cloud-based peri-operative registry in Ethiopia. Anaesthesia. 2021;76(7):933–939. pmid:33492690
  26. 26. Madzimbamuto FD, Chiware R. A critical incident reporting system in anaesthesia. Cent Afr J Med. 2001;47(11–12):243–247. pmid:12808775
  27. 27. Meshesha BR, Sibhatu MK, Beshir HM, Zewude WC, Taye DB, Getachew EM, et al. Access to surgical care in Ethiopia: a cross-sectional retrospective data review. BMC Health Services Research. 2022 Jul 30;22(1).
  28. 28. Mekebo M, Gobena T, Hawulte B, Tamiru D, Debella A, Yadeta E, et al. Level of implementation of district health information system 2 at public health facilities in Eastern Ethiopia. Digital health. 2022 Jan 1; 8:205520762211311–205520762211311. pmid:36249476
  29. 29. Sibhatu MK, Taye DB, Gebreegziabher SB, Mesfin E, Bashir HM, Varallo J. Compliance with the World Health Organization’s surgical safety checklist and related postoperative outcomes: a nationwide survey among 172 health facilities in Ethiopia. Patient Safety in Surgery. 2022 Jun 10;16(1). pmid:35689263
  30. 30. Adane A, Adege TM, Ahmed MM, Anteneh HA, Ayalew ES, Berhanu D, et al. Exploring data quality and use of the routine health information system in Ethiopia: a mixed-methods study. BMJ Open. 2021;11(12): e050356. Published 2021 Dec 23. pmid:34949613
  31. 31. Endriyas M, Alano A, Mekonnen E, Ayele S, Kelaye T, Shiferaw M, et al. Understanding performance data: health management information system data accuracy in Southern Nations Nationalities and People’s Region, Ethiopia. BMC Health Services Research. 2019 Mar 18;19(1). pmid:30885204
  32. 32. Adane A, Adege TM, Ahmed MM, Anteneh HA, Ayalew ES, Berhanu D, et al. Routine health management information system data in Ethiopia: consistency, trends, and challenges. Glob Health Action. 2021 Jan 1;14(1):1868961. pmid:33446081; PMCID: PMC7833046.
  33. 33. Ugwu-Olisa OA, Chidi EV, Emmanuel EU, Chinedu NG, Onyeyirichi O. Mortality audit in general surgery unit and lessons learned at a Nigerian tertiary hospital: a single centre observational study. Pan African Medical Journal. 2022;41.
  34. 34. Bindroo S, Saraf R. Surgical Mortality Audit-lessons Learned in a Developing Nation. Int Surg. 2015 Jun;100(6):1026–32. pmid:26414825