Background

Surgical site infections (SSIs) are associated with significant increases in morbidity, mortality, and hospital costs. One study found that SSIs at least doubled patients’ mortality risk and cost somewhere between $3.5 billion and $10 billion every year [1]. Because of its severe health and financial consequences, improving surgical care quality is one of the top priorities for various stakeholders in healthcare.

One strategy is public, and sometimes mandatory, reporting of SSI rates. At the state level, some local health authorities have started requiring hospitals to publicly report SSI rates for specific surgical procedures since 2004 [2]. At the national level, Hospital Care Compare, published by the Centers for Medicare & Medicaid (CMS), provides information on hospitals’ quality of care, including SSIs for abdominal hysterectomy and colon procedures.

Another strategy is using financial incentives [3]. The CMS initiated various pay-for-performance (P4P) programs by rewarding or penalizing hospitals based on their quality-of-care performance. Some of the widely publicized programs include the Hospital Value-Based Purchasing Program (HVBPP), Hospital-Acquired Condition Reduction Program (HACRP), and Hospital Readmissions Reduction Program (HRRP).

HACRP, the focus of this study, was launched by CMS in October 2014. Hospitals in the worst-performing quartile for targeted hospital-acquired conditions (HACs), based on Medicare fee-for-service (FFS) claims and chart-abstracted data, receive a 1% reduction in CMS payments for Medicare patients [4]. Since its start, a quarter of hospitals have been penalized, and the total penalties have been over $300 million annually [5]. These penalties could be sizeable for some financially struggling hospitals.

The HACs in the program include two domains. Domain one covers CMS Patient Safety Indicator 90, or PSI 90, based on Medicare claims data. Domain two in HACRP covers central line-associated bloodstream infection (CLABSI), catheter-associated urinary tract infection (CAUTI), SSIs for abdominal hysterectomy and colon procedures, methicillin-resistant Staphylococcus aureus (MRSA) bacteremia, and Clostridium difficile Infection (CDI). Rather than claim or other clinical documentation, domain two is from chart-abstracted surveillance data for all patients from the Centers for Disease Control and Prevention’s National Healthcare Safety Network (NHSN).

Previous studies on HACRP have not reached a consensus regarding the program’s effects. Although some studies [6–9] found reductions in overall HACs, others have yielded mixed results, generally pointing to limited impacts and sometimes counterproductive outcomes from HACRP [4,5,10–16]. This study examines two targeted conditions under HACRP: SSIs for abdominal hysterectomy and colon procedures.

Methods

Results

Table 1 lists the summary statistics for all observations and patient populations by procedures based on their primary (i.e., first) diagnoses. For categorical variables, the number of observations and proportions are reported. For patient age and Elixhauser mortality score, the mean and standard deviation are reported.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary Statistics for All Observations and Patient Populations by Procedures.

https://doi.org/10.1371/journal.pone.0337072.t001

Since this study focuses on Medicare patients, the average age is over 70 years. The majority of patients are non-Hispanic whites admitted to medium or large nonprofit hospitals in urban areas. Admission years and quarters are about evenly distributed. There are similar patterns for summary statistics by procedures.

Fig 1. presents the number of SSI cases per 1,000 discharges by procedures and admission year and quarter. The corresponding numbers and the percentage of total discharges that underwent the corresponding procedures are provided in the supporting information, S2 Table. The SSI rates for the targeted procedures and control procedures exhibit fluctuations rather than consistent increases or decreases. S3 Table lists the procedure rates and SSI rates when alternative procedures and SSI definitions are used.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. SSI per 1,000 discharges by procedures, admission year and quarter.

https://doi.org/10.1371/journal.pone.0337072.g001

Table 2 lists the DID estimates. Reported coefficients are changes per 1,000 discharges. Standard errors, confidence intervals, and the number of observations in each regression are also reported. The cut-off time in the DID models is October 2014, when the financial penalties began. The table only reported the DID estimates, i.e., the interaction term between the post-policy dummy variable and the dummy variable for the treatment procedure. Full regression results are available upon request. All models control for patient and hospital characteristics and time trends. Sample weights and robust standard errors are estimated to generate nationally representative results.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. DID estimates by comparing treatment procedures with control procedures.

https://doi.org/10.1371/journal.pone.0337072.t002

The top half of the table compares abdominal hysterectomy with each of the four control procedures (cardiac implantable electronic device, laparoscopic cholecystectomy and appendectomy, orthopedic procedures, and all other procedures); the bottom half compares colon surgeries with the four control procedures. The procedures in each comparison are defined by the primary (i.e., the first) ICD-9 procedure codes in each discharge.

Estimates in Table 2 are all insignificant. The DID estimates are largely insignificant in S4 Table when the alternative procedure and SSI identifications are used. The other two cut-off times, January 2013 and August 2013, also have similar results.

Results from additional sensitivity analysis are provided in the supporting information. S1 Fig. presents the number of SSIs per 1,000 discharges, stratified by procedure and admission year/quarter, during the sample period 2012–2017. S5 and S6 Tables report the DID estimates after adding two years of data following the ICD-9 to ICD-10 transition, and the DID estimates remain insignificant. In the results from ITS models (S7 and S8 Tables), neither the HACRP indicator nor its interaction with the trend variable is statistically significant.

Discussions and conclusions

Prior research on HACRP has yielded mixed evidence regarding the effects of this or similar programs [4–16]. These studies focus on two areas of HACRP: (1) the design and methodology of the program and (2) the impact evaluation of the program. In the first area, several studies found a disconnection between the penalty levied by CMS’ HACRP and the hospital’s quality of care [5,11–13]. Penalties were more likely to be imposed on large teaching hospitals that treated a large proportion of low-income patients [5,10,12]. For example, one study found that the penalty imposed by CMS was primarily driven by the recalibration of PSIs rather than changes in hospitals’ HAC performance [5]. This disconnection may lower hospitals’ incentives and ability to direct resources to reduce SSIs and other HACs. It was found that the risk adjustment method in the HACRP program was inferior to other models [11]. It thus led to the incorrect identification of outlier hospitals and unfairly punished hospitals not based on actual SSI performance [11].

In the second area, i.e., impact evaluation, one study used regression discontinuity design to examine changes in domain one of the HACRP from July 2014 to November 2016 [10]. Another study used pre-program data (January 2009 to September 2014) to simulate potential savings from lowering domain one’s eight HACs [4]. Then, the authors compared these savings with CMS’ savings from payment reduction. Both studies showed no significant relationship between HACRP and improvement in these HACs.

In a 2019 study, the authors used difference-in-differences (DID) and interrupted time series approaches to examine changes in HACs in Michigan hospitals before and after the announcement of HACRP in August 2013 [14]. Due to data limitations, the HACs under study included a subset of domains one and two. The authors did not find any significant associations between patient safety and the HACRP, and the findings questioned the reliability of measures used in the program. A recent study [9] using the DID method and hospital-level data for 2896 hospitals from 2013 to 2020 found significant reductions in HACs following the passage of HACRP. This study has some limitations, such as the relatively short pre-program period available to assess the parallel trend assumption and potential concerns regarding the accuracy of certain HAC scores.

This study examines two targeted conditions: SSIs for abdominal hysterectomy and colon procedures. Consistent with previous studies [4,5,11,14], this paper finds HACRP ineffective in reducing SSIs in the targeted procedures. The trend graph of SSIs in Fig. 1 and the corresponding infection rates in S2 Table show no evidence of a decreasing trend. For example, SSI rates for abdominal hysterectomy procedures fluctuated between 10.31 and 24.08 per 10,000 discharges. Fluctuations around specific mean values are observed for other procedures as well. S3 Table, which reports SSI rates under alternative definitions, likewise shows no clear downward trend. Although these visual observations are informative, formal statistical tests that control for confounding variables are needed for a more conclusive interpretation. The DID estimates in Table 2 are all insignificant, suggesting that HACRP is not significantly associated with changes in SSI rates.

Similar findings are observed in the sensitivity analyses reported in S4 Table, which use alternative definitions for procedures and SSIs. The DID estimates in S4 Table are largely insignificant, except for colon surgeries when SSIs are defined by the third definition: 998.5, 998.51, 998.59, 996.6–996.69. In this case, the DID estimates range from −5.447 to −7.670 depending on the control procedures in the model. These values suggest that HACRP is associated with a reduction of 5.447 to 7.670 SSIs per 1,000 discharges. However, this conclusion may be questioned because the third definition of SSI is more general and less procedure-specific than the other definitions. It omits several ICD codes (e.g., 567.21, 567.22, 567.29, 567.38, 569.5, 596.61, 596.81, 682.2, 879.9) that have been used in prior studies to identify SSI events [27–29].

As a result, SSI rates under the third definition range from 30.22 to 46.33 cases per 1,000 discharges, substantially lower than the 109.29 to 165.56 cases per 1,000 discharges observed under the other two definitions (see S3 Table). This underestimation of SSI rates may in part explain the negative and statistically significant DID estimates found in this sensitivity analysis.

Various sensitivity analyses, including alternative procedure and SSI identification methods, the addition of two years of data after the ICD-9 to ICD-10 transition, and alternative ITS model specifications, yield similar conclusions. Overall, these results suggest no direct effects from HACRP on SSIs in abdominal hysterectomy and colon surgeries.

An important assumption in the DID design is the parallel trend assumption, which requires the treatment and control groups to have similar trends before the intervention. If the parallel trend assumption holds, we have some confidence that both groups would continue to have similar trends had the intervention not occurred. Hence, any deviation from the expected trend can be taken as an indication of the impact of the intervention. The pre-policy trend between treatment and control procedures was analyzed by estimating models with interaction terms between the treatment indicator and discharge year-quarter indicators. The interaction terms are insignificant, suggesting that the trend between the two groups is not significantly different before the policy. Hence, the parallel trend assumption is valid.

This paper makes several contributions to the current literature. First, rather than using associations and trend analyses in some studies, the quasi-experimental DID design was adopted to provide causality analyses. Second, only one previous study used this design to analyze the SSIs in Michigan [14]. This study, by using the National (Nationwide) Inpatient Sample (NIS), provides nationally representative results. Third, to examine the robustness of the results, several definitions of SSIs, control groups, and pre-program periods were used in the analyses.

There are some limitations in this study. One limitation is the accuracy of the measurement by using patient discharge data and ICD-9 codes [4,37]. Unlike some studies that used CPT codes and chart-abstracted surveillance data reported to the NHSN [11], this study used claim-based data and its ICD-9 diagnosis and procedure codes. These codes may not accurately capture the SSI rates because of three problems. The first problem is their low positive predictive value. For instance, a prior study reported a low concordance between medical claims and NHSN surveillance data [37]. If ICD codes underreport (overreport) SSIs for the targeted procedures and overreport (underreport) for the control procedures, the DID estimates would overestimate (underestimate) the impact of HACRP. The second problem is the transition of ICD-9 codes to ICD-10 codes in the fourth quarter of 2015. Despite the use of methodological adjustment, this transition could still affect trend analysis and related DID estimates [18]. The third problem of using ICD codes is the comparability of control groups. Some control procedures may not be clinically appropriate. Moreover, if targeted and control procedures are performed by the same services and improve at similar rates after the passage of HACRP, the DID estimates would fail to capture these improvements.

To address these concerns, this study used four control groups instead of one, each selected based on prior literature in the same research area [3,14,20–27]. The DID models were estimated separately for each control group to provide robustness checks against potential bias from an incorrectly identified control group. In addition, a sensitivity analysis using the ITS models was conducted, offering an additional layer of robustness checks on the findings. Despite these methodological adjustments and sensitivity analyses, concerns remain regarding the accuracy of ICD codes for case identification and the appropriateness of selected control procedures.

Second, the NIS is discharge-level data rather than patient-level data, and it only tracks inpatient hospital stays and does not capture outpatient services. This data structure leads to two problems. First, it fails to account for the changing landscape of hospital care. Hospitals are increasingly shifting toward outpatient services such as emergency room visits and observational stays [38–40]. Because NIS excludes outpatient encounters, these shifts and any related SSIs are not reflected in the analysis. Second, each NIS record represents a single discharge, not an individual patient. As a result, repeated admissions by the same patient are treated as independent observations, since no variables exist to link them. This can bias SSI identification. For example, if a patient underwent colon surgery and was discharged. Later, they were readmitted because of SSI in the first admission. The second admission would not be linked to the first. Instead, its primary diagnosis would differ, preventing the SSI from being attributed to the earlier surgery. Future research could address these limitations by using alternative databases, such as the State Ambulatory Surgery and Services Databases (SASD) from HCUP, which is the largest national database for outpatient visits and outcomes. SASD includes patient identifiers, enabling researchers to track outcomes for the same patient over time, and thus to examine the impact of changes in hospital care on SSIs and other HACs at the patient level.

Finally, this study controls a broad set of patient and hospital characteristics consistent with previous research on HACRP and similar programs [4,8,10,14,22,30]. These confounding factors could affect the infection outcomes. However, the NIS data in this study lack several variables used in the CDC/NHSN risk-adjustment models for calculating SSI scores for HACRP [41]: BMI, procedure duration, wound class, and oncology hospital status. Omission of these variables could bias the estimation results.

Previous studies have proposed several policy recommendations to improve HACRP in three areas: enhancing measurement design, strengthening communication and follow-up, and revising penalty structure. Some studies recommended alternative risk-adjustment models to better account for patient and hospital heterogeneities [11,13,15]. Other studies recommended clearer and more timely communication with hospitals to allow them to make appropriate changes [5,12,14]. They also recommended consistent follow-ups and auditing to improve the program and prevent fraud and misuse. Finally, a 2019 study proposed graduated penalties as in HRRP rather than the all-or-nothing penalty structure in the current HACRP [10]. The authors also suggested changing the threshold for the penalty to accommodate patient mix and other differences in hospital characteristics. Although the NIS used in this study is “the largest publicly available all-payer inpatient care database in the United States [42]” and contains a rich set of patient and hospital-level variables, it lacks detailed information on hospital practices that would allow a deeper analysis of program impact. To address this gap, future research could draw on alternative data sources, such as the American Hospital Association Annual Survey, or employ methods such as in-person interviews with healthcare providers.

In conclusion, although the findings suggest that HACRP had no direct impact on SSIs in the two targeted conditions, these results should be interpreted with caution. This study is exploratory, and future research is needed to address its data limitations. Collecting more granular data and conducting in-person interviews in future studies could help validate the current findings and identify effective strategies for improving HACRP.

Supporting information

Acknowledgments

The author gratefully acknowledges the valuable contributions from Dr. Abdulaziz T. Bako (Academic Editor) and five anonymous referees. Their insightful comments and suggestions have greatly improved the quality of the paper.