Quality of Voluntary Medical Male Circumcision Services during Scale-Up: A Comparative Process Evaluation in Kenya, South Africa, Tanzania and Zimbabwe

Background The rapid expansion of voluntary medical male circumcision (VMMC) has raised concerns whether health systems can deliver and sustain VMMC according to minimum quality criteria. Methods and Findings A comparative process evaluation was used to examine data from SYMMACS, the Systematic Monitoring of the Voluntary Medical Male Circumcision Scale-Up, among health facilities providing VMMC across two years of program scale-up. Site-level assessments examined the availability of guidelines, supplies and equipment, infection control, and continuity of care services. Direct observation of VMMC surgeries were used to assess care quality. Two sample tests of proportions and t-tests were used to examine differences in the percent of facilities meeting requisite preparedness standards and the mean number of directly-observed surgical tasks performed correctly. Results showed that safe, high quality VMMC can be implemented and sustained at-scale, although substantial variability was observed over time. In some settings, facility preparedness and VMMC service quality improved as the number of VMMC facilities increased. Yet, lapses in high performance and expansion of considerably deficient services were also observed. Surgical tasks had the highest quality scores, with lower performance levels in infection control, pre-operative examinations, and post-operative patient monitoring and counseling. The range of scale-up models used across countries additionally underscored the complexity of delivering high quality VMMC. Conclusions Greater efforts are needed to integrate VMMC scale-up and quality improvement processes in sub-Saharan African settings. Monitoring of service quality, not just adverse events reporting, will be essential in realizing the full health impact of VMMC for HIV prevention.


Introduction
Given the scientific evidence of the benefits of male circumcision in reducing HIV incidence rates among men [1][2][3], several sub-Saharan African countries have begun to scale-up voluntary medical male circumcision (VMMC) services for adult and adolescent males [4][5][6]. High coverage of safe VMMC services could prevent up to 3.4 million new HIV infections [7]. However, the rapid expansion of VMMC programs has raised questions regarding the extent to which health systems can deliver VMMC according to minimum quality standards which are sustained atscale [8,9]. Ensuring correct performance of VMMC procedures during scale-up is essential in realizing the potential health impact of averted HIV infections and in reducing health systems costs for HIV-related care [10,11].
In 2009, the World Health Organization (WHO) developed a quality assessment toolkit to assess a range of quality standards relating to facilities' VMMC supply inventory, provider competence, surgical technique, infection prevention, and continuity of care [12]. Preliminary findings suggest that VMMC surgical complications have resulted from inadequate provider competence and training, poor clinical infrastructure, and limited patient information and follow-up [2,5]. Yet, there is a dearth of published data on other aspects of service quality among VMMC programs in resource-poor settings [5,13,14]. Most quality assessments have focused on occurrence of adverse events [8,[14][15][16], device malfunction rates [10], procedure duration [10], or patient acceptability and resumption of sexual activity [13]. Less attention has been given to the availability of minimum service components and provider performance of specific surgical tasks in contexts where countries are rapidly expanding the number of VMMC sites [2,13,14]. To our knowledge, this study is the first published todate to address this gap by examining quality of VMMC services among four African countries over two years of program scale-up.

Study Design
A comparative process evaluation was used to examine VMMC service provision over time. Data were drawn from SYMMACS, the Systematic Monitoring of the Voluntary Medical Male Circumcision Scale-Up, which assessed the evolution of VMMC scale-up in Kenya, South Africa, Tanzania, and Zimbabwe. SYMMACS conducted a two-stage multi-country evaluation with two cross-sectional data collection periods (one in 2011 and one in 2012) among health facilities providing VMMC and selected VMMC surgeries. A detailed description of the SYMMACS evaluation and sampling methodology is provided in this supplement [17].

Scale-up Setting
The scale-up model used to expand the number of VMMC sites varied by country. Kenya and Tanzania adopted a focused implementation strategy using outreach sites and fixed clinical facilities supported by multiple implementing agencies within regions with the highest HIV prevalence and lowest male circumcision rates. Kenya additionally provided services via mobile sites to deliver VMMC in underserved areas. South Africa adopted a national approach using outreach and fixed sites with multiple partners, while Zimbabwe's national program was implemented by public sector service providers in outreach and fixed sites supported by a single implementing partner. Kenya was the only country with a fully-scaled VMMC program in the evaluation's first and second years, while Tanzania, South Africa, and Zimbabwe substantially increased the number of VMMC sites from two-to five-fold over the evaluation period.

Measurement
The comparative process evaluation measured two outcomes: VMMC facility preparedness and quality of VMMC surgeries. VMMC facility preparedness was assessed by direct observation using a pre-tested observation checklist based on the WHO VMMC quality assessment tool [12]. The checklist included items grouped into five preparedness categories: ''guidelines and protocols'' comprised 3 standard operating protocols; ''equipment and supplies'' included 9 items relating to anesthesia, antibiotics, condoms, and other supplies; ''basic life support'' (BLS) consisted of 4 items: cardiopulmonary resuscitation bag and mask, oxygen, intravenous lines, and anaphylaxis medications; ''hygienic and safe infrastructure'' consisted of 4 items measuring adequacy of lighting, ventilation, cleanliness, and adverse event monitoring; and ''patient services'' included 5 items: circumcision, group education, individual counseling, HIV counseling and testing, and referral services. Quality of VMMC surgeries was evaluated by direct observation of randomly selected surgeries using a checklist of tasks across three surgical periods: pre-operative (1 item), operative (21 items), and post-operative (7 items). Example tasks included pre-operative examination, use of sterile processes, correct surgical techniques, and provision of post-operative monitoring and counseling. Operative period tasks were further separated into two categories: 'safe operative procedures' (8 items) relating to surgical safety and 'operative techniques' (13 items) relating to step-by-step circumcision procedures. Trained observers scored items as ''unsatisfactory'' (scored = 0), ''partially satisfactory'' (scored = 1), and ''satisfactory'' (scored = 2).

Statistical Analysis
Data were analyzed using STATA (Version 12). Two sample tests of proportions and t-tests were used to examine differences in facility preparedness and VMMC service quality by country between 2011 and 2012. Composite measures of facility preparedness and service quality were calculated based on the percent of facilities with all requisite items per category and the mean number and mean percent of items scored as ''satisfactory'' per category. Item-level differences between time points were also assessed using Pearson's chi-squared tests. A significance level of p,.05 was used in all analyses.   +0.1 (+2%) [A] Includes only percent of facilities with total score ( = 1 point). Items receiving a partial score ( = 0.5 point) at the facility were not counted as available.

VMMC Facility Preparedness
*  Table 2 shows item-level analyses of facility preparedness. Protocol availability varied among countries. Supplies most commonly unavailable were HIV post-exposure prophylaxis, followed by antibiotics and condoms. Absence of oxygen and anaphylaxis medications appeared to drive low BLS scores, while absence of adverse event monitoring systems and lighting drove lower infrastructure scores. With the exception of referral services, most VMMC facilities offered group education, individual counseling, and HIV counseling and testing. Table 3 summarizes the quality of observed VMMC surgeries. Pre-operative tasks were correctly done in 79.5% and 97.1% of observed surgeries in Kenya and Zimbabwe, respectively, during the initial year and were maintained during the second year: (Kenya: 28.5, 95% CI: 217.6, 0.6 and Zimbabwe: +2.9, 95% CI: 0.6, 5.2). Pre-operative tasks were omitted the most in Tanzania's observed surgeries with low scores during the initial (20.6%) and second (26.3%) year (+5.7, 95% CI: 23.4, 14.8). Pre-operative tasks were correctly done in half (52.1%) of observed surgeries in South Africa, although this significantly declined over time (239.8, 95% CI: 247.7, 232.0).

Quality of VMMC Surgeries
Quality scores were highest during the operative period for surgical techniques. On average, 11.5 to 12.5 of 13 tasks were correctly done across countries. High scores were maintained during the follow-up year in Kenya (20.1, 95% CI 20.3, 0.01) with significant improvements in Zimbabwe to 12.9 (+1.4, 95% CI 1.1, 1.7). Scores significantly decreased in South Africa to 11.6 (20.8, 95% CI: 21.1,20.5) and in Tanzania to 12.0 (20.5, 95% CI: 20.7, 20.3). Scores for specific surgical tasks are shown in Table 4. Except for Zimbabwe, tying the surgical knot was consistently done incorrectly across countries, resulting in lower percentages of observed surgeries during expansion for which all 13 operative tasks were done correctly (Kenya 50.2%, South Africa 33.7%, Tanzania 36.7%, Zimbabwe 94.1%). Other incorrectly performed operative tasks were identification of the skin to be excised, administration of local anesthesia, and securing mattress sutures.
While on average the majority of safe operative procedures were correctly done during the initial year (6.3 to 6.9 of the 8 minimum tasks), protective eyewear was often omitted in all 4 countries resulting in few surgeries where all 8 safety tasks were completed (Kenya: 4.1%, South Africa, 9.7%, Tanzania 9.8%, Zimbabwe 0%). Kenya maintained otherwise high safety scores (20.1, 95% CI: 20.2, 0.04), with significant improvements in Tanzania (+0.5, 95% CI: 0.3, 0.7) and Zimbabwe (+0.5, 95% CI: 0.4, 0.6). Significantly fewer safety tasks were done in South Africa's expansion year as compared to the initial year (21.0, 95% CI: 21.3, 20.7). Lower safety scores reflected lack of hand washing, not disinfecting hands between clients, or not maintaining sterile surgical fields.
Quality scores were lowest during the post-operative surgical period with significant decreases in three countries: Kenya, 4. This study's comparative process evaluation demonstrated that high quality VMMC services can be provided during rapid scaleup in sub-Saharan African settings. Nearly all VMMC operative tasks observed in the study were performed correctly, with the exception of tying surgical knots. However, the evaluation revealed several areas of sub-optimal VMMC performance levels. Lower scores in infection control during surgery and pre-and postoperative tasks highlighted the need to reinforce provider training and supervision in these areas. Pre-operative examinations are necessary to assess eligibility for surgery and identify surgical contradictions which may lead to adverse events. Post-operative tasks are essential in prevention, identification, and management of complications. Reinforcement of counseling messages during the post-operative period on the importance of risk reduction and sexual abstinence during wound-healing is also crucial. Lower scores among these items could influence the overall effectiveness of VMMC given that transmission rates are directly affected by these behaviors.
When examining overall scores, there were considerable variations across the two years of program implementation. Results showed that high levels of service quality can be maintained at-scale and, in some cases, dramatically improved during the process of scale-up, as was the experience of Kenya and Zimbabwe, respectively. However, analyses also underscored the challenge of replicating initial successes at-scale in the case of South Africa, and the challenge of managing concurrent performance improvement and deterioration within the VMMC program in Tanzania. Some safety measures improved, while surgical technique and post-operative care quality declined. During the second year in both countries, less than a third of sampled sites had all requisite protocols to equip staff with readily accessible information. Equally as few sites had all necessary Includes only percent of surgical observations with total score ( = 1 point). Items receiving a partial score ( = 0.5 point) during observation were not counted as correctly done. [B] Includes only facilities with no missing items. Thus, the sum of percentages does not equal zero. *Denotes statistically significant at p,0.05. doi:10.1371/journal.pone.0079524.t003 Table 4.  Includes only percent of surgical observations with total score ( = 1 point). Items receiving a partial score ( = 0.5 point) during observation were not counted as correctly done. *Differences from T equipment and supplies for normal and emergency surgeries. The implications for expanding VMMC services in South Africa are further detailed in this supplement [18]. In all four countries, the absence of HIV post-exposure prophylaxis was more severe during the second year, and shortages at both time points of antibiotics and BLS equipment can hinder the management of severe adverse events. Several lessons learned from this evaluation may inform future efforts to scale-up VMMC services in similar settings. One key finding is that while high quality VMMC programs can be achieved at-scale, the process of scaling-up does not guarantee compliance with all quality criteria. This was evidenced by the variability over time in facility preparedness and surgical care within countries. In some settings, quality measures declined over the two-year period, or the VMMC program was expanded in the second year with comparably low performance levels as observed during the first year. This evaluation emphasizes the need to routinely measure aspects of quality, and ensure that VMMC program expansion is linked with quality improvement initiatives.
Secondly, while the diversity of program experiences among countries precludes any definitive comparison, some insight may be drawn from the range of scale-up models which were used. All countries partnered with implementing agencies with variations in the implementation of fixed, mobile, and outreach sites. In addition, focused models provided the advantage of allocating resources to regions with greatest disparities in HIV prevalence and male circumcision, while national models benefited from population-wide implementation. Zimbabwe additionally delivered its VMMC program through a single implementing organization. There were also differences across countries in the volume of VMMC facilities added during the evaluation's twoyear period. A final lesson emerging from this analysis is the recognition that challenges in commodity procurement and provider capacity may have reflected broader health systems strengthening needs that were not unique to VMMC. Careful attention will be needed to ensure that the urgency of scaling up VMMC does not bypass efforts to strengthen health systems or hinder quality of existing health services in facilities where VMMC is introduced.

Limitations
The study's limitations deserve mention. VMMC facility preparedness was measured at the time of the study visit and does not reflect cumulative availability or adequacy of supplies relative to client load. Thus, use of point-in-time measures limited the evaluation's ability to examine seasonal trends in VMMC quality as the program expanded. Data were also collected during low volume service periods which may have confounded the study's observations, and the small number of facilities may have contributed to the wide confidence intervals of some measures. In addition, it is possible that in some instances the evaluation of VMMC surgeries was biased given that scores were based on the evaluator's interpretation. To minimize this, all observers were trained using a standardized observation guide. However, it was not possible to standardize ratings across countries which may have resulted in unequal scoring.
The evaluation's summary measures should likewise be interpreted carefully. Quality of VMMC services was examined from the perspective of the health system. Client perspectives of quality such as comfort, privacy, or communication adequacy are not reflected, and other systems-level quality metrics not measured in this study may have been informative. Furthermore, it can be argued that not all items were equally critical, and thus the analysis' equal weighting of items unduly magnified (or diluted) the clinical relevance of certain tasks. Although the development of differential weights was not feasible within the study's parameters, to reduce this possibility, we aimed to balance the interpretive weight of each score by including in the questionnaire only the minimally essential elements in each category. Thus, the range of scores is designed to reflect only items necessary to perform safe and effective circumcision. Finally, the results from this study are unique to the selected countries and may not be generalizable to other settings. This analysis was not designed to identify factors associated with differences in observed quality measures. Therefore, causal or comparative inferences cannot be drawn between countries or facilities.

Conclusions
To our knowledge, this study is the first to examine facility preparedness and quality of VMMC services in contexts of scale-up in four African countries. Lapses in high performance and, in some cases, expansion of considerably deficient services emphasize the need to better integrate VMMC scale-up and quality improvement processes. Monitoring service quality, not just adverse event reporting, will be essential in realizing the full health impact of VMMC for HIV prevention in African settings.