Supply-side interventions to improve health: Findings from the Salud Mesoamérica Initiative

Background Results-based aid (RBA) is increasingly used to incentivize action in health. In Mesoamerica, the region consisting of southern Mexico and Central America, the RBA project known as the Salud Mesoamérica Initiative (SMI) was designed to target disparities in maternal and child health, focusing on the poorest 20% of the population across the region. Methods and findings Data were first collected in 365 intervention health facilities to establish a baseline of indicators. For the first follow-up measure, 18 to 24 months later, 368 facilities were evaluated in these same areas. At both stages, we measured a near-identical set of supply-side performance indicators in line with country-specific priorities in maternal and child health. All countries showed progress in performance indicators, although with different levels. El Salvador, Honduras, Nicaragua, and Panama reached their 18-month targets, while the State of Chiapas in Mexico, Guatemala, and Belize did not. A second follow-up measurement in Chiapas and Guatemala showed continued progress, as they achieved previously missed targets nine to 12 months later, after implementing a performance improvement plan. Conclusions Our findings show an initial success in the supply-side indicators of SMI. Our data suggest that the RBA approach can be a motivator to improve availability of drugs and services in poor areas. Moreover, our innovative monitoring and evaluation framework will allow health officials with limited resources to identify and target areas of greatest need.


Introduction
The design of SMI as an RBA is different from previous studies. Participating countries receive 50% of the cost of program intervention from funders and contribute the remaining 50% themselves. At the end of each operation, pre-defined performance indicators are measured independently, and if 80% of these indicators are met, the country is awarded half of its contribution share back. Funds awarded to recipient countries for meeting targets can be spent on other health programs at the country's discretion. SMI's design enables IDB to dispense these pre-agreed funds to countries based on their compliance with country-specific health targets in selected geographic areas. SMI uses this RBA approach and delivers integrated evidenced-based supply-and demand-side interventions to improve maternal and child health for the poorest quintile of the population across Mesoamerica.
In this manuscript, we describe the improvement from the baseline to the first follow-up supply-side measurements of SMI.

Methods
The SMI methodology was previously published [35,36]. Briefly, the IDB worked with countries to select the poorest regions for the intervention. The selection was based on national poverty measures, roughly equivalent to the bottom 20% of the income or wealth distribution. A scale was created to order health jurisdictions and municipalities according to the countrydefined poverty metrics and to select municipalities eligible for intervention from the poorest quintile. Selection from these groupings of municipalities also considered additional criteria, such as the presence of gaps in health service coverage, the existence of ongoing health reforms, and the presence of complementary financing by other donors and the government. In each country, SMI is implementing two to three consecutive projects. Each project includes a set of performance (payment) indicators and targets that were set with governments, in line with country-specific priorities in maternal and child health. Key indicators include coverage of contraceptives, antenatal and postnatal care for women and newborns, in-facility delivery and skilled birth attendance, management of maternal and neonatal complications, complete vaccination coverage for age, prevalence of anemia in children, and quality of care for antenatal, delivery, postnatal, and child health care visits. As part of establishing the baseline performance for SMI indicators, surveys were conducted in households and health facilities in each country.
Written informed consent was obtained from all participants. The study received approval from the institutional review board (IRB) from the University of Washington, partnering data collection agencies, and the Ministry of Health in each country.
Interventions included implementing the Essential Obstetric and Neonatal Care (EONC) strategy, strengthening referral networks, improving the supply chain, encouraging the cultural adaptation of services for indigenous populations, supporting new service delivery platforms and community platforms, and the design and approval of updated country norms and protocols, among other activities. SMI design calls for follow-up measurements at the close of each of three intervention periods in each country to capture the impact of the interventions. Depending on whether targets for improvement are met at each of these critical junctures, countries will be reimbursed with funds corresponding to half of the counterpart investment that they had to put forward up front, to be used freely within the health sector.
SMI dispensed about $23 million for the first operation, with similar matching funds coming from the participating countries (S1 Table). A tranche equal to half of the government contribution was awarded if a country met the 18-month targets.
Indicator targets were set through negotiation with the countries' health officials and IDB based on literature reviews of intervention effectiveness from previous country-level studies, trend analysis using data from the Global Burden of Disease 2010 study [2,[37][38][39][40][41], expert advice, and a cost-benefit model developed by IDB. Several health facility indicators were used by SMI and measured at the baseline and 18-month junctures (S2 Table). Performance indicators for payment varied in their requirements by country (S3 Table shows the most comparable ones). For example, the indicator related to the supply of antenatal and postnatal care, "health facilities with permanent availability of inputs and equipment necessary for prenatal and postpartum care" had five components in El Salvador compared to 29 in Belize due to negotiation and agreement between each country and IDB. In order for a country to "pass," 80% of facilities must meet the indicator targets. For a facility to be declared as having met an indicator, every single component of that indicator must be met. For the 18-month measurement, countries committed to a predetermined target for each supply-side performance indicator. Targets were set specific to each country and were quantified as a percentage of facilities complying with all indicator components.
For each round, we selected a random sample of the facilities to be representative. All facilities were eligible for inclusion in each round. We conducted a third round of measurement for Chiapas and Guatemala, for the performance indicators only, after both implemented a performance improvement plan, given that they failed to achieve their targets at 18 months but they showed improvement. We only measured the performance indicators during the repeat to reduce the cost of the measurement. Though Belize also did not meet 18-month targets, a third round measurement was not conducted due to the limited SMI investment funds in Belize, as detailed in S1 Table. The health facility surveys at baseline and 18 months collected data on facility conditions, service provision and utilization, and quality of care. The survey involved three main components: an interview questionnaire, an observation checklist, and medical record reviews (MRRs). Health facilities were grouped according to three EONC levels-ambulatory, basic, and complete-as provided by SMI. Mainly, ambulatory level facilities have the capacity to provide outpatient care only; basic-level facilities are able to attend uncomplicated vaginal deliveries and provide immediate emergency obstetric and neonatal care; and complete-level facilities have a surgery room and the capacity to attend most obstetric and neonatal complications (not including intensive care). Different health facility indicator criteria were assessed depending on the EONC classification level. In the interview questionnaire, the facility director, manager, or other person in charge of the health facility was interviewed to capture information on general facility characteristics, infrastructure, human resource composition and trainings, supply logistics, infection control, child health care provision, vaccine availability, family planning service provision, availability of contraceptives, and antenatal, delivery, and postpartum care. Once these were completed, surveyors used an observation checklist to record direct observations of the availability and functionality, as applicable, of essential equipment and supplies for maternal and child health care, including pharmaceuticals. Surveyors also reviewed administrative records of pharmaceutical stocks in this module, capturing drug and contraceptive stock-outs occurring in the three months prior to the date of the survey.
We used MRRs to capture retrospective data on record-keeping and treatment practices of surveyed facilities. MRRs were selected at random from each facility using the records of encounters and applying systematic sampling to cover the entire study period. The MRRs covered various medical complications that mothers and infants experienced during delivery and how each sampled case was treated at a given health facility. The MRRs also captured the facilities' medical practices before, during, and after uncomplicated births. Depending on the country, other MRRs on diarrhea, pneumonia, low birth weight, child registration and growth charting, deworming, and family planning services offered were also implemented. Indicators from MRRs were collected for monitoring purposes during the 18-month measurement but will be included as performance-or payment-based indicators in the subsequent evaluation.
The SMI surveys were conducted using a computer-assisted personal interview (CAPI), and incoming data were continuously monitored by the Institute for Health Metrics and Evaluation (IHME). We conducted training sessions and pilots in each country before implementation. We also conducted supervisory visits during health facility data collection. We used Stata 13.1 for the analyses and to account for the complex study design. Baseline surveys were conducted from March 1, 2011, to August 31, 2013, while the 18-month follow-up surveys were conducted between January 20, 2014, and October 24, 2014. Dates of the surveys were designed to ensure an 18-month period between the baseline and the follow-up for each country. Performance improvement plan measurements (PIPM) in Chiapas and Guatemala were conducted between May 11, 2015, and June 25, 2015. Additional detail on SMI methodology and implementation is available elsewhere [35].

Results
In total, 365 health facilities were interviewed for the baseline measurement, and 6,332 medical records were reviewed, compared to 368 and 11,143, respectively, for the 18-month follow-up (Table 1). Moreover, 30 facilities were measured as part of the performance improvement plan nine to 12 months after the 18-month follow-up in Guatemala and Chiapas. Table 2 shows the progress in selected payment indicators from baseline to 18 months and related indicator targets. All countries showed progress in the indicators, although with different levels. El Salvador, Honduras, Nicaragua, and Panama reached their 18-month targets, while Chiapas, Guatemala, and Belize did not. The third measurement showed that both Guatemala and Chiapas reached their 18-month targets nine to 12 months later.
All countries showed improvement in SMI indicators from the baseline to the 18-month measurements (S1 Appendix). In total, 448 individual components were measured for the 18-month performance indicators for the seven countries, and only 47 showed a decline, 72 showed no change, and 279 improved from the baseline (Table 3). When we examined gains in equipment and supplies by health facilities, progress had been achieved in 18 months (Figs 2 and 3). Most facilities that did not meet the goal and qualify for

Discussion
To our knowledge, this is the largest RBA project to improve supplies and services at health facilities in poor areas of Mesoamerica. Our data show improvement in the availability of supplies and medicines at health facilities in a short time. SMI data suggest that the RBA approach is a motivator to improve availability of drugs and services in poor areas. Our results are encouraging for public health as this improvement happened in poor and remote areas, indicating that lessons learned from SMI could be applied in other regions of these countries and globally. SMI is a very ambitious project, and the targets set for each country were strict. For a facility to pass, it had to have all components of an indicator observed or achieved. The rule for disbursing the performance tranche was designed as a pass or fail, irrespective of the level of improvement in each facility, and was designed to offer a reward for extraordinary effort in a short time. Belize, Guatemala, and Chiapas did not pass the 18-month measurement, while Panama, Nicaragua, Honduras, and El Salvador did. The design of SMI ensures that the intervention continues in all countries, even though some countries did not receive a performance tranche at 18 months. However, improvement in supplies and equipment was observed in the three countries that did not meet their targets at the 18-month juncture. Our data show that most facilities in these countries did not pass because they were missing only one or two components of complex indicators. When both Guatemala and Chiapas were  given an extra nine to 12 months to implement a performance improvement plan, they were able to meet all targets.
Although RBA/RBF is increasingly used to incentivize action in health, most results-based evaluations provide evidence for schemes that apply financial incentives at the health facility or health provider level. Less evidence is available on the impact of RBA approaches that provide incentives at the national level, such as the Gavi ISS program [22].
Our supply-side results add to the limited available evidence of the effect of RBA schemes that apply financial incentives at the national level. Our study shows an improvement in the availability of supplies and equipment at facilities in 18 months, though we are not yet able to show an increase in uptake of services by the population, since the SMI measurement calls for another household survey in 2017-2018. However, increasing availability of supplies and equipment has been associated with increases in uptake previously [42]. Subsequent rounds of evaluation of SMI will provide more robust evidence to this area of study. Continued rigorous evaluation is needed to assess successes and failures in RBA schemes where financial incentives are applied at the national level.
SMI raised the bar on the requirement of supplies and equipment in poor and remote areas. These demanding indicators should be maintained and used in future SMI follow-up measurements. For example, the presence of stethoscopes was a requirement, though many facilities that reported not owning a stethoscope were relying on facility staff's personal stethoscopes. A strong argument could be made that stethoscopes should be available at the facilities since doctors or other facility staff may be absent on certain days (vacation or illness). Moreover, the improvement in such a strict and ambitious intervention in poor and remote areas is encouraging and suggests that such an approach could easily be adopted in other parts of each country. Showing decision-makers and health authorities that success can be achieved through proper strategies can ensure improvement in health elsewhere.
Our findings showed that countries had more success with certain components of an indicator. This was not surprising given the number of components and indicators we have in our evaluations. Our ability to provide each country with a detailed report on what are the missed targets and in what facilities will enable them to address these shortages. Moreover, our approach for evaluation could be used by countries to track their own progress in future years.
SMI provided a platform for action to improve health, giving all countries technical assistance to help remove bottlenecks, improve processes, analyze failures, and adjust strategies. Challenges and solutions for overcoming them should be documented, as should failures, to ensure that they are not repeated. After Chiapas and Guatemala did not meet targets and were given the external measurement results, they proved through the PIPM to be able to mobilize quickly in order to supply and equip facilities that were lacking. There is a need to include a qualitative component to this evaluation to complement the quantitative findings in order to better understand the drivers of success. Indeed, finding a way for people in remote areas to communicate their experience and provide the lessons learned is essential. Our findings call for creating a network of local health officials and residents to exchange ideas through the implementation of SMI.
Several factors have to be considered in SMI. First, as supplies increase in one area of the community (intervention areas), it is possible that the demand would increase from the community and neighboring ones, especially if they are not part of the intervention. Previous studies have shown that individuals may travel a longer distance to seek care at a better-equipped facility. Hence, these facilities could face a challenge in supplies and time going forward. Second, it is possible that the attention to comply with targets in the intervention areas has led to diverting resources to these areas at the expense of others, although we have no evidence that this occurred. Still, both of these scenarios have to be examined to ensure continued success.
Finally, a major concern for SMI is sustainability. Although the governments have agreed to provide funds to sustain the programs and intervention, it is possible that when donor investment and reimbursement end and the measurements stop things may change. A challenge for SMI is how to maintain these improvements once the funds from IDB are dispensed and IHME monitoring is complete. The sustainability of programs has been a major challenge in global health. However, we have already observed that governments have started their own monitoring of supplies and equipment. In fact, our interviewers in Chiapas and Guatemala documented how each component of country-specific indicators was numbered and labeled at some health facilities. Moreover, we hope that supply-side improvements will generate healthseeking behaviors and public and official awareness of the need to continue and maintain these achievements. Indeed, the population's expectations will result in increased demands for these supplies and availability at their health facilities.
Our study has several limitations. First, we did not conduct a household survey with the health facility survey this measurement round, and thus were not able to examine the change in demand. However, this is a planned activity for the next round of the evaluation. Second, we did not conduct a qualitative assessment in all countries to explain our quantitative results and identify the drivers or obstacles for change. Indeed, our qualitative work in Chiapas showed that the RBA model created a culture of accountability, which was strengthened following the failure to meet targets during the first measurement round. With the improvement plan, accountability trickled down to the level of health workers, who developed the habit of routinely checking supplies and ensuring shortages were avoided [36]. Third, it is possible that we have some self-reporting bias in our survey. However, this does not affect most of our indicators, which are based on observation and MRRs. Finally, we did not examine indicators in control areas during this round of health facility surveys to see if the same improvements in supplies occurred outside our study area. However, subsequent rounds of measurement include data collection activities in both our comparison and study areas.
Our study showed that when local staff, central health, and federal officials set a priority and are given a monetary and a professional incentive, positive changes could occur in a short time and in a challenging environment. We found that this type of RBF is the right approach to improve health in Mesoamerica. Our findings have great implications on health funding for reducing the burden of diseases.
Supporting information S1  Table. SMI selected 18-month performance indicators and requirements by country. (DOCX) S1 Appendix. Improvements in supplies and equipment from baseline to 18-month follow-up, by country. (DOCX) teams, and supervisors who conducted data collection, including El Colegio de la Frontera Sur-Mexico, UNIMER, and the University of Belize. We also thank Adrienne Chew at the Institute for Health Metrics and Evaluation for editing this manuscript.