Quantitative study component.

Quantitative design. The impact evaluation from which we drew quantitative data employed a quasi-experimental pre-test-post-test design with independent controls. We briefly summarize the design and analytical approach below, but would like to refer the reader to the publication by De Allegri et al. (2019) for details [24].

Quantitative samples. All 12 intervention districts in the six regions to which PBF was newly scaled up in 2014 were included in the evaluation. As control districts, PBF implementers identified two additional districts without PBF per region, judged as comparable to the intervention districts in terms of health indicators and service provision structures. The study sample consisted of all CSPS in the twelve PBF districts and a random sample of CSPS in the twelve control districts. This resulted in total of 523 CSPS, surveyed at both baseline and endline for a fully balanced panel. Of note, the decision to assign the PBF intervention at district rather than at CSPS level within districts was taken by local decision makers for feasibility reasons given the context.

Data collection. Data were collected immediately before (October 2013–March 2014) and about three years after (April–June 2017) implementation start (i.e., baseline and endline). At each time point, a comprehensive facility-based survey was conducted, which also included a health facility inventory checklist. This checklist included indicators measuring the availability and frequency of recent stockouts for a range of EM, including the 19 PBF tracer medicines in Table 2. Data collection teams spent one day at each sampled CSPS facility, collecting data on paper at baseline and with android mobile devices at endline.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Qualitative sample.

https://doi.org/10.1371/journal.pgph.0000212.t002

Outcome variables. We defined four outcome variables for this study. Based on the information collected in the facility inventory, we first grouped all medicines relevant to the CSPS level for which data was collected into two groups: tracer EM (i.e., the 19 incentivized by the PBF) and non-tracer EM (i.e., 22 non-PBF-incentivized medicines) (see Table 2). For each group, we then created binary variables representing the availability (defined as at least one unit in line with the PBF indicator definition) and the stockout frequency (defined as one or more interruptions of stock during the 30 days preceding each survey) of each medicine included in each group. For 60 CSPS, either baseline or endline data was missing for all medicines included in the analysis due to data collection errors. We therefore excluded these facilities entirely, resulting in a total facility sample of 465. Remaining missing data ranged from 0% to 4% per medicine for availability and from 3% to 30% for the stockout variable. Based on observations of how data collectors filled checklists during data collection supervision visits, we are confident that remaining missing data indicate non-availability or stockout, and replaced these data accordingly (i.e., conservative approach to replacing missing data). As a robustness check, however, we also calculated an “optimistic scenario” where we inversely assumed availability and no stockout. Resulting impact estimates differed slightly in magnitude, but not in direction and statistical significance, so that we are confident in the results obtained with the conservative imputation and will only present those in the following. Following data cleaning, we created four composite scores for each outcome variable (i.e., availability of tracer EM, availability of non-tracer EM, stockout of tracer EM, stockout of non-tracer EM) by additive aggregation of the respective binary variables.

Data analysis. We employed a Difference-in-Differences (DID) approach to identify the impact of PBF compared to status quo for each of the four outcome variables as expressed by the following regression equation: where Y_dft is the outcome variable for CSPS f in district d at time t with t  =  {2014, 2017}. 2017_t is a dummy variable indicating endline observations, thus coefficient β gives the time fixed effect. PBF_d is a dummy variable that equals one for CPSP in PBF districts and zero for CPSP in control districts. α_f are CSPS fixed effects, and ϵ_dft is the error term. Standard errors were clustered at the district level as the level of treatment assignment [26]. The coefficient δ gives the DID estimate for the effect of being located in a PBF district when compared to non-PBF districts. Our analysis is therefore challenged by a relatively low number of clusters (24 districts), which might lead to the estimation of downward-biased standard errors with an over-rejection of the null hypothesis [27, 28]. We addressed this threat to validity by using a ‘wild bootstrap’ to adjust confidence intervals [28, 29]. Unfortunately, routine data on medicine availability to test for parallel trends pre-intervention are not available. However, an interrupted time series study on the impact of PBF in Burkina Faso on various health care utilization indicators showed no difference in pre-intervention trends for the majority of investigated EM-dependent services, supporting our assumption that this was the case for EM availability as well [30].

Qualitative study component.

Qualitative design. Qualitative data were drawn from a cross-sectional study conducted in 2018 as a follow-up to the larger impact evaluation to better understand impact evaluation results, specifically what had allowed PBF to produce a positive impact on some indicators of service quality and utilization, but not on others. Data collection tools included specific questions regarding medicine availability, having emerged as an issue of interest in the quantitative impact evaluation and ensuing dialogue with implementation stakeholders.

Qualitative samples. Respondents were purposively sampled to represent key stakeholder groups directly or indirectly involved in PBF, as detailed in Table 2.

Given high staff turnover in light of the fact that qualitative data collection took place one year after the quantitative endline, we took care to select and/or trace respondents who had been in office during the impact evaluation study period.

Data collection. Data were collected in September 2018, four and a half years after the start of the intervention and one and a half years after the impact evaluation endline. PBF was still officially on-going at the time of data collection, but activities including incentive payments had been on hold due to financial issues. Data were collected through in-depth interviews (focus group discussions with community members given focus of the overall qualitative study), by the first and second author with support of trained interviewers along semi-structured interview guides. Interview guides for all stakeholder groups contained question related to role of and impact on EM availability. Interviews and focus group discussions were recorded, verbatim transcribed, and where not already conducted in French, translated into French.

Data analysis. Transcripts were coded by the first and second author using NVivo, and triangulated by MDA using a largely deductive thematic coding approach, with few additional emerging codes added as we proceeded through the material. For the purpose of this paper, the first author revisited all initial codes relevant to EM availability (including decision-making processes at CSPS level; observed changes in quality; change mechanism regarding purchasing medicines and supplies; factors affecting change in supply chains, understanding of PBF, leadership, financial aspects, gratuité, other factors) and translated material into a framework matrix [31], summarizing statements and key quotes per code for each respondent or focus group.

Mechanisms of change.

As outlined in the introduction, the assumption was that PBF would motivate facilities to work on EM availability both through the explicit incentives as part of the quality verification and reimbursement and through the implicit incentive of attracting and retaining patients with services of high quality, including availability of medicines.

Explicit incentives for quality improvement. Narratives indicate that the incentives for EM stock improvements built into the PBF quality component were not particularly powerful. Given the comparatively small amount of money to be gained by quality improvements, particularly in relation to effort and necessary investments, and in light of no sanctions for poor quality, many health facilities appeared to have contented themselves with the seemingly “effortless” earnings through the PBF quantity component, at least initially.

Implicit incentives for quality improvements. Initially, few facilities appeared to have realized the implicit incentive linked to offering services of high quality, namely that services of high quality would attract patients, while services of low quality would deter patients, particularly in an environment of enhanced competition in response to PBF. Over time, however, it appears that more and more facilities developed understanding of the importance of service quality in sustaining and increasing service use.

PBF impact on facility finances. In principle, CSPS should have been able to maintain sufficient stocks without PBF as they recovered expenses through user fees or–in the case of user fee exempted services–through reimbursements from the government. Interviews at health facility level, however, revealed that prior to PBF, EM stockouts had often been related to financial difficulties nonetheless. Respondents pointed in particular to the partial user fee removal policy for deliveries (SONU), citing difficulties with government reimbursements. The cash inflow through PBF helped facilities overcome these difficulties and allowed them to procure medicines in sufficient quantities.

In a similar fashion, the nationwide introduction of the gratuité in mid-2016 put many facilities throughout the country in financial trouble. Patient numbers increased dramatically, leading to higher medicine consumption. At the same time, the gratuité program, supposed to prefinance expected expenses for forgone user fees and medicine charges, incurred payment delays. The resulting cash flow challenges left many health facilities across the country unable to pay their medicine bills at times, often resulting in stockouts. PBF facilities were generally less affected as they tended to be more liquid thanks to PBF and could more easily bridge periods waiting for gratuité payments to arrive.

Barriers to profound and lasting change.

Despite perceived improvements in EM availability as a result of PBF, quantitative findings indicate that these were not sustained over time. Narratives converge on four main elements, which PBF was not or only partly able to change: Challenges related to the monopoly of the CAMEG in the medicine supply chain, inefficiency in proactive stock management, perceived unfairness of the quality verification process, and frustrations with PBF payment delays.

Monopoly of the CAMEG limiting financial autonomy unchanged with PBF. As mentioned above, EM purchasing and supply for the public sector is centrally managed by the CAMEG; facilities and districts are not authorized to procure from outside the CAMEG unless under very specific circumstances and with explicit MoH permission. For health facilities, this poses supply challenges in the event that CAMEG is unable to keep certain medicines in stock. Respondents reported that central-level EM stockouts happened from time to time during the PBF implementation period, but were particularly pronounced following the introduction of the gratuité in mid-2016. At CSPS level, this led to stockouts of significant proportions of EM at various points in time, significantly hindering facility operations.

Respondents deplored that the monopoly of the CAMEG limited the extent to which PBF could unfold its key principles of free market and financial autonomy of health facilities, which would have been pivotal in allowing facilities to improve their EM supply situation.

Regional directors reported the CAMEG monopoly having been subject of discussion during the design and implementation stages of PBF, but that fundamental changes were not possible for political and administrative reasons, including that the CAMEG is managed by multiple ministries and has a strong lobby within the health sector.

Cumbersome process of procuring from outside the CAMEG only slightly easier with PBF. Respondents conceded that in instances where the CAMEG faced stock outs, facilities were in principle allowed to procure medicines and supplies outside the CAMEG. The required procurement process to do so, however, was perceived as rather lengthy and complex, as facilities first needed to obtain administrative approvals from all levels of the procurement hierarchy. Some respondents, however, pointed out that in response more frequent requests to purchase from outside the CAMEG from PBF districts, supportive provisions within the hierarchy to speed up this approval process were put in place over time as requests.

Several respondents pointed out, however, that despite this provision, MoH administrators tended to be rather reluctant in approving too many requests for outside procurement given the powerful position of the CAMEG.

Inefficient proactive medicine stock management persisting with PBF due to implementation issues. One central level respondent pointed out that the CAMEG should not take all blame, but that they have been very reliable in supplying the health system with the exception of the crisis in 2016, after the introduction of the gratuité. Rather, they described how districts and health facilities themselves are oftentimes part of the stockout problem, for instance by not sufficiently anticipating that supply processes take time in the Burkinabè system, with or without PBF, or because of other shortcomings in medicine stock management.

Deficiencies in stock management in many CSPS were also observed by members of the ACV, who in addition to the quantity verification were tasked with supporting health facilities in identifying areas for quality improvement and effecting change. Specifically, ACV observed both stockouts and expiry of EM in consequence of facilities’ not anticipating their EM needs well when placing orders. In the context of PBF, health facilities were able to receive coaching in proactive stock management from the ACV. Respondents for the ACV admitted, however, that they generally barely acted on their coaching function as they used almost all of their time and budget to effect the quantity verification, conceding that this might have contributed to the lack of change in service quality.

Perceived unfairness in the quality verification process. In the above-described instances where facilities were unable to procure EM despite having the financial means to do so, frustrations were compounded by perceived unfairness in the verification process, leading to an erosion of motivation to make an effort to improve EM availability in the context of PBF. Specifically, respondents reported that PBF quality verifiers would rate facilities down for EM stockouts, even if these stockouts were not of their own fault, but rather due to higher-level supply chain challenges.

Frustrations with PBF payment delays. Further, the PBF scheme incurred quite severe payment delays at various points throughout the implementation period. Health workers were frustrated with the non-appearance of the anticipated recognition of effort as well as of anticipated cash to realize planned quality improvements. For many, this compromised motivation to make particular efforts particularly towards quality improvement, particularly where financial investment was necessary such as for EM stock improvement.

Heterogeneity in PBF impacts on EM availability.

Finally, respondents conceded that beyond the above-cited general mechanisms and issues, there was significant heterogeneity in how facilities and districts approached availability of EM in the context of PBF. Respondents highlighted three main factors: Engagement of managers at all levels, facility size, and availability of alternative suppliers.

Engagement of managers. First and foremost, respondents underlined that whether districts and facilities were able to make progress in EM availability and other qualities depended largely on the engagement, energy, leadership capacity, and motivation of the health facility in-charge to effect change with the enhanced means PBF offered to them.

Respondents further noted the important role of the District Health Officer and their engagement in enabling EM availability, and pushing quality improvement in the district more generally. Findings for instance revealed that District Health Officers handled issues with stockouts at the CAMEG in very different ways, with some being very reluctant to approve and pass on requests from health facilities due to the administrative burden and political dimension, and others being more supportive in either requesting official approval or allowing facilities the purchase of missing drugs from private pharmacies even in the absence of official approval.

Finally, respondents underlined the importance of cooperation and engagement of the COGES in improving EM availability, as it is the COGES that manages the health facility finances. It appears that understanding of PBF and reactions to PBF varied tremendously from COGES to COGES, with some realizing its potential in improving their CSPS for the better, while others were envious of the “gift” to health workers they had no part in–as COGES members were not entitled to individual financial incentives in the context of PBF. Ability of health facility staff to engage their COGES in the context of PBF was therefore reported as one important success factor.

Facility size. In relation to the above-discussed financial capacity issues, health facility in-charges pointed out that the size and frequentation of the facility had a major impact on whether facilities were able to cope with the payment delays in financial terms, with direct impact on EM availability. Respondents explained that the sale of medicines constitutes the primary mechanism of generating revenue for health facilities, as medicines are sold at a small profit. By principles of scale, large health facilities, turning over large amounts of medicines, are able to generate much larger revenue in absolute terms, allowing them to build up savings whereas small facilities spend their entire revenue on ordinary expenses.

Availability of alternative suppliers. Finally, in relation to the possibility of buying outside the CAMEG, respondents pointed out that even in the presence of approval, private pharmacies or other medical retailers are usually only available in urban centers, while more remotely located districts and CSPS often fully depend on the centralized system in absence of alternative procurement routes.