3.1.1 Understanding context and need.

Despite the varied contexts, the studies found the barriers to immunisation uptake to be somewhat similar. Most studies found that caregivers were aware of the benefits of immunisation and that vaccine refusal was not a major barrier to immunisation.

All studies reported strong political will to increase immunisation coverage in the country. Given the global push, the country governments are investing resources in innovative programmes for last-mile delivery of health services. In Ethiopia, Health Development Army (HDA) was mobilised to link communities with the formal health system. In Nigeria, the government launched the Reaching Every Ward (REW) strategy in 2005 to improve vaccination coverage.

To understand the barriers to immunisation uptake, studies used a combination of tools like extensive baseline surveys, rapid assessments, and social mapping exercises. The baseline surveys revealed a fair understanding of the importance of vaccination in preventing diseases among communities across contexts. In the Collaborative community checklists for immunisation project in Myanmar (CCCI Myanmar), the researchers found no evidence of vaccine refusal and attitudinal barriers were often related to issues of convenience or logistics. In the Vaccine Indicator Reminder project in Pakistan (VIR Pakistan), less than 2% of the respondents thought vaccines to be harmful for children. In the study promoting HDA among pastoralists in Ethiopia (Pastoralists Ethiopia), almost 93% of women believed vaccines prevented diseases. In the Participatory action research project in Nigeria (PAR Nigeria), 88% of respondents in Remo North Local Government Authority recognised the importance of full immunisation. In the Bunza Local Government Authority, however, VIR Nigeria respondents did not demonstrate such high awareness. Knowledge about immunisation schedule and timeliness, however, remained low across contexts. Most respondents were unable to either state the correct age of a child for different vaccines or the number of vaccines to be administered.

Overall, health providers remained an important source for immunisation-related information. In PAR Nigeria, over 90% of the respondents stated health facilities as the main information source. In VIR Nigeria, 46% of respondents expressed an explicit preference for receiving immunisation-related activities from Frontline Health Workers (FHWs). In VIR Pakistan, doctors were the main source of information (46%) after relatives. The mostly commonly discussed supply-side constraints included FHW availability, long wait times, and refusal by FHWs to vaccinate either due to vaccine shortages or there being too few children at the clinic to open a vial.

3.1.2 Fine-tuning interventions.

Almost all studies revisited intervention modalities to ensure they aligned well with contextual realities and participant preferences.

In VIR Pakistan, the baseline findings showed the need for extensive community engagement to facilitate community acceptance of the reminder bands. This finding informed the design of a comprehensive community engagement plan, which included the identification and creation of a cadre of community health champions to spread awareness about the bands and the need for timely vaccination. In response to caregiver preferences, it was decided that the bands be provided by the FHWs rather than CHWs, which was the original plan. VIR Nigeria also adopted this model to ensure that ‘tools were correctly targeted and used as per protocol’.

In CCCI Myanmar, responding to low baseline levels of community knowledge, the intervention scope was expanded to include a health education component to build caregiver understanding of immunisation and equip them to apply checklists as a tool to provide feedback on health service delivery.

In Pastoralists Ethiopia, based on stakeholder and community inputs, traditional leaders (“Abba olla”) were assigned a prominent role in organising the HDA network. Their representation in the kebele (village) administration council was also ensured. Other HDA implementation strategies were also tweaked to fit the pastoral context. For instance, the meeting frequency changed from weekly to fortnightly for 1–5 networks and monthly for development group team leaders. Oral narratives were allowed instead of written ones for reporting purposes. Flexibility was introduced in the 1–5 network membership to include less than six community women.

In PAR Nigeria, to prevent elite capture of stakeholder dialogues, the meeting venue was moved from wards to more neutral places to ensure that ‘different community sub-groups [were] well represented in training/dialogues, in an environment that is not disruptive’.

3.2.1 Intervention acceptability by frontline and community health workers.

An important concern with introducing new tools and approaches was that they could add to health worker workload and lead to demotivation or poor performance.

In the Fifth Child Project in Ethiopia (Fifth Child Ethiopia), FHWs felt that the defaulter tracing tool and the Enat Mastawesha calendar made them more efficient and freed up time to reach out to caregivers in remote locations. They eased communication with mothers and the community in general. They also helped better explain the benefits of immunisation. Using the defaulter tracing tool, FHWs successfully immunised 84% of dropout children.

In Pastoralists Ethiopia, health workers at all levels valued women’s involvement in the HDA. The FHWs reorganised 1–5 networks when necessitated by group member migration, and this process helped keep the group active.

In CCCI Myanmar, the provider checklist helped increase FHW confidence in providing immunisation services and improved interactions with clients. FHWs admitted to being more polite to caregivers and taking more time to explain things. The FHWs provided earlier notifications and more frequent reminders to community members about the upcoming immunisation sessions.

In PAR Nigeria, health workers acknowledged the importance of the participatory approach that did not ‘plan for’ but instead ‘planned with’ community members. Respondents across the board felt that ‘all stakeholder groups were actively involved in the PAR process and turnout at meetings was encouraging’. The Principal Medical Officer of Health in one of the wards was termed an ‘active change agent’.

VIR studies in Nigeria and Pakistan did not provide detailed insights into intervention acceptability among FHWs. In Nigeria, of the 14 FHWs trained on the use of VIR bands, only five implemented the intervention. In Pakistan, the CHW performance varied across cadres. The Female Community Volunteers of the Polio Eradication Programme participated actively and made referrals to the vaccination centres, and the Lady Health Workers made only 2% of the referrals.

3.2.2 Intervention acceptability by caregivers and other community members.

The community-engagement tools and approaches were successful in obtaining community buy-in, both among caregivers and community members at large. A key distinguishing aspect of every approach was centre staging the role of community members, particularly leaders, by putting in place systems and structures that ensured their participation in monitoring activities, setting up feedback loops and making immunisation an agenda item in community platforms. In doing so, most interventions either leveraged existing community institutions, like the ward development committees and the social mobilisation committees in PAR Nigeria or the kebele administration in Ethiopia studies, or set up their own platforms, like CCCI Myanmar, to create opportunities for dialogue between service providers and the community to facilitate collective ownership and action.

In Fifth Child Ethiopia, home visits by health workers and Enat Mastawesha calendars improved communication and built trust between caregivers and health providers. Village leaders actively participated in mobilising caregivers and monitoring outreach sessions for vaccine availability and FHW attendance. At kebele meetings, barriers to immunisation uptake and gaps in health service delivery were discussed to find a way forward.

In Pastoralists Ethiopia, of the 968 women of reproductive age surveyed, 96.3% became a network member. Meeting attendance, however, remained low with only 25% of women attending all or most of the meetings. Additionally, a greater onus was placed on the community to improve immunisation with community leaders, rather than FHWs, reporting on progress at meetings.

In CCCI Myanmar, 81% caregivers credited the checklist for improving their knowledge of immunisation, and 97% expressed willingness to use it again. Ability to provide feedback on health services made them more confident about using these services and more satisfied with the service quality.

Among the VIR studies, in Pakistan, for each DPT vaccine, close to 80% of the children returned to the vaccination centre with the VIR band. In Nigeria, of the 153 respondents at endline, close to 95% mentioned that they would recommend the VIR band to others. The interventions made significant outreach efforts to community influencers to improve band acceptance, though these efforts were better reported in Nigeria than Pakistan.

In PAR Nigeria, a ‘majority of the respondents felt that all stakeholder groups were actively involved in the PAR process and turnout for meetings was encouraging’. PAR enabled power-sharing among different groups of stakeholders. Interviews revealed positive participant perceptions about having a voice in the discussions, and overall satisfaction with the decision-making process in the meetings.

3.3.1 Working with the health systems.

No intervention was delivered as a parallel health service. Integration with the health systems required significant investments in relationship building at the top levels and health staff training at the frontlines. Extensive engagement with stakeholders at all levels of the health systems and trainings for CHWs and FHWs was key in ensuring that piloted interventions fit into the mandates of the health delivery systems. Designing and promoting tools and engagement approaches that eased health worker workload and were perceived as collaborative further helped secure buy in. Though financial incentives were offered to health workers in some contexts, they were limited and their role in motivating health workers was unclear.

The exact details of relationships with health systems varied by context. Burnet institute in Myanmar entered a memorandum of understanding with the Department of Public Health at the Ministry of Health to conduct health-related activities. In CCCI Myanmar and PAR Nigeria, government agency representatives joined the evaluation team as principal investigators, demonstrating the health department’s commitment to using evidence to improve immunisation uptake. In Fifth Child Ethiopia and VIR Nigeria and Pakistan, pre-existing relationships with in-country stakeholders was key to building collaborations. IRC in Ethiopia had a long-standing relationship with key stakeholders. In Pakistan, the National Immunization Technical Advisory Group chairperson was also a founding member of the Trust for Vaccines and Immunisation, the implementing agency. This relationship helped secure support from the government Expanded Programme on Immunization (EPI) to pilot VIR bands. In VIR Nigeria, the lead researcher had a long-standing relationship with the Nigeria Primary Health Care Development Agency, a parastatal body with statutory responsibility of improving access to primary health. In Pastoralists Ethiopia, the study was demand-led, as the Federal Ministry of Health was looking to develop a prototype of the HDA model for the pastoral context.

Buy-in at the top was complemented by significant investments in building the capacity of FHWs and CHWs to roll out the interventions. In the VIR studies, 53 CHWs and 14 FHWs were trained in Nigeria, and in Pakistan 50 participants consisting of Female Community Volunteers, Lady Health Workers and their supervisors, and vaccinators were trained. In Fifth Child Ethiopia, the International Rescue Committee (IRC) conducted a training of trainers for FHWs and their supervisors on the use of the Enat Mastawesha calendar and the defaulter-tracing tool. In CCCI Myanmar, the FHWs and township level supervisors were trained in the use of the provider checklist.

However, the cost implications of these interventions pose a challenge to feasibility. CCCI Myanmar nurtured a new and remunerated cadre of ‘checklist assistants’ tasked with collating findings from checklists to inform the monthly discussions in the community. VIR Pakistan paid CHWs US$3 for every child referred to a health facility for immunisation. In Nigeria, the CHWs could earn up to US$16 a month for making more than three referrals to a health facility, while FHWs were paid US$13 as an incentive to encourage adherence to the study protocol. Given that these studies did not discuss the cost implications of these intervention components, it is difficult to ascertain how these costs would encroach upon the limited health budgets in L&MIC contexts. In general, cost-related barriers to implementing interventions at scale have not been discussed in most of the studies. However, these barriers require careful consideration when assessing the feasibility of rolling these out through country health systems.

3.3.2 Improving intervention design and delivery.

The formative evaluations helped highlight design aspects that were critical to consider for intervention feasibility, including community health worker literacy and skill set, financial constraints, and tool design. They also pointed to improvements needed to better adapt interventions to their context. They also highlighted common vulnerabilities in the interventions’ theories of change related to participant capacity, institutional readiness, and financial resources that require further investment to enhance intervention feasibility and relevance.

Low literacy levels and limited facilitation skills among health workers posed problems in multiple contexts. In both VIR Nigeria and Pakistan studies, low literacy levels among CHWs affected their ability to complete paperwork to make referrals to health clinics for vaccination, as well as their ability to track and report defaulters. In CCCI Myanmar, checklist assistants required significant handholding to fulfil their role of collating data and communicating findings back to the community. In addition, while the intervention involved project staff conducting monthly health education sessions to build caregiver knowledge and awareness of vaccination, it is not clear how feasible it would be for regular health providers to take over this component. The PAR Nigeria evaluation found that the rural contexts with less cohesion needed stronger leadership for community-based groups. That evaluation pointed to the need to build participant capacity in specific domains such as conflict resolution. From a theory of change perspective, these examples allude to limited capacity among key players to take on new roles and responsibilities, in addition to the potential challenges of implementing similar interventions in low-resource settings.

In CCCI Myanmar, questions about the checklist suggested potential problems with the theory of change. Stakeholders suggested simplifying the community checklist and re-orienting it as a tool that would help the community own its role in immunisation, rather than as ‘a tool for critiquing’ the health system. The implied lack of clarity on the roles of the stakeholders involved and the key purposes of engagement undermines the intervention theory of change. The underlying distrust of the purpose of the community checklist helps explain why health providers were reluctant to participate in monthly stocktaking meetings at the community level and work collaboratively to resolve immunisation-related issues, thereby affecting implementation.

Financial constraints also arose as an issue. In PAR Nigeria, some interview subjects reported social mobilisation committees to be ‘not so active’ in the intervention due to various reasons, including financial constraints. Additionally, the cost of venues was a major cost driver necessitating identifying more cost-effective ways of organising stakeholder dialogues. Besides PAR Nigeria, at least two other studies suggest financial constraints inhibited the smooth implementation of the interventions. In these studies, these constraints affected the organisation of outreach services, which in hard-to-reach populations can be an important deterrent to timely vaccination. Though not discussed by researchers extensively, it may be worth considering how critical components of interventions may be adversely affected by financial barriers.

In addition to the above, the formative evaluations brought forth the need to enhance the functioning and customisation of the tools. For example, in VIR studies in Nigeria and Pakistan, there were serious technical glitches with several bands malfunctioning. In Fifth Child Ethiopia, while the health providers and the community members found the tools useful, suggestions were made to improve them by including pictures of men to encourage male participation in maternal and child health.

3.4.1 Intervention monitoring.

Systematic monitoring that captures process indicators can point to gaps in the intervention theory of change. The Pastoralist Ethiopia study uses indicators such as trainings and meetings held, groups organised, outreach services set up, supportive supervision visits conducted and so on to track intervention progress. The VIR teams in Nigeria and Pakistan track implementation using tools such as the band tracking register, enrolment summary and attendance registers to name a few. PAR Nigeria developed a checklist to assess implementation of joint actions plans, including meetings held, participant attendance, duration of participation, and achievement of targets set as part of the planning process. In Fifth Child Ethiopia, FHWs maintained a record of outreach sessions conducted, Enat Mastawesha calendars and defaulter-tracing tool distributed, and home visits conducted. The study also reports observing FHW-caregiver interactions to assess adherence to protocol.

Though extremely rich information was collected as part of the monitoring activities in almost all the studies, it was either not reported or reported selectively for only a subset of indicators or intervention components. While it is not clear how the teams determined what monitoring data to report on, systematic reporting, by drawing on existing tools such as the TIDieR intervention reporting guidelines [19], would have further enriched the findings and would have been especially useful to policymakers keen to test the intervention in their context.

3.4.2 Intrinsic challenges with establishing intervention feasibility.

All interventions were delivered in close collaboration with the existing health machinery, but the researchers and their implementing agency counterparts remained largely responsible for the intervention rollout and delivery. Researchers and their counterparts’ roles included sensitisation and awareness generation among community stakeholders, capacity building of CHWs and FHWs, setting up systems and protocols for intervention delivery, collecting monitoring data and providing supportive supervision. In both the Ethiopia studies, the study teams were involved in logistic support such as transporting vaccines, ensuring cold chain functionality, and ensuring EPI systems remained functional. In CCCI Myanmar, researchers stepped in to compensate for capacity shortfalls among the checklist assistants. In PAR Nigeria, the research team helped resolve conflicts at various dialogues to facilitate consensus on important service delivery issues.

While the very raison d’etre of formative evaluations is to allow an iterative approach to establish proof-of-concept, these roles raise concerns about the health systems’ commitment and capacity to deliver the interventions without support from research teams. The existing literature does not shed light on how a formative evaluation can address the seemingly paradoxical expectation of establishing a proof-of-concept with researcher involvement while testing for its appropriateness, acceptability and feasibility from a service provider perspective.