Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Realist evaluation to improve health systems responsiveness to neglected health needs of vulnerable groups in Ghana and Vietnam: Study protocol

  • Tolib Mirzoev ,

    Roles Conceptualization, Funding acquisition, Methodology, Project administration, Writing – original draft, Writing – review & editing (TM); (SK)

    Affiliation Nuffield Centre for International Health and Development, University of Leeds, Leeds, United Kingdom

  • Ana Manzano,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation School of Sociology and Social Policy, University of Leeds, Leeds, United Kingdom

  • Bui Thi Thu Ha,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Population and Reproductive Health, Hanoi University of Public Health, Hanoi, Vietnam

  • Irene Akua Agyepong,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Research and Development Division, Ghana Health Service, Accra, Ghana

  • Do Thi Hanh Trang,

    Roles Methodology, Project administration, Writing – review & editing

    Affiliation Department of Undergraduate Education, Hanoi University of Public Health, Hanoi, Vietnam

  • Anthony Danso-Appiah,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation School of Public Health, University of Ghana, Accra, Ghana

  • Le Minh Thi,

    Roles Methodology, Project administration, Writing – review & editing

    Affiliation Department of Population and Reproductive Health, Hanoi University of Public Health, Hanoi, Vietnam

  • Mary Eyram Ashinyo,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Quality Assurance, Institutional Care Directorate, Ghana Health Service, Accra, Ghana

  • Le Thi Vui,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Population and Reproductive Health, Hanoi University of Public Health, Hanoi, Vietnam

  • Leveana Gyimah,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Mental Health Authority, Accra, Ghana

  • Nguyen Thai Quynh Chi,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Department of Population and Reproductive Health, Hanoi University of Public Health, Hanoi, Vietnam

  • Lucy Yevoo,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Research and Development Division, Ghana Health Service, Accra, Ghana

  • Doan Thi Thuy Duong,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

    Affiliation Department of Population and Reproductive Health, Hanoi University of Public Health, Hanoi, Vietnam

  • Elizabeth Awini,

    Roles Methodology, Project administration, Writing – review & editing

    Affiliation Research and Development Division, Ghana Health Service, Accra, Ghana

  • Joseph Paul Hicks,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Nuffield Centre for International Health and Development, University of Leeds, Leeds, United Kingdom

  • Anna Cronin de Chavez,

    Roles Methodology, Project administration, Writing – review & editing

    Affiliation Nuffield Centre for International Health and Development, University of Leeds, Leeds, United Kingdom

  •  [ ... ],
  • Sumit Kane

    Roles Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing (TM); (SK)

    Affiliation Nossal Institute for Global Health, University of Melbourne, Melbourne, VIC, Australia

  • [ view all ]
  • [ view less ]



Socio-economic growth in many low and middle-income countries has resulted in more available, though not equitably accessible, healthcare. Such growth has also increased demands from citizens for their health systems to be more responsive to their needs. This paper shares a protocol for the RESPONSE study which aims to understand, co-produce, implement and evaluate context-sensitive interventions to improve health systems responsiveness to health needs of vulnerable groups in Ghana and Vietnam.


We will use a realist mixed-methods theory-driven case study design, combining quantitative (household survey, secondary analysis of facility data) and qualitative (in-depth interviews, focus groups, observations and document and literature review) methods. Data will be analysed retroductively. The study will comprise three Phases. In Phase 1, we will understand actors’ expectations of responsive health systems, identify key priorities for interventions, and using evidence from a realist synthesis we will develop an initial theory and generate a baseline data. In Phase 2, we will co-produce jointly with key actors, the context-sensitive interventions to improve health systems responsiveness. The interventions will seek to improve internal (i.e. intra-system) and external (i.e. people-systems) interactions through participatory workshops. In Phase 3, we will implement and evaluate the interventions by testing and refining our initial theory through comparing the intended design to the interventions’ actual performance.


The study’s key outcomes will be: (1) improved health systems responsiveness, contributing to improved health services and ultimately health outcomes in Ghana and Vietnam and (2) an empirically-grounded and theoretically-informed model of complex contexts-mechanisms-outcomes relations, together with transferable best practices for scalability and generalisability. Decision-makers across different levels will be engaged throughout. Capacity strengthening will be underpinned by in-depth understanding of capacity needs and assets of each partner team, and will aim to strengthen individual, organisational and system level capacities.


Responsiveness is a key goal of any national health system, and can be defined as “…when institutions… are cognisant and respond appropriately to the universally legitimate expectations of individuals… safeguarding of rights of patients to adequate… care” [1 p.3]. In addition to being an intrinsic goal, responsiveness is also a process involving multiple interactions within health systems [2] and is an integral value within health systems [3].

In the last two decades, large parts of the world have seen significant and rapid economic growth [4]. Between 2003 and 2018, 32 of 66 low-income countries have become middle-income (Fig 1), and today the vast majority (73%) of people live in middle-income countries. Such growth has led to major improvements in availability of healthcare in those contexts. Citizens are now no longer satisfied with mere availability of services but increasingly demand systems to be responsive to their medical and non-medical social expectations, for example treatment with dignity and confidentiality [1, 5].

Fig 1. Number of countries by income category, over time.

Improved responsiveness, while itself being a health systems goal, also contributes towards equitable improvements in population health. Responsive systems can improve service uptake, ensure adherence to treatment, and ultimately enhance patient welfare [6, 7]. People, especially the most vulnerable, are more likely to use services if health systems are responsive to their expectations. Conversely, people are unlikely to use services within unresponsive systems [5, 7, 8]. Yet of the four intrinsic goals of health systems i.e. improving health; ensuring fairness in financing, efficiency and responsiveness [9, 10], responsiveness is the least studied, particularly in low and middle-income countries (LMICs) [2, 11].

Improving health systems responsiveness requires targeting internal interactions (between healthcare staff and managers within health systems) and external interactions (between people and healthcare staff) [2]. Both interactions require, and in turn contribute towards, strengthening of different health systems components. This includes staff support and development, information exchange, communication and service delivery.

Vulnerability, defined as defencelessness, insecurity and exposure to risks [12], comprises two elements: exposure to risks (financial, physical, psychological) and inability to deal with risks without suffering further damage. Anyone can become vulnerable, but some groups in certain contexts such as pregnant women, children and the elderly are often considered as the most vulnerable in society [13].

Vulnerability of pregnant women is a particularly complex and contested issue [14]. Scholars cautioned against unnecessary medicalisation of pregnancies from the feminist position and sociology of health [15], highlighting a vicious cycle of exclusion of pregnant women from biomedical research leading to them losing out on potentially safe treatments [16] and raising vulnerability as a relational and context-specific issue which needs safeguarding within research studies [17].

Despite significant progress, improvements in maternal health remain highly inequitable in many LMICs. This includes Ghana and Vietnam where large disparities cut across income, ethnicity, residence and other aspects of intersectionality [18, 19] and there is limited utilisation of healthcare by the most vulnerable. For example, in Vietnam women from ethnic minorities have 30% fewer antenatal care visits and facility deliveries [19]. Yet, despite the vulnerable having major needs [20], their voices are often neglected in health systems [21].

Psychological wellbeing in pregnancy and postpartum is crucial to mother’s and child’s health [22]. However, mental health is often the most neglected aspect of maternal health [20, 23]. A truly responsive health system should recognise the intersectional nature of vulnerability and effectively identify and respond to the neglected health needs of vulnerable groups, such as maternal mental health. It is particularly important within the context of LMICs, where power differentials between health providers and service users are often heightened due to substantial information asymmetry between these groups, which contributes to perceptions of risks in pregnancy being most shaped by healthcare providers [17].

This paper shares a protocol for the RESPONSE study which aims to understand and improve health systems responsiveness to health needs of vulnerable groups in Ghana and Vietnam. RESPONSE seeks to contribute to advanced understanding and improving health systems responsiveness in LMICs through multiple case studies in Ghana and Vietnam. We will co-produce with service providers and service users, implement and evaluate context-sensitive interventions to improve systems responsiveness. The focus of our study is on improving health systems responsiveness to the needs of pregnant women, particularly those with mental health problems. Pregnant women were chosen as a priority group who meets the criteria for earlier-discussed vulnerability, due to the intersection of their gender, age, ethnicity, socio-economic status, health needs and other characteristics of intersectionality.


Study objectives and questions

Our overall health systems research question is: In what way can health systems become more responsive to neglected health needs of vulnerable groups within the contexts of lower-middle-income countries? This study has five objectives, shown in Table 1 alongside the corresponding research questions.

Key study’s outcomes will be: (1) improved health systems responsiveness to neglected health needs of vulnerable groups in Ghana and Vietnam, and (2) an empirically-grounded and theoretically-informed model of complex relations between the contexts, mechanisms and outcomes of the interventions, along with transferable best practices for scalability (i.e. expansion within similar contexts) and generalisability (i.e. to different contexts, such as other health areas and countries) for future health systems strengthening.

Theoretical framework

The most widely known framework for health systems responsiveness, developed by the WHO in 2000, comprises seven domains: dignity, autonomy, confidentiality, prompt attention, quality of amenities, access to support networks, and choice of service provider [1, 8, 24]. The WHO framework has been translated into a survey toolkit for self-assessments of patients’ use of healthcare [2426].

More recent work on the topic stressed that interactions between the people and their health systems are central to understanding this concept [2], and improving responsiveness should therefore address both the ‘people’ and the ‘systems’ sides of such interactions [2, 27, 28]. Evidence also shows that interpretations of responsiveness can be context-sensitive (e.g. expectations of dignity reflect political, democratic and policy climate [5]), and vary across actors (e.g. patients and providers with different powers [7, 29]) and health facilities (e.g. public/private [5, 7]). Responsiveness, therefore, is arguably a socially-constructed, rather than an ‘absolute’ and ‘universally normative’ concept.

Adaptations of the seven domains of responsiveness have been proposed for different health areas, such as HIV/AIDS or mental health [26, 30, 31]. Yet, we found no studies which sought to improve health systems responsiveness to health needs of vulnerable groups in LMICs, including to neglected maternal mental health needs. Furthermore, many theoretical and applied questions remain unanswered, such as the following. What does responsiveness mean to different actors on the ‘people’ and the ‘systems’ sides? How do these conceptualisations shape actors’ behaviours and practices? In what way improving responsiveness contributes to other health systems goals?

In RESPONSE, we will bridge some of these knowledge gaps while also responding to calls for improved understanding of health systems responsiveness to the needs of diverse health systems actors in LMICs [2, 11, 27, 28].

We approach ‘responsiveness’ as a dynamic social action which is produced via relationships between different actors within contexts of particular socio-economic arrangements as they negotiate experiences of professional, citizen, consumer and patient rights and responsibilities.

In our theoretical framework (Fig 2), health systems responsiveness is understood as comprising the socially-constructed processes of external and internal interactions. Experiences of these interactions determine the degree of health systems responsiveness across eight domains: dignity, autonomy, confidentiality, attention, access to networks, quality of amenities, choice of service provider and trust. People’s engagements with health systems (for example, to seek healthcare) and system’s responses to these engagements (for example, service delivery) are shaped by their initial expectations [2]. People’s initial expectations are influenced by their individual’s relations with their families and communities within the context of cultural and societal norms as well as their potential vulnerability. The system’s ability to be responsive is driven by the expectations of frontline service providers of responsive services, in the context of current structures, processes, priorities and targets set by policymakers and managers.

Conceptually, health systems responsiveness involves two socially-constructed interactions:

  • internal i.e. between policymakers, managers and service providers, for example as part of prioritisation, resource allocation, target-setting, staff supervision and performance appraisal
  • external i.e. between people and the system, typically during service provision [27, 28]

People’s interactions with their families and communities are conceptualised as a context which shapes their initial expectations and subsequent engagements with a health system, though these can also be interpreted as a possible third type of interactions.

Improving health systems responsiveness should, therefore, target both the internal (within the health systems) and external (between the people and health systems) interactions and be cognisant of the wider context which shapes people’s initial expectations of, and their subsequent engagements with, their health systems.

Study setting

We will conduct the study in two lower-middle-income countries: Ghana and Vietnam. Our choice of these two countries is driven by three considerations. First, their commonalities and diversity provide excellent value for this research. Key similarities are:

  • Both Ghana and Vietnam are rapidly transitioning LMIC contexts with stable political climate and socio-economic growth (Table 2), but with inequitable distribution of development gains [19, 32]. The fragile states index ranks them as 110th and 109th, respectively (
  • In both countries, the most vulnerable are the poor and those have been left behind amidst major developmental gains for the rest of the country. In response, policymakers in each country are committed to the agenda of health equity and universal health coverage.
  • Both countries have largely publicly funded national health systems, comprising four-tiered organisations with national, regional or province, district and local levels. Both health systems have a high degree of decentralization and a growing private sector.
Table 2. Key common indicators between Ghana and Vietnam.

The two health systems also have differences. Vietnam’s publicly-dominated system has increasing market competition, but the Ghanaian system has large private not-for-profit sector and a more substantial share of external funding for health (13% compared with 2% in Vietnam) [33]. With regards to decentralisation, in Vietnam most power is consolidated at the province (regional) level whereas in Ghana districts have greater autonomy. While the antenatal care coverage and skilled birth attendance levels are relatively high (over 85%) in each country, maternal mortality in Ghana is higher i.e. 319/100,000 live births compared with 54/100,000 in Vietnam [34]. This may reflect problems with service quality and poorer people-system interactions leading to delays in seeking healthcare. In Ghana, there are dedicated staff for local mental healthcare (e.g. community psychiatric nurses), whereas provision of grassroots level mental healthcare in Vietnam is through mainstream primary health care workers. Our experience shows that in Vietnam, vulnerability often relates to ethnicity and income, but in Ghana it seems to be mostly location and income-related.

The second reason which guided the choice of Ghana and Vietnam is that there is high interest from policymakers in making the health systems more responsive to health needs of the most vulnerable. Preliminary discussions with key decision-makers from the Ghana Health Service and Ministries of Health in each country held during the proposal development showed that all key policymakers are committed to improving health systems responsiveness. In each country, maternal health remains a priority [35, 36] with increasing attention to mental health and integrated and responsive services. Psychological well-being of mothers and new-borns is a part of Ghana’s 2014 National Reproductive Health Policy and Standards. In Vietnam, policymakers emphasise ensuring equity in access, integrating mental healthcare (for example, into community programmes [37]) and piloting models of support and rehabilitation for those with mental illnesses.

Finally, we have chosen these two countries because we have strong and longstanding academic collaborations between the Universities of Leeds and Ghana [38], and Universities of Leeds and Melbourne with Hanoi University of Public Health [39, 40]. We will leverage and extend these into effective South-South collaborations, exchange and learning, as well as building links amongst policymakers and practitioners in Ghana and Vietnam.

In each country, we have purposefully selected one region (in Ghana) or province (in Vietnam) with some of the largest inequities disproportionately affecting vulnerable groups. Within each region and province, we have further selected a rural and an urban district to allow comparisons between rural-urban contexts.

In Ghana, we will work in Greater Accra Region located in the Coastal zone. It is 90% urban with highest maternal mortality of 336/100,000 live births as compared with the national average of 310/100,000 live births according to the 2017 Maternal Health Survey, growing squatter urban slums and some of the most deprived rural communities. In Vietnam, we will work in Bắc Giang, a mountainous Province 50km to the east of the capital Hanoi, with large vulnerable groups and 12% of the population comprising ethnic minorities.

In each country, we have selected two district health systems as our case studies, based on consultations with policymakers and their demographic and health indicators (Table 3) and leadership’s commitment to and opportunities for change. We have selected a purely rural (Shai Osudoku and Yên Thế), a rural with urbanised periphery (Ningo Prampram) and urban (Hiệp Hoà) districts, to allow learning across rural-urban settings.

Both districts in Ghana have a Demographic Surveillance System run by the Dodowa Health Research Centre, thus giving a useful link to an existing dataset. The Vietnam’s MOH has identified the Bắc Giang Province to pilot models of support and rehabilitation for those with mental illnesses (including pregnant women), which provides an excellent opportunity and timing for our results to inform on-going reforms.

In each district, we will implement interventions in the district hospital, 2–4 public and private primary health care facilities and within respective communities. As we explain later in the paper, the interventions will aim to improve external interactions (i.e. between people and health systems) as well as internal interactions within health systems (i.e. between healthcare staff and managers). Simultaneously, in Ghana we will engage with Greater Accra Regional Health Directorate and Ghana Health Service, in Vietnam we will engage with Province Health Department, and national Ministry of Health. The project co-investigator Dr Ashinyo is Deputy Director in the Directorate of Clinical Care of the Ghana Health Service and therefore has direct links with national policymakers. In Vietnam, we will leverage project co-investigator Professor Bui’s strong research-policy links. This will ensure sustainability, replication and scaling up of the interventions.

Study design

We will use a mixed-methods realist theory-driven case study design, utilising our expertise in realist evaluations [4146] and established standards for reporting realist evaluations [43]. Health systems comprise heterogeneous interconnected actors (e.g. patients and providers) operating at multiple levels within a complex dynamic system. Ensuring systems responsiveness requires aligning multiple interpretations from actors with different powers and resources. A realist approach helps make sense of such complexity by identifying how the multiple components interact in non-linear ways [47, 48]. It recognises micro, meso and macro contexts (Cs) in triggering the mechanisms (Ms) to produce outcomes (Os) [47, 48], known as CMO configurations, to explore what works, in which circumstances, for whom and why [47], and therefore suits this study.

Realist researchers develop, test and refine middle-range theories which show causal pathways of how interventions work. Such an approach is similar to studies guided by theories of change or even initial hypotheses. Our draft initial theory, to be further developed, tested and refined is:

Context-sensitive interventions for better recognition of initial expectations of key actors, if co-produced by these actors to target internal and external interactions and implemented within favourable policy contexts, will improve health systems responsiveness to neglected health needs of vulnerable groups, ultimately contributing to better health outcomes for all.

This 42-months study will comprise three Phases as shown in Fig 3 (methods are in italics) and set out in detail in the following three sub-sections.

Phase 1: Theory development and baseline.

In phase 1 (first year) we will understand actors’ expectations of responsive health systems, identify key priorities for fine-tuning interventions, develop initial theory (using evidence from a realist synthesis) and generate a baseline of data (through primary data collection and analysis). This phase will particularly address the study’s objective 1 and will include two broad parts:

  1. conducting realist synthesis using established processes [49, 50] to understand how responsiveness works and advance our initial working theory for its testing, validation and refining and
  2. collecting and analysing primary data using interviews, focus group discussions and community surveys, in order to understand actors’ expectations of responsive health systems, identify issues for fine-tuning the interventions in Phase 2 and establish a baseline for the interventions.

We will rigorously synthesise knowledge on systems responsiveness in LMICs using realist synthesis, a “…systematic, theory-driven interpretative technique… to help make sense of heterogeneous evidence about complex interventions applied in diverse contexts” [51 p.2]. This review approach is well suited for explaining disparate data and conceptualisations across the academic disciplines [52] and unpacking specific pathways of how particular contexts may trigger (or interfere with) mechanisms to produce intended or unintended outcomes [53]. The realist synthesis will help understand how specific micro-meso-macro contexts shape certain pathways through which responsiveness works. While there is a substantial knowledge base on assessing health systems responsiveness using adaptations of the WHO survey toolkit, published literature on improving systems responsiveness is scarce. Therefore, in addressing key issues from Phase 1 and fine-tuning the interventions (in Phase 2), we will draw on broader relevant knowledge of health systems strengthening–for example, accountability [6, 54, 55], integration of mental and maternal services [20, 22, 23] and staff support [5658]–and will relate these to our understanding of vulnerability [1417], our understanding of responsiveness in our framework and our initial theory.

Our realist synthesis will utilise a four-step process [59], which will run alongside other components of the wider RESPONSE study, and will comprise:

  1. initial screening to map theoretical landscape of health systems responsiveness,
  2. formulating initial programme theories on the basis of theorisation of health systems responsiveness
  3. refining programme theories through purposeful screening of the literature and consultations with key stakeholders as part of the intervention co-production for improving health systems responsiveness in Ghana and Vietnam
  4. testing programme theories through empirical evaluations of implemented interventions in Ghana and Vietnam and against the substantive theories underpinning health systems responsiveness.

A detailed protocol for the realist synthesis is available from the International Prospective Register of Systematic Reviews, PROSPERO registration CRD42020200353 (

The aim of collecting and analysing primary data during Phase 1, will be to answer research questions 1a and 1b (see Table 1 earlier) through exploring:

  • what responsiveness means to people across intersectionality dimensions (e.g. income, gender, social strata) and how this shapes their initial expectations from and interactions with, health systems within local contexts (e.g. through patient feedback [6, 60, 61]);
  • how responsiveness is understood and enacted at different health systems levels (facility, district, regional, national) within the bureaucracies of health systems [27] and involving interactions between policymakers, managers and service providers [62];
  • what health systems responsiveness means within preventive and curative settings of public and private healthcare facilities. We recognise, however, that public-private distinctions are increasingly blurred, with many private (not-for-profit) facilities sharing public-sector values and principles, and public-sector facilities facing market pressures.

We will collect data using: (i) reviews of policy and facility documentation including analysis of facility records of service provision and use; (ii) in-depth interviews and focus group discussions with key actors from local communities and health systems and (iii) a community survey. Data will be analysed retroductively, meaning both inductively and deductively. We will draw upon established processes for data analysis such as thematic analysis of qualitative data, and regression models for analysis of quantitative data.

We will begin with a comprehensive review of policy and facility documentation to deepen our understanding of policy, regulatory and systems environments and healthcare practices—relating all back to the conceptualisations of health systems responsiveness. We will qualitatively review two purposefully-identified types of documents: key national-level policies, plans and guidelines; and relevant local-level documents within health facilities, such as minutes of management meetings, clinical reviews, and staff management and performance appraisal records. We will adopt a semi-structured template to summarise insights on the underlying conceptualisations, values and ideas around responsiveness, and approaches to ensuring health systems responsiveness. We will also quantitatively assess patterns of provision and utilisation of maternal and mental health services. To do so, we will analyse facility records for the last 2–3 years disaggregated by key dimensions of intersectionality such as age, income and residence.

Using the IDIs, we will understand what responsiveness means to different health systems actors including its importance, underlying principles, components, mechanisms and intended outcomes. We will also explore the actors’ framing of underlying fundamental issues (e.g. rights, agency) and practices (management, service provision, health-seeking), relating them to our continuously refined understanding of responsiveness. From our experience, 20–25 IDIs should sufficiently represent views of key actor groups in each district, with further 10–15 IDIs at the province and national levels each (total 60–80 IDIs per country). This may decrease if we reach data saturation earlier, i.e. when no major new themes will be emerging from subsequent IDIs.

We will also conduct 4–6 FGDs with key actors in each country. These will explore their understanding and expectations of responsive health systems, their framing of underlying fundamental issues and will understand group norms and dynamics. The FGDs will be conducted at the community, facility and province levels (1–2 at each level). Each group will comprise 6–8 participants to maximise engagement, will comprise similar participants (in terms of age, staff category) and separate FGDs may be conducted to reduce gender-related bias.

The participants for IDIs and FGDs will be identified through purposive sampling and will include: (a) patients and public across their intersectionality dimensions (such as different genders, incomes and social strata); (b) health facility staff including service providers and support staff; and (c) health policymakers and managers. An initial participant list will be drawn by month 3 and we will identify further ones through snowballing. All FGDs and IDIs will be guided by a semi-structured topic guide to explore causal pathways of responsiveness [42]. Questions will be adapted to specific individuals’ backgrounds and roles. All IDIs and FGDs will be conducted in local languages as appropriate, audio-recorded, transcribed and either translated verbatim to English for thematic analysis or analysed in local languages with relevant extracts to be translated for cross-country comparisons.

We will conduct a baseline community survey to help us understand which attributes of responsive health systems community members value and expect most, and which drivers determine health-seeking behaviours of vulnerable groups. These attributes will be based on the WHO’s seven domains of health systems responsiveness: dignity, autonomy, confidentiality, prompt attention, quality of amenities, access to support networks, and choice of service provider [1, 8, 24]. We will explore how the values and expectations that respondents hold about these attributes, differ across key characteristics, such as users vs non-users of the health system, and across their different intersectionality dimensions, such as gender and socio-economic status. We will use insights from the preceding qualitative methods to guide the design of the survey questionnaire. We will measure how much relative importance respondents place on the different attributes of responsiveness, that is how much they value and expect each attribute, using Likert scales or discretely coded visual analogue scales. We will pilot test the questionnaire among a small number of purposively selected respondents to explore its acceptability and their understanding of a range of different scales, to allow us to select the most suitable scale. During the pilot we will also explore respondents’ understanding of the rest of the questionnaire, along with the feasibility and acceptability of the survey methodology to data collectors. We will also collect a range of key socio-demographic details from respondents on factors that are likely related to their access and use of the health system and their experiences of engaging with the health system, and which will allow us to categorise them in terms of key determinants of intersectionality like gender and socio-economic status.

Depending on the sampling frame data that is available in each country, we will use a multi-stage clustered household survey sampling approach [63] to allow us to select a statistically representative population sample from the relevant communities around our intervention facilities. Sample size: To understand respondents’ views on the importance of the different attributes of responsiveness, we will treat each discrete point on our chosen responsiveness attribute “importance scale” as a binary outcome and estimate the percentage of respondents selecting that point on the given scale (accounting for the complex survey design). We estimate that for the survey in each country we will require 562 respondents (assuming a response rate of 95%) to estimate the percentage of respondents selecting each point on the scale (assuming the most conservative percentage of 50% when estimating the sample size for a binary outcome based on precision) with a margin of error of ± 6 percentage points (95% confidence intervals), which we judged to be a suitable balance between precision and resources, and assuming a moderate design effect of 2 given we have no existing data (the mean design effect in the Ghana Demographic and Health Survey 2014 was 1.5). We will aim to understand the relationships between the socio-economic factors and respondents’ views on the relative importance of the different attributes of responsiveness, such as differences between men and women. In doing so, we will use appropriate multiple linear regression models that treat the “importance scale” outcomes as continuous variables (and which adjust for any complex survey design features such as clustering, stratification and weighting). Based on this approach, the above sample size (562) would also allow us to detect differences between binary subgroups, such as men and women (assuming a maximum between-subgroup size ratio of ≤6:1), of 0.65 or greater points in their responses to any of the “importance scale” questions with 80% power (based on two-sided hypothesis testing at the 5% significance level, assuming t-distributed data, a response rate of 95%, a design effect of 2).

The outcome of Phase 1 will be an initial theoretical model explaining how health systems responsiveness works. It will explain how different contexts shape and trigger the mechanisms through which health systems responsiveness is enacted (or not) by the different actors to produce the intended and unintended outcomes.

Phase 2: Intervention co-production.

In Phase 2 (months 13–18) we will co-produce the context-sensitive interventions to improve health systems responsiveness, addressing study objective 2. The co-production will be through 2–3 meetings in each district involving key actors (communities, service providers, facility managers, regional/province and national-level actors). These meetings themselves can also be seen as interventions, and will therefore involve elements of capacity strengthening and knowledge uptake. The meetings will be led by district (or regional) health leadership and facilitated by researchers who will present evidence from Phase 1, document causal pathways of how the interventions are intended to work (i.e. refine initial theory) and using participant observations following a semi-structured template reflect on the co-production processes in terms of clarity, inclusivity, transparency and effectiveness.

In each country, the interventions will seek to improve two key components of responsiveness:

  1. internal interactions i.e. within and across facility, district, province and national levels
  2. external interactions i.e. between the health systems and vulnerable groups.

The rationale is that these two components, when combined, will ultimately contribute to improving people’s experiences across the different domains of health systems responsiveness specified in our theoretical framework. Our focus on the processes of interaction is also driven by our intention to enact systemic change rather than target specific individual domains in a discreet manner.

The overall design of the interventions is shown in Table 4.

Table 4. Two components of the interventions in Ghana and Vietnam.

During the intervention co-production meetings we will fine-tune this general design (i.e. finalise themes, refine facilitation guidance and produce required materials), to be informed by better understanding of people’s initial expectations of responsive health systems which inform their interactions with their health systems from Phase 1.

In fine-tuning the interventions, we will consolidate, adapt and extend our work in Health Workers for Change (HWFC) workshops [58, 64], Continuous Quality Improvement (CQI) [65, 66], patient feedback systems [61, 67] and acceptability of maternal healthcare by vulnerable groups [39, 40]. Relevant published and unpublished results will be presented during intervention co-production meetings as a possible ‘menu’ of interventions for considerations by the key stakeholders.

To improve internal interactions, in each district we will conduct sets of six thematic workshops, following the HWFC approach, comprising frontline health workers and managers in the district hospital and PHC facilities. These are participatory, interactive 2-hour workshops, moderated by a skilled facilitator–usually a social scientist with experience in moderating group discussions. The workshop series stems from Paulo Freire’s work on transformational learning [68] and aims to help staff surface and critically reflect on their experiences, strengths and constraints in delivering responsive and quality healthcare. From these reflections, staff are then encouraged and supported, to develop relevant and feasible local solutions [69]. The six workshops are usually titled: “why I am a health worker”; “how do our clients see us”; “women’s status in society”; “unmet needs”; “overcoming obstacles at work” and “solutions”. The manual for the HWFC series was developed from initial work in South Africa and then refined following experiences in other African contexts. During co-production, we will align the themes with our initial theory of systems responsiveness specifically targeting the internal (and external) interactions and adapt facilitation guidance as appropriate.

Experiences of using HFWC approach in Ghana and other countries, show them as an effective platform for institutionalising a process of Continuous Quality Improvement (CQI) in healthcare facilities [58, 64]. Evidence from Vietnam also shows that similar facilitated stakeholder groups can contribute towards improved health outcomes [70]. The CQI emphasises the process of systemic change underpinned by gradual optimisation and improvement and organisational learning; and that healthcare quality needs to be satisfied for both service users and providers [65, 66]. Both CQI and HWFC have been shown to improve communication within health facilities [58, 65] and thus focus on both the people and the systems sides of health systems responsiveness.

To improve external interactions, to establish group consensus during these workshops, we will use a Nominal Group Technique (NGT)–a structured, multistep, group consensus building technique which comprises 6 steps: (a) individual writing of ideas; (b) group review and feedback; and (c) discussion, clarification and evaluation of each idea (d) ranking ideas in order of their significance; (e) discussion of the preliminary vote; and (f) final individual voting on significance of each idea.

We will draw upon the NGT’s documented effects on consensus building, including in Ghana and in Vietnam [71, 72], to improve external interactions through empowering people to engage with their health systems. To raise people’s awareness of possible options, we will utilise our knowledge of improving patient feedback (as channels for people to convey their expectations from and reflections on, systems performance [61, 67]) and improving use of maternal healthcare [39, 40]. Evidence shows that effective interventions to improve feedback systems should target all three steps in the process: collection of feedback (e.g. raising patients’ awareness), analysis within facilities (e.g. improving staff skills), and acting on the information (e.g. integrating learning into service quality improvement) [67]. We will also draw upon the knowledge that use of available healthcare by vulnerable groups requires people’s cultural acceptance of these services [39]–and which can be improved through raising awareness about, and increasing confidence in, health workers [40].

The interventions are intentionally designed as low-cost activities to be embedded within the current structures and processes to ensure their sustainability, replication and scaling up. Although the HFWC workshops and NGT intend to improve internal and external interactions respectively, each is also likely to bridge the external-internal boundaries. We also anticipate that cumulatively these two intervention components will raise awareness and empower health workers and communities, and consequently will provide a sustainable platform for problem analysis and solution seeking. Such a platform will contribute to systemic improvements in responsiveness as a key attribute of health systems strengthening.

Phase 3: Intervention implementation and evaluation.

In Phase 3 (months 19–42) we will implement and evaluate the interventions within local contexts. The interventions will be implemented for one full year and through existing systems’ structures and processes. Such duration should help embed the interventions within annual health planning and budgeting cycles and thus ensure their integration within routine practices and longer-term sustainability.

Using realist evaluation [47, 73] we will test and refine our initial theory and intended intervention pathways from Phases 1 and 2. These will be compared to the actual performance of the interventions in improving internal and external interactions. We will relate results to any changes in key domains of responsiveness from our theoretical framework (e.g. dignity), within the complex context of vulnerability and assessed against the baseline. We will repeat our community survey within the same areas and asking the same questions. See Phase 1 section for proposed details of the survey design, questionnaire topics and format, and sample size/analysis. With the baseline and follow-up community surveys we can then explore whether respondents’ views on the relative importance of different attributes of responsiveness have changed subsequent to the intervention. This will be done using appropriate multiple linear regression models that adjust for the complex survey design, and also control for likely important confounding variables.

We will also assess the interventions’ feasibility and acceptability by key actors and processes within and between Ghana and Vietnam, utilising the UK Medical Research Council’s framework for process evaluation of complex interventions [74]. In doing so, we will modify, extend and reuse the Phase 1 methods.

Our intra- and cross-country comparative analyses will allow us to develop transferable best practices for other areas and LMICs which also experience similar socio-economic growth and face pressures to effectively address the needs of vulnerable groups. These best practices will be in a form of a theory-informed and empirically-grounded model of health systems responsiveness to neglected health needs of vulnerable groups, specifically addressing the study objectives 3 and 4. The model will guide a deeper understanding of how the contexts shape and trigger specific mechanisms through which health systems responsiveness works for different actors. We will also produce detailed practical guidance notes on further adaptations of this model to inform future policy and practice on improving health systems responsiveness in LMICs.

Ethics and research governance

Ethics approvals for this study were obtained from the University of Leeds School of Medicine Research Ethics Committee (ref: MREC19-051), Ghana Health Service Ethics Review Committee (ref GHS-ERC 012/03/20) and Hanoi University of Public Health Institutional Review Board (ref 020-149/DD-YTCC). All primary data will be collected after obtaining written informed consent (or thumb print for those who can’t write) and while preserving participants’ anonymity and confidentiality.

The project will be carried out with full respect of current relevant legislation such as the General Data Protection Regulation. The methods development, data collection and analysis will consider:

  • anonymity of all study respondents will be preserved where possible and will be ensured at all times if respondent(s) request. Unnecessary collection of personal data will be avoided. Where personal data is collected it will be coded, removed from the data for analysis and stored separately from transcripts. Only PIs and designated research personnel in each organisation will have access to the keys linking the data with the personal information.
  • informed consent will be obtained from all study participants and in the case of refusal, alternative means of data collection will be explored (e.g. alternative respondents)
  • specific emphasis will be placed on confidentiality and other data protection issues, which will include secure data storage and clear access rights. Only members of teams identified by the PIs in each institution will have access. Where data (e.g. transcripts) is stored on a server it will be password-protected and only research teams will have access to the passwords. Availability of documents on the internet will be following the consent of all collaborators.

The project will be implemented according to established robust research governance practice standards at the University of Leeds for implementing collaborative projects. This includes ensuring: regular communication between the partners and engagement with policymakers and practitioners; quality assurance through regular peer-review both within and between the teams; appropriate mentoring and coaching support to early career researchers; and equal opportunities as part of the Leeds University’s commitment to the Athena Swan equal opportunities initiative.

Communication of project results

We will embed the research into policy and practice working with facility, district, regional and national actors, and through extending our academic collaborations into South-South research and policy partnerships. Engagements of decision-makers will facilitate implementation and scaling-up of these and similar interventions within and across Ghana and Vietnam.

We will communicate research findings through combinations of:

  1. developing policy briefs addressed to national and international policymakers and practitioners and designed as short and practical documents;
  2. delivering presentations at review meetings at local, district, regional/province and national levels in Ghana and Vietnam (e.g. quarterly, semi-annual and annual reviews involving national and international policy actors) and relevant regional meetings;
  3. regular (e.g. quarterly) project review meetings with national, regional and district actors and continuous engagement with key decision-makers through for example sharing draft policy implications for discussion of feasibility before including them in official reports;
  4. developing newsletters and press-releases aimed at communicating the key project findings in simple ways accessible to the general public in each country and their respective regions;
  5. interviews in the national media (e.g. radio and television) as well as articles for national newspapers, communicating our findings and educating the public on effective engagements and ensuring health systems responsiveness as appropriate and feasible;
  6. creating a website where all materials will be easily accessible by (inter)national decision-makers and practitioners, with mutual links to GHS and MOH websites in Ghana and Vietnam
  7. delivering presentations at national, regional and international conferences and publication of articles in peer-reviewed journals with specific emphasis on open access where feasible
  8. developing project reports for key stakeholders, with a publishable executive summary.

There is a high interest from key policymakers in this topic, some of whom are the members of the project team. We will maintain equal research-policy partnerships [75] and will embed research into policy and practice to facilitate integrating interventions within district health systems and ensure their sustainability.


This paper summarised the protocol for a mixed methods realist evaluation study which aims to understand and improve health systems responsiveness to health needs of vulnerable groups in Ghana and Vietnam.

A key study’s theoretical outcome will be an empirically-grounded and theory-informed model of complex contexts-mechanisms-outcomes relations, together with transferable best practices for scalability (i.e. expanding within similar contexts) and generalisability (i.e. expanding to other contexts e.g. other health areas and countries) for future health systems strengthening. The model will advance the current understanding of health systems responsiveness and will form an overall theoretical contribution from this study to the published knowledge on the subject.

Decision-makers from facility, district, regional and national levels will be engaged through embedded research in policy and practice and equal research-policy partnerships [75]. This will ensure the ownership and buy-in from the key stakeholders throughout the study and sustainability of project impacts in the longer-term. The economic and societal impacts will be generated through improvements in: health systems responsiveness, health policy implementation and health service provision, health systems performance and ultimately health outcomes and equity.

Capacity strengthening is a distinct project objective and is therefore an important aspect of this study. We will deploy a systematic approach to capacity strengthening which will be underpinned by our in-depth understanding of current capacity needs and assets of each partner team from previous studies [38, 40, 61, 76], and will comprehensively target individual, organisational and system levels of capacity. At the individual level we will seek to improve knowledge and skills of early-career researchers particularly in Ghana and Vietnam through mentoring support and on-the-job training (e.g. to lead academic publications). We also aim to improve individual capacities of key decision-makers to effectively set research priorities and understand and apply research results in their routine practices, through their continuous engagements in the study. At the organisational level, we will improve partners’ processes for research governance (e.g. finance management, reporting) and their ability to effectively communicate results. Systems level work will target partners’ networks including strengthening and expanding existing research-policy links in Ghana and Vietnam. During the first three months of the project, we will consolidate our in-depth knowledge of partners’ current capacities and will develop a clear, coherent and feasible capacity strengthening plan. The South-South exchange and learning will be central to our capacity strengthening approach with collaborators in Ghana and Vietnam owning and driving capacity strengthening. Senior co-Is in Ghana and Vietnam will mentor early-career researchers and create organisational environment where researchers’ enhanced expertise can be effectively applied.


The authors would like to thank all key stakeholders in Ghana and Vietnam for preliminary discussions and expressing interest in engaging with this study. All views are of the authors only.


  1. 1. de Silva A. A framework for measuring responsiveness. GPE Discussion Paper series No 32. Geneva: World Health Organization; 2000.
  2. 2. Mirzoev T, Kane S. What is health systems responsiveness? Review of existing knowledge and proposed conceptual framework. BMJ Global Health. 2017;2(4):e000486. pmid:29225953
  3. 3. Agyepong IA, Kwamie A, Frimpong E, Defor S, Ibrahim A, Aryeetey GC, et al. Spanning maternal, newborn and child health (MNCH) and health systems research boundaries: conducive and limiting health systems factors to improving MNCH outcomes in West Africa. Health research policy and systems. 2017;15(Suppl 1):54–. pmid:28722556.
  4. 4. World Bank. World Development Indicators Washington: The World Bank; [cited 2019 9 May]. Available from:
  5. 5. Robone S, Rice N, Smith PC. Health Systems' Responsiveness and Its Characteristics: A Cross-Country Comparative Analysis. Health Serv Res. 2011;46(6pt2):2079–100. pmid:21762144
  6. 6. Lodenstein E, Dieleman M, Gerretsen B, Broerse J. A realist synthesis of the effect of social accountability interventions on health service providers' and policymakers' responsiveness. Systematic Reviews. 2013;2(1):98. pmid:24199936
  7. 7. Valentine N, Prasad A, Rice N, Robone S, S C. Health Systems Responsiveness–A Measure of the Acceptability of Health Care Processes and Systems. In: Smith P, Mossialos E, S L, editors. Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. London: WHO European Regional Office; 2009. p. pp. 138–86.
  8. 8. Darby C, Valentine N, Murray CJL, De Silva A. World Health Organization: Strategy on measuring responsiveness. The Journal of Medicine and Philosophy. 2000;39(1):2.
  9. 9. WHO. Everybody business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: World Health Organization; 2007.
  10. 10. van Olmen J, Marchal B, Van Damme W, Kegels G, Hill P. Health systems frameworks in their political context: framing divergent agendas. BMC Public Health. 2012;12(1):774. pmid:22971107
  11. 11. Lodenstein E, Dieleman M, Gerretsen B, Broerse JEW. Health provider responsiveness to social accountability initiatives in low- and middle-income countries: a realist review. Health Policy Plan. 2016;32(1):125–40. pmid:27375128
  12. 12. Chambers R. Vulnerability, Coping and Policy—Introduction. IDS Bulletin. 1989;20(2):1–7.
  13. 13. UNDP. Human Development Report 2016 Human Development for Everyone: UNDP; 2016.
  14. 14. Klode K, Ringer A, Hølge-Hazelton B. Interprofessional and intersectoral collaboration in the care of vulnerable pregnant women: An interpretive study. Journal of Interprofessional Care. 2020:1–10. pmid:32436756
  15. 15. Luna F. Elucidating the Concept of Vulnerability: Layers Not Labels. International Journal of Feminist Approaches to Bioethics. 2009;2(1):121–39.
  16. 16. Krubiner CB, Faden RR. Pregnant women should not be categorised as a 'vulnerable population' in biomedical research studies: ending a vicious cycle of 'vulnerability'. J Med Ethics. 2017;43(10):664–5. Epub 2017/07/19. pmid:28716976.
  17. 17. Den Hollander GC, Browne JL, Arhinful D, van der Graaf R, Klipstein-Grobusch K. Power Difference and Risk Perception: Mapping Vulnerability within the Decision Process of Pregnant Women towards Clinical Trial Participation in an Urban Middle-Income Setting. Developing World Bioethics. 2018;18(2):68–75. Epub 2016/10/21. pmid:27761986; PubMed Central PMCID: PMC6001438.
  18. 18. Adua E, Frimpong K, Li X, Wang W. Emerging issues in public health: a perspective on Ghana's healthcare expenditure, policies and outcomes. The EPMA journal. 2017;8(3):197–206. pmid:29021830.
  19. 19. Chuong NC, Van Minh H, Thi Thuy Duong D, Duc DM, Anh Dao LT, Duy Anh N. Progress on maternal health care in Vietnam: Findings from Vietnam Multiple Indicator Cluster Survey 2000, 2006, 2011, and 2014. Health Care Women Int. 2018;39(4):368–76. pmid:29161191
  20. 20. Cheung FK, Snowden LR. Community mental health and ethnic minority populations. Community Ment Health J. 1990;26(3):277–91. pmid:2354624
  21. 21. Gilson L, Lehmann U, Schneider H. Practicing governance towards equity in health systems: LMIC perspectives and experience. International Journal for Equity in Health. 2017;16(1):171. pmid:28911320
  22. 22. National Collaborating Centre for Mental Health. NICE: Guidance. Antenatal and Postnatal Mental Health: The NICE Guideline on Clinical Management and Service Guidance. Leicester (UK): British Psychological Society. The British Psychological Society & The Royal College of Psychiatrists.; 2007.
  23. 23. Bagadia A, Chandra PS. Starting the conversation—Integrating mental health into maternal health care in India. The Indian journal of medical research. 2017;145(3):267–9. pmid:28749389.
  24. 24. Letkovicova H, Prasad A, Vallée RL, Valentine N. The health systems responsiveness analytical guidelines for surveys in the multi-country survey study. Geneva: World Health Organization; 2005.
  25. 25. Valentine N, Darby C, Bonsel GJ. Which aspects of non-clinical quality of care are most important? Results from WHO's general population surveys of “health systems responsiveness” in 41 countries. Social Science & Medicine. 2008;66(9):1939–50.
  26. 26. Njeru MK, Blystad A, Nyamongo IK, Fylkesnes K. A critical assessment of the WHO responsiveness tool: lessons from voluntary HIV testing and counselling services in Kenya. BMC Health Services Research. 2009;9(1):1–11. pmid:20028540
  27. 27. Cleary S, Molyneux S, Gilson L. Resources, attitudes and culture: an understanding of the factors that influence the functioning of accountability mechanisms in primary health care settings. BMC Health Services Research. 2013;13(1):320. pmid:23953492
  28. 28. Berlan D, Shiffman J. Holding health providers in developing countries accountable to consumers: a synthesis of relevant scholarship. Health Policy Plan. 2012;27(4):271–80. pmid:21551121
  29. 29. Valentine NB, Bonsel GJ. Exploring models for the roles of health systems’ responsiveness and social determinants in explaining universal health coverage and health outcomes. Global Health Action. 2016;9:10.3402/gha.v9.29329. PMC4778385. pmid:26942516
  30. 30. Forouzan AS, Ghazinour M, Dejman M, Rafeiey H, San Sebastian M. Testing the WHO responsiveness concept in the Iranian mental healthcare system: a qualitative study of service users. BMC Health Serv Res. 2011;11:325. Epub 2011/11/26. pmid:22115499; PubMed Central PMCID: PMC3280196.
  31. 31. Rottger J, Blumel M, Fuchs S, Busse R. Assessing the responsiveness of chronic disease care—is the World Health Organization's concept of health system responsiveness applicable? Soc Sci Med. 2014;113:87–94. Epub 2014/05/24. pmid:24852659.
  32. 32. Fenny AP, Asuman D, Crentsil AO, Odame DNA. Trends and causes of socioeconomic inequalities in maternal healthcare in Ghana, 2003–2014. Int J of Social Economics. 2019;46(2):288–308.
  33. 33. WHO. Global Health Expenditure Database [cited 2019 9 May]. Available from:
  34. 34. WHO. Global Health Observatory Geneva: World Health Organization; [cited 2019 9 May]. Available from:
  35. 35. MOH. National Health Policy Creating Wealth through Health. Accra, Ghana: MOH; 2007.
  36. 36. MOH. National Strategy of Reproductive Health Care. Hanoi, Vietnam: Ministry of Health; 2011.
  37. 37. Minas H, Edington C, La N, Kakuma R. Mental Health in Vietnam. In: Minas H, Lewis M, editors. Mental Health in Asia and the Pacific: Historical and Cultural Perspectives. Boston, MA: Springer US; 2017. p. 145–61.
  38. 38. Agyepong IA, Anniah K, Aikins M, Akweongo P, Esena R, Mirzoev T. Health Policy, Health Systems Research and Analysis Capacity Assessment of the School of Public Health, University of Ghana. Ghana Med J. 2015;49(3):200–13. PMC4676591. pmid:26693197
  39. 39. Doan DTT, Bui HTT, Le TM, Duong DM, Luu TH, Dinh TA, et al. Utilization of services provided by village based ethnic minority midwives in mountainous villages of Vietnam. International Journal of Women's Health. 2016;8:571–80. pmid:27799825
  40. 40. Doan DTT, Mirzoev T, Nguyen CC, Bui HTT. Utilization of Services Provided by Village-Based Ethnic Minority Midwives in Vietnam: Lessons From Implementation Research. J Public Health Manag Pract. 2018;24:S9–S18. 00124784-201803001-00003. pmid:29369252
  41. 41. Mirzoev T, Etiaba E, Ebenso B, Uzochukwu B, Manzano A, Onwujekwe O, et al. Study protocol: realist evaluation of effectiveness and sustainability of a community health workers programme in improving maternal and child health in Nigeria. Implementation Science. 2016;11(1):1–11. pmid:27268006
  42. 42. Manzano A. The craft of interviewing in realist evaluation. Evaluation. 2016;22(3):342–60.
  43. 43. Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Medicine. 2016;14(1):96. pmid:27342217
  44. 44. Kane SS, Gerretsen B, Scherpbier R, Dal Poz M, Dieleman M. A realist synthesis of randomised control trials involving use of community health workers for delivering child health interventions in low and middle income countries. BMC Health Services Research. 2010;10:286–. PMC2964693. pmid:20942900
  45. 45. Ebenso B, Mbachu C, Etiaba E, Huss R, Manzano A, Onwujekwe O, et al. Which mechanisms explain the motivation of primary health workers? Insights from the realist evaluation of a maternal and child health programme in Nigeria. BMJ Global Health. 2020;5(8):e002408. pmid:32843524
  46. 46. Mirzoev T, Etiaba E, Ebenso B, Uzochukwu B, Ensor T, Onwujekwe O, et al. Tracing theories in realist evaluations of large-scale health programmes in low- and middle-income countries: experience from Nigeria. Health Policy Plan. 2020.
  47. 47. Pawson R, Tilley N. Realistic Evaluation: London: SAGE Publications; 1997.
  48. 48. Stame N. Theory-Based Evaluation and Types of Complexity. Evaluation. 2004;10(1):58–76.
  49. 49. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. ESRC Research Methods Programme, Working Paper Series. 2004.
  50. 50. Rycroft-Malone J, McCormack B, Hutchinson AM, DeCorby K, Bucknall TK, Kent B, et al. Realist synthesis: illustrating the method for implementation research. Implementation Science. 2012;7(1):33. pmid:22515663
  51. 51. Greenhalgh T, Wong G, Westhorp G, Pawson R. Protocol—realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES). BMC Medical Research Methodology. 2011;11(1):115. pmid:21843376
  52. 52. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005;61(2):417–30. Epub 2005/05/17. pmid:15893056.
  53. 53. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34. Epub 2005/08/02. pmid:16053581.
  54. 54. Bukenya B, Hickey S, King S. Understanding the Role of Context in Shaping Social Accountability Interventions: Toward an Evidence-Based Approach. Washington DC: Social Accountability and Demand for Good Governance Team Report, World Bank; 2012.
  55. 55. George AS, Erchick DJ, Zubairu MM, Barau IY, Wonodi C. Sparking, supporting and steering change: grounding an accountability framework with viewpoints from Nigerian routine immunization and primary health care government officials. Health Policy Plan. 2016;31(9):1326–32. pmid:27198980
  56. 56. Ballard M, Montgomery P. Systematic review of interventions for improving the performance of community health workers in low-income and middle-income countries. BMJ Open. 2017;7(10). pmid:29074507
  57. 57. Cometto G, Ford N, Pfaffman-Zambruni J, Akl EA, Lehmann U, McPake B, et al. Health policy and system support to optimise community health worker programmes: an abridged WHO guideline. The Lancet Global Health. 2018. pmid:30430994
  58. 58. Onyango-Ouma W, Laisser R, Mbilima M, Araoye M, Pittman P, Agyepong I, et al. An evaluation of Health Workers for Change in seven settings: a useful management and health system development tool. Health Policy Plan. 2001;16 Suppl 1:24–32. Epub 2001/10/16. pmid:11599666.
  59. 59. Cooper C, Lhussier M, Shucksmith J, Carr SM. Protocol for a realist review of complex interventions to prevent adolescents from engaging in multiple risk behaviours. BMJ Open. 2017;7(9):e015477. pmid:28939568
  60. 60. Jiang Y, Ying X, Zhang Q, Tang SR, Kane S, Mukhopadhyay M, et al. Managing patient complaints in China: a qualitative study in Shanghai. BMJ Open. 2014;4(8). pmid:25146715
  61. 61. Ha BTT, Mirzoev T, Morgan R. Patient complaints in healthcare services in Vietnam’s health system. SAGE Open Medicine. 2015;3. pmid:26770804
  62. 62. Brinkerhoff DW, Bossert TJ. Health governance: principal–agent linkages and health system strengthening. Health Policy Plan. 2014;29(6):685–93. pmid:23411121
  63. 63. ICF International. Demographic and Health Survey Sampling and Household Listing Manual. MEASURE DHS. Calverton, Maryland, USA: ICF International; 2012.
  64. 64. Agyepong IA, Anafi P, Asiamah E, Ansah EK, Ashon DA, Narh-Dometey C. Health worker (internal customer) satisfaction and motivation in the public sector in Ghana. Int J Health Plann Manage. 2004;19(4):319–36. Epub 2005/02/04. pmid:15688876.
  65. 65. Agyepong IA, Sollecito WA, Adjei S, Veney JE. Continuous quality improvement in public health in Ghana: CQI as a model for primary health care management and delivery. Qual Manag Health Care. 2001;9(4):1–10. Epub 2001/08/14. pmid:11499347.
  66. 66. Agyepong I, Sollectio WA, McLaughlin CP. CQI for Malaria Control in Ghana. In: McLaughlin CP, Johnson JK, Sollecito WA, editors. Implementing Continous Quality Improvement in Health Care A Global Case Book: Jones and Bartlett Learning; 2012.
  67. 67. Mirzoev T, Kane S. Key strategies to improve systems for managing patient complaints within health facilities–what can we learn from the existing literature? Global Health Action. 2018;11(1):1458938. pmid:29658393
  68. 68. Haaland A, Vlassoff C. Introducing Health Workers for Change: from transformation theory to health systems in developing countries. Health Policy Plan. 2001;16(suppl_1):1–6. pmid:11599663
  69. 69. Fonn S, Xaba M. Health Workers for Change: developing the initiative. Health Policy Plan. 2001;16(suppl_1):13–8. pmid:11599664
  70. 70. Persson LÅ, Nga NT, Målqvist M, Thi Phuong Hoa D, Eriksson L, Wallin L, et al. Effect of Facilitation of Local Maternal-and-Newborn Stakeholder Groups on Neonatal Mortality: Cluster-Randomized Controlled Trial. PLOS Medicine. 2013;10(5):e1001445. pmid:23690755
  71. 71. Bates I, Ansong D, Bedu-Addo G, Agbenyega T, Akoto AYO, Nsiah-Asare A, et al. Evaluation of a learner-designed course for teaching health research skills in Ghana. BMC Medical Education. 2007;7(1):18. pmid:17596260
  72. 72. Dang VH. The Use of Nominal Group Technique: Case Study in Vietnam. World Journal of Education. 2015;5(4):11–25.
  73. 73. Wong J, Greenhalgh T, Pawson R, Westhorp G, Greenhalgh J, Manzano A, et al. Protocols and realist evaluation: The RAMESES II Project; 2016.
  74. 74. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ: British Medical Journal. 2015;350:h1258. pmid:25791983
  75. 75. Mirzoev T, Omar M, Green A, Bird P, Lund C, Ofori-Atta A, et al. Research-policy partnerships—experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. Health Research Policy and Systems. 2012;10(1):30. pmid:22978604
  76. 76. Mirzoev T, Lê G, Green A, Orgill M, Komba A, Esena RK, et al. Assessment of capacity for Health Policy and Systems Research and Analysis in seven African universities: results from the CHEPSAA project. Health Policy Plan. 2014;29(7):831–41. pmid:24038107