Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Association between a complex community intervention and quality of health extension workers’ performance to correctly classify common childhood illnesses in four regions of Ethiopia

  • Theodros Getachew ,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing

    tedi.getachew@yahoo.com

    Affiliations Health System and Reproductive Health Research Directorate, Ethiopian Public Health Institute, Addis Ababa, Ethiopia, College of Medicine and Health Sciences, Institute of Public Health, University of Gondar, Gondar, Ethiopia

  • Solomon Mekonnen Abebe,

    Roles Conceptualization, Methodology, Supervision, Validation, Writing – review & editing

    Affiliation College of Medicine and Health Sciences, Institute of Public Health, University of Gondar, Gondar, Ethiopia

  • Mezgebu Yitayal,

    Roles Conceptualization, Methodology, Supervision, Validation, Writing – review & editing

    Affiliation College of Medicine and Health Sciences, Institute of Public Health, University of Gondar, Gondar, Ethiopia

  • Lars Åke Persson,

    Roles Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing

    Affiliations Health System and Reproductive Health Research Directorate, Ethiopian Public Health Institute, Addis Ababa, Ethiopia, Department of Disease Control, London School of Hygiene & Tropical Medicine, London, United Kingdom

  • Della Berhanu

    Roles Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing – review & editing

    Affiliations Health System and Reproductive Health Research Directorate, Ethiopian Public Health Institute, Addis Ababa, Ethiopia, Department of Disease Control, London School of Hygiene & Tropical Medicine, London, United Kingdom

Association between a complex community intervention and quality of health extension workers’ performance to correctly classify common childhood illnesses in four regions of Ethiopia

  • Theodros Getachew, 
  • Solomon Mekonnen Abebe, 
  • Mezgebu Yitayal, 
  • Lars Åke Persson, 
  • Della Berhanu
PLOS
x

Abstract

Background

Due to low care utilization, a complex intervention was done for two years to optimize the Ethiopian Health Extension Program. Improved quality of the integrated community case management services was an intermediate outcome of this intervention through community education and mobilization, capacity building of health workers, and strengthening of district ownership and accountability of sick child services. We evaluated the association between the intervention and the health extension workers’ ability to correctly classify common childhood illnesses in four regions of Ethiopia.

Methods

Baseline and endline assessments were done in 2016 and 2018 in intervention and comparison areas in four regions of Ethiopia. Ill children aged 2 to 59 months were mobilized to visit health posts for an assessment that was followed by re-examination. We analyzed sensitivity, specificity, and difference-in-difference of correct classification with multilevel mixed logistic regression in intervention and comparison areas at baseline and endline.

Results

Health extensions workers’ consultations with ill children were observed in intervention (n = 710) and comparison areas (n = 615). At baseline, re-examination of the children showed that in intervention areas, health extension workers’ sensitivity for fever or malaria was 54%, 68% for respiratory infections, 90% for diarrheal diseases, and 34% for malnutrition. At endline, it was 40% for fever or malaria, 49% for respiratory infections, 85% for diarrheal diseases, and 48% for malnutrition. Specificity was higher (89–100%) for all childhood illnesses. Difference-in-differences was 6% for correct classification of fever or malaria [aOR = 1.45 95% CI: 0.81–2.60], 4% for respiratory tract infection [aOR = 1.49 95% CI: 0.81–2.74], and 5% for diarrheal diseases [aOR = 1.74 95% CI: 0.77–3.92].

Conclusion

This study revealed that the Optimization of Health Extension Program intervention, which included training, supportive supervision, and performance reviews of health extension workers, was not associated with an improved classification of childhood illnesses by these Ethiopian primary health care workers.

Trial registration

ISRCTN12040912, http://www.isrctn.com/ISRCTN12040912.

Background

Respiratory infections, diarrheal diseases, and neonatal illnesses are major causes of mortality in children below the age of five years in Sub-Saharan Africa, including Ethiopia [1]. The correct classification of these illnesses is a prerequisite for further reduction of mortality [2]. However, insufficient quality of the care provided at the primary care level is a major problem in low- and middle-income countries [3]. We have previously shown that the Ethiopian health extension workers did not satisfactorily classify these common childhood diseases [4].

The integrated Community Case Management is a strategy to reduce childhood mortality and morbidity [5]. In Ethiopia, these services are delivered through a community-based health care delivery system at health posts called the Health Extension Program [6]. The programme aims at improving access to evidence-based classification, management and treatment of sick children.

In order to reach the Sustainable Development Goals by 2030, the rate of reduction in under-five mortality needs to be accelerated through rapid scale-up of effective interventions [7]. Community-based interventions have the potential to reduce morbidity and mortality in children under the age of five years [8]. Performance reviews and mentoring of primary health care workers has been effective in improving the clinical management of childhood illnesses [9]. Regular supportive supervision, in-service training, and community ownerships are also interventions that potentially could improve the performance of community health workers [10]. In addition, ascertaining necessary equipment and supplies are needed for the provision of high-quality services at the primary health care level [11].

Some studies have found that training and supervision did not improve the quality of care provided [12]. Different approaches to supervision and support to community health workers have been tried. There is no consensus regarding who should provide the supervision, where it should be done, at what frequency, with what content, and with what tools [13]. Supervision approaches should be context-specific [14]. Further operational research has been recommended to increase the understanding how to sustain improvements in performance over time [15].

Due to a low level of care seeking for childhood illnesses [1618], the Ethiopian Ministry of Health aimed to optimize the health extension program with a complex intervention to increase the utilization of health care services for children below the age of five years. This intervention had three components: community engagement activities that aimed to increase the utilization of services, capacity building and support to the health extension workers that intended to increase the quality of services, and activities at the district health system level, which aimed at increasing the ownership and accountability of child health services.

Although the primary outcome of this intervention was utilization of child health services, the second component included activities that potentially could improve the quality of the primary care of sick children. Hence, the objective of this study was to analyse the association between the intervention and the health extension workers’ correct classification of common childhood illnesses in intervention and comparison areas at baseline and endline surveys in the four regions of Ethiopia, where the study was implemented.

Methods

Study design and area

The assessment was based on a pragmatic design with 26 intervention districts and 26 comparison districts with baseline and end-line surveys performed before and after the intervention. In spite of not being a proper randomized trial, the study was registered in a trial registry for transparency reasons (http://www.isrctn.com/ISRCTN12040912). The study was implemented in four regions (Amhara, Tigray, Oromia, Southern Nations, Nationalities and Peoples) of Ethiopia. The baseline survey was conducted from December 2016 to February 2017, and the endline survey was done in the same season after two years, i.e., from December 2018 to February 2019.

Participants

A total of 200 enumeration areas had been selected to represent the districts in the four regions. A stratified cluster sampling was used to select the enumeration areas. The sampling frame was based on the population and housing census conducted for Ethiopia in 2007 [19]. Enumeration Areas (EAs) were considered as the Primary Sampling Units (PSUs) for this survey. An enumeration area had on average 150–200 households [19]. Health posts serving these areas were included in the survey. In each health post’s catchment area mobilization was done to recruit sick children from 2 to 59 months to seek care at the health post. The study aimed to assess four children per health posts, who were considered ill by their caretakers and brought to the health post for assessment, disease classification, and management by the health extension workers.

Interventions

The intervention, labelled Optimizing the Health Extension Program, was a collaborative effort to increase the utilization of child health services. To achieve this goal the project implemented three interlinked strategies, i.e., community engagement, capacity building of health extension workers, and strengthening the district health administration’s ownership and accountability to the child health service program. The intervention was based on evidence generated through a rigorous barrier analysis and literature review. The intervention was implemented with support from four non-governmental organizations (PATH, UNICEF, Save the Children, and Last 10 Kilometres/John Snow Incorporated). The intervention had a preparatory phase in the second half of 2016. From the start of 2017 to October 2018, it was fully implemented, although in eight districts implementation activities were ceased between December 2017 and April 2018 for administrative reasons.

The intervention activities that composed the capacity building component, which may have an effect on quality of care, were gap-filling training on the integrated community case management of childhood illnesses, provision of registers, job aids and program toolkits, supportive supervision of health extension workers, performance reviews and mentorship meetings for health extension workers at the district level, and training of trainers for these health workers on community-based data for decision making. The theory of change for this intervention is illustrated in a S1 Table.

Illness classification/outcome

The outcome variable in this analysis was disease classification, measured based on the World Health Organization health facility survey tools [20] that had been pilot-tested and customized to the local context. The assessment tool comprised of an observation of the health extension workers’ assessment and classification of sick children and re-examination of these children by health officers. The health officer re-examined the ill child and classified the illnesses, i.e., fever or malaria, acute respiratory tract infection, diarrhoea or dysentery, malnutrition, or ear infection according to the integrated community case management chart booklet, which displayed an integrated decision-making algorithm [21].

Sampling

The sample size was based on the requirement that the surveys should measure changes of a fixed number of percentage points between intervention and comparison areas at baseline and endline. A sample size of 400 per group, 800 in total, would have 80% power to detect a difference of at least 15 percentage points of correct classification of illnesses as statistically significant when comparing the health extension workers’ and the re-examiners’ classifications.

Assignment method

The intervention districts had been selected based on a low coverage of key maternal, newborn and child health indicators. The Regional Health Bureaus in collaboration with the study team selected similar comparison districts. Selection criteria included burden of diseases, size of the population, number of primary health care units, health service coverage data and duration of service delivery to sick children and newborns. Absence of organizations that were implementing other demand generation activities was also a selection criterion. In addition, rural districts were considered and urban administration districts were excluded, since these neither had health posts nor implementation of the integrated community case management of childhood illnesses.

Blinding

The health extension worker uses the health post for her regular services to sick children. The health extension workers and the implementing partners were naturally not blinded to the intervention. In contrast, the field workers and health officers performing the baseline and endline surveys were blinded to the status of intervention and comparison study areas.

Data management and analyses

Data cleaning included the checking of ranges of key variables, and checks for internal consistency. Data collection was done on Computer Assisted Field Editing, which is a data capturing approach using paper questionnaire then entering the data to a computer tablet while the data collectors were in the field using the Census and Survey Processing System (CSPro) software (US Census Bureau, Suitland, Maryland, USA). Data were transferred using Internet File Streaming System on a daily basis. Data analyses were done using STATA v14.1 (Stata Corp LP, College Station, TX, USA).

The classifications done by the health extension workers and the re-examiners were cross-tabulated to compute sensitivity and specificity [22]. Difference-in-Difference (DID) analysis was used to estimate the effect of the intervention on the classification of childhood illnesses over time between intervention and comparison groups. In this quasi-experimental design, we used baseline and endline data from intervention and comparison groups to obtain an appropriate counterfactual estimate of a plausible causal effect. Difference-in-Differences was represented by the interaction term between time and intervention group dummy variables in a Difference-in-Differences regression model [23]. Multilevel mixed logistic regression was used to analyze if differences-in-difference of correct classification of childhood illnesses were associated with the interaction term between time and intervention group in the regression model. The fixed effects, which is a measure of association, and random effects that is a measure of variation for correct disease classification, were determined by considering the clustering effect at the health posts and health extension workers as the level of variation at 0.05 significant level. In the analysis, we controlled the child age as a covariate that could affect correct classification.

Where

  • Y: health extension workers’ ability to correctly classify common childhood illnesses
  • β₁: Average change in y from the baseline to the endline time period that is common to both groups.
  • β₂: Average difference in y between the two groups that is common in both time periods.
  • β₃: Average differential change in y from the baseline to the endline time period of the intervention group relative to the comparison group.
  • β4: child age considered as a covariate that was controlled in the model.

In order to analyze the health extension workers’ ability to correctly classify common childhood illnesses in the Difference-in-Differences analysis, we generated a new variable, coded as 1, if there was agreement between the health extension worker’s and the re-examiner’s classification of the child’s illness, and coded as 0, if these disagreed.

Ethical considerations

Ethical approvals were obtained from the University of Gondar (Ref O/V/P/RCS/05/371/2018), the Ethiopian Public Health Institute (Ref 613/52) and the London School of Hygiene and Tropical Medicine (Ref 16117). Information sheets of the surveys were translated into the local languages Amharic, Oromiffaa, and Tigrigna and read to caregivers to obtain written informed consent. Participants with an acute illness that could not be managed at the health post were referred to a higher-level health facility. Re-examiners informed the health extension workers of any missing diagnoses in need of treatment for immediate additional action. This trial was registered once the optimization of Health Extension program intervention had started but prior to the end of the program and evaluation.

Results

Participant flow diagram

The study flow chart is presented in Fig 1. Protocol deviations from study as planned was observed. Out of a total of 200 eligible health posts, six were excluded from the intervention at baseline due to security reasons and therefore not included in the endline survey. At endline, additional four and ten health posts were excluded in intervention and comparison areas, respectively, also due to security reasons. A total of 1,325 sick child consultations were observed and re-examined in intervention (n = 710) and comparison areas (n = 615) (Fig 1).

thumbnail
Fig 1. Study flow diagram of intervention and comparison areas by baseline and endline data collection period.

https://doi.org/10.1371/journal.pone.0247474.g001

Characteristics of sick under-five children

Table 1 depicts the characteristics of sick children that were assessed for common childhood illnesses in intervention and comparison areas. The characteristics of intervention and comparison districts at baseline and endline surveys were similar. A majority (75%) of the sick children in the intervention and comparison areas were from Amhara and Oromia regions. Two-thirds of the sick children were in the age group 2–23 months in the intervention as well as in the comparison areas. Among the complaints presented, cough and difficult breathing dominated in both arms.

thumbnail
Table 1. Characteristics of sick 2–59 months old children at health posts who were offered integrated community case management services at baseline (December 2016- February 2017) and endline surveys (December 2018-February 2019) in intervention and comparison areas.

https://doi.org/10.1371/journal.pone.0247474.t001

Disease classification

Fig 2 shows the health extension workers’ ability to correctly classify common childhood illnesses (sensitivity) at baseline and endline surveys. The baseline assessment indicated that the health extension workers were poor at classifying the diseases in the intervention as well as in the comparison areas, except for diarrheal diseases, which had a sensitivity of around 90% in both arms. Similarly, the health extension workers poorly classified the diseases at the endline assessment, except for diarrheal diseases that had a sensitivity of 85% (Fig 2).

thumbnail
Fig 2. Sensitivity of the health extension workers’ disease classification among 2–59 months of age children at baseline (December 2016- February 2017) and endline surveys (December 2018-February 2019) in intervention and comparison areas.

https://doi.org/10.1371/journal.pone.0247474.g002

Fig 3 shows the health extension workers’ ability to correctly classify the healthy child (specificity) at the baseline and endline. Consequently, the baseline and endline assessment imply that these health workers were good at classifying the healthy child in the intervention as well as comparison areas (Fig 3).

thumbnail
Fig 3. Specificity of the health extension workers’ disease classification among 2–59 months of age children in baseline (December 2016- February 2017) and endline surveys (December 2018-February 2019) in intervention and comparison areas.

https://doi.org/10.1371/journal.pone.0247474.g003

Table 2 displays difference-in-differences analyses of correct classification of illnesses in intervention and comparison areas at baseline and endline surveys. Accordingly, there was no change in the correct classification of any of the childhood illnesses that was associated with the intervention.

thumbnail
Table 2. Correct classification of childhood illnesses among children 2–59 months of age in baseline (December 2016- February 2017) and endline surveys (December 2018-February 2019) in intervention and comparison areas.

https://doi.org/10.1371/journal.pone.0247474.t002

In addition, we present the fixed effects and random effects for correct classification of childhood illnesses. The Intraclass Correlation Coefficient (ICC) estimated in the model ranged from 5.4% to 16.2%, indicating that the differences in the disease classification could be attributed to the health extension workers and health posts level.

Discussion

We found that there was no association between the Optimizing of Health Extension Program intervention and the health extension workers’ ability to correctly classify sick children. The poor ability to correctly classify sick children noted at the baseline survey was not changed. An exception was diarrheal diseases, which were appropriately diagnosed at baseline as well as at endline surveys. The specificity of the classification, i.e., the ability to identify healthy children, was good at both times.

The sampling of health posts included in this study was based on a pre-planned intervention activity. Thus, the sample was not selected to represent the regions, but we have no reasons to believe that these health posts and health extension workers were different from the average of these regions. The selection of intervention and comparison districts provided similar study population characteristics at baseline, except for the age category. This difference was considered in the analysis. The characteristics of the sick under-five children were also similar at baseline. The re-examiners of the sick children, the health officers, were well trained in the integrated community case management algorithms, and re-trained as part of study preparations. Still, they may theoretically have over-estimated classification of some sick children. However, a study conducted in Ethiopia indicated that the health officers had adequate and higher knowledge than midwifes and nurses in child health services [24]. Therefore, we believe that the health officers provided an adequate comparison in relation to the standard integrated community case management classification algorithm.

This study has some limitations that should be considered. The health extension workers might have altered their behaviour when being observed, a phenomenon labelled the Hawthorne effect [25, 26]. This potential bias might overestimate the performance and has been discussed in our report of the baseline findings [4]. In spite of this potential bias, a considerable proportion of sick children with common childhood illness were not appropriately classified based on their symptoms. The exclusion of some health posts due to security issues could theoretically also introduce a bias. It should, however, be noted that these facilities were few and with equal proportions excluded from both arms.

This study found no improvement of the correct disease classification linked to the intervention. In contrast, a study conducted in rural Liberia suggested significant improvements in childhood illness treatment from qualified providers after implementation of a comprehensive intervention geared to capacitate community health workers [27]. The content, intensity and frequency of that intervention as well as the measures of outcome were different from the intervention in our study.

There is a growing awareness that quality improvements are needed to strengthen the health systems [28]. Training alone may not be sufficient to improve the performance of health care providers [3]. A majority of the health extension workers were trained in the integrated community case management of childhood illnesses that should have equipped them for the provision of good quality care to sick children.

The goal of this evaluation was to inform policymakers and underline the importance of continued efforts to improve the primary care services. In an implementation science context, a failure to achieve the targeted change can be due to failure in the implementation or deficiencies in the theory of change [29]. A recent study conducted in Tanzania showed that a multi-faceted intervention, which included in-service training, mentorship, supportive supervision and infrastructure support resulted in no improvement on quality of services. Also in that study the lack of effect could be due to failure in implementation as well as failure in the suggested theory of change [30].

A recently published evaluation of the effectiveness of the Optimizing the Health Extension Program intervention showed no effect on the utilization of services for sick children [31]. The lack of effect could partly be attributed to delays, interruptions and an overall short implementation period of a complex intervention. Complex interventions that aim at behaviour changes in the health services as well as in care seeking for sick children may need an extended implementation period [32]. Previous Ethiopian studies have shown that the health extension workers were insufficiently prepared for all the tasks that fall within their scope of work [33]. This could be associated with knowledge, job satisfaction, availability of clinical guidelines, and frequency and quality of supervision [34]. A qualitative study suggested that clearly defined roles might improve the performance of the health extension workers [35]. There is currently an uncertainty about which implementation strategies work where, for whom, and under which circumstances [36]. Public health practitioners should assess the significance and feasibility of interventions in health setting prior to implementation [14]. Therefore, it is important to assess the context in which the interventions are implemented to understand the variation in effects.

Conclusion

This evaluation in four regions of Ethiopia showed that optimizing the health extension program intervention was not associated with any positive change in the health extension workers’ ability to correctly classify common childhood illnesses within the integrated community case management program. The intervention included evidence-based components that potentially could influence the quality of care provided [9, 10]. However, when combined in the upscaled intervention, there was no effect. A process evaluation to examine the quantity and quality of what was implemented showed that lack of effect could partly be attributed to delays in implementation of some initial activities that would create the condition for the successful implementation of the next activity, interruptions and an overall short implementation period of a complex intervention [32]. This could shed light on the fidelity of the implementation, the context in which this complex intervention was delivered, and explain why it did not work.

Supporting information

S1 Table. Theory of change for OHEP interventions.

https://doi.org/10.1371/journal.pone.0247474.s002

(DOCX)

S1 Protocol. Protocol for the evaluation of a complex intervention aiming at increased utilisation of primary child health services in Ethiopia.

A before and after study in intervention and comparison areas.

https://doi.org/10.1371/journal.pone.0247474.s003

(DOCX)

Acknowledgments

We would like to acknowledge the Institute of Public Health, University of Gondar staff members and PhD students for their constructive criticism of this manuscript during a PhD workshop.

References

  1. 1. Deribew A, Tessema GA, Deribe K, Melaku YA, Lakew Y, Amare AT, et al. Trends, causes, and risk factors of mortality among children under 5 in Ethiopia, 1990–2013: findings from the Global Burden of Disease Study 2013. Popul Health Metr. 2016;14(1).
  2. 2. Mubiru D, Byabasheija R, Bwanika JB, Meier JE, Magumba G, Kaggwa FM, et al. Evaluation of integrated community case management in eight districts of Central Uganda. PLoS One. 2015 Aug 12;10(8). pmid:26267141
  3. 3. Rowe AK, Rowe SY, Peters DH, Holloway KA, Chalker J, Ross-Degnan D. Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review. Lancet Glob Heal. 2018;6(11):e1163–75. pmid:30309799
  4. 4. Getachew T, Mekonnen S, Yitayal M, Persson LÅ, Berhanu D. Health Extension Workers’ diagnostic accuracy for common childhood illnesses in four regions of Ethiopia: a cross-sectional study. Acta Paediatr Int J Paediatr. 2019;108(11):2100–6. pmid:31162734
  5. 5. Miller JS, English L, Matte M, Mbusa R, Ntaro M, Bwambale S, et al. Quality of care in integrated community case management services in Bugoye, Uganda: a retrospective observational study. Malar J [Internet]. 2018;17(1). Available from: pmid:29486773
  6. 6. Assefa Y, Gelaw YA, Hill PS, Taye BW, Van Damme W. Community health extension program of Ethiopia, 2003–2018: Successes and challenges toward universal coverage for primary healthcare services. Global Health. 2019;15(1):1–11. pmid:30606214
  7. 7. Boerma T, Requejo J, Victora CG, Amouzou A, George A, Agyepong I, et al. Countdown to 2030: tracking progress towards universal coverage for reproductive, maternal, newborn, and child health. Lancet. 2018;391(10129):1538–48. pmid:29395268
  8. 8. Das JK, Lassi ZS, Salam RA, Bhutta ZA. Effect of community based interventions on childhood diarrhea and pneumonia: uptake of treatment modalities and impact on mortality. BMC Public Health. 2013;13(Suppl 3). pmid:24564451
  9. 9. Feyissa GT, Balabanova D, Woldie M. How Effective are Mentoring Programs for Improving Health Worker Competence and Institutional Performance in Africa? A Systematic Review of Quantitative Evidence. J Multidiscip Healthc. 2019;12:989–1005. pmid:31824166
  10. 10. Woldie M, Feyissa GT, Admasu B, Hassen K, Mitchell K, Mayhew S, et al. Community health volunteers could help improve access to and use of essential health services by communities in LMICs: An umbrella review. Health Policy Plan. 2018;33(10):1128–43. pmid:30590543
  11. 11. Capblanch XB, Marceau C. Training, supervision and quality of care in selected integrated community case management (iCCM) programmes: A scoping review of programmatic evidence. J Glob Health. 2014;4(2):1–13.
  12. 12. Leslie HH, Gage A, Nsona H, Hirschhorn LR, Kruk ME. Training and supervision did not meaningfully improve quality of care for pregnant women or sick children in sub-Saharan Africa. Health Aff. 2016;35(9):1716–24.
  13. 13. Whidden CE, Kayentao K, Liu JX, Lee S, Keita Y, Diakité D, et al. Improving Community Health Worker performance by using a personalised feedback dashboard for supervision: A randomised controlled trial. J Glob Health. 2018;8(2). pmid:30333922
  14. 14. Hill Z, Dumbaugh M, Benton L, Källander K, Strachan D, ten Asbroek A, et al. Supervising community health workers in low-income countries—a review of impact and implementation issues. Glob Health Action. 2014;7(1):1–10. pmid:24815075
  15. 15. Ballard M, Montgomery P. Systematic review of interventions for improving the performance of community health workers in low-income and middle-income countries. BMJ Open. 2017;7(10). pmid:29074507
  16. 16. Tadesse Y, Eniyew A, Mengistu B, Eniyew A, Marsh DR. Utilization of integrated community case management services in three regions in Ethiopia after two years of implementation. Ethiop Med J. 2014 Oct;52(3):47–55. pmid:25845073
  17. 17. Amouzou A, Hazel E, Shaw B, Miller NP, Tafesse M, Mekonnen Y, et al. Effects of the integrated community case management of childhood illness strategy on child mortality in Ethiopia: A cluster randomized trial. Am J Trop Med Hyg. 2016;94(3):596–604. pmid:26787148
  18. 18. Shaw B, Tsui AO, Bryce J, Amouzou A, Miller NP, Surkan PJ, et al. Determinants of Utilization of Health Extension Workers in the Context of Scale-Up of Integrated Community Case Management of Childhood Illnesses in Ethiopia. Am J Trop Med Hyg. 2015;93(3):636–47. pmid:26195461
  19. 19. Central Statistical Agency. 2007 Population and housing census of Ethiopia: administrative report. 2007.
  20. 20. World Health Organization. Health Facility Survey: Tool to evaluate the quality of care delivered to sick children attending outpatients facilities (using the Integrated Management of Childhood Illness clinical guidelines as best practices). IWorld Health Organization. 2003.
  21. 21. World Health Organization. Integrated Management of Childhood Illness: Chart Booklet. World Health Organization. Geneva (Switzerland):; 2014.
  22. 22. Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas. 1960;20(1):37–46.
  23. 23. Gertler Paul J., Martinez Sebastian, Premand Patrick, Rawlings Laura B. and CMJV. Impact Evaluation in Practice: second edition. World Bank Group. 2007. 129–141 p.
  24. 24. Zemedu TG, Teshome A, Tadesse Y, Bekele A, Keyes E, Bailey P, et al. Healthcare workers ‘ clinical knowledge on maternal and newborn care in Ethiopia: findings from 2016 national EmONC assessment. BMC Health Serv Res. 2019;19(219):1–9.
  25. 25. Anthony M. Ho H., Phelan R, Mizubuti GB, Murdoch JAC, Wickett S, Ho AK, et al. Bias in before?after studies: Narrative overview for anesthesiologists. Anesth Analg. 2018;126(5):1755–62. pmid:29239959
  26. 26. Rowe AK, Lama M, Onikpo F, Deming MS. Health Worker Perceptions of How Being Observed Influences Their Practices during Consultations with Ill Children. Trop Doct. 2002;32(3):166–7. pmid:12139161
  27. 27. White EE, Downey J, Sathananthan V, Kanjee Z, Kenny A, Waters A, et al. A community health worker intervention to increase childhood disease treatment coverage in rural Liberia: A controlled before-and-after evaluation. Am J Public Health. 2018;108(9):1252–9. pmid:30024811
  28. 28. Leatherman S, Fortune T, Ruelas E, Go O, Gay-molina JG. Strengthening the quality agenda in health care in low- and middle-income countries: questions to consider. 2012;24(6):553–7.
  29. 29. Linnan L, Steckler A, editors. Process Evaluation for Public Health Interventions and Research. In: Process Evaluation for Public Health Interventions and Research. 2004. p. 739–739.
  30. 30. Larson E, Mbaruku GM, Cohen J, Kruk ME. Did a quality improvement intervention improve quality of maternal health care? Implementation evaluation from a cluster-randomized controlled study. Int J Qual Heal Care. 2019 Dec 12;1–10.
  31. 31. Della B, Yemisrach BO, Atkure D, Abebe B, Ephrem Tekle L, Araya Abrha M, et al. Does a complex intervention targeting communities, health facilities and district health managers increase the utilisation of community- based child health services? A before and after study in intervention and comparison areas of Ethiopia. BMJ Open. 2020;10(e040868).
  32. 32. Okwaraji YB, Hill Z, Defar A, Berhanu D, Wolassa D, Persson LÅ, et al. Implementation of the ‘optimising the health extension program’ intervention in ethiopia: A process evaluation using mixed methods. Int J Environ Res Public Health. 2020;17(16):1–20. pmid:32796574
  33. 33. Desta FA, Shifa GT, Dagoye DW, Carr C, Van Roosmalen J, Stekelenburg J, et al. Identifying gaps in the practices of rural health extension workers in Ethiopia: A task analysis study. BMC Health Serv Res. 2017;17(1):1–9. pmid:28049468
  34. 34. Edward A, Kumar B, Niayesh H, Naeem AJAN, Burnham G, Peters DH. The association of health workforce capacity and quality of pediatric care in Afghanistan. Int J Qual Heal Care. 2012;24(6):578–86. pmid:23081907
  35. 35. Kok MC, Kea AZ, Datiko DG, Broerse JEW, Dieleman M, Taegtmeyer M, et al. A qualitative assessment of health extension workers’ relationships with the community and health sector in Ethiopia: Opportunities for enhancing maternal health performance. Hum Resour Health. 2015;13(1). pmid:26423049
  36. 36. Duc D, Bergström A, Eriksson L, Selling K, Ha BTT, Wallin L. Response process and test-retest reliability of the Context Assessment for Community Health tool in Vietnam. Glob Health Action. 2016;9(1). pmid:28157008