Figures
Abstract
Background
Africa bears 24% of the global burden of disease but has only 3% of the world’s health workers. Substantial variation in health worker performance adds to the negative impact of this significant shortfall. We therefore sought to identify interventions implemented in sub-Saharan African aiming to improve health worker performance and the contextual factors likely to influence local effectiveness.
Methods and Findings
A systematic search for randomised controlled trials of interventions to improve health worker performance undertaken in sub-Saharan Africa identified 41 eligible trials. Data were extracted to define the interventions’ components, calculate the absolute improvement in performance achieved, and document the likelihood of bias. Within-study variability in effect was extracted where reported. Statements about contextual factors likely to have modified effect were subjected to thematic analysis. Interventions to improve health worker performance can be very effective. Two of the three trials assessing mortality impact showed significant reductions in death rates (age<5 case fatality 5% versus 10%, p<0.01; maternal in-hospital mortality 6.8/1000 versus 10.3/1000; p<0.05). Eight of twelve trials focusing on prescribing had a statistically significant positive effect, achieving an absolute improvement varying from 9% to 48%. However, reported range of improvement between centres within trials varied substantially, in many cases exceeding the mean effect. Nine contextual themes were identified as modifiers of intervention effect across studies; most frequently cited were supply-line failures, inadequate supervision or management, and failure to follow-up training interventions with ongoing support, in addition to staff turnover.
Conclusions
Interventions to improve performance of existing staff and service quality have the potential to improve patient care in underserved settings. But in order to implement interventions effectively, policy makers need to understand and address the contextual factors which can contribute to differences in local effect. Researchers therefore must recognise the importance of reporting how context may modify effect size.
Citation: Blacklock C, Gonçalves Bradley DC, Mickan S, Willcox M, Roberts N, Bergström A, et al. (2016) Impact of Contextual Factors on the Effect of Interventions to Improve Health Worker Performance in Sub-Saharan Africa: Review of Randomised Clinical Trials. PLoS ONE 11(1): e0145206. https://doi.org/10.1371/journal.pone.0145206
Editor: Eric Brian Faragher, Liverpool School of Tropical Medicine, UNITED KINGDOM
Received: September 1, 2015; Accepted: November 30, 2015; Published: January 5, 2016
Copyright: © 2016 Blacklock et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This review received no external funding. All contributors received support from their host institutions.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Africa bears 24% of the global burden of disease but has only 3% of the world’s health workers [1]. This relative shortage will not be corrected without a redistribution of global economic resources and human capital. However, quality of medical care produced by existing staff varies substantially between facilities [2]. Addressing the substantial variation in care quality produced by existing health workers may be a more feasible immediate-term solution. Robust evidence about which interventions to improve health worker performance are likely to be most effective is required for implementation. By health worker performance, we mean the effectiveness with which existing health workers perform their professional tasks, as measured by their demonstrated skill (e.g. to tie a surgical knot), their care quality (e.g. adherence to clinical guidelines), or the impact of their care (e.g. case-fatality rate).
An overview of strategies to maintain and improve health worker performance in low and middle income countries was previously published [3], but a subsequent paper recognised that the evidence to support policy-making is weak [4]. We were motivated to undertake the current review because our own research (on the shortage of human resources to deliver primary care in sub-Saharan Africa) echoed these findings. It not only suggests that quality of healthcare provision may be of greater importance than quantity of health workers in addressing the inverse care law [5] but also highlighted the difficulty faced by policy makers in selecting evidence-led solutions to improve performance in their local context.
A number of Cochrane reviews address the effectiveness of specific interventions to improve healthcare systems and delivery. While some focus on low and middle-income countries, most include studies conducted in high income western countries and so their applicability to sub-Saharan Africa is questionable [6]. A recent qualitative review applying realist methodology also concluded that the effect of interventions targeted on health worker performance is very context specific, with similar interventions producing very different outcomes when implemented in different circumstances [7].
We have therefore systematically identified trials carried out in sub-Saharan Africa and extracted available data on context, as well as absolute effect, to facilitate the use of the evidence by policy makers. By context, we mean local issues (both within and external to the health care system) reported by authors as having impacted on the effectiveness of the intervention assessed.
Methods
In conducting the review we adhered to PRISMA guidelines [8]. The aim was to identify trials meeting the following criteria: participants were existing formal health workers; intervention aimed to improve performance; undertaken in sub-Saharan Africa; randomised controlled (RCT) design. Studies recruiting solely health workers in training (e.g. nursing students), informal health workers (e.g. traditional birth attendants), and reported only in conference abstracts were excluded.
The initial electronic search is detailed in S1 Table. The titles and abstracts of the 7109 papers identified by the electronic search were reviewed by two authors (CB, DGB) who selected 365 for full text review, assessing eligibility for inclusion against a pre-piloted checklist. Disagreements were resolved by discussion after joint re-review of appropriate manuscripts. The flow chart in Fig 1 gives further detail of this review process. As a final step, papers selected were compared with those registered on the database of trials of interventions to improve health worker performance in low and middle income countries compiled by Rowe et al [9].
The Cochrane EPOC Group classification of interventions was used to categorise the components of each intervention [10]. Risk of bias was also assessed against Cochrane criteria [11]. The extent to which interventions had been customised to local circumstances was assessed against two criteria: whether the intervention had been designed or adapted in response to prior assessment of local circumstance, and whether the intervention was based on a specified theory and mechanism for achieving change.
A small number of trials were multi-arm, comparing a number of interventions, often cumulative (i.e. each extra intervention arm including the interventions in other arms). For these studies we report the comparison between the most and least intensive intervention. Many of the studies reported more than one outcome. In selecting the outcome to report in this analysis we applied the following criteria: the stated main objective of the study, the objectivity of the measurement of performance (e.g. observed performance was preferred to self-reported), and the anticipated impact (on performance or health). Three authors (CB, DM, SM) independently identified the outcomes which they felt most closely reflected the stated objective and, if more than one, the most serious outcome in terms of impact. Initial inter-rater agreement in this assessment was high (35/41 studies complete agreement). If outcome at more than one time point was reported, we used the last time point in reporting effect.
In line with our intention to make our findings useful to policy makers, we reported and defined four unique measures. The baseline performance is the pre-intervention performance in the intervention arm (or, if not measured, the performance in the control arm). The absolute change in the intervention group is the difference in the pre- and post-intervention performance in the intervention arm. The absolute difference between the change in the intervention group and the change in the control group is the difference in pre-and post-intervention performance or, if not measured, the difference in end performance. The within study range of change is the minimum and maximum change in performance in the different study sites where reported (or, if not measured, the treatment success rate in the best and worst performing centres in the intervention group).
Extracted data were entered and analysed using Microsoft Excel (2010). The statistical significance reported is the 2-tailed p-value calculated by the authors of the original paper to assess the likelihood that the observed difference in performance between the intervention and control arms could have arisen by chance. The precise value given therefore reflects the way in which the authors chose to report the chosen outcome (e.g. as an absolute difference, risk ratio or odds ratio) and so is an approximation of the statistical significance of the difference between baseline and absolute improvement in performance reported in the tables. However, in all but two trials this p-value was adjusted for clustering and co-variance when appropriate. If the authors reported only confidence intervals, the p-value was calculated from the standard error implicit in the reported intervals (assuming the log of the standard error of an odds ratio is normally distributed).
Contextual effect modifiers were identified when authors incidentally reported factors beyond the intervention, which may have led to differences in outcome between randomisation units (mainly health facilities) within the trial, or contributed to the overall reported trial effect size. Data were identified from the results or discussion text of the reports. In five instances where study authors referred to qualitative publications, they were accessed to confirm and provide additional contextual data. For one identified trial, a post-hoc analysis report was also accessed [12], and related reports were accessed for two further trials [13,14]. The identified context modifiers were analysed by inductive content analysis [15]. Initially three authors (CB, SM and DGB) read each paper thoroughly to become familiar with each author’s language. Through reading and coding, they identified commonality of meaning within and between papers. Through this process of constant comparison, key themes emerged. Higher organisational categories represented a final level of synthesis that facilitated meaningful reporting.
Results
We report the characteristics of trials, the main trial outcome data, and the contextual data reported by authors. We then present a qualitative analysis of themes emerging from the extracted contextual data.
Descriptive analysis
The 42 manuscripts reporting the 41 eligible trials identified are shown in Table 1 [16–57]. Two of the included manuscripts reported the same trial (Pirkle [45] and Dumont [28]); however each manuscript reported different aspects of the intervention and different outcomes (one assessing the impact of multi-factorial training on obstetric skills and the other assessing the effect on maternal mortality of focusing training on the lessons learned from maternal death audit). We therefore decided to include both of these manuscripts in this review.
Although seven trials were hospital-based, the majority of trials were conducted in primary care facilities, including some private clinics or dispensaries and some specialist clinics (for TB, HIV and STD, family planning and addiction). The analysis of the elements of the intervention according to the EPOC classification shows that just over half the interventions (23/41) were multi-faceted, including three or more elements. The most frequent elements were educational material and meetings. About one-third (17/41) of the trials reported some form of prior assessment of local context to customise the intervention, but only in five was a formal theory of change presented. Five of the trials referred to a total of six accompanying qualitative papers in their discussions, of which four were available for analysis [58–61]. We also accessed a post-hoc analysis report [12] relating to one trial [19], and additional reports [13,14] related to a further two trials [35,33].
Half of the trials (23/41) were conducted in South Africa or Kenya with most of the rest in other East African countries (Fig 2). Only eight trials were conducted in West Africa (one trial had sites in both Mali and Senegal [28]) and one in Central Africa, in Cameroon [40]. Fig 3 shows the overall assessment of risk of bias. In many cases it was difficult or impossible to make assessment of outcome blinded to the intervention arm and most trials were incompletely reported.
Reprinted from SmartDraw Software LLC under a CC BY license, with permission from SmartDraw Software LLC, original copyright 2013.
Effect of interventions aimed at improving skills (Table 2)
There was substantial variation in the effect of these interventions (Table 2); for example, one intervention aimed at improving diagnosis of depression in Kenya was associated with no improvement [16] while another in Malawi reported a 20-fold improvement in diagnostic sensitivity from 3% to 57% (p<0.001) [35]. Other interventions with substantial effect sizes were internet-video based training to improve ability to tie surgical knots (61% greater improvement in the intervention group compared to control, p<0.05, though local suture-conserving techniques influenced outcome) [17] and a management intervention in Kenya to encourage supervisors to improve their effectiveness (increase in proportion directly observing staff performance from 31% to 53%, p<0.01) [47].
The most frequent contextual factors cited by authors as influencing outcome were staff motivation, supervision and turnover. None of these studies reported the within-study heterogeneity between centres.
Effect of interventions to improve the quality of the care process (Table 3)
There was again substantial variation in effect (Table 3), ranging from no impact of an intervention in a South African hospital to promote childbirth companions, as an incentive to improve the performance of obstetric staff [26], to a Kenyan study showing an improvement in correct IV fluid prescribing in intervention paediatric wards of 30%, compared to controls (p<0.01) [19]. All but two [38,48] of the interventions to improve prescribing were effective in achieving a clinically important improvement, most (8/12) showing a statistically significant effect compared to control, ranging from 9 to 48%.
Five of the studies reported within study heterogeneity. Where the range of difference between paired intervention and control sites was reported [40,44], this exceeded mean effect. The most frequent contextual factor cited as influencing within-study effect were stock-outs of drugs and adequacy of supervision. High staff turnover was also mentioned as a contextual modifier in four trials and personal motivation in two.
Effect of interventions to improve impact of care (treatment outcomes) (Table 4)
Two interventions to improve health worker performance achieved important reductions in mortality (Table 4)–in child mortality (case fatality age<5 years reduced from 10% to 5%, p<0.01, in Guinea-Bissau [24]) and in-hospital maternal mortality (from 10.3/1000 to 6.8/1000, p<0.05, in Mali/Senegal [28]). Increasing the involvement of the community in monitoring local health services was also associated with a substantial reduction in <5 child mortality (from 144 to 97/1000) but the statistical power of the trial was very limited as even an effect of this magnitude was non-significant [25].
The within-study variability was reported for five trials. Where reported, the range in the difference between intervention and control sites in different strata exceeded mean effect [28,31]. The most frequently cited contextual modifiers were again drug stock-outs and supervision / continuing in-service support. The three trials with the greatest within-study range of improvement all cited the quality of local management and leadership as key issues.
Thematic analysis of context
The results of the thematic analysis of contextual effect modifiers are summarised in Fig 4, and included in full in S2 Table with illustrative quotes. Nine unique themes have been synthesised into three organisational headings for ease of understanding: management, staffing, and local environment. We have provided detailed quotes for each of the nine key themes, within the higher order organisational headings. This illustrates the depth of meaning for each contextual theme as reported by each trial (see S2 Table: Thematic analysis of contextual effect modifiers, for direct quotations from the trials).
Management.
The management theme which emerged most frequently was the effectiveness of local leadership. In more than one case, the lack of leadership was attributed to the frequent rotation of supervisory staff. When interventions failed to achieve the intended improvement in performance, it was frequently attributed to lack of subsequent local supervision, support or follow up (“…inadequately supported training, supervision and follow-up, which has resulted in their infrequent use by the prescribers” [18]). The other strong management theme was organisational inertia, poor team-working and resistance to change (“…poor team working and staff conflicts were common; task-orientated care was entrenched” [37]).
Staffing.
The three staffing themes identified were: the absolute shortage of staff, the adequacy of existing knowledge and skills, and erosion of personal motivation or of perceived personal agency to effect change. The shortage of staff (“the redistribution of patients to the primary level resulted in increasing patient numbers but this was unaccompanied by increases in staff numbers and clinic facilities” [50]) was exacerbated by high staff turnover and inadequacy of basic, induction, and follow-up in-service training (“Training …should be followed up by support and reinforcement” [24]). A lack of personal motivation to change, and a failure of individuals to see themselves as possible agents for change, was expressed in relation to both supervisors and staff (“……staff did not see themselves as having the agency to initiate workplace change” [37].)
Local environment.
The environmental themes identified related to: material resources, patient and community factors, and concurrent health system interventions. The most frequently cited issue was “stock-outs”, such as failure of the supply chain for drugs (“Regular availability of drugs is an important factor influencing the credibility, confidence and utilization of health services among the population” [23]) although lack of other material supplies, such as electricity failure or poor telephone/internet reception, was sometimes also an issue. Many patient and community factors (such as poverty) were reported which cannot be easily reformed, but the impact of customising interventions to reflect community issues such as language, cultural expectation and transport issues was widely recognised (“… [intervention materials] were adapted to the [local] environment and... used locally relevant graphics to enhance engagement with low-literate women” [33]). Finally, concurrent programmes implemented by other agencies (particularly NGOs) were often mentioned as having an important impact on the measured local effect of the trialled intervention (“The poor effect in [regional] hospitals outside the capital could be due to potential contamination bias” [28]).
Discussion
Our review shows three important things. Firstly, it is possible to improve the performance of existing health workers serving in under-resourced health systems. Secondly, the success of an intervention in improving performance depends very much on local context, with differences reported between centres within trials. Thirdly, few trials have been undertaken in sub-Saharan Africa and even fewer have adopted methodologies which allow the effect of local context to be assessed (e.g. only a minority applied a theory of change model or reported difference in effect between participating centres).
The wide range of interventions assessed and outcomes measured in very different settings means that a simple comparison of relative effect has limited policy relevance. However, the positive performance of interventions to improve prescribing in 8/12 included trials is policy relevant (even if this is because it is a straightforward care function to target and measure). The parallel contextual observation that drug “stock-outs” were a frequent cause of variation in the within-study effect also emphasises the policy importance of securing the medicines supply-chain.
The fact that the range of performance improvement exceeded the mean in many trials reporting this information is perhaps the most important observation. However, we are not the first to note the modifying effect of local context on outcome and the limited policy utility of focusing on mean effect. Others have previously stressed that triallists should give better information on contextual effects [62] and tools to facilitate their measurement have been developed applicable to high-income settings [63]. Bonnell [64] and Michie [65] have both suggested mechanisms for better specifying the process by which the intervention achieves performance improvement, allowing the impact of key contextual modifiers to be estimated. The growing emphasis on process evaluation for trials of complex interventions is an important step. Process evaluations can explore intervention theory and the influence of context on its implementation and effect. Data on recruitment, delivery, response, maintenance, effectiveness and unintended consequences, are gathered during and after the trial using mixed methods [66]. The new MRC guidance ‘Process evaluation of complex interventions’, emphasises the need to design and evaluate key contextual components [67]. However, challenges faced by some have included maintaining methodological quality [68], costs, how to integrate evaluation findings with traditional trial outcomes [69], and making communication of findings accessible [66].
The informal way in which contextual data were reported in our included trials weakens the generalisability of our findings while also reflecting the lack of systematic effort to report and understand aspects of context which influence the effect of interventions. We could only extract what was reported incidentally, and this is likely to be incomplete. Despite this, the frequency with which the same contextual issues recurred in different manuscripts is consistent with the existing literature and echoes the context elements of the PARIHS framework [70]. The themes also support the contextual factors perceived to influence the implementation of interventions as measured by the Alberta Context Tool [63] and the Context Assessment for Community Health tool [71]. Thirteen of our included trials were conducted in South Africa. Although South Africa differs in some ways to other sub-Saharan countries, we found that the emerging themes were consistent between countries. Representation of particular countries and settings in this study should be taken into consideration when interpreting the findings.
In order to know what will work locally, and to implement any intervention effectively, policy makers need to understand and address the specific contextual factors which contribute to differences in effect sizes seen in trials. To implement the “knowledge to action cycle” promoted by the Canadian Institute of Health Research, it is essential to identify the local barriers and facilitators likely to impact on the effectiveness of an intervention [72], and to also consider the underlying theory and mechanisms of achieving change [64, 65]. Evidence about key effect modifiers not only informs the choice of intervention, but also facilitates the context to be modified, and the intervention “tailored” [73], as part of the implementation process.
In other words, those who undertake trials need to recognise the importance of providing sufficient evidence to make clear whether intervention X will work in setting Y. Our results show that interventions to improve the performance of existing health workers have the potential to impact very positively on patient morbidity and mortality in an underserved sub-Saharan context. The importance of local contextual influence in intervention trials should be reflected in both trial reporting, and in intervention planning and design.
Supporting Information
S2 Table. Thematic Analysis of contextual effect modifiers (full).
https://doi.org/10.1371/journal.pone.0145206.s002
(DOCX)
S2 Text. Revisions made to registered protocol.
https://doi.org/10.1371/journal.pone.0145206.s005
(DOCX)
Acknowledgments
Sincere thanks to Alexander Rowe, Malaria Branch, Centers for Disease Control and Prevention, Atlanta, Georgia, USA, for his helpful comments on a previous draft of the manuscript, and for sharing his group’s database of trials.
Author Contributions
Conceived and designed the experiments: CB DCGB NR. Performed the experiments: CB DCGB NR SM DM. Analyzed the data: CB DCGB SM DM MW. Wrote the paper: CB DCGB SM AB MW DM.
References
- 1.
World Health Report, Working together for health. 2006, WHO. Available from www.who.int/whr/2006/whr06_en.pdf: Geneva.
- 2. Nolan T, Angos P, Cunha AJLA, Muhe L, Qazi S, Simoes EAF, et al. Quality of hospital care for seriously ill children in less-developed countries. Lancet 2001;357(9250):106–110 pmid:11197397
- 3. Rowe AK, de Savigny D, Lanata CF, Victora CG. How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet. 2005;366(9490):1026–1035. pmid:16168785
- 4. Chopra M, Munro S, Lavis J, Vist G, Bennett S. Effects of policy options for human resources for health: An analysis of systematic reviews. Lancet. 2008;371(9613):668–674. pmid:18295024
- 5. Moosa S, Wojczewski S, Hoffmann K, Poppe A, Nkomazana O, Peersman W, et al. Why there is an inverse primary-care law in Africa. Lancet Glob Health. 2013;1(6):e332–3. pmid:25104594
- 6. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. IS. 2012;7:50 pmid:22651257
- 7. Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers' performance in low and middle income countries: A realist review. Health Res Policy Syst. 2009;7:7–7. pmid:19374734
- 8. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7): e1000097 pmid:19621072
- 9.
Rowe AK, Rowe SY, Peters DH, Holloway KA, Chalker J, Ross-Degnan D. Database of the Health Care Provider Performance Review: A systematic review of the effectiveness of strategies to improve health care provider performance in low- and middle-income countries, in US Centers for Disease Control and Prevention, Provided by Rowe Samantha, Editor. March 23 2015: Atlanta, Georgia.
- 10.
Effective Practice and Organisation of Care (EPOC). Data collection checklist. 2014; Available from: https://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/datacollectionchecklist.pdf.
- 11.
Higgins JPT, Altman DG (editors). Chapter 8: Assessing risk of bias in included studies, In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org
- 12. Opondo C, Ayieko P, Ntoburi S, Wagai J, Opiyo N, Irimu G, et al. Effect of a multi-faceted quality improvement intervention on inappropriate antibiotic use in children with non-bloody diarrhoea admitted to district hospitals in Kenya. BMC Pediatr. 2011;11:109. pmid:22117602
- 13. Okwen M, Lewallen S, Courtright P. Primary eye care skills scores for health workers in routine and enhanced supervision settings. Public Health 2014;128(1):96–100 pmid:24359760
- 14. Jennings L, Yebadokpo A, Affo J, Agbogbe M. Use of job aids to improve facility-based postnatal counselling and care in rural Benin. Matern Child Health J 2015;19(3):557–565. pmid:24916207
- 15. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–115. pmid:18352969
- 16. Alexander CL, Arnkoff DB, Glass CR, Kaburu AW. Detecting depression in rural primary care clinics in central Kenya: Impact of a brief training intervention. Int Perspect Psychol. 2013;2(1):14–28.
- 17. Autry AM, Knight S, Lester F, Dubowitz G, Byamugisha J, Nsubuga Y, et al., Teaching surgical skills using video internet communication in a resource-limited setting. Obstet Gynecol. 2013;122(1):127–131. pmid:23743458
- 18. Awad AI, Eltayeb IB, Baraka OZ, Changing antibiotics prescribing practices in health centers of Khartoum State, Sudan. Eur J Clin Pharmacol. 2006;62(2):135–42. pmid:16389536
- 19. Ayieko P, Ntoburi S, Wagai J, Opondo C, Opiyo N, Migiro S, et al., A multifaceted intervention to implement guidelines and improve admission paediatric care in Kenyan district hospitals: A cluster randomised trial. PLOS Med. 2011;8(4):e1001018. pmid:21483712
- 20. Bachmann MO, Fairell LR, Lombard C, Timmerman V, van der Merwe S, Bateman ED, et al. Effect on tuberculosis outcomes of educational outreach to South African clinics during two randomised trials. Int J Tuberc Lung Dis. 2010;14(3):311–7. pmid:20132622
- 21. Basinga P, Gertler PJ, Binagwaho A, Soucat ALB, Sturdy J, Vemeersch CMJ. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: An impact evaluation. Lancet. 2011;377(9775):1421–8. pmid:21515164
- 22. Baumgartner JN, Morroni C, Mlobeli RD, Otterness C, Buga G, Chen M. Impact of a provider job aid intervention on injectable contraceptive continuation in South Africa. Stud Fam Plann. 2012;43(4):305–314. pmid:23239249
- 23. Bexell A, Lwando E, von Hofsten B, Tembo S, Eriksson B, Diwan VK. Improving drug use through continuing education: A randomized controlled trial in Zambia. J Clin Epidemiol. 1996;49(3):355–7. pmid:8676185
- 24. Biai S, Rodrigues A, Gomes M, Ribeiro I, Sodemann M, Alves F, et al. Reduced in-hospital mortality after improved management of children under 5 years admitted to hospital with malaria: Randomised trial. BMJ. 2007;335(7625):862. pmid:17954513
- 25. Bjorkman M, Svensson J. Power to the people: Evidence from a randomized field experiment on community-based monitoring in Uganda. Q J Econ. 2009;124(2):735–769.
- 26. Brown H, Hofmeyr GJ, Nikodem VC, Smith H, Garner P. Promoting childbirth companions in South Africa: A randomised pilot study. BMC Med. 2007;5:7. pmid:17470267
- 27. Buchanan H, Siegfried N, Jelsma J, Lombard C. Comparison of an interactive with a didactic educational intervention for improving the evidence-based practice knowledge of occupational therapists in the public health sector in South Africa: A randomised controlled trial. Trials. 2014;15:216. pmid:24916176
- 28. Dumont A, Fournier P, Brahamowicz M, Traore M, Haddad S, Fraser WD, et al. Quality of care, risk management, and technology in obstetrics to reduce hospital-based maternal mortality in Senegal and Mali (QUARITE): A cluster-randomised trial. Lancet. 2013;382(9887):146–57. pmid:23721752
- 29. Fairall LR, Zwarenstein M, Bateman ED, Bachmann M, Lombard C, Majara BP, et al. Effect of educational outreach to nurses on tuberculosis case detection and primary care of respiratory illness: Pragmatic cluster randomised controlled trial. BMJ. 2005;331(7519):750–754A. pmid:16195293
- 30. Gilroy K, Winch PJ, Diawara A, Swedberg E, Thiero F, Kane M, et al. Impact of IMCI training and language used by provider on quality of counseling provided to parents of sick children in Bougouni District, Mali. Patient Educ Couns. 2004;54(1):35–44. pmid:15210258
- 31. Grosskurth H, Mosha F, Todd J, Mwijarubi E, Klokke A, Senkoro K, et al. Impact of improved treatment of sexually-transmitted diseases on HIV-infection in rural Tanzania: Randomized controlled trial. Lancet. 1995;346(8974):530–536. pmid:7658778
- 32. Harrison A, Karim SA, Floyd K, Lombard C, Lurie M, Ntuli N, et al. Syndrome packets and health worker training improve sexually transmitted disease case management in rural South Africa: Randomized controlled trial. AIDS. 2000;14(17):2769–79. pmid:11125896
- 33. Jennings L, Yebadokpo AS, Affo J, Agbogbe M. Antenatal counseling in maternal and newborn care: Use of job aids to improve health worker performance and maternal understanding in Benin. BMC Pregnancy Childbirth. 2010;10:75. pmid:21092183
- 34. Kalua K, Gichangi M, Barassa E, Eliah E, Lewallen S, Courtright P. A randomised controlled trial to investigate effects of enhanced supervision on primary eye care services at health centres in Kenya, Malawi and Tanzania. BMC Health Serv Res. 2014;14(Suppl 1):S6. pmid:25079942
- 35. Kauye F, Jenkins R, Rahman A. Training primary health care workers in mental health and its impact on diagnoses of common mental disorders in primary care of a developing country, Malawi: a cluster-randomized controlled trial. Psychol Med. 2014;44(3):657–66. pmid:23721658
- 36. Larke N, Cleophas-Mazige B, Plummer ML, Obasi AI, Rwakatare M, Todd J, et al. Impact of the MEMA kwa Vijana adolescent sexual and reproductive health interventions on use of health services by young people in rural Mwanza, Tanzania: Results of a cluster randomized trial. J Adolesc Health. 2010;47(5):512–522. pmid:20970087
- 37. Lewin S, Dick J, Zwarenstein M, Lombard CJ. Staff training and ambulatory tuberculosis treatment outcomes: A cluster randomized controlled trial in South Africa. Bull World Health Organ. 2005;83(4):250–9. pmid:15868015
- 38. Liambila W, Obare F, Keesbury J. Can private pharmacy providers offer comprehensive reproductive health services to users of emergency contraceptives? Evidence from Nairobi, Kenya. Patient Educ Couns. 2010;81(3):368–73. pmid:20934833
- 39. Loevinsohn BP, Gareaballah E. Missed opportunities for immunization during visits for curative care: A randomized cross-over trial in Sudan. Bull World Health Organ. 1992;70(3):335–339. pmid:1638661
- 40. Mbacham WF, Mangham-Jeffries L, Cundill B, Achonduh OA, Chandler CIR, Ambebila JN, et al. Basic or enhanced clinician training to improve adherence to malaria treatment guidelines: A cluster-randomised trial in two areas of Cameroon. Lancet Glob Health. 2014;2(6):e346–58. pmid:25103303
- 41. Mbonye MK, Burnett SM, Burna A, Colebunders R, Crozier I, Kinoti SN, et al. Effect of integrated capacity-building interventions on malaria case management by health professionals in Uganda: A mixed design study with pre/post and cluster randomized trial components. PLoS ONE. 2014;9(1):e84945 pmid:24416316
- 42. Meyer JC, Summers RS, Moller H. Randomized, controlled trial of prescribing training in a South African province. Med Educ. 2001;35(9):833–840. pmid:11555220
- 43. Opiyo N, Shepperd S, Musila N, Allen E, Nyamai R, Fretheim A, et al. Comparison of alternative evidence summary and presentation formats in clinical guideline development: A mixed-method study. PLoS ONE. 2013;8(1):e55067. pmid:23372813
- 44. Pattinson RC, Arsalo I, Bergh AM, Malan AF, Patrick M, Phillips N. Implementation of kangaroo mother care: A randomized trial of two outreach strategies. Acta Paediatr. 2005;94(7):924–927. pmid:16188816
- 45. Pirkle CM, Dumont A, Traore M, Zunzunegui M, QUARITE Group. Effect of a facility-based multifaceted intervention on the quality of obstetrical care: A cluster randomized controlled trial in Mali and Senegal. BMC Pregnancy & Childbirth. 2013;13:24.
- 46. Rawson RA, Rataemane S, Tataemane L, Ntlhe N, Fox RS, McCuller J, et al. Dissemination and implementation of Cognitive Behavioral Therapy for atimulant dependence: A randomized trial comparison of 3 approaches. Subst Abus. 2013;34(2):108–17. pmid:23577903
- 47. Reynolds HW, Toroitich-Ruto C, Nasution M, Beaston-Blaakman A, Janowitz B. Effectiveness of training supervisors to improve reproductive health quality of care: A cluster-randomized trial in Kenya. Health Policy Plan. 2008;23(1):56–66. pmid:17942447
- 48. Skarbinski J, Ouma PO, Causer LM, Karinki SK, Barnwell JW, Alaii JA, et al. Effect of malaria rapid diagnostic tests on the management of uncomplicated malaria with artemether-lumefantrine in Kenya: A cluster randomized trial. Am J Trop Med Hyg. 2009;80(6):919–26. pmid:19478249
- 49. Stanback J, Griffey S, Lynam P, Ruto C, Cummings S. Improving adherence to family planning guidelines in Kenya: An experiment. Int J Qual Health Care. 2007;19(2):68–73. pmid:17277011
- 50. Steyn K, Lombard C, Gwebushe N, Fourie JM, Everett-Murphy K, Zwarenstein M, et al. Implementation of national guidelines, incorporated within structured diabetes and hypertension records at primary level care in Cape Town, South Africa: A randomised controlled trial. Glob Health Action. 2013;6:20796. pmid:24070181
- 51. Thiam S, LeFevre AM, Hane F, Ndiaye A, Ba F, Fielding KL, et al. Effectiveness of a strategy to improve adherence to tuberculosis treatment in a resource-poor setting—A cluster randomized controlled trial. JAMA. 2007;297(4):380–386. pmid:17244834
- 52. Trap B, Todd CH, Moore H, Laing R. The impact of supervision on stock management and adherence to treatment guidelines: A randomized controlled trial. Health Policy Plan. 2001;16(3):273–80. pmid:11527868
- 53. Weaver MR, Crozier I, Eleku S, Makanga G, Mpanga Sebuyira L, Nyakake J, et al. Capacity-building and clinical competence in infectious disease in Uganda: A mixed-design study with pre/post and cluster-randomized trial components. PLoS ONE. 2012;7(12):e51319. pmid:23272097
- 54. Were MC, Nyandiko WM, Huang KT, Slaven JE, Shen C, Tierney WM, et al. Computer-generated reminders and quality of pediatric HIV care in a resource-limited setting. Pediatrics. 2013;131(3):e789–96. pmid:23439898
- 55. Zurovac D, Sudoi RK, Akhwale WS, Ndiritu M, Hamer DH, Rowe AK, et al. The effect of mobile phone text-message reminders on Kenyan health workers' adherence to malaria treatment guidelines: A cluster randomised trial. Lancet. 2011. 378(9793): p. 795–803. pmid:21820166
- 56. Zwarenstein M, Bheekie A, Lombard C, Swingler G, Erlich R, Eccles M, et al. Educational outreach to general practitioners reduces children's asthma symptoms: A cluster randomised controlled trial. IS. 2007;2:30. pmid:17892588
- 57. Zwarenstein M, Fairell LR, Lombard C, Meyers P, Bheekie A, English RG, et al. Outreach education for integration of HIV/AIDS care, antiretroviral treatment, and tuberculosis care in primary care clinics in South Africa: PALSA PLUS pragmatic cluster randomised trial. BMJ. 2011;342:d2022. pmid:21511783
- 58. Nzinga J, Ntoburi S, Wagai J, Mbindyo P, Mbaabu L, Migiro S, et al. Implementation experience during an eighteen month intervention to improve paediatric and newborn care in Kenyan district hospitals. IS. 2009;4:45. pmid:19627594
- 59. Nzinga J, Mbindyo P, Mbaabu L, Warira A, English M. Documenting the experiences of health workers expected to implement guidelines during an intervention study in Kenyan hospitals. IS. 2009;4:44. pmid:19627591
- 60. Plummer ML, Mshana G, Wamoyi J, Shigongo ZS, Hayes RJ, Ross DA, et al. 'The man who believed he had AIDS was cured': AIDS and sexually-transmitted infection treatment-seeking behaviour in rural Mwanza, Tanzania. AIDS Care. 2006;18(5):460–466. pmid:16777638
- 61. Stein J, Lewin S, Fairell L, Mayers P, English R, Bheekie A, et al. Building capacity for antiretroviral delivery in South Africa: A qualitative evaluation of the PALSA PLUS nurse training programme. BMC Health Serv Res. 2008;8:240. pmid:19017394
- 62. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ. 2008;337:a1655. pmid:18824488
- 63. Estabrooks CA, Squires JE, Cummings GG, Birdsell JM, Norton PG. Development and assessment of the Alberta Context Tool. BMC Health Serv Res. 2009;9:234. pmid:20003531
- 64. Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–2306. pmid:22989491
- 65. Michie S, van Stralen MM, West R. The behaviour change wheel: A new method for characterising and designing behaviour change interventions. IS. 2011;6:42. pmid:21513547
- 66. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013;14:15 pmid:23311722
- 67.
Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions. UK Medical Research Council Guidance. Available online: https://www.ioe.ac.uk/MRC_PHSRN_Process_evaluation_guidance_final(2).pdf (accessed 09.11.2015)
- 68. Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ 2009;339:b3496 pmid:19744976
- 69. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ;332:413–416. pmid:16484270
- 70. Bergstrӧm A, Peterson S, Namusoko S, Waiswa P, Wallin L. Knowledge translation in Uganda: A qualitative study of Ugandan midwives' and managers' perceived relevance of the sub-elements of the context cornerstone in the PARIHS framework. IS. 2012;7:117. pmid:23206670
- 71. Bergström A, Skeen S, Duc DM, Blandon EZ, Estabrooks C, Gustavsson P, et al. Health system context and implementation of evidence-based practices—development and validation of the Conext Assessment for Community Health (COACH) tool for low- and middle-income settings. IS. 2015;10:120 pmid:26276443
- 72. Straus SE, Tetroe J, Graham I. Defining knowledge translation. CMAJ. 2009;181(3–4):165–168. pmid:19620273
- 73. Harrison MB, Legare F, Graham ID, Fervers B. Adapting clinical practice guidelines to local context and assessing barriers to their use. CMAJ. 2010;182(2):e78–84. pmid:19969563