Effectiveness of Onsite Nurse Mentoring in Improving Quality of Institutional Births in the Primary Health Centres of High Priority Districts of Karnataka, South India: A Cluster Randomized Trial

Background In India, although the proportion of institutional births is increasing, there are concerns regarding quality of care. We assessed the effectiveness of a nurse-led onsite mentoring program in improving quality of care of institutional births in 24/7 primary health centres (PHCs that are open 24 hours a day, 7 days a week) of two high priority districts in Karnataka state, South India. Primary outcomes were improved facility readiness and provider preparedness in managing institutional births and associated complications during child birth. Methods All functional 24/7 PHCs in the two districts were included in the study. We used a parallel, cluster randomized trial design in which 54 of 108 facilities received six onsite mentoring visits, along with an initial training update and specially designed case sheets for providers; the control arm received just the initial training update and the case sheets. Pre- and post-intervention surveys were administered in April-2012 and August-2013 using facility audits, provider interviews and case sheet audits. The provider interviews were administered to all staff nurses available at the PHCs and audits were done of all the filled case sheets during the month prior to data collection. In addition, a cost analysis of the intervention was undertaken. Results Between the surveys, we achieved coverage of 100% of facilities and 91.2% of staff nurse interviews. Since the case sheets were newly designed, case-sheet audit data were available only from the end line survey for about 80.2% of all women in the intervention facilities and 57.3% in the control facilities. A higher number of facilities in the intervention arm had all appropriate drugs, equipment and supplies to deal with gestational hypertension (19 vs.3, OR (odds ratio) 9.2, 95% C.I 2.5 to33.6), postpartum haemorrhage (29 vs. 12, OR 3.7, 95% C.I 1.6 to8.3); and obstructed labour (25 vs.9, OR 3.4, 95% CI 1.6 to8.3). The providers in the intervention arm had better knowledge of active management of the third stage of labour (82.4% vs.35.8%, AOR (adjusted odds ratio) 10, 95% C.I 5.5 to 18.2); management of maternal sepsis (73.5% vs. 10.9%, AOR 36.1, 95% C.I 13.6 to 95.9); neonatal resuscitation (48.5% vs.11.7%, AOR 10.7, 95% C.I 4.6 to 25.0) and low birth weight newborn care (58.1% vs. 40.9%, AOR 2.4, 95% C.I 1.2 to 4.7). The case sheet audits revealed that providers in the intervention arm showed greater compliance with the protocols during labour monitoring (77.3% vs. 32.1%, AOR 25.8, 95% C.I 9.6 to 69.4); delivery and immediate post-partum care for mothers (78.6% vs. 31.8%, AOR 22.1, 95% C.I 8.0 to 61.4) and for newborns (73.9% vs. 32.8%, AOR 24.1, 95% C.I 8.1 to 72.0). The cost analysis showed that the intervention cost an additional $5.60 overall per delivery. Conclusions The mentoring program successfully improved provider preparedness and facility readiness to deal with institutional births and associated complications. It is feasible to improve the quality of institutional births at a large operational scale, without substantial incremental costs. Trial Registration ClinicalTrials.gov NCT02004912


The on-site MNCH mentoring (MM) for improved clinical care and service delivery intervention Intervention background and rationale
In the PHCs (primary health care centres) in the Sukshema project districts, only 63% of staff nurses have been trained in skilled birth attendance (SBA) and 12% of medical officers have been trained in BEmONC (Basic Emergency Obstetric and Newborn Care). In addition, there are no mechanisms after training for follow-up of staff to ensure good clinical practice and maintenance of skills. While basic training may provide the knowledge required, it often does not provide the confidence to translate the knowledge into practice, or there may be service delivery issues that hamper staff ability to practice good quality care. Service provider assessments conducted by the Sukshema project suggest that knowledge levels are low, and actual skills and abilities are even lower. In an assessment of intranatal care knowledge, less than 70% of staff nurses knew that AMSTL (active management of third stage of labour) was essential for all deliveries and only 28% knew the proper steps in AMSTL. For postnatal care, providers scored only 52% on knowledge questions, and on observation, their practice was correct in just 31% of provider/client interactions. Only 13% of providers were able to perform all the components of a postpartum physical exam. Clinical mentoring can complement in-service training to build competencies and confidence of providers on-site -that is, in the actual context where they are providing care.
However, clinical competency alone is not sufficient to ensure quality service delivery. The delivery of quality services is influenced by such things as staffing patterns and schedules, infection prevention practices, referral practices, equipment and supply logistics, record keeping, lack of a supportive environment for staff, lack of teamwork, and lack of staff attention to patient rights to information, respect and dignity. Of particular concern is weak referral processes and followthrough once referrals are made, which compromise providing a continuum of care for mothers and newborns. Attention to these aspects of service delivery requires that all staff at the PHC be involved in developing a common understanding of what it means to provide a quality service, their roles in providing these services across the continuum of care, and how to support each other and problem-solve to improve services.
Project Sukshema proposes to introduce an innovative strategy of providing on-site clinical mentoring and supportive supervision to staff nurses in PHCs (year 1 focus) through a dedicated cadre of nurse mentors. The aim is to develop and pilot a replicable model of on-site mentoring to ensure clinical competencies, strengthen referral processes and improve MNCH service quality.

Hypothesis
By providing on-site mentoring for improved clinical care and service delivery, we hypothesize that the quality of services and care will improve and patients will have better clinical outcomes. Specifically we believe that routine on-site clinical and service delivery mentoring will strengthen clinical knowledge, skills and practices among staff nurses, will strengthen appropriate referrals and follow up care, will improve other aspects of service quality, and provider and client satisfaction. The evaluation will assess change during the first year of the intervention that focuses on clinical mentoring of staff nurses and on quality improvement in the PHCs as a whole. The evaluation will focus on success indicators as shown in Appendix 1. Due to the short time frame for evaluation, success indicators will be principally at the levels of inputs, outputs, and outcomes focusing on adherence to the Sukshema Critical Care package, although we also hope to assess some key health outcomes, in terms of reduction of "near-miss" cases.

Intervention overview
Geography and site selection Using a cluster randomized control study design, the MM (MNCH mentoring) intervention will be evaluated in PHCs in two districts, Gulbarga and Bellary that have a total of 122 24x7 PHCs (Gulbarga 70; Bellary 52). Sixty one PHCs (Gulbarga 35; Bellary 26) will be randomly assigned to an MM intervention or experimental group (stratified by taluka and 6-monthly patient delivery load), while the rest will serve as control sites.

Pre-baseline (SBA+) training
The first part of the intervention involves all the 122 PHC 24/7 facilities in the two districts for two reasons: first we feel that this training is generally needed and will help to put the intervention and control site staff on the same level playing field in terms of basic MNCH knowledge; second, we know from the Sukshema baseline survey conducted in 2011 that few staff use adequate reporting mechanisms such as case sheets. As these are needed both programmatically as job aids, and for programme evaluative purposes, we will use this training as an opportunity to introduce new case sheets. After the first phase of baseline survey (described below) and before the mentoring programme begins in the intervention sites, Sukshema project staff will conduct a 3 day training update (called SBA+) for all staff nurses and a one day SBA+ for all M.O.'s from all 122 PHCs in the two districts, irrespective of whether they have previously attended the NRHM Skilled Birth Attendant (SBA) training. The 3 day training will focus on the essential components on intra-partum and postpartum (mother and baby) care identified in the Sukshema critical care package that evidence shows can significantly reduce maternal and newborn morbidity and mortality. The one day training for doctors will focus on identification of danger signs, pre-referral management of complications, quality of care, case audits and improvement of management and documentation. At these trainings, staff will be given, and instructed in the use of enhanced case sheets for women and babies from labour through to discharge, including use of the partograph. After this training, there will be no further interventions in the control sites, except that Sukshema staff will visit every 3 months to collect copies of the case sheets.

MNCH Mentoring
In each district MNCH mentors (MMs), a new staff cadre, will be recruited and trained (Gulbarga 5; Bellary 4), each responsible for the intervention in 6-7 experimental/intervention PHCs. We will also train 3 additional MMs to act as a back-up (2 in Gulbarga and one in Bellary), total 12 mentors. The MMs will attend some of the PHC staff training but will also be trained for 4 weeks by project staff at St John's Medical College in essential clinical competencies and in how to mentor staff in clinical skills, team building and problem solving, service delivery improvement, as well as in how to be effective mentors. The MNCH mentors will be senior nurses, selected based on specific criteria such as clinical experience, training and mentoring abilities, and experience/familiarity with government health systems. They will be guided by a Mentoring Coordinator in each district who will support the mentors' activities and liaise between them, the PHCs, district government staff and KHPT district staff. During the first year, the mentors will be trained by the Project to (a) provide clinical mentoring in safe delivery to staff nurses by hands-on coaching; (b) provide mentoring in how to improve the quality of services by building teamwork and introducing problem solving initiatives to address other aspects of service delivery (including especially referral processes); and in year 2 to mentor ANMs who will come to the PHC during the MNCH mentor visits, in providing antenatal, postpartum and newborn care, and referrals and follow up.
After the trainings, the MMs will visit each of their assigned intervention PHCs for 2-3 days every 2 months during the first year of the programme. The mentors will use tools and approaches such as observations, clinical audits, and interviews as aids to make an assessment of capacities in the facilities. In year one they will attempt to upgrade staff nurse skills through case reviews and audits, demonstrations and modelling of good practice, bed-side case discussions and small group teachings on such issues as partograph use, infection prevention, family planning, breastfeeding and complementary feeding, baby care, referral practices and record-keeping. Mentors will bring mannequins and other training aids to the site to provide skills practice for mentees in the event patients are not available or if additional practice is required. Although the focus of attention during the evaluation will be on the staff nurses, other staff who are involved in delivery and postpartum care will also be helped to improve patient care, for example the M.O.'s who are called to assist with complicated cases, the ward attendants who are responsible for infection prevention and other nurses who might help with postpartum care. Improvements in this area will be also be documented. Apart from clinical mentoring, the mentors will focus on problem-solving around all aspects of the provision of quality MNCH services. A team approach will be adopted to address specific problem areas of concern to staff such as (but not limited to): staffing patterns and schedules; infection prevention practices; referral practices; equipment and supply logistics; record keeping; staff support; teamwork; and staff attention to patient rights to information, respect, dignity and friendly services. Part of the quality improvement work will include review of all case sheets and log books/registers, collation of data with staff and discussion of case management, focusing especially on cases where there have been problems or complications requiring referral. Thus the intervention/experimental sites will have the full package of interventions (SBA+ training for staff nurses and doctors plus on site mentoring), whereas the control sites will merely have the short SBA+ training for staff nurses and doctors.
The State Institute of Health and Family Welfare, Government of Karnataka, has already expressed an interest in adding mentoring into existing capacity building systems and has requested the Sukshema project to demonstrate sustainable and effective mentoring systems. The project will continue to involve the state and district health officials and programme officers in the development/adaptation of manuals and curriculum, training and support for mentors, and monitoring the pilot intervention. The evaluation of the intervention will be shared with them to help in integrating this model in the remaining project districts.

Monitoring and Evaluation overview
Very little data exist on the utility of such a model and thus rigorous evaluation of this new intervention is required. Below, we outline an experimental design to answer the critical question of whether a training update plus regular on-site mentoring as described above, is more effective in improving services and achieving better patient outcomes, than merely provision of the training update. Using a cluster randomized control study design; the evaluation will be at two levels: routine on-going monitoring; and pre and post MM intervention evaluation.
The overall key objectives of the evaluation of this pilot project are: 1. To document operational details and the implementation processes, and evaluate the effectiveness of the intervention model to inform scale-up of the intervention, using 61 intervention sites and 61 matched control sites. The evaluation will focus on the list of success indicators shown in Appendix 1. 2. To accumulate data that will inform any modification of the intervention model and then institutionalize it outside of the pilot sites and districts.

Evaluation design
Experimental study design (cluster randomized control trial) will be used to evaluate the effectiveness of mentoring intervention.

Site selection and randomization
Bellary and Gulbarga districts that have a total of 122 PHCs will provide the sites for intervention. The sites will be randomized into intervention and control after the initial (SBA+) training.

Evaluation phases
The "pre-intervention" evaluation will have two phases. The first phase will take place in all 122 sites before any intervention activities begin (Figure 1). This evaluation will focus on site readiness to provide quality services, provider knowledge, attitudes and reported or observed (or simulated) practices, supervision and mentoring, record keeping, and provider and client satisfaction. The NRHM SBA guidelines, PHC 24/7 guidelines, MHFW NSSK guidelines, and the Sukshema critical care package will all be used to determine expectations of clinical quality of care at this level.
The focus of the first year of the MM intervention is enhanced clinical skills in the intra-partum and postpartum periods and thus assessments of actual clinical practices are important. However, data on actual practices in the intrapartum and postpartum periods will be difficult to collect by observation during this baseline (especially those with complications). Most of the PHCs conduct only a few deliveries each week, and it will be therefore almost impossible to obtain a representative sample of observations. Therefore we will assess intrapartum and postpartum clinical practices largely though the audit of case sheets and registers. However, these case sheets are currently either not available or not used. Thus our strategy necessitates delay of this part of the second phase of the baseline until after the update training where all staff will be given and instructed in the use of new and improved case sheets. All sites will be given the opportunity to practice using these patient case sheets for one month before they are collected by project staff and used as a basis for assessing both compliance with the new case sheets, adequacy of reporting, and actual case management. Actual case observations will be done during routine MM visits where possible. The post intervention phase of the assessment will take place in all sites once year after the first (planned 6) MM visits and will repeat the baseline survey.

Pre and post MM intervention evaluation survey tools
The survey will use four types of survey tools pre and post MM intervention and one additional tool after one year in the intervention sites only: A1. Questionnaires for all staff nurses on duty (including night duty) at the PHCs during the evaluation that will assess knowledge of intrapartum, postpartum, newborn care and infection prevention, and solicit provider perspectives on mentoring and supervision. In some cases, the questions will be in the form of case studies or asking for simulated demonstrations of practices.
A2. Facility audits and observations in all PHCs to assess site readiness to provide quality services and provide checks of drugs used: equipment and drugs, staffing and training levels, availability and correct use of case sheets, registers and referral slips. This tool will also have a section for observing practices in the postnatal ward.
A3. Postpartum client questionnaires with all women who have delivered up to one month before the survey traced to their homes that will assess client recall of information and services given and client satisfaction. Delivery registers will be checked and all women who delivered in the PHC n the previous month will be traced to their homes for interview.
A4. Case sheet and register audits for all cases for the one month before survey dates to assess completeness of reporting and appropriate client management A5. Focus Group Discussions with SNs in a sample of intervention PHCs: During the post MM intervention survey we will conduct focus group discussions (FGDs) with staff at a selection of intervention sites to understand their experiences with the MM intervention and to gather their suggestions for improvement.

Data collection
The field staff will consist of three supervisors per district, plus 13 teams of 2 interviewers, one of whom will manage the facility audits and staff surveys, and one the postpartum client questionnaires for each of the 122 PHCs. It will be necessary for each team to spend 2 days in each PHC, thus each team will be able to complete the survey work in 3-4 weeks. Simultaneous data collection will be carried out in both districts.
A week's training will be provided to the field teams that will include lectures on technical topics, demonstration interviews, mock interviews and field practice. The training will be provided to about 10% more persons than actually required for the survey, and the required number will be selected based on their performance during the training. KHPT and SJMC staff will conduct the training and will monitor the field work activities and quality of data.

Informed consent
Written, witnessed informed consent will be sought from the head of the PHC who will be asked to give permission to survey the PHC; from staff nurses who will be interviewed in the PHCs and from postnatal women traced to the community

Data processing
Double data entry will be done at KHPT, who will prepare a report of the pre-intervention survey and end-line survey comparison.

Data analysis
Changes in quality of care (facility readiness and staff competencies) will be compared before and after intervention within and between the intervention and control sites. Appropriate statistical tools will be used to control for any external factors.

B. Routine continuous monitoring
First, project staff will routinely document staffing, training, financial and other project inputs, data that will be crucial for assessing the feasibility of scale up to other districts. During bi-monthly MM visits, staff will have a proscribed list of data to collect. As well as keeping details of staff at the sites their training background and any clinical mentoring given, the mentors will have a proscribed list of clinical skills to observe; wherever possible during their 3 day visits, they will observe women in labour, after delivery and in the postpartum ward, and where this is not possible, they will use models and case studies to assess staff skills. There will be continuous monitoring of the MM interventions through reports of MM visits that will include: documentation of visits through trip reports that detail facility meetings, talks given, happenings in the PHC or area and notes on any morbidity and mortality discussions; completed clinical competency checklists for mentored individuals. They will also photocopy the case sheets, which will have been used as a basis for discussion during the visit and facility action plans, for review by KHPT clinical staff. The case sheets will also be collected quarterly from the control sites by the KHPT district staff. These staff will encourage use of the case sheets at each of their quarterly visits without any additional training or mentoring.

Routine monitoring survey tools
B1. MM trip reports B2. MM labour and newborn care observations checklists (or use of models where cases not available) B3. Facility action plans developed during self-assessment exercises during visits