Introduction

Although great strides have been made in maternal and newborn health in Nepal, the maternal mortality ratio (MMR) is still high at 186 per 100,000 births [1]. The rate of decline in the MMR has slowed down significantly despite increased utilization of maternal and neonatal health (MNH) services and visits. For example, between 2006 and 2016, institutional delivery by a skilled birth attendant (SBA) increased from 18 to 59% while the MMR only decreased from 281 to 259 per 100,000 [2]. In addition, substantial disparities exist in access to MNH services, with the urban population with higher socio-economic status benefiting while leaving poor and rural communities behind [3–5]. One factor for the slow improvement in MMR and in-country disparities is the actual quality of care being provided to mothers and newborns. Evidence suggests the MMR is highest among disadvantaged groups, who have poorer coverage of routine MNH visits and receive poorer quality of care during those routine visits [2]. Furthermore, our prior study showed that while countries like Nepal have made important investments in SBA programs, these healthcare workers are failing to receive either effective training or sufficient practice to stay clinically competent and knowledgeable in the field [6, 7].

In order to address issues of the quality of skilled birth attendants and their training, in 2011, the National Health Training Center (NHTC) in collaboration with the Nick Simons Institute (NSI) devised a novel program called the Follow-Up Enhancement Program (FEP). The goal of FEP was to cultivate a pioneer program of supportive supervision where the clinical knowledge and skills of SBAs who have received in-service training are assessed directly at their working sites [8]. The objectives of FEP are to assess the knowledge and skills of SBAs after training, explore the reasons for skill and knowledge weaknesses, determine how much practical experience SBAs have in conducting deliveries and key maternity procedures, describe what factors affect the ability of SBAs to implement tools acquired in training and provide future recommendations. In addition, the FEP allows on-site coaching and feedback so that trainees may continue to improve upon any gaps in their skills and knowledge. FEP furthermore includes an assessment of the trainees’ working environment in terms of essential equipment, drug supply, and team support, collectively termed the Enabling Environment Score. A good enabling environment is crucial to overcoming poor service utilization or poor service provision for quality maternal and neonatal health services [9, 10]. The clinical skills, knowledge, and enabling environment assessments from FEP are then fed back to the Health Facility Operation and Management Committees (HFOMC) at district and central levels so that gaps can be fulfilled [8].

One of the main components used for the assessment is a paper FEP tool (see Supporting Information), administered to individual SBAs by SBA trainers, who are doctors and nurses who have completed an SBA course, Clinical Training Skills course, and National Health Training Center certification, and have 1–4 years of SBA training experience, with the assistance of NSI and public health nurses. The SBA FEP tool was drafted by SBA trainers, Support for Safer Motherhood Project, and the Family Health Division and was pretested in five districts, the findings of which were disseminated to stakeholders and used to revise the tool. The assessment, which included background information, knowledge and clinical assessment, clinical decision-making, infection prevention, and an enabling environment checklist, took approximately 5 hours for each SBA to complete [6, 8].

As of 2021, NSI and NHTC had implemented FEP in 47 districts throughout Nepal, visiting 762 birthing centers with a follow-up of 1408 SBAs [11, 12]. However, to date, there has not been a study to assess the actual effectiveness of FEP. This study, therefore, aims to:

1. Assess the impact of FEP on the knowledge and clinical skills of SBAs and the enabling environment of the health facilities in which they work
2. Assess the impact of FEP on the number of deliveries being conducted by SBAs
3. Assess stakeholder perceptions regarding FEP delivery, design, implementation, and follow-up

Methods

We used a convergent mixed method approach whereby we triangulated qualitative and quantitative data. For the quantitative component, the clinical knowledge and skills and enabling environment of 73 SBAs working in birthing centers who had undergone FEP as well as 93 SBAs working in birthing centers who had never had FEP were assessed. All the SBAs included in the study were nurses (auxiliary nurse midwife or staff nurse). Furthermore, we assessed clinical knowledge and skills of 22 SBAs working in a facility with other SBAs that had previously undergone FEP. Four external consultants who were SBA trainers were hired to assess the clinical knowledge, skills and enabling environment. The knowledge of procedures and managing complications was assessed using a 20-question multiple-choice survey [6, 8]. The clinical competence level was evaluated using standardized checklists on anatomical models and clinical case vignettes for the use of partograph, normal and vacuum delivery, newborn resuscitation, referral, and care of pregnancy complications [6, 8]. The enabling environment was assessed by a FEP trainer using a thorough checklist that included each facility’s equipment, infrastructure, and medications [6, 8]. Furthermore, the total monthly deliveries for each SBA was assessed by reviewing each facility’s delivery logs, and information on the number of normal deliveries, vacuum deliveries, breech deliveries, and the management of any of the following seven complications over a 3- month period were recorded. These complications included postpartum hemorrhage, antepartum hemorrhage, eclampsia, episiotomy, manual placenta removal, post-abortion care, and newborn resuscitation [6, 8]. The statistical significance level was set at a p-value threshold of 0.05, indicating that any result with p-values less than 0.05 was considered statistically significant.

Qualitative data were collected simultaneously using semi-structured interviews and focus group discussions from 16 birthing centers in 3 districts (Bardiya, Makawanpur, Bajhang) where SBAs received FEP intervention. Semi-structured interviews were conducted with 30 SBAs, and 3 District Health Officers (DHO), 3 Public Health Nurses(PHN) and 1 central level stakeholder. Focus group discussions were conducted with 16 HFOMC members and 2 focus group discussions were conducted with SBA trainers at 2 training sites. There were 6 to 8 people in each focus group discussion, and discussions took up to 45–90 minutes in duration. Topic guides were developed in an iterative process throughout data collection to guide the interviews and discussions. An external team of trained Nepali field researchers collected qualitative data. They were oriented to the purpose of study and topic guides for 2 days prior to field visits. Participants gave written informed consent to participate. All the discussions were digitally recorded in the local Nepali language and transcribed and translated into English. 5 pages of 3 randomly selected transcripts were back translated into Nepali and compared with recordings to check the quality of translation.

Sampling

Altogether 6 districts (3 have had FEP intervention and 3 never had FEP intervention) were selected purposefully from three ecological areas (Mountain, Hill and Plain) across Nepal (Fig 1). During the year 2016, the FEP team visited 5 districts. For this study, we took a purposive sample of 3 districts, in which 2 districts included the highest number of FEP SBAs working in the same birthing site in 2017 as when the FEP baseline survey was conducted in 2016. The three districts were Bardiya (n = 27), Bajhang (n = 20), and Makwanpur (n = 26) and included a total of 73 SBAs who had a pre-FEP assessment in 2016 (A2016 Cohort, n = 73) and who were then followed up for a post-FEP assessment in 2017 (Cohort A2017, n = 73). In addition, there were 22 new SBAs (Cohort C2017, n = 22) who had never received FEP but worked alongside the other 73 SBAs who had previously had FEP in 2016. We also collected data in 2017 on SBAs from 3 additional districts that had not previously had FEP—Dadeldhura (n = 37), Syangja (n = 26), and Siraha (n = 30)—(Cohort B2017: total n = 93) to compare knowledge, skills, enabling environment, and deliveries in FEP implemented and not implemented SBA districts (Fig 2). We excluded districts where FEP was implemented before the year 2016 and districts where only a small number of SBAs who previously participated in FEP are working in the same district. A total of 188 SBAs remained after all inclusion and exclusion criteria were applied. Data was collected from December 2017 to May 2018.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Nepal Map with study sites.

(A) License: Creative Commons Attribution–ShareAlike 4.0 International License. Caption Credit: geoBoundaries-NPL-ADM2-PREVIEW.png (640×480) (raw.githubusercontent.com). (Accessed March 26, 2023), and Runfola, D. et al. (2020) geoBoundaries: A global database of political administrative boundaries. PLoS ONE 15(4): e0231866. https://doi.org/10.1371/journal.pone.0231866. (Accessed March 23, 2023) [13, 14].

https://doi.org/10.1371/journal.pone.0285653.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Flow diagram depicting a timeline of training and assessments of SBAs.

https://doi.org/10.1371/journal.pone.0285653.g002

Data management and analysis

Quantitative data was collected in paper form and entered into Stata and analyzed.

Participants were assigned identification numbers and data was made anonymous. The pre-post analysis was done to assess if there is a difference between the same SBAs a year after they undergo FEP in terms of knowledge, clinical skills, and the number of deliveries and whether there was a change in their enabling environment. The intergroup comparative analysis was done to assess any difference between SBAs that had a one-time FEP and those who never had it, and any trickle-down effect of working in a FEP facility (Fig 3). For qualitative data, we conducted a thematic content analysis, whereby the research team read a sample of the transcripts. A descriptive report of the preliminary analysis was written by researchers after which the research team independently generated themes from the data and came to a consensus through discussion. The data were coded according to the themes in NVivo 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Flow chart depicting the analysis framework of the study.

https://doi.org/10.1371/journal.pone.0285653.g003

Results

Quantitative findings

Pre-post analysis.

A total of 188 SBAs were sampled. 73 SBAs were still present at the facility where prior FEP had been conducted and underwent pre- and post- FEP analysis. Another 22 SBAs were working at the same facility where prior FEP had been conducted and were also assessed. 93 SBAs were from 3 districts where FEP had not been conducted. SBAs on average exhibited an improvement in knowledge, clinical skills, enabling environment, and total deliveries following a one-time FEP. As seen in Table 1, of the total 73 SBAs still present at the facility where FEP had been conducted, total knowledge scores improved with one-time FEP by approximately 9%. SBAs who received FEP achieved a total mean score of 84% on the written knowledge assessment compared to the 75% total mean score achieved by SBAs before receiving FEP (p<0.001). After receiving FEP, SBAs on average achieved 8.22% higher on partograph knowledge (p = 0.03), 10.96% higher on PPH knowledge (p = 0.16), 14.38% higher on infection prevention knowledge (p<0.001), and 18.72% higher on vacuum delivery knowledge (p<0.001).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Pre-post analysis of differences in knowledge, clinical skills, and enabling environment scores.

https://doi.org/10.1371/journal.pone.0285653.t001

Total clinical skills scores improved with one-time FEP by approximately 29%. SBAs who received FEP achieved a total mean score of 75% on the clinical skills assessment compared to the 46% total mean score achieved by SBAs before receiving FEP (p<0.001). After receiving FEP, SBAs on average achieved 32.19% higher in the management of eclampsia (p<0.001) and 37.75% higher in the management of shock (p<0.001). However, there was a 3.03% decrease in vacuum delivery skills following a one-time FEP, but this was not found to be statistically significant (p = 0.48).

An examination of the total enabling environment scores revealed that a one-time FEP improved the enabling environment scores by approximately 7%. SBAs who received FEP achieved a total mean score of 78% on the enabling environment assessment compared to the 71% total mean score achieved by SBAs before receiving FEP (p = 0.09). Upon analysis, the components of the enabling environment score that improved significantly with one-time FEP were labor equipment, general equipment, supplies, and team support.

In addition, as seen in Table 2, we also assessed whether there was a difference in the number of deliveries conducted by SBAs pre-and post-FEP. After receiving FEP, SBAs conducted 4.7 additional total deliveries (mostly normal deliveries) over 3 months, although this increase was not found to be statistically significant (p = 0.17). There was an increase of 4.76 normal deliveries (p = 0.15), an increase of 0.02 breech deliveries (p = 0.79), and a decrease of 0.1 vacuum deliveries (p = 0.39) following a one-time FEP, but these were not found to be statistically significant.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Pre-post analysis of differences in deliveries between sbas who had pre-assessment in 2016 and post-assessment in 2017.

https://doi.org/10.1371/journal.pone.0285653.t002

Intergroup comparative analysis.

Table 3 examines if there is any difference between 73 SBAs that underwent FEP compared to 93 SBAs that had not yet had FEP in 2017. The SBAs that had already undergone FEP had total knowledge scores of approximately 84% compared to the 80% total knowledge scores achieved by the SBAs that were yet to undergo FEP (p = 0.04). In terms of clinical skills, the SBAs that had undergone FEP achieved total scores of 75% compared to 41% scored by those that were yet to undergo FEP (p<0.001). Additionally, there is an observed difference in the enabling environment of SBAs that underwent FEP compared to the SBAs that had yet to undergo FEP. Those that already underwent FEP achieved total enabling environment scores of 77% compared to the 57% scored by those that were yet to undergo FEP (p<0.001). In terms of total deliveries, there is a 5.46 increase in deliveries over 3 months for the 73 SBAs that underwent FEP compared to the 93 SBAs that were yet to undergo FEP. This difference, however, is not found to be statistically significant (p = 0.10).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Differences in knowledge, clinical skills, enabling environment, and deliveries between SBAs who were post-FEP in 2017 compared to SBAs that were yet to undergo FEP in 2017.

https://doi.org/10.1371/journal.pone.0285653.t003

Table 4 examines the effect of working at a site that had previously undergone FEP. It compares 93 SBAs who had not yet undergone FEP in 2017 with 22 SBAs who had also never undergone FEP but were working at a site where FEP was previously conducted. SBAs working at a prior FEP site had total knowledge scores of approximately 87% compared to 80% total knowledge scores achieved by SBAs not working at a prior FEP site (p = 0.01).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Differences in knowledge, clinical skills, enabling environment, and deliveries for non-FEP SBAs in 2017 compared to non-FEP SBAs that worked at FEP facility in 2017.

https://doi.org/10.1371/journal.pone.0285653.t004

There is an observed difference in clinical skills with SBAs working at a prior FEP facility having skills scores of 81% compared to 41% for those not working at a prior FEP site (p<0.001). This is also true in the observed difference in enabling environment with SBAs working at a prior FEP site achieving enabling environment scores of 83% compared to 58% for those not working at a prior FEP site (p<0.001). In terms of total deliveries, SBAs working at a prior FEP site conducted 7.88 additional deliveries over 3 months compared to SBAs that worked at non-FEP sites. However, this difference was not found to be statistically significant (p = 0.39).

Qualitative findings

Positive perceptions of FEP.

Our qualitative interviews and data found that FEP is an ideal monitoring and support program that is highly accepted and appreciated (Fig 4). It was found that the FEP is beneficial to all stakeholders involved—SBAs, trainers, and the HFOMC. FEP updates knowledge and skills, energizes and builds confidence, and establishes linkages between SBA trainers and HFOMC, with one SBA from mountain district (Bajhang) stating, “It’s been a long time since we took SBA training. We may forget. Patient flow is low here. That is why it [FEP] refreshes us. We can revise whatever we have forgotten. We can correct our mistakes.” Additionally, SBAs felt that FEP enhanced their ability to conduct deliveries according to protocol. In the words of one SBA working in the birthing center of the Plain area (Bardiya) “Now we can successfully conduct deliveries using the methods of infection prevention in a skilled way. We used to do deliveries in a haphazard way prior to the [FEP] training but after the training, we are doing deliveries in a systematic way.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Positive aspects of the follow up and enhancement program (FEP) from the perspective of skilled birth attendants, the health management committee, district level managers, and trainers.

https://doi.org/10.1371/journal.pone.0285653.g004

FEP also updates knowledge and skills, provides an opportunity to see real-world scenarios, and establishes linkages between the different stakeholders involved, with one trainer from Bharatpur recounting, “Providing messages only is not enough. If we go to the FEP, then we can understand how the SBA trainees are doing at their sites of work, whether they are doing what we taught them or not.” FEP is beneficial to the HFOMC because they become aware of what their responsibilities are. It energizes the HFOMC and establishes a linkage between the HFOMC and SBAs, with one HFOMC member in Hill area (Makawanpur) reporting, “Communication is frequently made with the staff. We are also aware of the various aspects of their work so that a better working environment can be created, and maximum service can be provided. We focus our concentration on making things easier for the staff.”

Improvements needed in FEP.

Although FEP was generally well received, informants offered several suggestions to improve the program (Fig 5). Most participants wished that FEP occurred at more regular time intervals, with every six to twelve months as an ideal timeframe. In addition, more frequent intervals would also keep the health management committee more responsible for completing their tasks. One member of the HFOMC of the Plain area (Bardiya) stated, “If you visited our organization every six months…we can remember. We cannot remember about the activities which were conducted 1.5 years ago.” In addition, participants generally would have preferred FEP to occur over a few days with a shorter working day, with one SBA of Hill area (Makawanpur) lamenting, “This one-day duration is too little, and we have to do everything in a hurry.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Improvements for the follow-up and enhancement program (FEP) from the perspective of skilled birth attendants, the health management committee, district level managers, and trainers.

https://doi.org/10.1371/journal.pone.0285653.g005

Some respondents wanted to receive equipment and incentives after participating in FEP. For instance, one member of the HFOMC from the Plain area (Bardiya) said, “As they come here and observe and tell us the things we need to improve, but it would be good if they manage (help us get or give us) materials too.” In terms of the training, participants also wanted more educational materials, training during live deliveries, training more applicable to the environment of uncomplicated deliveries, and in some cases better teaching methods. Besides providing educational materials in person, a few participants mentioned sending updated materials throughout the year.

Discussion

Our study evaluated the effect of FEP on the knowledge, clinical skills, enabling environment, and number of deliveries of SBAs in Nepal. We found that a one-time FEP significantly increases the knowledge and clinical skills scores of SBAs. Additionally, we found that a one-time FEP significantly improves the enabling environment of the health facilities in which the SBAs work. There was also an improvement in the delivery volume conducted by SBAs following a one-time FEP, although this was not statistically significant. Our study found that there is even a trickle-down effect of working in a facility that has had prior FEP, with SBAs that have never undergone FEP showing improved clinical skills.

The improvement in knowledge and clinical skills through an on-site mentoring program like FEP has been confirmed in other studies. A study conducted in Karnataka, India from 2012–2014 implemented a nurse mentor program at PHCCs to improve quality services [15]. Using a similar evaluation tool, the study found that the nurse mentoring program improved the staff’s knowledge and skills. Similar to our study, the checklist evaluation tool implemented in this study was also used to improve the working environment in terms of infrastructure, equipment, and drug supply. However, this study did not analyze any quantitative data and the results were all qualitative. A more recent study conducted in Nepal between 2016–2018 assessed an on-site clinical mentoring program and its effect on mentees’ knowledge and skills [16]. Of the 308 nurses assessed, clinical skills scores increased significantly, with SBA-trained mentees having better knowledge scores and performance on the 12 core clinical skills assessed by the program. There has been one randomized controlled trial also in Karnataka, India that used a checklist evaluation tool in conjunction with on-site nurse mentoring as the study intervention [17]. This study found that on-site mentoring, used with a checklist evaluation tool, improved knowledge, and skills by 2.3 times, compared to just a checklist evaluation tool alone, further supporting the value of hands-on mentoring.

In addition, both the qualitative and quantitative data suggested that the FEP program establishes communication linkages between the HFOMC and the SBAs, which is crucial for improving the enabling environment. One SBA working in the birthing center of Bajhang district stated, “During FEP feedback, we discussed and met with the management committee. Now, it (HFOMC) has become active. They came to know their responsibilities. They listen to us and fulfill the essential materials we need”. A study conducted in Nepal between 2014 and 2015 found that the HFOMC’s role was confined to mobilizing additional resources for the health facility, supporting infrastructure, and raising health awareness [18]. There was limited communication between the HFOMC members and the SBAs. There is a strong need for a program that establishes linkages between the HFOMC and SBAs and other staff at health facilities so that they can be involved in health needs assessment, planning, staffing, and fund management to promote a positive enabling environment at the health facility.

Many of the SBAs and other stakeholders involved in FEP reported that a one-time FEP was not enough and that it must be carried out with greater frequency. This need for continuous mentoring has been supported by other studies including a study conducted in Bihar, India that evaluated 319 public PHCCs and compared the intrapartum quality of care between 179 mentored and 80 non-mentored PHCCs [19]. This study found that the task completion scores were higher among mentored nurses but plateaued after one year, suggesting that mentoring needs to be ongoing. Other studies, such as one conducted in Nepal, looked at 104 SBAs and found that SBAs that underwent repeat training demonstrated greater clinical skills than those who did not [20].

The investigation boasts several strengths, including a mixed methods research design that integrates both quantitative and qualitative research methods to provide a more comprehensive understanding of the effectiveness of the Follow-Up Enhancement Program (FEP). The quantitative section of the study is strong in that it includes not only a pre-post analysis but also a comparative analysis of data from three additional districts that had not previously had FEP, enabling a comparison of skilled birth attendants (SBAs) in FEP versus non-FEP districts. Importantly, the study draws actionable conclusions that recommend frequent and continuous FEP to retain the knowledge and clinical skills of SBAs, motivate them through on-site coaching, and improve their working environment through direct feedback to the Ministry of Health and Population. These conclusions provide clear recommendations for policymakers and practitioners to improve maternal and neonatal health in Nepal.

Limitations

Though we sampled a total of 188 SBAs for our study, this is a relatively small sample size which may affect generalizability. Moreover, we assessed SBAs only in Nepal, and the results may be specific to this region which could further impede generalizability. As with any observational study, we cannot determine causality. Residual confounding could also be present even after controlling for other factors.

Policy implications

Although the SBA program is currently being phased out in Nepal to be replaced by a midwifery curriculum, many of the lessons from FEP for SBAs can be applied to FEP for midwives as well. FEP allows SBAs who have been trained at various SBA training sites in the country to be assessed with regard to their knowledge and clinical skills as well as the number of deliveries they are conducting. Our prior study of the quality of SBAs in Nepal found that SBAs are conducting very few deliveries, with only 7 percent (38/511) meeting the minimal volume recommended to maintain competence by the WHO, and a substantial fraction (70/511, 14%) performing an average of no monthly deliveries at all [6]. On multivariable analysis, one statistically significant predictor of clinical skills scores and monthly delivery volume was training at a continuously monitored and supervised site. Thus, the FEP provides some measure of accountability for various SBA training programs and sites. FEP also allows for ongoing and effective continuing medical education. By comparing the clinical knowledge and skills of SBAs who received FEP to non-FEP SBAs it was found that FEP significantly increased knowledge and skill level. If a one-time FEP can result in this level of improvement, a continuous FEP program will surely bring about improved and ongoing competency of SBAs.

A frequent and continuous FEP program establishes linkages between SBAs, trainers, and HFOMCs. As one HFOMC in Bajhang reported, “…the discussion can be made regarding the strengths and weaknesses of the health facility. It has also acted as a connecting medium between public representatives and health personnel at the facility, district, and central level as well.” Creating a system of feedback is vital to ensuring that knowledge, skills, and the enabling environment meet the marks required for proficiency and practice. This allows for an open discussion regarding the strengths and weaknesses of health facilities which then opens the window to discussions regarding improvement. By connecting health care personnel, the HFOMC, and the central government, the FEP program is successful in cultivating communication and awareness of the responsibilities among the stakeholders involved. Thus, utilizing the findings from the FEP tool, policies to improve and standardize the working environment in turn improve the quality of SBAs and the care that they can provide, ultimately improving health outcomes.

Conclusion

Overall, FEP is a highly effective program by both quantitative and qualitative evaluation. A one-time FEP significantly increases the knowledge and clinical skills scores of SBAs as well as the enabling environment of the health facilities in which they work. Our study suggests that FEP should be frequent and continuous. Continuation and scaling up of the FEP program are vital to retaining the knowledge and clinical skills of SBAs, motivating them through on-site coaching, and improving the working environment through direct feedback to the MoHP. FEP allows for an intimate understanding of the local working environment of SBAs and thereby leads to more effective health policies.

Supporting information