Background

Before Odysseus leaves on his journey in Homer’s ancient poem, The Odyssey, he entrusts his son Telemachus’ welfare to his closest and oldest friend, Mentor. As guardian, friend, sponsor, and counselor to Telemachus, Mentor then defines the oldest documented practice of mentorship which continues to play a critical role in society today [1]. Mentorship may be described most generally as having the foundational elements of knowledge sharing and learning promotion throughout its many forms [2]. Research mentorship can help mentees improve scientific and grant-writing skills, promote career development, and help the mentee transition to scientific independence [3]. Although mentorship often describes a specific relationship between two people, institutionalizing research mentorship is important [4].

Institutionalizing research mentorship nurtures research capacity in organizations (e.g., universities, professional associations, and research institutes) to improve research effectiveness and health equity [5]. Comprehensive research mentorship resources such as toolkits and practical guides can promote effective institutional research mentorship [6]. However, most of the available resources for research mentorship are tailored for high-income countries (HICs) [7]. Research institutions in HICs may have a longer tradition of supporting research mentorship and have greater institutional resources to sustain mentorship compared to LMICs [8].

Existing mentorship resources focus on the mentor-mentee relationship [8]. They often do not take into consideration the many forms of mentorship and the wider institutional factors (such as existing resources, cultural norms) that contribute to research mentorship [8]. There is a need for the development of an institutional research mentorship guide for LMIC researchers [9]. This paper answers the research question: Using a Modified Delphi process, how can we develop and build consensus for items within a practical guide focused on enhancing institutional research mentorship in LMICs? The Delphi method is a consensus-building process which is widely used by researchers to achieve consensus in diverse fields. The Delphi method is based off of the assumption that aggregated group opinions are more accurate than individual opinions [10]. The objective of our research is to build consensus on a core set of items on enhancing and institutionalizing research mentorship in LMICs to be included in a research mentorship guide.

Methods

This study employed a two-round modified Delphi process informed by a related scoping review [11] crowdsourcing open call and an in-person conference [12]. The Delphi process has been widely used to achieve consensus in a structured way. The traditional Delphi process uses iterative survey stages with controlled feedback, statistical group responses including “agreement” levels, as well as solicits the opinions of topic experts [10]. Since the advent of the traditional Delphi process, many researchers have adapted the process to suit their research needs and address potential limitations of the traditional method [13]. While the traditional Delphi method involves anonymous responses, our study was not anonymous as we were interested in measuring the demographics of respondents. Another modification to our study was that we employed a group of panelists to revise the statements instead only one facilitator which is traditional for the Delphi method. Additionally, this modified Delphi process employed a community-based participatory research (CBPR) approach which emphasizes the importance of equity and knowledge sharing in research [14]. We chose to use the Delphi method instead of only in person focus groups at the conference because we wanted to involve as many people as possible. Not everyone on the expert Delphi panel attended the in-person meeting. As such, limiting consensus building to the in person only process would have likely decreased total responses and engagement. Additionally, we wanted to compare conference and non-conference participants to see if being in person had an effect on the level of engagement.

The modified Delphi consensus process included the administration of two online surveys. The surveys were administered two weeks apart from each other in June 2022. We asked participants to indicate their level of agreement with each statement using a 5-point Likert scale (5 = strongly agree, agree, neutral, disagree and 1 = strongly disagree) and participants had the option to provide additional comments on each statement item. We chose to use a 5-point Likert scale due to its simplicity to construct and ease for participants to complete. A 5-point Likert scale produces a simple yet reliable measure of agreement which was necessary for this study [15].

We received a diverse mix of participants across age, gender, role, and work experience. Of the 36 total participants, 61% (22/39) of our participants identified as female while 39% (14/36) identified as male. Additionally, 66% of our participants identified as being from a low-income country or lower-middle income country. Additional information on participants can be found in Table 1. Given the focus of the study, for the Delphi we invited persons with interest and experience regarding institutional health research mentorship in LMIC settings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Expert participant demographics for the first-round modified Delphi survey.

https://doi.org/10.1371/journal.pone.0291816.t001

Consensus panel participants (n = 22) included people identified from the original crowdsourcing open call, steering committee members, and others with expertise. The role of the consensus panel was to put together sections of the consensus document for the research guide and to edit the first draft. Participants included LMIC researchers, mentors, mentees, institutional leaders, and funders. We invited panelists from the open call steering group, finalists and participants on the open call, and members from the collaborating network (TDR Global, AHRI, SESH). There is not enough data from these groups to do a sub analysis of each of these groups, but we asked their identifiers in order to have an idea of the diversity of participants.

Prior to the administration of the modified Delphi surveys, a consensus level of ≥80% was agreed. This is including a response to “agree” and/or “strongly agree” to any statement item on the surveys based on recommendations from the literature. Any statement item from the surveys that achieved 100% agreement were graded as ‘U’ (unanimous); 90%–99% agreement was graded ‘A’; and 80%–89% agreement were graded ‘B’. The consensus panel reviewed the second-round survey grading to determine which items would be included within the final consensus statement for the research mentorship guide.

Round 1

The development of the first-round Delphi survey was based on the statements generated from the open call and scoping review findings by the virtual working group (Fig 1). This was based on existing guidelines on Delphi statement development [16]. The first survey consisted of 39 statement items. The first four items were the participant demographic questions, and the remaining 35 items were the consensus statement items for the research mentorship guide. The facilitator invited participants to the first online survey created with JotForm via email in June 2022.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. An overview of the four main stages of the institutional research mentorship guide development.

https://doi.org/10.1371/journal.pone.0291816.g001

After participants completed the first-round survey, the core team members compiled the survey results, analyzed consensus level for each statement item and graded them based on the grading scheme described above. We were interested in identifying key themes from the first survey comments. Given that modified Delphi processes are founded on the importance of more than one opinion, we analyzed comments on statement items where two or more participants wrote in similar themes whether it agreed, disagreed, or was neutral with the statement item.

Following the first-round survey, Armauer Hansen Research Institute (AHRI) in partnership with The Special Programme for Research and Training in Tropical Diseases (TDR) hosted an in-person ‘Institutional Research Mentorship Development’ conference in Addis Ababa, Ethiopia from June 23rd to June 24th, 2022. The purpose of the in-person conference was to gather the expert participants to work together on finalizing the research mentorship guide. Having an in-person discussion is useful for consensus building as point of disagreement could be more easily solved in an in person setting. Additionally, we wanted to make sure there was representation from AHRI as they were partners in the study.

Results

There were 61 expert participants recruited for this modified Delphi process however we received survey responses from 36 participants. The full participant demographics can be shown in Table 1 for the first survey.

Round 2

There were 14 people who joined both days of the conference held in Addis Ababa, Ethiopia. These conference attendees included the crowdsourcing open call finalists, virtual working group members and AHRI researchers. The conference participants met for two days to finalize the institutional research mentorship practical guide and complete the consensus process by revising the first survey through an open discussion.

The second-round survey was an iteration of the first survey with the comments incorporated from each statement item and from the discussion at the in-person conference in Ethiopia. While all the items achieved consensus after Round 1, we wanted to incorporate the feedback received in order to achieve a higher level of consensus. We wanted to ensure that all statement items had been reviewed and revised by our participants. This survey had 42 items: 1–4 were the participant demographic items, 5–39 were the consensus items and 40–42 were items to correctly acknowledge participants for their contributions to the final practical guide. We emailed out the survey to the 61 participants and we received 32 responses back for a response rate of 52%. The participants also reached consensus (≥ 80% agreement level) on all consensus items in the survey. The development and revision of each statement was a multi-step process and therefore listing the changes at each phase would be beyond the scope of the study. The final consensus statement items and their associated agreement levels is shown in Table 2 below. Fig 2 shows the definition and three main sections of institutional research mentorship that we created through the results of the modified Delphi process [5]. Both Fig 2 and the items in Table 2 were included in the full consensus document for the practical guide (S1 File).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Infographic of the three sections of institutional research mentorship, adapted from the TDR guide.

https://doi.org/10.1371/journal.pone.0291816.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Final abbreviated consensus statement items with associated degrees of consensus from the second-round modified Delphi survey1.

https://doi.org/10.1371/journal.pone.0291816.t002

Discussion

We developed a consensus document (Appendix 1) to enhance research mentorship in LMIC institutions. In both rounds of the modified Delphi surveys all consensus items reached the predetermined consensus level of ≥ 80%. Through the Delphi study, we were able to achieve our key objective which was to develop a consensus document on institutionalizing research mentorship. The HERMES guide was published in October of 2022. We were also able to meet our secondary objective which was to compare engagement levels of conference and non-conference participants to see if being in person had an effect on engagement. This study expands the literature by using crowdsourcing in a consensus process.

In our analyses we found an increased survey response rate and engagement rate for the conference participants compared to the non-conference participants. This is consistent with the findings of other studies that measured engagement rates of online conferences [17–19]. Both groups were informed that they would be able to get access to the final guide and receive credit for their contributions if desired. The response rate may have been higher for the in-person group because they were given dedicated time to complete the survey. Additionally, by virtue of this group being physically at the conference with the common goal to produce a guide from which they would all benefit, these participants may have been more invested in seeing the final product come to fruition. We also found that the in-person conference did not artificially increase the strength of agreement by means of collective conversation or other group mentality, but rather both the in-person and at-home participants were equally strong in their agreement levels. This stands in contrast to other reported observations, where the sole use of conferences dedicated to consensus can take longer to execute and can be more susceptible to group conformity [20]. However, we are recommending future Delphi studies to include both virtual and in-person aspects. As previously mentioned, an in-person aspect is important for connection and consensus building while a virtual aspect ensures diversity and inclusion of all participants.

This modified Delphi process helped show key aspects of institutional research mentorship that researchers in LMICs found important. These key aspects included the three sections that informed the definition of institutional research mentorship: the research mentorship life cycle, leveraging of existing resources and the measurement of institutional research mentorship (Fig 2). Based on the preliminary analysis of the key themes from the first-round survey (Table 2) it was evident that the participants valued the inclusion of equity and fair, dynamic processes within the final practical research mentorship guide. There was also an emphasis placed on considering current cultural practices within institutions as the institutions themselves begin to foster research mentorship. Additionally, the participants acknowledged that institutions should consider appointing a mentorship leader or dedicating an office to mentorship in order to truly embed mentorship into the institution and sustain the practice.

Our study has several limitations. First, there is no standardized way to measure the reliability of a modified Delphi process [21]. However, we used established methods [22] and complemented them with participatory crowdsourcing methods. Second, not all participants attended the in-person conference in Ethiopia. To mitigate this problem, we organized a hybrid meeting session during the in-person conference where working group members not attending could join remotely. Third, the framing of the open call focused on strategies to enhance research mentorship and not barriers to research mentorship. As a result, less can be stated about barriers to institutional research mentorship. Specific to our application of this modified Delphi process was the potential limitation of the survey being too long for participants to remain engaged or to be willing to complete both surveys. This limitation is often referred to as survey fatigue [23]. However, our results were encouraging in that the response rate only decreased by 7% between the two surveys with over 50% of participants responding in both.

This study has implications for public health policy and research. From a policy perspective, the use of remote data collection capabilities reduced the cost and time-to-consensus of the decision-making process [24]. Including both an in-person and virtual option was key to the success of our consensus process. Additionally, this consensus process helped in the development of the HERMES practical guide which contains methods for enhancing institutional research mentorship. The guide has been disseminated across the various TDR Global partner organizations. The HERMES guide provides general ideas for enhancing research mentorship in LMICs, however, there are many areas that could be further explored such as gender and age. We suggest that further research be conducted into understanding and quantifying the value of having multiple modalities in the consensus building process. A gap currently exists in the literature regarding clear guidelines for qualitative data analyses of modified Delphi processes [25]. Assessment of the Delphi process could help to iteratively improve the method.

Conclusion

With a 2 round modifies Delphi process, we achieved consensus on strategies to improve research mentorship among LMIC organizations. The results from this study have directly informed a World Health Organization practical guide that is now being disseminated [5]. However, there remains unanswered questions and areas that require further research and policy attention.

Supporting information