Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Comparing undergraduate research experiences before, during, and after the COVID-19 quarantine: The successful adaptation of the BUILD PODER Summer JumpStart program

  • Patricia Escobedo,

    Roles Supervision, Writing – original draft

    Affiliation Department of Psychology, California State University, Northridge, Northridge, California, United States of America

    ⨯
  • Daniel Garcia,

    Roles Writing – original draft, Writing – review & editing

    Affiliations Department of Psychology, California State University, Northridge, Northridge, California, United States of America, School of Education, University of California, Irvine, Irvine, California, United States of America

    ⨯
  • Liam Cascelli,

    Roles Data curation, Formal analysis, Writing – review & editing

    Affiliations Department of Psychology, California State University, Northridge, Northridge, California, United States of America, Center for Assessment, Research, and Evaluation (CARE), California State University, Northridge, Northridge, California, United States of America

    ⨯
  • Gabriela Chavira ,

    Roles Conceptualization, Funding acquisition, Supervision, Writing – review & editing

    gabriela.chavira@csun.edu

    Affiliation Department of Psychology, California State University, Northridge, Northridge, California, United States of America

    ⨯
  • Gilberto E. Flores,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of Biology, California State University, Northridge, Northridge, California, United States of America

    ⨯
  • Jodi L. Constantine Brown,

    Roles Data curation, Methodology, Writing – review & editing

    Affiliations Center for Assessment, Research, and Evaluation (CARE), California State University, Northridge, Northridge, California, United States of America, Department of Social Work, California State University, Northridge, Northridge, California, United States of America

    ⨯
  • David Boyns,

    Roles Data curation, Methodology, Writing – review & editing

    Affiliations Department of Social Work, California State University, Northridge, Northridge, California, United States of America, Department of Sociology, California State University, Northridge, Northridge, California, United States of America

    ⨯
  • Andrew T. Ainsworth

    Roles Data curation, Formal analysis, Methodology, Writing – review & editing

    Affiliations Department of Psychology, California State University, Northridge, Northridge, California, United States of America, Center for Assessment, Research, and Evaluation (CARE), California State University, Northridge, Northridge, California, United States of America

    ⨯

Abstract

In March 2020, the COVID-19 pandemic forced many in person undergraduate research experiences (UREs) to pivot to remote online training. To investigate how the COVID-19 quarantine disrupted student URE outcomes over time, the current study examines Building Infrastructure Leading to Diversity (BUILD) Promoting Opportunities for Diversity in Education and Research (PODER) URE outcomes across different platforms (in-person, remote, and hybrid models) by comparing student survey data from 2019 to 2021. Participants consisted of three cohorts: 2019 (n = 26 students), 2020 (n = 33), 2021 (n = 34). The BUILD PODER Summer JumpStart program (SJS), which aims to increase diversity in Science, Technology, Engineering, and Mathematics (STEM) by recruiting mostly underrepresented students, was conducted in person in 2019, remotely in 2020 and using a hybrid model in 2021. All students completed an online survey on the first and last day of the four-week SJS program. We used one-way and mixed ANOVA models to analyze Cohort, Time (pre-test vs. post-test scores), and interaction of Cohort and Time for Research Self-Efficacy, Sense of Belonging, Mentor Relationship, Mentee Knowledge, Health, Stress, and Student Program Satisfaction measures. Despite the platform changes, student scores increased significantly over time for all measures. There was a significant main effect of Time for Research Self-Efficacy, Sense of Belonging, Mentor Relationship, Mentee Knowledge, Health Assessment, and Stress Management. Findings indicate that URE programs that are implemented remotely and using a hybrid format can provide students with experiences similar to in-person URE programs. In addition, remote UREs may provide added benefits compared to in-person programs. For instance, remote UREs could engage more historically minoritized students, who may experience barriers to access, such as work/family commitments, financial constraints, and geographic limitations.

Citation: Escobedo P, Garcia D, Cascelli L, Chavira G, Flores GE, Constantine Brown JL, et al. (2023) Comparing undergraduate research experiences before, during, and after the COVID-19 quarantine: The successful adaptation of the BUILD PODER Summer JumpStart program. PLoS ONE 18(12): e0295901. https://doi.org/10.1371/journal.pone.0295901

Editor: Fatma Refaat Ahmed, University of Sharjah College of Health Sciences, UNITED ARAB EMIRATES

Received: December 27, 2022; Accepted: December 1, 2023; Published: December 28, 2023

Copyright: © 2023 Escobedo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This work is supported by the National Institutes of Health (NIH) BUILD PODER (Building Infrastructure Leading to Diversity: Promoting Opportunities for Diversity in Education and Research) at California State University, Northridge, grants TL4GM118977, RL5GM118975, and UL1GM118976. All awards granted to GC. The funders had no role in study design, data collection, analysis, decision to publish, or manuscript preparation.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Over the past few decades, the scientific community has made great efforts to diversify the scientific workforce [1, 2]. In particular, Black and Latinx students are less likely to pursue science, technology, engineering, and mathematics (STEM) majors and less likely to pursue STEM-related careers than their white or Asian counterparts [3]. To address these disparities, many institutions have developed initiatives to increase access to STEM education, which have increased the enrollment of minoritized students [4]. Yet despite some gains, students from ethnic-racial minoritized communities remain less likely to persist in STEM postsecondary majors [3]. High-impact practices (HIPs) positively affect student retention, success, and graduation [5, 6]. HIPs are active learning approaches that foster deep learning by promoting student engagement and positively impacting historically underserved students. Undergraduate research experiences (URE) are powerful HIPs that unequivocally show improvements in student retention and success [2, 5].

Short-term (i.e., summer) and longer-term (i.e., academic year) undergraduate research experiences (UREs) are traditionally delivered in person [7]. This format has effectively retained undergraduates in STEM through their graduate degrees and careers, especially students from minoritized communities [8, 9]. In March 2020, the COVID-19 pandemic forced UREs across the U.S. to cancel their summer programs, with few programs pivoting to remote online experiences for summer training [7, 10–12]. The pandemic allowed programs to examine the efficacy of online/remote undergraduate research training [11–13]. While some research has compared pre-pandemic to pandemic program implementation [14], to our knowledge, no studies have examined programmatic outcomes across different methods of implementation (i.e., in-person pre-pandemic, remote during COVID-19 quarantine, hybrid after some restrictions were lifted) to investigate the extent to which the quarantine created disruptions in student outcomes. Moreover, examining these student outcomes across platforms is important to demonstrate if the core objectives of traditional in-person UREs can be accomplished virtually, especially when unforeseen disruptions occur.

Disparities in STEM majors and careers

During the past 30 years, U.S. federal agencies increased efforts to diversify the STEM workforce, which yielded successful equity initiatives and research training opportunities for underrepresented scholars [2, 15]. However, disparities persist. Black, Latinx, American Indian/Alaska Natives, and Hawaiians/Pacific Islanders comprise 32% of the US population, yet they are historically minoritized; that is, they remain underrepresented in the sciences compared to Asians and Whites [16]. For instance, in 2018, only 22.5% of STEM bachelor’s degrees were awarded to historically minoritized groups [3]. This is concerning given that in 2011, the National Academy of Sciences (NAS) reported that most of the growth in new jobs will require science and technology skills [17]. To create a diverse STEM workforce, NAS recommended doubling the number of historically minoritized undergraduates earning STEM degrees [17]. To accomplish this goal, there is an urgent need to target and train students from minoritized groups with the research skills and self-efficacy necessary to earn undergraduate and graduate STEM degrees.

Undergraduate Research Experiences (UREs)

Interventions focused on increasing the number of students from minoritized groups earning undergraduate STEM degrees should offer students opportunities to participate in URE programs. Such programs include Building Infrastructure Leading to Diversity (BUILD) [18], Research Experiences for Undergraduates (REU) [19], Ronald E. McNair Scholars Program [20], Research Initiative for Scientific Enhancement (RISE) [2], Supporting Undergraduate Research Experiences (SURE) [14], the Meyerhoff Scholars Program [21], Biomedical Career Enrichment Programs (BCEPs), Einstein/Montefiore program and Woods Hole Partnership Education (PEP) [11, 22, 23]. UREs offer undergraduates exposure to research projects, faculty, and advanced peers who can provide access to professional connections and acquaint students with scientific norms [24–26]. For instance, URE faculty and peer support systems have been shown to promote gains among student participants in active learning, self-confidence, and pursuit of science careers [27, 28]. URE programs also provide spaces for demonstrating science discourse and practice, facilitating undergraduates’ scientific identities [29]. As a result, URE participants are better prepared to become science professionals and succeed in STEM programs than their peers [8, 30–32]. Moreover, UREs increase students’ critical thinking and communication skills than course-based research projects [31, 33–35].

Research on URE outcomes show there are positive effects on students’ intentions to enroll and actual enrollment in STEM graduate and professional programs [8–10, 19, 36, 37]. For example, Wilson and colleagues examined five NSF-funded UREs [19]. They found that URE participants were more likely to pursue a Ph.D. program and produce more presentations, publications, and awards than students who applied to the URE but did not participate. UREs provide undergraduates opportunities to develop scientific professional skills, explore possible career options, and refine post-undergraduate goals [26].

Increasing ethnic-racial minoritized groups’ participation in STEM

Some of the long-term goals of URE programs is to help increase the diversity of the scientific workforce in the U.S. by broadening the pool of student participants, improving retention, and increasing matriculation into competitive graduate programs [38, 39]. UREs typically provide undergraduate training through a multi-institution consortium that includes several research partners and various pipeline partners [37, 40, 41]. Key features of these programs include mentor-driven practices to support undergraduate student researchers as well as training faculty in research pedagogy and mentoring techniques and providing faculty with resources for increasing their research productivity [19, 40–43].

Research shows that UREs aimed at increasing the number of underrepresented minoritized students in the sciences can improve students’ likelihood of enrolling in doctorate programs [37, 38, 44]. For example, a previous study examined 13 student cohorts of the Meyerhoff Scholarship Program, which focuses on increasing the number of ethnic-racial minoritized students who earn doctorate degrees and pursue research careers in STEM fields [38]. This study found that URE participation was associated with a significant increase in the probability of minoritized students pursuing a science or engineering doctorate [38]. Further, many UREs are designed to support minoritized undergraduates, are successful at helping these students graduate with a STEM degree [39, 45] and can facilitate intentions to pursue a scientific research career [15, 36, 41, 44]. In addition, UREs tailored for minoritized undergraduates help improve retention in STEM [19, 44, 46]. For minoritized students to persist in the sciences, UREs should consider incorporating a culturally-affirming curriculum that fosters the development of a science identity and interest in pursuing a scientific career [36]. In addition, UREs that focus on critical mentorship and asset-based curricula that affirm the strengths of student’s culture/ethnicity are being designed to recruit and train minoritized students [36, 47–49]. Yet, it wasn’t until 2017 that the National Institute on General Medical Sciences (NIGMS) required that all training grant applications state that program mentors will receive training in mentorship [50]. Similarly, in 2022, the Research Experiences for Undergraduates (REU), funded by the National Science Foundation, strongly encouraged their REUs to provide training for research mentors [51], as they previously only encouraged training for new research mentors [52].

Differences in learning platform

Previous research exploring differences between the traditional classroom and remote or hybrid learning options has focused on student satisfaction [53] or grades [54] as outcomes, with few studies exploring student research self-efficacy or well-being. In a longitudinal study of social work graduate students’ research self-efficacy, Constantine Brown and Park found that student knowledge and research self-efficacy improved between pre-test and post-test on standardized measures and remained significantly improved at follow-up one year later, with no significant difference between online learners and traditional on-site students [55]. URE program designs remained largely traditional until the COVID-19 pandemic forced a rapid shift to remote online training [7, 10–12]. Three studies that measured student and program outcomes after training was completed, found that students reported the virtual program was a positive experience [11, 13], and students reported increased interest in pursuing biomedical careers, increases in research skills, positive relationship with mentors, and increased mental wellbeing after virtual program participation [12]. To date, no studies of URE have compared student outcomes by learning platform across different cohorts.

BUILD PODER: Objectives and Summer JumpStart (SJS)

Founded in 2014 through a common fund from the National Institutes of Health (NIH), the Building Infrastructure Leading to Diversity (BUILD) initiative aims to increase the diversity of the research workforce by preparing undergraduate student researchers from diverse groups in preparation for graduate studies in biomedical sciences. The BUILD program based at California State University, Northridge program is known as BUILD Promoting Opportunities for Diversity in Education and Research (PODER) and is one of ten BUILD sites across the United States. BUILD PODER employs Critical Race Theory (CRT) and related theories rooted in a social justice perspective throughout the student training curriculum and takes community-based, holistic, and transdisciplinary approaches to identify and resolve social problems such as health disparities. For example, BUILD PODER students use the National Science Foundation’s National Center for Science and Engineering Statistics (NCSES) [56] to review data on diversity in STEM and track their own trajectories through higher education and into science careers to highlight the disparities in the STEM workforce. BUILD PODER also incorporates discussions of current real-world health disparities, such as unequal access to healthcare during the COVID-19 pandemic that resulted in higher cases of infection and deaths related to COVID-19 among low-income and ethnically minoritized communities, as well as discussions centered around historical accounts of ethical violations and mistrust of medical scientists. BUILD PODER extends Solórzano and Yosso’s critical race analytical approach [57] to research in education by using the theory’s tenets as guides in all aspects of the research training curriculum and programming for faculty and students. BUILD PODER also employs Yosso’s Cultural Community Wealth Model [58] as an asset-based approach to challenge traditional forms (e.g., wealth, education) of capital by valuing non-traditional forms of capital (e.g., language and cultural knowledge).

Newly accepted BUILD PODER students begin training by participating in a four-week intensive entry-to-research Summer JumpStart (SJS) program (see S1 Fig for student recruitment and program timeline). The SJS program is an important component of the BUILD PODER URE as the training program is designed to provide students with biomedical research skills. However, SJS is also designed to help students build positive mentee-mentor relationships, increase feelings of research self-efficacy, develop a sense of belonging, and increase overall well-being to help underrepresented students persist in STEM majors, earn STEM degrees, and enter the STEM workforce.

Summer JumpStart objectives

Research Self-Efficacy.

Self-efficacy is an individual’s belief about their ability to perform tasks or activities related to a specific goal or outcome [59]. Research self-efficacy, therefore, is a student’s self-perception of their ability to perform research-related tasks and activities [60]. Research on science self-efficacy indicates it is a predictor of a students’ persistence in science-related fields [61]. Research self-efficacy and research skills were found to predict students’ aspirations for research careers, and the effects of research skills are partially mediated through self-efficacy beliefs [62]. Similarly, Pajares argued that students’ research self-efficacy is a vital link between their acquisition of research skills and knowledge, and how they apply these skills [63]. Furthermore, studies found that UREs can positively impact a research self-efficacy among students, which included the ability to think and work like a scientist, understanding how science is conducted and the ability to complete research-related tasks (e.g., laboratory or fields work, analyzing and interpreting data, research presentations) [34, 64, 65].

Mentoring relationship.

Faculty mentorship provides instrumental psychosocial, emotional, and networking functions for students [66–68]. A previous study found that research and mentoring experiences with faculty significantly improve students’ self-rated science efficacy [69]. Previous research also found an association between faculty mentoring, academic success, and persistence [70]. Research suggests mentoring is especially critical for minoritized students [49, 71–73]. For instance, in a study of 277 minoritized students, Castellanos and colleagues found that students who reported having a mentor had higher cultural fit (i.e., cultural congruity in combination with a perception of the university environment), more support (i.e., psychosocial support, and networking), and higher college and life satisfaction [72]. Furthermore, minoritized students’ perceptions of mentorship have been linked to increased retention [74], academic goal definition, and college adjustment [75].

Sense of Belonging and health.

Students who report feeling a sense of community in the classroom are more likely to attend class, participate during class, and graduate from college [76–78]. Furthermore, a sense of community has been shown to correlate positively with a student’s likelihood to contribute during class discussions and negatively with a student’s anxiety in the classroom [79]. These factors are particularly salient among undergraduate, first-generation, and minoritized populations [80–82]. Previous research indicates that success in STEM relies on a sense of belonging, especially among minoritized students [11, 81]. Previous studies demonstrate that UREs can influence students’ feelings of connectedness and well-being [83, 84], and previous research also indicates that science identity is facilitated by a sense of community and affiliation [29].

Individuals have an innate need to belong to communities, motivating them to maintain lasting, positive relationships and strongly affecting emotional and cognitive functioning [85]. Failure to make connections with others can negatively affect mental health and behavior [85]. For the community to impact an individual student, the student must perceive themselves as part of that community [86]. Therefore, the present study investigates students’ sense of community within the summer URE program and its role in increasing positive student outcomes.

Purpose

To develop effective program implementation and increase URE accessibility, especially when unexpected disruptions occur, the current study examined if student outcomes from the summer UREs differed significantly across formats (in-person, remote, or hybrid). The following research questions guided the study:

  1. How does the URE program format impact changes in Research Self-Efficacy, Sense of Belonging, Mentor Relationship, Mentee Knowledge, Health, and Stress Management scores across time?
  2. How does the URE program format impact ratings of Program Satisfaction among student participants?

Methods

Procedure

A total of 93 undergraduate students participated in the summer entry-to-research training program. The participants included 26 students in 2019, 33 in 2020, and 34 in 2021. To join BUILD PODER at California State University, Northridge (CSUN), individuals needed to be currently enrolled at one of four Pipeline Partners (PP) community colleges in the university’s service area or be a CSUN undergraduate student in their sophomore or junior year. The selection process consists of three steps: 1) students apply, 2) applications are screened for inclusionary criteria by a review panel, and 3) selected students are interviewed. BUILD PODER is a two to three-year program, depending on the coursework needed for graduation. All newly accepted BUILD PODER students complete the Summer JumpStart (SJS), a four-week intensive summer research experience held during the month of July that includes training on: research skill-building, building positive mentor-mentee relationships, research ethics, community-building, CRT principles and maintaining physical and mental health. Upon completion of SJS, students enter their first year in BUILD PODER.

Since its inception in 2015, BUILD PODER SJS has been conducted in person on the university campus, and since 2017, all participants have been provided complimentary on-campus housing. SJS participants completed community-building activities, attended workshops, research presentations, and field trips, and worked on projects in their mentor’s research labs. However, in 2020 all BUILD PODER SJS programming was moved entirely online in response to the COVID-19 pandemic. To complete lab work and other hands-on training or activities during remote programming, students picked up complimentary supplies provided by program staff from the university as needed (curb-side pickup), or it was mailed to participants if they were not within driving distance. In 2021, SJS began as an in-person program, with five students beginning the training remotely due to not being fully vaccinated for COVID-19 per BUILD PODER’s policy. However, all students were in-person by week three of the 2021 SJS program.

The current study used SJS data from 2019 to 2021. Trained research investigators administered all pre-test and post-test surveys. Only students who provided informed consent were given a link to complete an online survey on a personal device (e.g., laptop, tablet, or cell phone). Pre-test surveys were administered on the first day of SJS, and post-test surveys were administered four weeks later on the last day of SJS. In 2019, all surveys were administered on the CSUN campus, and in 2020 all surveys were administered remotely. In 2021, the pre-test survey was administered remotely, but the post-test survey was completed on the university campus. All students completed the same survey. The full survey took around 60 minutes to complete. The CSUN Committee for the Protection of Human Subjects approved all procedures.

Participant characteristics

The survey asked SJS participants to provide demographic information, including gender, first-generation college status, and Federal Pell Grant status. The U.S. Department of Education defines Pell Grants as awards to first-time undergraduate students with “exceptional financial need,” and award status is often used as an indication of low-income status [87]. Additional demographic information, such as ethnicity, was obtained from program records. Ethnicity was coded as Asian, Black, Latinx, White, and Other (multi-ethnic). First-generation college status and Pell Grant recipient status were coded as dichotomous variables (1 = “yes,” 0 = “no”).

Measures

Research Self-Efficacy.

Research Self-Efficacy was assessed using a 9-item scale. The questions were adapted from the 30-item Undergraduate Research Student Self-Assessment (URSSA) survey [65]. The nine items used in this survey focused on personal gains related to research work. Students were asked to rate their level of confidence or comfort level with various research activities, which included “comfort in discussing scientific concepts with others,” “confidence in my ability to do well in future courses,” and “confidence in engaging in real-world science research.” Responses were coded 1 = “none”, 2 = “little”, 3 = “moderate”, 4 = “quite a bit”, and 5 = “a lot” (see S1 Appendix all measure items and response options). Composite scores were created by averaging item responses to form a single composite score. Composite scores were calculated for pre-test items and post-test items. Cronbach’s alpha [α] statistics were calculated for pre-test items (α = 0.879) and post-test items (α = 0.906).

Sense of Belonging.

Sense of Belonging was assessed using a 10-item scale. The questions were adapted from Rovai’s 20-item scale focused on measuring classroom community [88]. The ten items used in this study focused on connectedness. Students were prompted to think about the program and asked, for example, whether they feel “the students care about each other,” “I trust other students,” “isolated,” or “other students depend on me” (see S1 Appendix all measure items and response options). Responses were coded “strongly disagree” = 1, “disagree” = 2, “neutral” = 3, and “agree” = 4. Composite scores were created by averaging item responses to form a single composite score. Composite scores were calculated for pre-test and post-test items. Composite scores were calculated for pre-test items (α = 0.879) and post-test items (α = 0.884).

Mentor relationship.

Students’ relationship with their mentor was assessed using a 9-item scale. The questions were adapted from Rovai’s 20-item scale [88]. The ten items used in this study focused on the connectedness between the student and mentor. Students were prompted to think about their mentor and asked, for example, whether they feel “connected to my mentor,” “isolated,” or “confident that my mentor will support me” (see S1 Appendix all measure items and response options). Responses were coded “strongly disagree” = 1, “disagree” = 2, “neutral” = 3, and “agree” = 4. Composite scores were created by averaging item responses. Composite scores were calculated for pre-test and post-test items. Composite scores were calculated for pre-test items (α = 0.871) and post-test items (α = 0.904).

Mentee Knowledge.

The student’s understanding of their responsibilities as a mentee was assessed with an 8-item scale. The measure was created by research team members using the wording from the mentor-mentee contract, a form they completed before the Summer JumpStart program. Students were asked if they knew, for example, “how many hours I will work on [their] project with [their] mentor,” “what I must prepare to present to my mentor prior to our meetings,” and “what my mentor expects of me” (see Supplemental File for all items). Responses were coded “strongly disagree” = 1, “disagree” = 2, “neutral” = 3, and “agree” = 4. Composite scores were created by averaging item responses. Composite scores were calculated for pre-test and post-test items. Composite scores were calculated for pre-test items (α = 0.897) and post-test items (α = 0.924).

Health Assessment.

The measure was created by research team members based on the BUILD PODER’s SJS Health and Well-being curriculum. For example, students were asked how often they do the following, “exercise or go to the gym,” “get enough sleep,” “make time for self-reflection, prayer or meditation,” and “stay in contact with important people in your life” (see Supplemental File for all items). Responses were coded “it will probably never occur to me” = 1, “never” = 2, “rarely” = 3, “sometimes” = 4, and “frequently” = 5. Composite scores were created by averaging item responses. Composite scores were calculated for pre-test items (α = 0.744) and post-test items (α = 0.826).

Stress Management.

Student Stress Management was assessed using one item. Students were asked to “rate their ability to manage the overall level of stress they experience.” Responses were coded as “poor” = 1, “fair” = 2, “neutral” = 3, “good” = 4, and “excellent” = 5, and the research team created the measure.

Student satisfaction.

Student satisfaction with the Summer JumpStart program was assessed using a 7-item measure adapted from the Client Satisfaction Questionnaire (CSQ-8) [89], which was divided into three subscales (Satisfaction 1, Satisfaction 2, and Satisfaction 3) (see S1 Appendix all measure items and response options). Averaging item responses to each question created composite scores. Students were only asked about student satisfaction in the post-test survey. Therefore, composite scores were calculated for post-test items only. Cronbach’s alpha statistics were computed for Satisfaction 1 (α = 0.852), Satisfaction 2 (α = 0.724), and Satisfaction 3 (α = 0.867) subscales.

Analysis strategy.

Participant demographics for the analytic sample (n = 92) were calculated by cohort (Table 1). The average mean and standard deviation were calculated for each measure (pre-test and post-test) by cohort (Table 2). We conducted a one-way ANOVA analysis and Tukey HSD post hoc tests to examine how student satisfaction scores differed by cohort. To examine Cohort (i.e., 2019, 2020, 2021) and Time (i.e., pre-test versus post-test) and interaction of Cohort and Time for Research Self-Efficacy, Sense of Belonging, Mentor Relationship, Mentee Knowledge, Health Assessment, and Stress Management measures, we conducted a two-way mixed ANOVA analysis and Tukey HSD post hoc tests. Due to less than 5% missing data, participants with missing data on any measure used in the analysis were removed from the sample [90]. Therefore, the final analytic sample consisted of 92 participants, with one participant in 2021 removed due to not answering mentor-related questions in the post-test survey. Statistical analyses were performed using Stata (version 17) and SPSS (version 28.0).

thumbnail
Download:
Table 1. Participant characteristics by cohort.

https://doi.org/10.1371/journal.pone.0295901.t001

thumbnail
Download:
Table 2. Descriptive statistics for study measures (pre-test and post-test) by cohort.

https://doi.org/10.1371/journal.pone.0295901.t002

Ethics statement.

All participants provided informed written consent online before completing any surveys for this study. This study was approved by the CSUN Committee for the Protection of Human Subjects under IRB-FY21-222.

Results

Among the participants, 71.73% identified as female and 28.26% as male (Table 1). Among the participants, 54.34% were Latinx, 16.3% were Asian, 13.04% were White, 6.52% were Black, 5.43% were Other (multi-ethnic), 60.86% were first-generation, and 50% were Federal Pell Grant recipients. Among the participants, 13.04% identified as health and human development majors (e.g., public health, nutrition), 39.13% identified as social and behavioral science majors (e.g., psychology), 36.95% identified as science and mathematics majors (e.g., biology, chemistry), and 10.86% identified as engineering and computer science majors.

Research Self Efficacy

There was a significant main effect of Time on Research Self-Efficacy scores overall (F(1,89) = 61.29, p < .001, ηp2 = 0.408), such that scores increased significantly from pre-test to post-test among participants in all cohorts (see Table 2 for means and standard deviations of all measures). However, the effect for Cohort was not significant (F(2,89) = 0.67, p = 0.512, ηp2 = 0.015), and the interaction between Time and Cohort on Research Self-Efficacy scores was not significant (F(2,89) = 0.22, p = 0.807, ηp2 = 0.005).

Sense of Belonging

There was a significant main effect of Time on Sense of Belonging scores overall (F(1,89) = 170.123, p < 0.001, ηp2 = 0.657), such that average scores increased significantly from pre-test to post-test among participants in all cohorts. There was also a significant Cohort effect on Sense of Belonging scores (F(2,89) = 4.158, p = 0.019, ηp2 = 0.085). Tukey post hoc tests show scores in the 2019 cohort (i.e., entirely in-person) were significantly higher (0.328 mean difference) than in the 2020 cohort (i.e., entirely online) at p < .05, but no other difference was significant. There was no significant interaction between Time and Cohort on Sense of Belonging scores (F(2,89) = .56, p = 0.571, ηp2 = 0.013).

Mentor Relationship

There was a significant main effect of Time on Mentor Relationship scores overall (F(1,89) = 69.86, p < 0.001, ηp2 = 0.440), such that scores increased significantly from pre-test to post-test among participants in all cohorts. However, the main effect of Cohort was not significant (F(2,89) = 2.44, p = 0.093, ηp2 = 0.052), and the interaction between Time and Cohort on Mentor Relationship scores was also not significant (F(2,89) = 1.78, p = 0.17, ηp2 = 0.038).

Mentee Knowledge

There was a significant main effect of Time on Mentee Knowledge scores overall (F(1,89) = 79.06, p < 0.001, ηp2 = 0.470), such that scores increased significantly from pre-test to post-test among participants in all cohorts. However, the main effect of Cohort was not significant (F(2,89) = 1.17, p = 0.315, ηp2 = 0.026), and the interaction between Time and Cohort on Mentee Knowledge scores was also not significant (F(2,89) = 0.352, p = 0.704, ηp2 = 0.008).

Health Assessment

There was a significant main effect of Time on Health Assessment scores overall (F(1,89) = 104.57, p < 0.001, ηp2 = 0.540), such that scores increased significantly from pre-test to post-test among participants in all cohorts. The main effect of Cohort on Health Assessment scores was not significant (F(2,89) = 1.33, p = 0.267, ηp2 = 0.029). However, there was a significant interaction between Time and Cohort on Health Assessment scores (F(2,89) = 6.34, p = .03, np2 = 0.125), indicating that the scores in the 2019 cohort increased at a significantly faster rate than observed in the 2020 and 2021 cohorts.

Stress Management

There was a significant main effect of Time on Stress Management scores overall (F(1,89) = 37.50, p < .001, ηp2 = .296), indicating that students in all cohorts significantly increased their ability to manage stress from pre-test to post-test. However, the main effect of Cohort was not significant (F(2,89) = 1.12, p = .331, ηp2 = .025), and the interaction between Time and Cohort on Stress Management scores was not significant (F(2,89) = .474, p = .624, ηp2 = .011).

Student satisfaction

There was no statistically significant difference between the average Satisfaction 1 score by Cohort (F(2,90) = 0.612, p = .545, ηp2 = .013). There was no statistically significant difference between average Satisfaction 2 scores by Cohort (F(2,90) = 1.55, p = .217, ηp2 = .033). There was no statistically significant difference between average Satisfaction 3 scores by Cohort (F(2,90) = 2.31, p = .104, ηp2 = .049).

Discussion

Findings from this study highlight URE student outcomes over time and across different platforms (in-person pre-pandemic, remote during COVID-19 quarantine, and hybrid) by comparing student surveys conducted in 2019, 2020, and 2021. This study found that despite the change in platforms, student scores across the three cohorts increased significantly from pre-test to post-test for Research Self-Efficacy, Sense of Belonging, Mentor Relationship, Mentee Knowledge, Health Assessment, and Stress Management. In addition, there were no significant differences between program satisfaction scores by year. Our findings indicate that URE programs implemented remotely and using a hybrid format can provide students with an effective and impactful URE experience.

During the summer of 2020, several UREs pivoted from in-person to remote online training in response to the COVID-19 pandemic [7, 10–12]. Of these programs, three studies examined the efficacy of remote online training by measuring student and program outcomes after training was completed [11–13], and only one study compared pre-pandemic URE program implementation to pandemic program implementation [14]. The current study built on previous research by measuring URE student outcomes across three different types of program implementation in 2019, 2020, and 2021. Examining student scores across different URE program design and implementation strategies is critical. The current study provides program design recommendations to URE principal investigators (PIs), faculty, and staff who may need to shift in-person URE programming to remote or hybrid program models, especially UREs serving historically minoritized groups [7].

The current study found that despite the change in URE platforms, student Research Self-Efficacy scores increased significantly from pre-test to post-test. These results indicate that participants felt more confident engaging with research activities by the end of the SJS program. These findings parallel a study by Ajayi and colleagues, who compared student outcomes before and during the COVID quarantine [14]. Ajayi and colleagues found that students in the 2019 and 2020 URE programs reported significant gains in confidence conducting research [14]. Moreover, previous studies used a pre-test and post-test design in 2020 to examine remote URE programs and found that remote URE participants reported gains in research self-efficacy [12, 83].

It is well established that traditional in-person UREs can increase research self-efficacy among undergraduates [34, 64, 65]. The current study builds on these findings by showing that online UREs can also increase research self-efficacy among undergraduates. In the current study, students in the SJS program were matched with research mentors and received research training in person and remotely. Our results are notable because they suggest that remote UREs can also support persistence in STEM and encourage undergraduates to pursue careers in research [61, 63].

The current study also found that student Sense of Belonging scores increased significantly from pre-test to post-test. However, the 2019 (in-person) cohort scores were significantly higher than the 2020 (online) cohort scores. These findings are consistent with past research, which found that students felt that online UREs could improve outcomes by focusing on cohort building programming and activities [7]. Differences in Sense of Belonging may be influenced by cohort characteristics and/or the program platform. For example, the 2020 cohort was the first wave of students required to shift from in-person to remote learning, therefore students in this cohort may not have found online socialization as impactful as in-person socialization, or students needed time to adjust to informal interactions with peers online.

Findings from previous research indicate that remote URE participants can develop a sense of belonging [14, 83, 91]. A study by Alaee and colleagues noted that remote URE participants experienced a sense of belonging by communicating and socializing with lab mates and other program participants [83]. Additionally, intimate conversations and whole-group discussions help remote URE participants feel connected with each other [7, 12]. Sense of Belonging is critical, as it greatly affects emotional and cognitive functioning [85]. In the current study, activities to build relationships with peers and program staff were implemented in-person or remotely in 2019, 2020, and 2021. Program leadership implemented weekly or monthly events for all students in the program so participants could socialize and interact with their cohort and the larger group. In addition, across all cohorts and platforms, SJS students participated in team-building activities, movie nights, and tours of local museums or communities (in-person or online). Virtual engagement has been shown to foster feelings of belonging in historically minoritized STEM female faculty [92]. It is worth noting that Sense of Belonging in the current study captured participants’ growth in feelings of acceptance by their research team. Future research should examine how URE participants develop feelings of acceptance by the broader scientific research community.

The current study found that student Mentor Relationship and Mentee Knowledge scores increased significantly from pre-test to post-test. These results are consistent with past research indicating that remote UREs can complement traditional forms of mentorship [10] and that mentees completing remote UREs can have meaningful and productive virtual mentoring experiences [12, 93]. A growing body of research demonstrates that virtual mentoring can facilitate social, academic, and career success and enables the development of transferable and technical skills similar to the benefits of in-person mentoring [94–100]. In the current study, all SJS program participants had multiple opportunities to interact with their mentors, potentially affecting their mentoring relationships positively. Moreover, all participants were required to submit a mentor-mentee contract, establishing clear expectations at the program’s start. These program components were required of in-person and remote participants, which may explain why SJS mentees perceived their mentoring relationships favorably.

The results from the current study and previous research indicate that remote mentorship offers an ability to continue training in STEM research and promotes feelings of interpersonal connection. Further, remote mentoring can also have added benefits, such as increased flexibility in meeting times/locations and a more comfortable environment for mentee communication [97, 101, 102]. These benefits may be essential to historically minoritized students with financial or familial obligations. Future studies will be necessary to determine the long-term effects of remote research mentorship, such as job placement and retention in STEM.

The current study found that student Health Assessment and Stress Management scores increased significantly from pre-test to post-test. The results indicate that students felt more capable of managing their health and stress by the end of the program. These findings parallel past research which involved 170 remote URE participants in the summer of 2020 and found that participants reported gains in research self-efficacy and shifts in mental well-being. For example, participants reported decreased stress, resilience gains, and improved life satisfaction [12]. The results from the current study suggest that SJS sessions focused on stress and physical were impactful. The SJS program provided students with the tools to decrease stress and improve resilience and mitigate burnout [103–105]. Furthermore, undergraduate researchers’ shifts in mental well-being, such as improved life satisfaction, promote STEM excellence in a wellness environment [106].

Limitations

The study was limited to college students in Southern California. Therefore, results may not be generalizable to college students living in other regions of the United States. Most participants in the current study were female, lower-income, and first-generation college students, which may limit the generalizability of findings to other sociodemographic groups. In addition, some academic disciplines may be better adapted to an online URE experience than others. The BUILD PODER online URE was designed for students majoring in science and mathematics and social and behavioral sciences, which limits the generalizability of this program to other academic disciplines.

Conclusion

The results from the current study demonstrate that the core objectives of traditional in-person UREs can be accomplished virtually while also impacting students’ well-being. In addition, these results may provide added benefits compared to in-person programs. For instance, remote UREs could engage more historically minoritized students, who may experience barriers to access, such as work commitments, financial constraints, geographic limitations, childcare, or family obligations. As the scientific research community confronts its lack of diversity, remote experiences could be leveraged to address existing inequities. Altogether, the results indicate that remote URE programs afford many of the same opportunities and benefits as in-person programs. Virtual programming can target and improve the student well-being necessary for academic, career, and life satisfaction and success, bolstering science pathways for all students. Future studies should investigate how to scale up remote URE offerings and examine whether remote URE participants pursue graduate education and research-related careers at the same rates as in-person program participants.

Supporting information

S1 Fig. URE recruitment & program timeline.

https://doi.org/10.1371/journal.pone.0295901.s001

(PDF)

S1 Appendix. JumpStart study measures.

https://doi.org/10.1371/journal.pone.0295901.s002

(DOCX)

S1 Dataset. Jumpstart minimal data set deidentified.

https://doi.org/10.1371/journal.pone.0295901.s003

(SAV)

Acknowledgments

We want to acknowledge and thank Dr. Scott Appelrouth, who helped develop the BUILD PODER Summer JumpStart questionnaire. We also want to acknowledge Martha Carreon and Kyle Moreno for their review and formatting of the manuscript. Finally, we want to acknowledge the students of BUILD PODER for their hard work and dedication to becoming the next generation of STEM scholars.

References

  1. 1. Estrada M, Hernandez PR, Schultz PW. A longitudinal study of how quality mentorship and research experience integrate underrepresented minorities into STEM careers. CBE Life Sciences Education. 2018 Mar 01;17(1). pmid:29351912
  2. 2. Hernandez P, Woodcock A, Estrada M, Schultz P. Undergraduate research experiences broaden diversity in the scientific workforce. BioScience. 2018 Mar 01;68(3):204–211.
  3. 3. National Center for Science and Engineering Statistics. 2021. Women, minorities, and persons with disabilities in science and engineering: 2021. Special Report NSF 21–321. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/wmpd.
    • 4. Sithole A, Chiyaka ET, McCarthy P, Mupinga DM, Bucklein BK, Kibirige J. Student attraction, persistence and retention in STEM programs: Successes and continuing challenges. Higher Education Studies. 2017;7(1):46–59
    • 5. Kuh G. High-impact educational practices: What they are, who has access to them, and why they matter. Peer Review. 2012 Jun 22;14(3):44.
    • 6. Kuh G O’Donnell K, Reed SD. Ensuring quality and taking high-impact practices to scale. AAC & U. 2013 Mar 22;15(2):56.
    • 7. Erickson OA, Cole RB, Isaacs JM, Alvarez-Clare S, Arnold J, Augustus-Wallace A, et al. How do we do this at a distance?! A descriptive study of remote undergraduate research programs during Covid-19. CBE life sciences education. 2022 Mar 01;21(1).
    • 8. Eagan MK Jr, Hurtado S, Chang MJ, Garcia GA, Herrera FA, Garibay JC. Making a Difference in Science Education: The Impact of Undergraduate Research Programs. American Educational Research Journal. 2013 Aug;50(4):683–713. pmid:25190821
    • 9. Lopatto D. Survey of Undergraduate Research Experiences (SURE): First Findings. Cell Biology Education. 2004 Dec;3(4):270–7. pmid:15592600
    • 10. Samad T, Fleming HE, Bhatia SN. Virtual Undergraduate Research Experiences: More Than a Pandemic Stopgap. Med. 2021 Feb 12;2(2):118–21. pmid:35187514
    • 11. Scott Price O, Luis K, Price AL, Harden B, Howard J, Valentin LE, et al. Same Program Different Delivery: Adapting the Woods Hole Partnership Education Program for a Virtual Era. Limnology and Oceanography Bulletin. 2020 Nov;29(4):117–24.
    • 12. Yang Yowler J, Knier K, WareJoncas Z, Ehlers SL, Ekker SC, Guasp Reyes F, et al. Rapid Adaptation and Remote Delivery of Undergraduate Research Training during the COVID-19 Pandemic. Sustainability. 2021 Jan;13(11):6133.
    • 13. Malayil L, Negahban-Azar M, Goldstein RR, Sharma M, Gleason J, Muise A, et al.”Zooming” Our Way through Virtual Undergraduate Research Training: A Successful Redesign of the CONSERVE Summer Internship Program. Journal of Microbiology & Biology Education. 2021 Mar 31;22(1):22.1.90. pmid:33953822
    • 14. Ajayi TB, Mueller AL, Okwuosa IS, Barshilia A, Wu JC, Benjamin EJ, et al. Innovations in Undergraduate Research Training Through Multisite Collaborative Programming: American Heart Association Summer Undergraduate Research Experience Syndicate. Journal of the American Heart Association. 2022 Apr 19;11(8). pmid:35388707
    • 15. Schultz PW, Hernandez PR, Woodcock A, Estrada M, Chance RC, Aguilar M, et al. Patching the Pipeline: Reducing Educational Disparities in the Sciences Through Minority Training Programs. Educational Evaluation and Policy Analysis. 2011 Mar;33(1):95–114. pmid:24285910
    • 16. Fry R, Kennedy B, Funk C. STEM Jobs See Uneven Progress in Increasing Gender, Racial and Ethnic Diversity. Pew Research Center; 2021. [cited 2022 Aug 16] Available at https://www.pewresearch.org/science/2021/04/01/stem-jobs-see-uneven-progress-in-increasing-gender-racial-and-ethnic-diversity/
    • 17. National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. Expanding Underrepresented Minority Participation: America’s Science and Technology Talent at the Crossroads. Washington, DC: The National Academies Press; 2011. 286 p. Available from: https://doi.org/10.17226/12984.
      • 18. Norris KC, McCreath HE, Hueffer K, Aley SB, Chavira G, Christie CA, et al. Baseline Characteristics of the 2015–2019 First-Year Student Cohorts of the NIH Building Infrastructure Leading to Diversity (BUILD) Program. Ethnicity & Disease. 2020 Sep 24;30(4):681–92. pmid:32989368
      • 19. Wilson AE, Pollock JL, Billick I, Domingo C, Fernandez-Figueroa EG, Nagy ES, et al. Assessing Science Training Programs: Structured Undergraduate Research Programs Make a Difference. BioScience. 2018 Jul 01;68(7):529–34.
      • 20. Posselt J, Black K. Developing the research identities and aspirations of first-generation college students. International Journal for Researcher Development. 2012 May 18;3(1):26–48.
      • 21. Maton KI, Hrabowski FA III, Schmitt CL. African American college students excelling in the sciences: College and postcollege outcomes in the Meyerhoff Scholars Program. Journal of Research in Science Teaching. 2000 Sep;37(7):629–54.
      • 22. Bhatt R, West B, Chaudhary S. Biomedical career enrichment programs: Exploring women and minority participants’ motivators and outcomes. PLOS ONE. 2020 Feb 14;15(2). pmid:32059042
      • 23. Stephenson-Hunter C, Franco S, Martinez A, Strelnick AH. Virtual Summer Undergraduate Mentorship Program for Students Underrepresented in Medicine Yields Significant Increases in Self-Efficacy Measurements During COVID-19 Pandemic: A Mixed Methods Evaluation. Health Equity. 2021 Sep 01;5(1):697–706. pmid:34909539
      • 24. Hurtado S, Cabrera NL, Lin MH, Arellano L, Espinosa LL. Diversifying Science: Underrepresented Student Experiences in Structured Research Programs. Research in Higher Education. 2009 Mar;50(2):189–214. pmid:23503690
      • 25. Krim JS, Coté LE, Schwartz RS, Stone EM, Cleeves JJ, Barry KJ, et al. Models and impacts of science research experiences: A review of the literature of cures, ures, and tres. CBE Life Sciences Education. 2019 Dec 01;18(4). pmid:31782694
      • 26. McSweeney JC, Hudson TJ, Prince L, Beneš H, Tackett AJ, Miller Robinson C, et al. Impact of the INBRE summer student mentored research program on undergraduate students in Arkansas. Advances in Physiology Education. 2018 Mar 01;42(1):123–9. pmid:29446316
      • 27. Lopatto D. Undergraduate Research Experiences Support Science Career Decisions and Active Learning. CBE Life Sci Educ. 2007 Dec;6(4):297–306.
      • 28. Lopatto D, Tobias S. Science in solution: the impact of undergraduate research on student learning. 1st ed. Washington DC: Council on Undergraduate Research; 2010. 116 p.
        • 29. Carlone HB, Johnson A. Understanding the science experiences of successful women of color: Science identity as an analytic lens. Journal of Research in Science Teaching. 2007 Sep 11;44(8):1187–218.
        • 30. Laursen S, Hunter A, Seymour E, Thiry H, Melton G. Undergraduate Research in the Sciences: Engaging Students in Real Science. John Wiley & Sons; 2010 Jun 15. 320 p.
        • 31. Thiry H, Laursen SL, Hunter AB. What Experiences Help Students Become Scientists? A Comparative Study of Research and other Sources of Personal and Professional Gains for STEM Undergraduates. The Journal of Higher Education. 2011 Jul 01;82(4):357–88.
        • 32. Vincent-Ruz P, Grabowski J, Schunn C. The impact of early participation in undergraduate research experiences on multiple measures of premed path success. Scholarship and Practice of Undergraduate Research. 2018 Apr 01;1(3).
        • 33. Bauer KW, Bennett JS. Alumni Perceptions Used to Assess Undergraduate Research Experience. The Journal of Higher Education. 2003 Mar 01;74(2):210–30.
        • 34. Hunter AB, Laursen SL, Seymour E. Becoming a scientist: The role of undergraduate research in students’ cognitive, personal, and professional development. Science Education. 2007 Jan 01;91(1):36–74.
        • 35. Seymour E, Hunter AB, Laursen SL, DeAntoni T. Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education. 2004 Jul;88(4):493–534.
        • 36. Camacho TC, Vasquez-Salgado Y, Chavira G, Boyns D, Appelrouth S, Saetermoe C, et al. Science Identity among Latinx Students in the Biomedical Sciences: The Role of a Critical Race Theory–Informed Undergraduate Research Experience. CBE Life Sciences Education. 2021 May 03;20(2). pmid:33938764
        • 37. Lessard L, Smith C, O’Connor S, Velasquez SE, Benson J, Garfield J, et al. Collaborative Assessment Of Collective Reach And Impact Among INBRE Supported Summer Undergraduate Research Programs Across The United States. J STEM Educ. 2021 Apr;22(2):46–51. pmid:34413711
        • 38. Carter FD, Mandell M, Maton KI. The Influence of On-Campus, Academic Year Undergraduate Research on STEM Ph.D. Outcomes: Evidence From the Meyerhoff Scholarship Program. Educational Evaluation and Policy Analysis. 2009 Dec 01;31(4):441–62.
        • 39. Salto LM, Riggs ML, Delgado De Leon D, Casiano CA, De Leon M. Underrepresented minority high school and college students report STEM-pipeline sustaining gains after participating in the Loma Linda University Summer Health Disparities Research Program. PloS One. 2014 Sep 24;9(9).
        • 40. Collins TW, Aley SB, Boland T, Corral G, Cox MB, Echegoyen LE, et al. BUILDing SCHOLARS: enhancing diversity among U.S. biomedical researchers in the Southwest. BMC Proceedings. 2017 Dec 04;11(12):12. pmid:29375655
        • 41. Ghee M, Keels M, Collins D, Neal-Spence C, Baker E. Fine-Tuning Summer Research Programs to Promote Underrepresented Students’ Persistence in the STEM Pathway. CBE Life Sciences Education. 2016 Sep 15;15(3):ar28. pmid:27496359
        • 42. Kendricks K, Nedunuri K, Arment A. Minority Student Perceptions of the Impact of Mentoring to Enhance Academic Performance in STEM Disciplines. Journal of STEM Education: Innovations and Research. 2013 Mar 26;14(2).
        • 43. Chou AF, Hammon D, Akins DR. Impact and Outcomes of the Oklahoma IDeA Network of Biomedical Research Excellence Summer Undergraduate Research Program. Journal of Microbiology & Biology Education. 2019 Oct 31;20(3):20.3.50. pmid:31768211
        • 44. Kendricks KD, Arment AR. Adopting a K-12 family model with undergraduate research to enhance stem persistence and achievement in underrepresented minority students. 2010 Nov 01; Available from: https://hdl.handle.net/11299/180922
        • 45. Crawford MB, Wilson-Kennedy ZS, Thomas GA, Gilman SD, Warner IM. LA-STEM Research Scholars Program: A Model for Broadening Diversity in STEM Education. Technology and innovation. 2018;19(3):577–92.
        • 46. Fakayode SO, Yakubu M, Adeyeye OM, Pollard DA, Mohammed AK. Promoting Undergraduate STEM Education at a Historically Black College and University through Research Experience. Journal of Chemical Education. 2014 May 13;91(5):662–5.
        • 47. Ben-Zeev A, Paluy Y, Milless KL, Goldstein EJ, Wallace L, Márquez-Magaña L, et al. “Speaking Truth” Protects Underrepresented Minorities’ Intellectual Performance and Safety in STEM. Education Sciences. 2017 Jan 16;7(2):65. pmid:28835879
        • 48. Estrada M, Eroy-Reveles A, Ben-Zeev A, Baird T, Domingo C, Gómez CA, et al. Enabling full representation in science: the San Francisco BUILD project’s agents of change affirm science skills, belonging and community. BMC Proceedings. 2017 Dec 04;11(12):25. pmid:29375666
        • 49. Vargas JH, Saetermoe CL, Chavira G. Using critical race theory to reframe mentor training: theoretical considerations regarding the ecological systems of mentorship. Higher Education. 2021 May;81(5):1043–62. pmid:35935004
        • 50. Yellin J, Field A. NIGMS Announces New Institutional Research Training Grant [Internet]. AAMC. 2017. Available from: https://www.aamc.org/advocacy-policy/washington-highlights/nigms-announces-new-institutional-research-training-grant
        • 51. National Science Foundation. Research Experiences for Undergraduates (REU): Program Solicitation 22–601 [Internet]; 2022 [cited 2022 Oct 15]. Available from https://www.nsf.gov/pubs/2022/nsf22601/nsf22601.pdf
        • 52. National Science Foundation. Research Experiences for Undergraduates (REU): Program Solicitation 19–582 [Internet]; 2019 [cited 2022 Oct 15]. Available from https://www.nsf.gov/pubs/2019/nsf19582/nsf19582.pdf
        • 53. York RO. Comparing Three Modes of Instruction in a Graduate Social Work Program. Journal of Social Work Education. 2008 Apr 22;44(2):157–72.
        • 54. Jones SJ, Long V. Learning Equity between Online and On-Site Mathematics Courses. MERLOT Journal of Online Learning and Teaching. 2013 Mar;9(1):1–13.
        • 55. Constantine Brown J, Park HS. Longitudinal Student Research Competency: Comparing Online and Traditional Face-to-Face Learning Platforms. Advances in Social Work. 2016 Apr 25;17(1):44–58.
        • 56. National Center for Science and Engineering Statistics (NCSES). 2023. Diversity and STEM: Women, Minorities, and Persons with Disabilities Data Tables. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/pubs/nsf23315/data-tables#top.
          • 57. Solórzano DG, Yosso TJ. Critical Race Methodology: Counter-Storytelling as an Analytical Framework for Education Research. Qualitative Inquiry. 2002 Feb 1;8(1):23–44.
          • 58. Yosso TJ. Whose culture has capital? A critical race theory discussion of community cultural wealth. Critical Race Theory in Education. 2020 Jul;114–36.
          • 59. Bandura A. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ, US: Prentice-Hall, Inc; 1986. xiii, 617 p. (Social foundations of thought and action: A social cognitive theory).
            • 60. Bieschke KJ, Bishop RM, Garcia VL. The utility of the research self-efficacy scale. Journal of career assessment. 1996 Jul;4(1):59–75.
            • 61. Britner SL, Williams B, Pecore JL, Gagne P, Demetrikopoulos MK, Poh R, et al. Portraits of science self-efficacy: Four undergraduate women in a summer research experience. JWM [Internet]. 2012;18(3). Available from: https://www.dl.begellhouse.com/journals/00551c876cc2f027,174fe1882ce7b914,2f58a2566ba32c85.html
            • 62. Adedokun OA, Bessenbacher AB, Parker LC, Kirkham LL, Burgess WD. Research skills and STEM undergraduate research students’ aspirations for research careers: Mediating effects of research self‐efficacy. Journal of Research in Science teaching. 2013 Oct;50(8):940–51.
            • 63. Pajares F. Self-efficacy beliefs, motivation, and achievement in writing: A review of the literature. Reading & Writing Quarterly: Overcoming Learning Difficulties. 2003;19:139–58.
            • 64. Kardash CM. Evaluation of undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology. 2000;92:191–201.
            • 65. Hunter A, Weston T, Laursen S, Thiry H. Undergraduate Research Student Self-Assessment (URSSA): Evaluating Student Gains from Undergraduate Research in the Sciences. CUR Quarterly. 2009 Mar 01;29(3):15–9.
            • 66. Johnson WB, Zlotnik S. The frequency of advising and mentoring as salient work roles in academic job advertisements. Mentoring & Tutoring: Partnership in Learning. 2005 Apr 1;13(1):95–107.
            • 67. Mathews P. Academic mentoring: Enhancing the use of scarce resources. Educational Management & Administration. 2003 Jul 1;31(3):313–34.
            • 68. Ortiz-Walters R, Gilson LL. Mentoring in academia: An examination of the experiences of protégés of color. Journal of Vocational Behavior. 2005 Dec 1;67(3):459–75.
            • 69. Chemers MM, Zurbriggen EL, Syed M, Goza BK, Bearman S. The role of efficacy and identity in science career commitment among underrepresented minority students. Journal of Social Issues. 2011 Sep;67(3):469–91.
            • 70. Torres V, Hernandez E. Influence of an identified advisor/mentor on urban Latino students’ college experience. Journal of College Student Retention: Research, Theory & Practice. 2009 May;11(1):141–60.
            • 71. Bordes V, & Arredondo P. (2005). Mentoring and 1st-year Latina/o college students. Journal of Hispanic higher education, 4(2), 114–133.
            • 72. Castellanos J, Gloria AM, Besson D, & Harvey LOC. (2016). Mentoring matters: Racial ethnic minority undergraduates’ cultural fit, mentorship, and college and life satisfaction. Journal of College Reading and Learning, 46(2), 81–98
            • 73. Gloria AM, Castellanos J, Lopez AG, Rosales R. An examination of academic nonpersistence decisions of Latino undergraduates. Hispanic Journal of Behavioral Sciences. 2005 May;27(2):202–23.
            • 74. Good JM, Halpin G, Halpin G. The Affective and Academic Benefits for Mentors in a Minority Engineering Program [Internet]. 1998 Nov. Available from: https://eric.ed.gov/?id=ED429488
            • 75. Santos SJ, Reigadas ET. Latinos in higher education: An evaluation of a university faculty mentoring program. Journal of Hispanic Higher Education. 2002 Jan;1(1):40–50
            • 76. Collett S. The Road to Completion. Community College Journal. 2013 Apr 01; 83(5):40.
            • 77. Tinto V. Through the Eyes of Students. Journal of College Student Retention: Research, Theory & Practice. 2017 Nov;19(3):254–69.
            • 78. Hausmann LRM, Schofield JW, Woods RL. Sense of Belonging as a Predictor of Intentions to Persist Among African American and White First-Year College Students. Research in Higher Education. 2007 Feb 16;48(7):803–39.
            • 79. Freeman TM, Anderman LH, Jensen JM. Sense of Belonging in College Freshmen at the Classroom and Campus Levels. The Journal of Experimental Education. 2007 Apr 01;75(3):203–20.
            • 80. Fisher AJ, Mendoza-Denton R, Patt C, Young I, Eppig A, Garrell RL, et al. Structure and belonging: Pathways to success for underrepresented minority and women PhD students in STEM fields. PloS One. 2019 Jan 09;14(1). pmid:30625173
            • 81. Marín-Spiotta E, Barnes RT, Berhe AA, Hastings MG, Mattheis A, Schneider B, et al. Hostile climates are barriers to diversifying the geosciences. Advances in Geosciences. 2020 Jul 28;53:117–27.
            • 82. Stebleton MJ, Soria KM, Huesman RL Jr. First-Generation Students’ Sense of Belonging, Mental Health, and Use of Counseling Services at Public Research Universities. Journal of College Counseling. 2014 Apr 02;17(1):6–20.
            • 83. Alaee DZ, Campbell MK, Zwickl BM. Impact of virtual REU experiences on students’ psychosocial gains during the COVID-19 pandemic. arXiv:2108.07620 [Preprint]. 2021 Aug 17.
            • 84. Miller A, Williams L, Silberstein S. Found my place: the importance of faculty relationships for seniors’ sense of belonging. Higher Education Research & Development. 2019 Apr 16;38(3):594–608.
            • 85. Baumeister RF, Leary MR. The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin. 1995 May;117(3):497–529. pmid:7777651
            • 86. Brown S, Burdsal C. An exploration of sense of community and student success using the national survey of student engagement. The Journal of General Education. 2012 Dec 01;61(4):433–60.
            • 87. U.S. Department of Education. Federal Pell Grants are usually awarded only to undergraduate students [Internet]. Federal Student Aid. n.d. Available from https://studentaid.gov/understand-aid/types/grants/pell
              • 88. Rovai AP. Development of an instrument to measure classroom community. The Internet and Higher Education. 2002 Sep 01;5(3):197–211.
              • 89. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Evaluation and Program Planning. 1979 Jan 01;2(3):197–207. pmid:10245370
              • 90. Dong Y, Peng CY. Principled missing data methods for researchers. Springerplus. 2013 May 14;2(1):222. pmid:23853744; PMCID: PMC3701793.
              • 91. Alford RF, Leaver-Fay A, Gonzales L, Dolan EL, Gray JJ. A cyber-linked undergraduate research experience in computational biomolecular structure prediction and design. PLOS Computational Biology. 2017 Dec 7;13(12):e1005837. pmid:29216185
              • 92. Petersen S, Pearson BZ, Moriarty MA. Amplifying Voices: Investigating a Cross-Institutional, Mutual Mentoring Program for URM Women in STEM. Innov High Educ. 2020 Aug 1;45(4):317–32.
              • 93. Speer JE, Lyon M, Johnson J. Gains and Losses in Virtual Mentorship: A Descriptive Case Study of Undergraduate Mentees and Graduate Mentors in STEM Research during the COVID-19 Pandemic. CBE Life Sci Educ. 2021 Jun;20(2):ar14. pmid:33734867
              • 94. Adams C, Hemingway C. What Does Online Mentorship of Secondary Science Students Look Like? BioScience. 2014 Sep 16;64(11):1042–51.
              • 95. Breck S, Carr K, Davis DM, Nordhagen JN, Nye BD. Virtual Mentors in a Real STEM Fair: Experiences, Challenges, and Opportunities.
                • 96. Ensher EA, Heun C, Blanchard A. Online mentoring and computer-mediated communication: New directions in research. Journal of Vocational Behavior. 2003 Oct 1;63(2):264–88.
                • 97. Gregg N, Wolfe G, Jones S, Todd R, Moon N, Langston C. STEM E-Mentoring and Community College Students with Disabilities. Journal of Postsecondary Education and Disability. 2016;29(1):47–63.
                • 98. Gregg N, Galyardt A, Wolfe G, Moon N, Todd R. Virtual Mentoring and Persistence in STEM for Students With Disabilities. Career Development and Transition for Exceptional Individuals. 2017 Nov 1;40(4):205–14.
                • 99. Mack NA, Cummings R, Huff EW, Gosha K, Gilbert JE. Exploring the Needs and Preferences of Underrepresented Minority Students for an Intelligent Virtual Mentoring System. In: Stephanidis C, Antona M, editors. HCI International 2019 –Late Breaking Posters. Cham: Springer International Publishing; 2019. p. 213–21. (Communications in Computer and Information Science).
                  • 100. Oppenheim W, Knott A. Digital Mentorship, Global Service-Learning, and Critically-Engaged Undergraduate Research: Case Studies from Omprakash EdGE. Perspectives on Undergraduate Research and Mentoring. 2018;7(1); Available from: http://www.elon.edu/u/academics/undergraduate-research/purm/wp-content/uploads/sites/923/2019/06/Oppenheim-Knott_Main-1.pdf
                  • 101. Cantrell K, Fischer A, Bouzaher A, Bers M. The role of E-mentorship in a virtual world for youth transplant recipients. J Pediatr Oncol Nurs. 2010;27(6):344–55. pmid:20966162
                  • 102. Owen HD. Making the Most of Mobility: Virtual Mentoring and Education Practitioner Professional Development. Research in Learning Technology. 2015;23.
                  • 103. Dyrbye LN, Lipscomb W, Thibault G. Redesigning the Learning Environment to Promote Learner Well-Being and Professional Development. Academic Medicine. 2020 May;95(5):674–8. pmid:31789840
                  • 104. Jordan RK, Shah SS, Desai H, Tripi J, Mitchell A, Worth RG. Variation of stress levels, burnout, and resilience throughout the academic year in first-year medical students. PLOS ONE. 2020 Oct 15;15(10):e0240667. pmid:33057410
                  • 105. Mousavi MPS, Sohrabpour Z, Anderson EL, Stemig-Vindedahl A, Golden D, Christenson G, et al. Stress and Mental Health in Graduate School: How Student Empowerment Creates Lasting Change. J Chem Educ. 2018 Nov 13;95(11):1939–46.
                  • 106. Amdurer E, Boyatzis RE, Saatcioglu A, Smith ML, Taylor SN. Long term impact of emotional, social and cognitive intelligence competencies and GMAT on career and life satisfaction and career success. Front Psychol. 2014;5:1447. pmid:25566128