https://orcid.org/0000-0002-7744-5337
Figures
Abstract
Background
Scholarly publications are important indicators of research productivity and investigator development in Centers of Biomedical Research Excellence (COBREs). However, no information is available to describe implementation and evaluation of writing development programs within COBREs. Therefore, this paper aimed to evaluate the first year of a campus-wide COBRE-supported writing program.
Methods
A convergent parallel mixed-methods design (QUAN + QUAL) was used. All writing program participants were invited to complete post-participation surveys, and a subgroup was selected using purposive sampling to complete individual semi-structured interviews. Descriptive statistics were used to characterize survey data, and qualitative content analysis was employed to analyze interview data. Self-determination theory served as the theoretical framework by which themes were developed and interpreted.
Results
Professional staff, post-doctoral fellows, and faculty from all academic ranks (n = 29) participated in the writing program during its first year. Survey respondents (n = 18, response rate 62%) rated social support (89%), group accountability (89%), hearing group members’ writing goals (78%), receiving group advice (67%), and setting a weekly writing schedule (56%) as beneficial program components. Participants rated program benefits such as breaking away from other responsibilities, staying on task with writing goals, and receiving social support as most beneficial. During interviews, participants (n = 14) described five major themes related to the benefits received: 1) belonging to a community of writers; 2) managing writing-related emotions; 3) improved productivity; 4) establishing helpful writing habits; and 5) improved motivation for scholarly writing.
Conclusions
This first-year programmatic evaluation demonstrates the writing program’s effectiveness as a campus-level development resource supported by a research center. Both survey and interview data affirmed that participants perceived autonomy, competence, and relatedness were supported through participation in the writing program. Participants placed particular emphasis on the writing program’s successful development of a community of scholarly writers.
Citation: Franks AM, Teeter BS, Davis P, Allred M, Landes RD, Koturbash I, et al. (2024) First-year evaluation of a campus-wide, cross-disciplinary scholarly writing development program supported by a center for biomedical research excellence (COBRE). PLoS ONE 19(10): e0312322. https://doi.org/10.1371/journal.pone.0312322
Editor: Yolanda Malele-Kolisa, University of the Witwatersrand Johannesburg Faculty of Health Sciences, SOUTH AFRICA
Received: February 26, 2024; Accepted: September 17, 2024; Published: October 29, 2024
Copyright: © 2024 Franks et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files are available in the online Mendeley Data repository cited below: Franks, Amy; Teeter, Benjamin (2024), “Data for: evaluation of CCOP scholarly writing program”, Mendeley Data, V1, doi: 10.17632/mjbf9srd32.1.
Funding: Authors AMF, MA, RDL, IK, JW were supported by the Center for Childhood Obesity Prevention (CCOP), funded by the National Institute of General Medical Sciences of the National Institutes of Health (https://www.nigms.nih.gov/) under Award Number P20GM109096 (Arkansas Children’s Research Institute, PI: Weber). The content (program design, evaluation, decision to publish, and preparation of the manuscript) is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The National Institutes of Health-funded Centers of Biomedical Research Excellence (COBREs) offer institutions within Institutional Development Award (IDeA) states three phases of multi-year support to advance the development and sustainability of thematic multidisciplinary research centers. Previous evaluations of COBREs have underscored the value of publications as indicators of faculty research advancement [1–4]. In a 2023 comprehensive evaluation of funded COBREs, Schaller found that COBRE investigators had published over 30,000 papers since the COBRE program’s initiation in 2000, with a median of 130 papers published per COBRE [3]. However, despite this collective success of COBREs, faculty of all academic ranks encounter barriers to writing that impede their scholarly productivity. Mentors from one COBRE’s evaluation identified scientific publications as the area in which junior investigators had made the least progress [1].
One COBRE, the Center for Childhood Obesity Prevention (CCOP), is highly committed to the professional development of the center’s faculty and staff members, and its scholarly writing program is central to this philosophy of developing and advancing its personnel. The CCOP’s writing program launched in fall 2021 as an adaptation of a program previously developed by one of the authors (AMF) for faculty within a single department at the same university [5]. The writing program was designed using the framework of self-determination theory (SDT) to support faculty and staff members’ needs of autonomy, competence, and relatedness in the area of scholarly writing [6]. The program’s three major components are described in Table 1. These components and their individual activities were intentionally developed to enhance intrinsic motivation for scholarly writing by supporting fulfillment of the three basic psychological needs associated with SDT.
Initially, the writing program was designed to support CCOP-funded junior faculty as part of the stated COBRE goal of enhancing faculty development towards funding independence. However, over time, as the program’s popularity grew, the opportunity to participate was expanded to include faculty and staff in any program or college at the University of Arkansas for Medical Sciences (UAMS) as well those affiliated with the Arkansas Children’s Research Institute, thereby achieving campus-wide reach. As a result, the CCOP now offers a multi-component, cross-disciplinary, campus-wide program to support scholarly writing that is based on SDT. Few studies have evaluated the impact of a writing program through the theoretical lens of SDT. In this paper, we describe the early programmatic evaluation of this CCOP-supported, campus-wide writing program. Our mixed-methods analysis provides insight about how the basic psychological needs of autonomy, competence, and relatedness were supported by the program and how the program further supported their scholarly writing.
Methods
We evaluated the process, structure, and initial offering of the CCOP writing program using a convergent parallel mixed-methods design (QUAN + QUAL) [7]. The quantitative phase used post-participation surveys, and the qualitative phase employed semi-structured interviews. Results from each phase were analyzed and interpreted to explain and triangulate the perspectives of writing program participants. The University of Arkansas for Medical Sciences institutional review board determined this program evaluation did not involve human research and waived the requirement for consent. All interview participants agreed to audio recording of the interview.
Study setting and study population
The programmatic evaluation was conducted at the end of the first year of the CCOP writing program. Demographic characteristics of writing program participants were obtained from data collected during the program’s registration process. These demographic data were used to characterize the study population as shown in Table 2.
All participants of the Writer’s Block longitudinal program and/or the multi-day, off-campus writing retreat (n = 29) were invited to complete the post-participation survey(s) for the writing program component(s) they attended. Following the surveys, a purposive sampling approach was used to invite a subgroup of writing program participants to complete individual semi-structured interviews. Purposive sampling ensured participation by a representative sample of participants across writing program components, colleges, faculty ranks, and roles.
Quantitative evaluation
Post-participation survey data were collected following the completion of the Writer’s Block longitudinal program and the off-campus writing retreat. Participants were invited to complete survey(s) via email invitation. Survey data were collected using REDCap electronic data capture tools hosted at the University of Arkansas for Medical Sciences [8,9].
Survey items were developed by the authors to elicit information from the participants’ perspectives about their participation in the writing program (S1 File). Survey instruments explored different aspects of satisfaction and productivity associated with writing program participation and were not pilot tested in advance. Survey instruments included independent Likert items, ordered response items, and dichotomous response items (yes/no). For the Writer’s Block longitudinal program post-participation survey, there were 16 Likert items (what resulted from participation in program, satisfaction with program), 3 ordered-response items (writing habits, experience level as a scholarly writer), and 11 dichotomous response items (recommend program, identify helpful aspects of program, identify productivity resulting from program participation). For the writing retreat post-participation survey, there were 18 Likert items (what resulted from participation in retreat, satisfaction with retreat), 1 ordered-response item (experience level as a scholarly writer), and 8 dichotomous response items (participate again, identify new collaborators, identify productivity resulting from retreat participation). For both survey instruments, Likert items included 5 response options (1 = strongly disagree to 5 = strongly agree or 1 = very dissatisfied to 5 = very satisfied). Survey data were downloaded to a Microsoft Excel spreadsheet for review and analysis. Descriptive statistics were used to characterize the data. Likert items were analyzed independently using descriptive statistics (median, frequency, percent) and not analyzed as an aggregated or summed scale. Ordered response and dichotomous response items were also analyzed using descriptive statistics (frequency, percent).
Qualitative evaluation
Semi-structured interviews were performed to collect data for the qualitative component of the evaluation. One author (BST), who was experienced in qualitative methods and not involved in the design or facilitation of in the writing program, conducted in-person or videoconference interviews with individual participants. All interviews took place within one month of the individual’s participation in the writing program.
The interview guide (S2 File) was developed by the authors using concepts deemed important to the design of the writing program. Using open-ended prompts, the interviewer solicited information about each participant’s experiences in the writing program, focusing on which elements of the program were most beneficial and how the program benefitted participants. Questions were developed a priori to probe participants’ perspectives as well as generated in response to participants’ previous answers to elicit greater detail. Interviews were continued until data saturation was reached. Interviews were digitally recorded, transcribed verbatim by a professional transcriptionist, and checked for accuracy by one of the authors. Interview transcripts were de-identified and uploaded to MaxQDA 2020 (VERBI Software GmbH, Berlin, Germany) for analysis using a qualitative content analysis approach.
Two authors experienced in qualitative analysis developed a coding scheme using both inductive and deductive approaches (S3 File). The initial coding scheme included codes derived from the writing program components and activities, interview guide, and literature review. This coding scheme was further refined by a run-in review of interview transcripts by two authors to identify new concepts not included on the initial coding scheme. Codes and subcodes centered around three major categories: 1) activities and characteristics of the writing program that were beneficial, 2) the benefits experienced by writing program participants, and 3) suggestions for how the writing program could be improved. The coding scheme was entered into a MaxQDA project file, and all subsequent coding was performed using this software. Two authors independently coded each transcript using a constant comparison approach to ensure coding agreement, increase trustworthiness, and minimize bias. Discrepancies in coding were resolved by review of coded segments and discussion between coders. When needed, a third author assisted in reaching agreement between coders. After coding was completed, codes were grouped into categories, and themes were developed and interpreted drawing from the framework of SDT. Code frequencies were generated to further describe trends in qualitative data. Considerations according to the Consolidated Criteria for Reporting Qualitative Research (COREQ) [10] were used to guide the evaluation and preparation of this work (S4 File).
Results
Eighteen of the 29 writing program participants completed post-participation survey(s), yielding a response rate of 62%. Survey participants were broadly representative of all participants in the writing program according to position/faculty rank. Table 2 provides demographic information about participants completing post-participation survey(s).
All survey respondents agreed that they would participate in the writing program again or recommend the program to others (n = 18, 100%). Survey responses affirmed that the writing program supported the three basic needs of autonomy, competence, and relatedness (Figs 1 and 2). The top four most highly rated writing program elements from the surveys specifically supported relatedness through group interactions, thus highlighting the value of developing community with other scholarly writers (Fig 1). Survey results also indicated wide-ranging benefits resulting from participating in the writing program. Agreement with specific program benefits is depicted in Fig 2. Collectively, survey respondents indicated that the program helped them manage their own writing habits, increase their motivation for and productivity in writing, and develop a community of scholarly writers.
Response to survey item “What aspect(s) of [writing program] was/were helpful to you? Please choose one or more options”.
Response to the Likert item “Please report your agreement with each statement about [the writing program] below using a scale of 1 to 5, with 1 = strongly disagree and 5 = strongly agree. The [writing program]…”.
Fourteen writing program participants completed individual semi-structured interviews, with interviews lasting an average of ~28 minutes (range 17–39 minutes). Demographic characteristics of interview participants were similar to those of all writing program participants and are shown in Table 2.
The five program elements most frequently described as beneficial by interview participants were submitting word counts (n = 12, 86%), having dedicated writing time with others (n = 11, 79%), listening to other participants’ successes and challenges (n = 11, 79%), setting monthly writing goals (n = 10, 71%), and reducing distractions during social writing sessions (n = 10, 71%) (Table 3). It was evident that participants valued the program elements that helped them self-regulate, since setting goals, submitting evidence of progress (word counts), and limiting distractions all enhanced participants’ self-reported writing productivity. These program elements contributed to fulfilling the need for competence and autonomy. Participants’ discussion of activities that were social in nature, such as writing with others and engaging in discussions with fellow participants during writing group meetings, were strong indicators of the writing program’s support for relatedness.
Table 4 shows themes related to the benefits received by interviewed participants. All participants (100%) reported a sense of connection and belonging with other writers as a benefit of participating in the writing program. This relatedness occurred despite participants working on separate writing projects. Within this theme, several participants spoke about psychological safety among group members, and many described increased energy for writing when participating in social writing. All interview participants (100%) also discussed positive changes in their emotions related to scholarly writing. Participants described increased enjoyment of the writing process as well as reduction in negative emotions such as a fear and anxiety about their writing. Most (86%) also described improvement in their writing productivity as a result of their participation, thus realizing the progress they had made toward their writing projects or a reinvigoration for writing. Eleven (79%) participants described the development of new habits as a benefit to their participation in the writing program, and these new habits contributed to greater enjoyment and/or productivity in writing. Comments from nine participants (64%) also pointed to improved motivation to engage in scholarly writing, including placing a higher priority on writing as a responsibility of their work.
Interview participants provided suggestions for how the writing program may be improved (Table 5). Overall, participants reported being satisfied with the quality and benefits of the writing program, and suggestions for improvement varied considerably among interview participants. For example, one participant advocated for lengthening the Writer’s Block longitudinal program from a 6-month program to a year-round offering, while another recommended shortening the program to four months. Interview participants most commonly voiced support for offering more opportunities for social writing (n = 6, 43%) and enhancing the accountability measures of individuals (n = 6, 43%).
Discussion
In this early program evaluation, we focused on how the program was received by participants during its first year. Because the program was designed using an SDT framework, it was important to learn how the multidisciplinary group of participants viewed the program in terms of supporting their needs and how it benefited their scholarly writing. Data from survey responses and interviews show that participants viewed the program as supportive across all three basic psychological domains of autonomy, competence, and relatedness. The writing program was successfully expanded from a department-level approach within a single college to a cross-disciplinary campus program offered broadly to faculty and staff. The program represents a model for other institutions as a first step toward creating an institution-wide writing community.
The basic premise of SDT in an academic work context is that when the work environment helps faculty or staff members meet their basic psychological needs related to autonomy, competence, and relatedness, their intrinsic motivation to complete difficult self-directed tasks increases [6]. According to SDT, this should result in less work-related fatigue, a sense of value and ownership related to work responsibilities, greater enjoyment and engagement in writing, and enhanced motivation to pursue and complete scholarly writing projects [6,11,12]. Our evaluation demonstrated evidence of supporting all three basic psychological needs. The program’s emphasis on autonomy (prioritizing writing projects, setting writing goals, and instilling positive writing habits), competence (learning new writing-related skills, gaining writing experience, and developing confidence toward identity as a scholarly writer), and relatedness (discussing practices and progress with other writers, embracing writing as a social activity) equipped participants to adopt strategies to create and maintain regular progress toward publications. Participants valued opportunities to connect with others, demonstrating the program’s emphasis on a less traditional view of writing as a socially connected activity. As a result, participants reported experiencing a sense of belonging, enhanced self-regulation, and improved productivity. These are important lessons that help solidify the effectiveness of an SDT-based writing development program on instilling an institutional culture that supports scholarly writing.
We found only one other description of an SDT-based writing program in the published literature. Winnie and colleagues described a longitudinal writing support program for first-time physician authors who had previously presented their work at a national conference [13]. Similar to our findings, their evaluation of 9 participants’ comments indicated support of needs related to autonomy, competence, and relatedness. However, their program focused on the technical production of a manuscript through deadline-setting, peer feedback, and group discussion, which is in stark contrast to our program aimed at supporting scholarly writers through encouraging the establishment of productive writing habits and creating a supportive environment for engaging in social writing.
We are using the feedback received from this first-year evaluation to further enhance ongoing and future offerings of the writing program. Participants’ suggestions for further improving the program included more opportunities for social writing and additional focus on improving participants’ accountability to writing progress. In response, we are offering more social writing events and expanding opportunities for participants to learn more about each other and their work after receiving these suggestions. These suggestions highlight the participants’ need for relatedness and belonging to a community with similar interests, needs, and expectations for success. In addition, previous work supports enhancing accountability to the writing group. Individual commitment, and its effects on trust and development of relationships among group members, has been identified as a key factor for the success of a writing group [14]. Olszewska and colleagues highlighted the importance of fostering relational, communal, and institutional commitment and sustainability as writing programs are developed [14]. Grzybowski and colleagues found that faculty who frequently attended a writing group published more than those who infrequently attended or did not participate [15]. We believe increased accountability will not only enhance the individual’s experience in the program, but will also enhance the group’s interactions, sense of relatedness, and the overall sustainability of active participation in the program [14].
Our early evaluation provides insight on the acceptability and benefits of the writing program as perceived by its participants, as well as direction on how to further support the development of a writing community among faculty and staff at the institution. Although we have not yet evaluated the impact of the writing program on participants’ output of scholarly publications, the behaviors valued and demonstrated by participants suggest that writing habits have improved and may lead to an improvement in scholarly writing productivity. Ongoing evaluations of the writing program will consider the longer-term effects of the program on writing productivity, namely paper submissions and publications in peer-reviewed journals. While this initial evaluation also supports the expansion of a department-level development program to a campus-wide, multidisciplinary initiative, it is not yet known whether this is generalizable to other institutions or other types of support mechanisms. Finally, our data is limited by selection bias, since participants elected to enroll in the writing program and may have been more motivated to increase writing productivity than those who did not enroll in the program. Because the writing program emphasizes the development of intrinsic motivation by meeting basic psychological needs, we feel that mandatory participation would not be an effective strategy to broaden participation. However, helping potential participants such as junior investigators in COBREs better understand the goals and components of the writing program may encourage more participation by those who may not have otherwise enrolled in the program.
Conclusions
We successfully demonstrated the expansion of a department-level scholarly writing faculty development program to a research center-supported, campus-wide, cross-disciplinary model for a diverse group of faculty and staff. This is unique in that the writing program leveraged the resources of a research center (the CCOP COBRE) to build a faculty development program open to faculty and staff from across all campus units. Participants perceived the writing program to be beneficial in supporting all three basic psychological needs described by SDT. Further, participants reported developing a sense of community, experiencing more positive emotions and motivation, and improving habits and productivity related to writing as a result of participating in the program.
Supporting information
S1 File. Post-participation surveys for writing program.
https://doi.org/10.1371/journal.pone.0312322.s001
(PDF)
Acknowledgments
The authors gratefully acknowledge the participants of the writing program as well as those participants who provided valuable information during the program evaluation.
References
- 1. Kimberly JA, Rounds SI, Harrington EO, McNamara S. Formative evaluation results of a phase 2 Center of Biomedical Research Excellence (COBRE). J Clin Transl Sci. 2020;4(6):493–7. pmid:33948225
- 2.
National Center for Research Resources, National Institute of General Medical Sciences (NIGMS). Process evaluation of the Centers of Biomedical Research Excellence (COBRE) Program. 2008 [cited 23 Feb 2024]. https://www.nigms.nih.gov/Research/DRCB/IDeA/Documents/2008_evaluation_key_findings.pdf.
- 3.
Shaller MD. Efficacy of Centers of Biomedical Research Excellence (CoBRE) grants to build research capacity in underrepresented states. bioRxiv:2023.08.02.551624 [Preprint]. 2023 [cited 2023 Oct 30]. https://www.biorxiv.org/content/biorxiv/early/2023/08/05/2023.08.02.551624.full.pdf.
- 4. von Bartheld CS, Houmanfar R, Candido A. Prediction of junior faculty success in biomedical research: comparison of metrics and effects of mentoring programs. PeerJ. 2015;3:e1262. pmid:26421238
- 5. Franks AM. Design and evaluation of a longitudinal faculty development program to advance scholarly writing among pharmacy practice faculty. Am J Pharm Educ. 2018;82(6):6556. pmid:30181674
- 6. Deci EL, Olafsen AH, Ryan RM. Self-determination theory in work organizations: the state of a science. Annu Rev Organ Psychol Organ Behav. 2017;4:19–43.
- 7.
Creswell J, Plano Clark V. Designing and conducting mixed methods research. 3rd ed. Los Angeles, CA: SAGE; 2018.
- 8. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208. pmid:31078660
- 9. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. pmid:18929686
- 10. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. pmid:17872937
- 11. Stupnisky RH, BrckaLorenz A, Laird TFN. How does faculty research motivation type relate to success? A test of self-determination theory. Int J Educ Res. 2019;98:25–35.
- 12. Silva MN, Marques MM, Teixeira PJ. Testing theory in practice: the example of self-determination theory-based interventions. Eur Health Psychol. 2014;16(5):171–80.
- 13. Winnie K, Jackson JT, Ledford CJW. Writing Rounds: an innovation to increase physician scientific dissemination. PRiMER. 2021;5:34. pmid:34841209
- 14. Olszewska K, Lock J. Examining success and sustainability of academic writing: a case study of two writing-group models. Can J High Educ. 2016;46(4):132–45.
- 15. Grzybowski SCW, Bates J, Calam B, Alred J, Martin RE, Andrew R, et al. A physician peer support writing group. Fam Med. 2003;35(3):195–201. pmid:12670113