Introduction

Survey researchers often face many challenges to recruit respondents and to ensure data quality. However, the processes for developing surveys are not very often discussed [1–3]. Part of the controversy is related to random versus convenience samples. Defenders of random samples are always concerned with better coverage and unbiased data, while other researchers believe that convenience samples may better reflect local school reality [1,4–6].

While technology facilitates data collection, it is not a panacea. Electronic resources allow the involvement of a greater number of research centers and instruments [3,4,7–12], while other apply printed questionnaires to a few research centers and instruments [13–16]. A noteworthy exception is the series of studies of Roh et al. [17], on depression and academic performance, which utilized only printed copies involving 21 centers and 7,357 students, achieving over 52% response rate with the participation of 36 out of 41 existing medical schools in South Korea.

The medical course is long and demands full-time dedication, with frequent exposure to patient morbidity and mortality, which always raises in researchers the suspicion that, when using a convenience sample, those students experiencing greater levels of distress predominate among the volunteers responding to surveys, thus distorting the conclusions of studies. Applying random sampling adds complexity to the already challenging task of recruiting and motivating respondents, so this study aims to assess whether such additional effort is worthwhile.

This task must, therefore, be delegated to a network of local researchers which should, in turn, be committed to efficient data collection. However, the larger the study, the more complicated its implementation may be. In order to simplify operations and improve the adherence and compliance of local collaborators, most study designs choose a more straightforward convenience sampling, without randomization [11,18,19].

Here we evaluate Brazilian medical students’ quality of life, emotional competencies, and educational environment of random-selected students from 22 schools and volunteer students from 51 schools.

This study explores the rare opportunity to use an electronic platform recruiting a large number of students who were randomized at the same time that the project was publicized to medical schools nationwide, yielding a convenience sample comparable in size to the randomized group. For this reason, this study aimed to verify whether the assessment of psychophysical health is affected by conditions related to the recruitment method.

Literature review

To place the present work in context, we provide a brief review of the literature with emphasis on studies that describe the situation of medical education in Brazil, as well as evaluate quality of life, well-being, or related psychological and social metrics applied to undergraduate medical students as primary participants, including the sampling methods used, sample size, and the types of questionnaire administered.

Concerns about the quality of life of medical students in Brazil have existed for some time, particularly because there has been a pronounced expansion in the number of medical schools in the country over the last two decades.

For perspective, between 2010 and 2021 the number of schools changed from 183 to 366, with 54,870 vacancies available by the end of 2021 [20]. Due to the rapid expansion, some numerical divergence is observed depending on the time of year when the count is made; Tempski et al. reported a total of 201 active medical schools with an enrollment of 110,000 undergraduate students [21], of which 48 were newly established and still had no graduating class. The remaining 153 schools, with a combined enrollment of 86,000 students, were noted by Nassif et al. [22]. During this year in which the data collection period for the present study was performed, Brazil held the second position globally in the number of medical schools, trailing behind India, which had 346 schools [23].

This expansion continues, although with conflicting numbers. Official data should be published by INEP’s Higher Education Census and the federal e-MEC system; it is announced that data collection was concluded but its results have not yet been released, and the last reports seem to be related to 2022 [24]. Estimates from other sources for 2024/2025 are around 416 schools (373 active with 40,234 admission vacancies, and another 43 awaiting authorization to begin their activities) [22] or 448 schools with 44,491 admission vacancies per year [25]. Whatever the exact numbers, Brazil surpassed India (392 schools) and China (158 schools) [22], countries that have populations six times larger than ours, and the United States, which has 184 medical schools.

Approximately 87% of the new vacancies created between 2013 and 2022 were in private institutions [26], with tuition fees ranging from R$ 5,000 to R$ 16,000 [22] (for perspective, the Brazilian minimum wage in 2025 was increased to R$ 1,518). Due to this predominantly private expansion, federal programs were created for students from rural areas and lower-income backgrounds, thereby altering sociodemographic profiles in medical training; ProUni provides full or partial scholarships that do not require repayment, while FIES is a government student loan program that must be paid back under subsidized conditions.

Considering that the medical course in Brazil lasts six years, we are dealing with more than 260,000 students facing structural weaknesses in medical education that are not only relevant to academic quality and professional preparedness, but also have direct implications for the well-being and quality of life of students, who face disparities, high financial burden, uneven learning environments, and variable support across institutions [27]. Many institutions face precarious infrastructure, overloaded preceptors, and enduring regional inequalities [26]. These changes reinforce the need for public policies that aim to regulate and monitor educational quality, infrastructure, faculty development, socioeconomic inclusion, and quality of life.

In Brazil, a system of accreditation of medical schools was established in Brazil in 2015 by the Brazilian Federal Council of Medicine (Conselho Federal de Medicina, CFM) in collaboration with the Brazilian Association of Medical Education (ABEM). The evaluation includes 80 domains with several items in educational management, programs, faculty, students, and educational resources, classifying each item as sufficient or insufficient, in 76 medical schools. Although the study considered about 71.7% of these domains sufficient, it pointed out weaknesses in mentoring programs (26.3% sufficient), student permanence support (45.3%), health care (55.3%), and quality of life programs (43.4%) [28]. In addition to the fact that this initiative has reached only a small fraction of schools in operation, the accreditation process is voluntary. Therefore, it represents a convenience sample and, since schools are evaluated only if they request it, the overall scenario is probably more deficient than what is observed in this report.

Several publications from our research group approached different aspects of quality of life, emotional well-being, and academic performance. The present article compares the metrics of a sample obtained by convenience, which was conducted in parallel and had not yet been explored before, with another of a randomized sample. From this randomized sample of Brazilian medical students we can summarize the main findings. Paro et al. [29] applied the Davis’s Multidimensional Scale of Interpersonal Reactivity (IRI), the World Heath Organization Quality of Life Questionnaire (short form, WHOQOL-BREF), and the Maslach Burnout Inventory (MBI), finding that female students reported greater empathic concern but also greater personal distress, in addition to higher burnout scores. Emotional exhaustion and depersonalization were associated with lower empathy, whereas higher personal accomplishment correlated with greater cognitive empathy. Tempski et al. [30] used a Resilience Scale (RS-14), WHOQOL-BREF, the Dundee Ready Education Environment Measure (DREEM), Beck Depression Inventory (BDI), and the State-Trait Anxiety Inventory (STAI), and observed that students with low resilience reported poorer quality of life in all WHOQOL domains, more negative perceptions of the educational environment (DREEM), and more symptoms of anxiety and depression, showing a clear dose–response relationship. Enns et al. [31] evaluated global quality of life, WHOQOL-BREF, and DREEM, concluding that the perception of the educational environment was a key determinant of quality of life and mediated its association with depressive and anxious symptoms. Mayer et al. [32] applied the BDI and STAI, showing high prevalence of depression and anxiety, especially among women and mid-course students, highlighting the need for psychological support during training. Finally, Perotta et al. [33] used the Pittsburgh Sleep Quality Index (PSQI) and the Epworth Sleepiness Scale, and found that poor sleep quality and excessive daytime sleepiness were frequent and associated with worse quality of life and increased risk of burnout. Taken together, these studies highlight the multifactorial nature of medical students’ quality of life, involving emotional, cognitive, social, and environmental dimensions, and converge in pointing to the importance of institutional attention to students’ mental health, educational environment, and living conditions.

A cross-sectional survey of 129 Brazilian medical students found very high levels of psychological distress and burnout. Women scored higher on burnout, while men more often reported previous mental disorders. Anxiety was more frequent during the course, depression predominated before entering medical school. Alcohol misuse, cannabis and ecstasy use were more prevalent in comparison with other courses, thus suggesting that medical students represent a particularly vulnerable group in Brazil [34]. A more recent Brazilian survey used a convenience sample of 10,844 undergraduate students of several fields, including 3,659 medical students that showed sharper declines in quality of life, particularly in the physical and psychological domains, compared with peers from other fields. Anxiety and depression were highly prevalent, with a substantial proportion of untreated cases, and medical students reported greater use of antidepressants and cognitive enhancers. Although they expressed higher academic satisfaction, they also faced more time-management difficulties. The group presented a more advantaged socioeconomic profile because the sample was obtained from 32 private institutions, but revealed marked racial underrepresentation, showing that structural inequalities in access to medical education still remains [35]. Only one longitudinal study with a two-year follow-up was identified, in which 312 respondents completed questionnaires every semester approaching quality of life (WHOQOL-BREF), mental health (DASS-21), and religiousness (DUREL) instruments. As in other studies, medical students showed high levels of emotional disorders with depression, anxiety, stress, low income, female gender, early stages of medical training, and non-white ethnicity associated with poorer mental health and quality of life at follow-up [36].

Recent evidence underscores that these concerns remain unresolved not only in Brazil. For instance, Galgam et al. (2024) applied the WHOQOL-BREF among health students (n=349) and reported that dentistry students tend to achieve higher QoL scores compared with pharmacy and nursing students, with younger students in early years showing lower scores [37]. Putri et al. (2025) found a decline in QoL among nursing students (n=147), particularly in selected domains, using descriptive statistics and Pearson correlation [38]. Among health professionals in Gaza (n=1850), QoL was considered moderate despite the use of validated scales [39]. Other investigations reported strong associations between burnout, traumatic stress, and the practice environments of nursing students (n=341) [40]. In Austria, Huber et al. (2024) evaluated medical students before and after the COVID-19 pandemic, finding an increase in depressive and anxious symptoms, deterioration in subjective well-being, and perception of heavier workload and stress, with course year emerging as a predictor of QoL, particularly lower scores among those in early years [41].

Meta-analyses reveal similar trends. Pacheco et al. synthesized 59 studies with over 18,000 students and reported high prevalences of depression (30.6%), common mental disorders (31.5%), burnout (13.1%), problematic alcohol use (32.9%), stress (49.9%), poor sleep quality (51.5%), and anxiety (32.9%), with academic overload and lack of support as major correlates [42]. Soares et al. analyzed 14 studies and found a pooled prevalence of 43.3% for common mental disorders, associated with dissatisfaction with the course, desire to drop out, and sleep problems [43]. Solis and Lotufo-Neto identified predictors of quality of life in Brazilian medical students, showing lower scores among women and a decline across the years of training; negative predictors included depression, burnout, sleep difficulties, and chronic illness, while resilience, empathy, and physical activity were protective factors [44]. A recent meta-analysis identifying five eligible studies including 1,819 participants. The analysis compared students in the pre-clinical and clerkship cycles using the WHOQOL-BREF and found that medical training was associated with a decline in quality of life, particularly in the psychological and social domains. These findings indicate that the most critical impact of medical education on students’ well-being is concentrated in mental health and social aspects, suggesting the need for institutional measures to mitigate these effects [45].

Internationally, a meta-analysis by Erschens et al. including studies from multiple countries confirmed the heavy burden of burnout among medical students, with prevalence rates ranging from 7% to 75% depending on the context and measurement instruments [46], which suggests that this sort of problems is not a Brazilian uniqueness. A review mapped the literature on mental health among medical students in South Africa and identified eight eligible studies published between 2010 and 2023 [47]. Despite the limited evidence, the review also showed high rates of anxiety, depression, and burnout, with risk factors including academic overload, financial stress, cultural and linguistic isolation, and exposure to social inequalities and campus unrest.

In summary, many studies rely on non-probabilistic, convenience-based sampling (often via online or social media recruitment) without a systematic comparison of random versus volunteer respondents [35]. In Brazil, although numerous investigations have been conducted, most rely on convenience samples, often online, which limits representativeness. Cross-sectional designs with nonprobabilistic recruitment predominate, and there is substantial heterogeneity in instruments and limited standardization, hampering direct comparisons across studies. Systematic reviews and meta-analyses confirm these patterns and emphasize gender and year-of-course as key predictors of Quality of Life (QoL). However, few studies directly examine how recruitment strategy itself (randomized versus volunteer participation) may affect findings, leaving a gap regarding the validity and generalizability of survey-based evidence. The present study addresses this gap by analyzing large, parallel samples of randomized and volunteer medical students in a nationwide Brazilian survey.

Methods

In Portuguese, the acronym VERAS stands for “Vida do Estudante e Residente na Área da Saúde" (Life of the Student and Resident in the Health Area). This project began in 2010 with support of the Brazilian Ministry of Education, aiming at a cross-sectional, nationwide, multicenter study involving only medical undergraduate students, carried out by a core coordinating team and local researchers at each collaborating center.

The present study was classified as low-risk, and the research protocol number 181/11 was approved by the Research Ethics Committee of the School of Medicine, University of São Paulo (Comitê de Ética em Pesquisa da Faculdade de Medicina da Universidade de São Paulo). All participating medical schools were notified of this approval to ensure adherence to the study, including: Universidade Federal do Rio de Janeiro, Universidade Federal de Ciências da Saúde de Porto Alegre, Universidade Estadual do Piauí, Faculdade de Medicina de Petrópolis, Faculdade de Ciências Médicas da Paraíba, Pontifícia Universidade Católica de São Paulo, Universidade Federal do Ceará, Universidade Federal de Goiás, Universidade Federal de Mato Grosso do Sul, Escola Baiana de Medicina e Saúde Pública, Faculdade de Medicina de Marília, Faculdade de Medicina de São José do Rio Preto, Faculdade Evangélica do Paraná, Faculdade de Medicina do ABC, Fundação Universidade Federal de Rondônia, Pontifícia Universidade Católica do Rio Grande do Sul, Universidade Federal do Tocantins, Universidade Federal de Uberlândia, Universidade Estadual Paulista Júlio de Mesquita Filho, Centro Universitário Serra dos Órgãos, Universidade de Fortaleza, and Universidade de Passo Fundo.

Results

Participant recruitment and sample composition

A total of 1,650 random students were selected. They were from 31 schools initially committed; 9 schools were excluded (4 could not initiate data collection due to lack of institutional support, and 5 would not reach a minimum response rate of 50%), thus remaining 22 schools from which 1,350 random-selected students completed all questionnaires and were enrolled in this study.

There were at least 2 schools from each of the 5 geographic Brazilian regions; 13 are public schools and 9 are private schools; half of the schools are from noncapital areas and half from large metropolitan areas. This is a mixed group of schools, whether applying traditional, PBL, or blended methods. The number of years in operation for each school also varied: 6 are new schools founded during the past 15 years, 14 were founded between 43 and 63 years ago, and 2 others are the most traditional medical institutions in Brazil founded over a century ago.

In addition to the randomized sample, many non-selected students also chose to participate after the platform was announced to medical schools, resulting in a voluntary sample that was nearly balanced with the randomized one. Among 2,353 volunteers, 1,224 responded to all 14 questionnaires, but 23 answered that had already completed their undergraduation. Consequently, 1,201 volunteers were included in the present analysis for comparability purposes.

This criterion of completeness was applied to volunteers to ensure analytic consistency and comparability between groups because randomized students were required to complete the full set to be enrolled, while volunteer participants could freely desist at any point. In addition, among the excluded 1,129 incomplete submissions from volunteers, only 182 reached at least half of the questionnaires. It was also taken into account that partial responses would lead to inconsistently missing data and artificially lower scores because the instruments were presented in aleatory order.

These included 1,201 volunteers were students from 50 different schools, 646 from 21 schools in common with random-selected students, and 555 from 29 schools scattered across the country. Among these 29 schools, 19 are represented by only 1 to 3 volunteers, while another 9 have between 7 and 94 volunteers. One school, on the initiative of its administrator, recruited almost all of its students (263 respondents). These schools are also distributed between public and private schools and are located in non-capital or metropolitan areas.

The location of the 22 schools from which randomized students and 50 schools that contributed with volunteers is shown in Fig 1. A brief descriptive comparison of randomized and volunteer students is shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Location of Brazilian medical schools for randomized and volunteer students; circle diameter corresponds to the number of respondents.

Base map: Natural Earth, 1:50m cultural vectors, public domain (https://www.naturalearthdata.com/, accessed in August 13th, 2025 [58]).

https://doi.org/10.1371/journal.pone.0332850.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sample characteristics of randomized (R) and volunteer (V) students.

Biological sex, Age in years, BMI: body mass index, Activity: hours per week dedicated to physical activity, METs: metabolic equivalent estimated of Kcal consumed per day, Year: undergraduation current medical course year, School: type of school, Location: capital or non-capital areas, City Pop.: population of city where the school is located.

https://doi.org/10.1371/journal.pone.0332850.t001

Response effort

For randomized students, a collaborative effort was undertaken by the core team and local researchers, which manifested in the ongoing monitoring of the questionnaires obtained throughout the study. This effort to recruit randomized students was comprehensive and employed various outreach methods, including personal invitations, emails, letters, phone calls, and messages through social media. Members of the core group dedicated time to verify and contact school administrators, encouraging them to participate in data collection with their students.

Data collection occurred between August 2011 and August 2012. Throughout this period, additional support strategies were implemented, such as regular meetings, weekly reports displaying student registrations and questionnaire follow-ups, guidance for data collection, and monthly newsletters to local researchers. Despite these measures, the pace of data collection was not consistent (Fig 2). Towards the end of 2011, as expected, there was a decline in respondents, likely influenced by approaching holidays and summer vacation in Brazil. However, this slower pace persisted into the first months of 2012. It was during this period that we introduced the “adopt a school" strategy, recognizing that local researchers were losing track of their students or feeling disconnected from the core group of researchers. To foster a continuous connection between remote researchers and the core group, this strategy involved personalized and continuous contact between each local researcher and a specific member of the core group responsible for providing tailored follow-up or solutions based on the local researcher’s needs. We attribute the observed second wave of respondents to the implementation of this strategy. Some schools, in fact, only became active after the “adopt a school" strategy was implemented.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data collection of randomized students (number of respondents along time) from the three medical course cycles (basic, 1st and 2nd; clinical, 3rd and 4th; and clerkship, 5th and 6th years).

Two waves of respondents, were observed, the second coincident with the “adopt a school" strategy (see text). It was also observed that randomized respondents from the last course cycle, along the second wave, seemed to show a slower and progressive reaction to the estimuli than that obtained with the basic and clinical science cycles.

https://doi.org/10.1371/journal.pone.0332850.g002

Questionnaire completion

Students who answered the whole set of questionnaires at once spent an average of 73 minutes to complete the task, but time distribution (not shown) was highly asymmetrical due to students who discontinued answering a questionnaire, returning to it a few days later (a few students scattered their access along the allowed 10 days). Median is the most appropriate measuring, showing that each questionnaire was typically answered in 1 to 5 minutes; the most time-consuming questionnaires were VERAS-Q and DREEM (median of 7 minutes).

Table 2 summarizes randomized and volunteer responses. In addition to the sociodemographic and physical activity questions, the questionnaires and their subdomains are as follows:

1. GQol: a subjective rating from 0 to 10 given by the respondent for their overall quality of life.
2. CQol: a subjective rating from 0 to 10 given by the respondent for their quality of life in relation to the medical course.
3. WHOQOL: the World Health Organization Quality of Life Questionnaire (short form), with the following subdomains:
   • QoL: a rating from 1 to 5 for quality of life.
   • health: a rating from 1 to 5 for satisfaction with health.
   • phys: physical health.
   • psy: psychological health.
   • soc: social relationships.
   • env: environmental health.
4. DREEM: the Dundee Ready Educational Environment Measure, with the following subdomains:
   • learn: perception of learning.
   • teacher: perception of teachers.
   • self: academic self-perception.
   • env: perception of atmosphere.
   • soc: social self-perception.
5. VERASQ: Student Quality of Life Questionnaire, developed by the local researcher group, with the following subdomains:
   • time: time management and dedication to activities other than the course.
   • psy: positive and negative feelings, concentration, support, level of demand, and self-esteem.
   • phys: health care, sleep, leisure, physical activity, and appearance.
   • env: organization of courses, relationships with peers, teachers, and the institution.
6. IRI: Davis Interpersonal Reactivity Index, with the following subdomains:
   • empathy: empathetic concern.
   • perspec: perspective-taking.
   • distress: personal distress.
7. RS14: Wagnild and Young’s Resilience Scale, with the following subdomains:
   • selfreliance: self-reliance.
   • meaningful: meaningfulness.
   • equanimity: equanimity.
   • perseverance: perseverance.
   • aloneness: existential aloneness.
8. BURNOUT: Maslach Burnout Inventory, with the following subdomains:
   • exhaut: emotional exhaustion.
   • deperson: depersonalization.
   • accomp: personal accomplishment.
9. PSQI: Pittsburgh Sleep Quality Index, with the following subdomains:
   • quality: subjective sleep quality.
   • latency: sleep latency.
   • duration: sleep duration.
   • efficiency: sleep efficiency.
   • disturbance: sleep disturbance.
   • medication: use of sleep medication.
   • dysfunction: daytime dysfunction.
10. STAI_state and STAI_trace: State-Trace Anxiety Inventory.
11. BDI: Beck Depression Inventory.
12. EPWORTH: Epworth Sleepiness Scale.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of randomized (R) and volunteer (V) responses to questionnaire global and subdomain scores: mean (standard deviation).

See text for questionnaire descriptions.

https://doi.org/10.1371/journal.pone.0332850.t002

Effects of recruitment methods

We adopted a standardized layout across Tables 3 to 10. These tables present results of general linear models (GLM) applied to recruitment strategies (outcome: randomized or volunteer) and questionnaires (variables of interest: global scores or domains) under two conditions: without and with control variables, the latter showing which effects remain after adjustment for covariates. The lines labeled GLM correspond to the omnibus model test, associated with the F statistics, corresponding p value and ; here, rejection of the null model indicates the existence of all proposed models. In addition, for every variable of interest, we report the statistical significance and effect sizes. While statistical significance unnecessary verb informs that an effect exists (p-value), the effect size through the 95% confidence interval of Cohen’s eta-squared evaluates its magnitude (, expressed in percentage); statistically significant results and non-null effect sizes are highlighted in gray.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Global scores: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. WHOQOL: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. DREEM: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. VERASQ: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. IRI: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. RS14: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. BURNOUT: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. PSQI: general linear model without and with control variables; statistical significance (p values) and effect sizes (confidence interval of Cohen’s partial eta-squared, in percentage) with Bonferroni correction for 5% significance level.

Statistically significant differences and the presence of effect sizes are shaded in gray. Outcome is the recruitment methods (randomized or volunteers). GLM line is the overall model test (F, associated with ).

https://doi.org/10.1371/journal.pone.0332850.t010

The variables of interest in Table 3 are the global scores of the questionnaires. There is no effect of recruitment (randomized or volunteer groups), except for Maslach Burnout Inventory (Table 3, left panel). However, with the addition of control variables the difference between randomized and volunteers responses are mainly affected by the type of school (public or private), the size of the city population, and the school’s location (rural or metropolitan). These variables showed the largest effect sizes and are attributable to differences in questionnaire response patterns (in this example, only Burnout). Table 1 reveals that the convenience sample had a higher proportion of volunteers from private schools (presumably fee-paying students with better financial conditions), living in larger cities and in schools located in rural regions (Table 11 shows what differences are statistically significant). This example illustrates that, if statistical analysis were performed without accounting for variables known to influence questionnaire responses, we would risk identifying associations that could misleadingly suggest causal explanations for the observed differences. This does not deny that there is an average difference in the response pattern to the Burnout questionnaire or that the questionnaire is inadequate in its measurement (Table 2), but rather emphasizes that the type of school and the city of residence may play a dominant role in affecting the quality of life of students (in this case, specifically influencing the Burnout scores).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Estimated marginal means for main control variables: emmean indicates the adjusted proportion of volunteers and randomized respondents; cld shows the compact letter display of all pairwise comparisons to indicate grouping similarity; significant differences are reported with Bonferroni correction among categories.

https://doi.org/10.1371/journal.pone.0332850.t011

For this particular example, Table 9 details Burnout questionnaire, showing their domains as variables of interest. Statistical differences were observed in all three domains of Emotional Exhaustion (higher among volunteers), Depersonalization (higher among randomized students), and Reduced Personal Accomplishment domain (higher among randomized students) as shown in Table 2. However the Reduced Personal Accomplishment domain has negligible effect size (it includes zero in the confidence interval).

Similarly, the psychological subdomain of VERAS-Q (Table 6) showed effect of recruitment method, which vanished when control variables were included. In all other questionnaires no mean difference was observed between randomized and volunteer respondents (Tables 4, 5, 7, 8, and 10). Nevertheless, the control variables revealed impact on the response patterns of randomized versus volunteer students, usually in the following order of importance (i.e., effect size): type of school, city population, city location, medical school year, and sex (with negligible effect size) for WHOQOL, DREEM, IRI, RS14, and PSQI.

Therefore, it is suggested that the difference between the two groups lies in these five control variables. The recruitment effect, if it exists, is not detectable by the questionnaires. Thus, it is necessary to check if the randomized and volunteer student groups differ in one or more of the control variables using a post hoc test with estimated marginal means (R packages emmeans [51] and multcomp [52]). Table 11 shows that the groups differ in all these variables, but not in all categories: there are more women among volunteers and more men among randomized participants. A trend was observed for volunteers to be more frequent in the early years of medical school, gradually shifting so that responses from the final years required active recruitment by randomization. In the three categories of smaller city populations, randomized participants were more prevalent, whereas in cities over 5 million inhabitants, volunteers predominated. Finally, more randomized participants were from schools located in capitals while more volunteers came from schools located in rural areas.

Discussion

This study constitutes a groundbreaking initiative in Brazil and could be one of the first globally to utilize a distinctive sampling technique, innovative data collection strategies, and widespread participation from numerous schools and students. The instruments used in the present study have been previously described in publications from our research group with details on validation, scoring, and properties; namely, WHOQOL, Burnout, and IRI [29]; DREEM, WHOQOL, BDI, and RS-14 [30]; DREEM, WHOQOL, GCol, and CQol [31]; BDI and STAI [32]; WHOQOL, VERASQ, and METs [53]; and PSQI and Epworth [33]. In these studies the results of the randomly selected medical students have been previously published, but the data from volunteers have not. Here, there was a rare opportunity for comparison between recruitment strategies, having a data collection platform that obtained two large samples of comparable sizes of a convenience sample while simultaneously encouraging randomized students to respond to the questionnaires. Consequently, this enabled a more robust and meaningful comparison between the two groups. Given that the opportunity to collect the analyzed data preceded the substantial increase in medical schools in the country, this study can serve both as a baseline for comparison with future measurements and as a source of recommendations for the most effective strategies designed to reach a sufficient number of students.

SWOT analysis showed, as expected, that the large size of the Brazilian country and the unreliability of the telecommunication infrastructure hindered the study somewhat, as well as the small core research team, local researchers’ work overload and difficulties in motivating students to answer a large number of items. Perhaps, our relatively small losses may be related to our main strengths, which were threefold: support network (provided remotely by the core group to local researchers, and by personal contact between local researchers and volunteers), institutional support, and a customized electronic platform for data collection and follow-up. These strategies were a mixture of the electronic platform customization and successful human-driven actions.

Somewhat surprisingly, the number of volunteers was close to that of randomized respondents. In this study, we lack sufficient information to understand why this was the case. Concretely, many volunteers that started answering completed the all set of questions. On the other hand, we can only speculate on their motivations. Perhaps, volunteers had additional factors propelling them to undertake the task; it is conceivable that the intensive campaign was perceived by volunteers who, trusting the researchers, chose to contribute; or the electronic platform may have been considered attractive as it provided feedback at the conclusion of all responses, thus encouraging adherence.

A risk for the current study was the substantial volume of questions. While it may be more convenient to collect and analyze data through electronic systems, there is a risk of losing sight of the overall number of questions that respondents need to answer. As noted by Smith and Huntington [54], studies with 80 to 200 items experience a 10 to 15% loss due to respondent exhaustion, and those with over 200 items have a significantly reduced response rate. Therefore, all the already mentioned efforts to reach randomized students may be what contributed to achieve 81.8% of adherence (1350 out of 1650 respondents), a remarkably high proportion.

The question, therefore, is whether such effort pays off. Our expectation, in advocating for randomization, was to secure more reliable data with fewer biases concerning the perceptions of medical students. It is often assumed that students with poorer quality of life perceptions are more likely to volunteer, yet our findings do not support this. Randomization efforts are often associated with smaller samples and number of instruments, while larger samples and extensive evaluations tend to rely on convenience. For example, Mohammadi et al. [55] randomized 60 paramedics to assess the impact of bioethical education using two questionnaires (adherence or completion rates are not reported, which limits the evaluation of participant engagement). In contrast, a survey including 3,330 respondents out of 10,622 invitations (31% adherence), which applied multiple instruments to assess functional status, mental health, health literacy, social support, among other domains, used convenience sampling [56].

In general, studies rely on convenience samples. Quality of life among students and health professionals continues to appear in more recent publications, indicating that this concern is not a resolved issue. Examples include the application of WHOQOL-BREF showing that, among health students (n=349), dentistry students tend to report higher quality of life compared to pharmacy and nursing students, younger students in the early years showing lower scores [37]; and among nursing students (n=147) reporting declines in some domains using descriptive statistics and Pearson’s correlation [38]. Quality of life was considered moderate among health professionals (n=1850) in the Gaza Strip [39]. Using various instruments, nursing students (n=341) showed burnout and traumatic stress related to the type of environment in which activities are carried out [40]. Among medical students evaluated before and after the COVID-19 pandemic in Austria, increased symptoms of depressive and anxiety were observed, together with a decrease in subjective well-being and the perception of increased study load and stress. The study also indicated that the study year is a predictor of quality of life, with the early years associated with worse indicators. In our study, the group is homogeneous in terms of environments, since all medical students are exposed to a variety of settings, including outpatient care, wards, and emergency departments, throughout their training, with the emphasis shifting progressively from outpatient care to emergency attendance as they approach the end of the course. However, in our study, we did not find a clear and statistically significant trend related to the isolated effect of the study year on the questionnaire results among randomized or volunteer students, despite having considerable sample sizes.

Interestingly, we did not find important differences between the measurements obtained from the questionnaires, either in their overall scores or in their specific subdomains, when comparing the groups of randomized students and those who voluntarily responded to the questionnaires. What we identified were differences in the composition of the groups, leading to a higher proportion of women, fewer students in the later years of the course, more students from private schools, and more students from outside the capitals among the volunteers.

This study comes with certain limitations that need to be acknowledged before any interpretation. For instance, when observing an increase in exhaustion or a decrease in the perception of environmental quality with advancing course years, especially if it aligns with our preconceptions regarding the escalating stress in medical education, there might be a temptation to infer a causal explanation. Although this study stands among the largest conducted to date, we must emphasize that it adopts a cross-sectional approach. While this awareness is not new in the realm of science, it is crucial to recognize that it constitutes a potential source of cognitive biases. Additionally, from a statistical standpoint, the size of the respondent pool can easily yield significant differences in p-values, as they are highly sensitive to larger sample sizes. Consequently, even when p-values indicate statistical significance, it is imperative to critically evaluate the practical significance of each finding. Conversely, despite the substantial number of respondents, no significant differences were observed concerning the questionnaire measurements with respect to the recruitment methods.

The ultimate choice between a randomized or convenience sample hinges on one’s objectives. Even if potential bias exists, opting for convenience sampling based on volunteers, considering the worst-case scenario for a situational diagnosis, is justified. However, biases are more likely to occur in smaller samples. In larger samples, such as the one in the present study, personal bias may be diluted, resulting in similarity between the two student groups. Smaller studies may benefit from a bias toward overestimating a negative scenario, especially if it leads to interventions aimed at improving the quality of life, fostering a healthier environment, or enhancing relationships that could benefit all students, including those who may not respond to questions due to a more optimistic outlook. However, determining the effectiveness of any intervention may become challenging. It is well-established that individuals under burnout or depression, even after recovery, may persist with more negative views or express more critical opinions for an extended period, potentially longer than the duration of the medical course. Consequently, measuring outcomes among volunteers may erroneously show no effect of an intervention, leading to the abandonment of promising initiatives.

The present analysis includes only a limited number of control variables intended to address the observed differences between randomized and voluntary participants, thereby lessening the potential impact that the questionnaires might have revealed. One of the purposes of randomization is to eliminate the need for controls, assuming that perturbations by confounding variables are randomly dispersed and, consequently, their effects are neutralized in the final results. This rationale underlies the previous publications of our group, which focused solely on the 1350 randomized subjects. Opting for convenience samples necessitates the collection of additional data as potential controls, along with corresponding statistical analyses. Our findings suggest that, in large samples, life circumstances influence participation but do not significantly affect questionnaire outcomes.

Since the present study suggests that the two recruitment methods did not lead to disparate conclusions, studies using samples obtained through probabilistic and nonprobabilistic methods may be integrated in the future, and initial proposals in this regard already exist in the literature [57]. Sustainability of the research network is required to ensure future longitudinal studies and deepen our understanding of the universe of medical students, in order to contribute to the development and improvement of the medical school curricula.

Conclusion

These are the ten lessons learned from the VERAS project which can serve as guidelines to researchers interested in designing similar studies:

1. A user-friendly online platform enabled recruitment at reduced cost;
2. Random presentation of questionnaires is assumed to mitigate order effects;
3. Respondents must be able to pause and resume lengthy questionnaires;
4. Selection of questionnaires is essential to meet study objectives;
5. Immediate personal feedback is ethically advisable;
6. The electronic system enforced sequential completion, preventing missing data and incomplete responses;
7. Local researchers were required to increase adherence of respondents;
8. Weekly progress reports may motivate local researchers and support network cohesion;
9. Respondents should be informed and able to consult local researchers for support;
10. The combination of local researchers’ engagement and web-based accessibility likely contributed to large sample sizes.

Finally, our data suggest that the use of convenience samples may not introduce biases to study results. Therefore, the decision to randomize or persuade students to answer questionnaires should be left to the discretion of each researcher.