http://orcid.org/0000-0001-6283-5546
Figures
Abstract
Millions of people worldwide use online learning for post-secondary education and professional development, but participation from historically underrepresented groups remains low. Their choices to enroll in online courses can be influenced by course features that signal anticipated success and belonging, which motivates research to identify features associated with sociodemographic variation in enrollments. This pre-registered field study of 1.4 million enrollments in 159 online courses across 20 institutions identifies features that predict enrollment patterns in terms of age, gender, educational attainment, and socioeconomic status. Among forty visual and verbal features, course discipline, stated requirements, and presence of gender cues emerge as significant predictors of enrollment, while instructor skin color, linguistic style of course descriptions, prestige markers, and references to diversity do not predict who enrolls. This suggests strategic changes to how courses are presented to improve diversity and inclusion in online education.
Citation: Kizilcec RF, Kambhampaty A (2020) Identifying course characteristics associated with sociodemographic variation in enrollments across 159 online courses from 20 institutions. PLoS ONE 15(10): e0239766. https://doi.org/10.1371/journal.pone.0239766
Editor: Vitomir Kovanovic, University of South Australia, AUSTRALIA
Received: March 25, 2020; Accepted: September 13, 2020; Published: October 14, 2020
Copyright: © 2020 Kizilcec, Kambhampaty. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files are available from the OSF repository (https://osf.io/z2g4p/).
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Human capital development in higher and continuing education relies increasingly on online learning. Total enrollment at intuitions of higher education in the United States (US) is declining whilst more and more people turn to online education for post-secondary degrees and professional development [1]. Fifteen percent of US college students are currently enrolled in fully online degree programs, and a third of on-campus students take at least one online course during their studies [2]. Worldwide online learning is poised to expand access to higher and continuing education, especially in underserved communities [3–6]. However, there is mounting evidence of online courses perpetuating historical participation and achievement gaps, including the underrepresentation of people with lower socioeconomic status and women in STEM disciplines [7–11]. Researchers have argued that the presence of visual and verbal cues in online environments can activate psychological stereotypes and reduce the sense of belonging in the learning environment for certain groups of students [12–16]. In particular, enrollment pages that provide descriptions and contextual information about courses are known to impact enrollment decisions [16, 17] and students’ eventual course grades [18]. An important unknown, however, is which course features contribute to sociodemographic variation in enrollment. Identifying influential features can provide instructors and online learning providers with a lever to adjust enrollment patterns and encourage participation from members of traditionally underrepresented groups.
Prior research has studied specific multimedia cues expected to raise identity-based concerns and influence anticipated belonging, including gender-stereotypic TV commercials [19, 20], promotional videos for STEM events [21], artifacts in computer science classrooms [16, 22, 23], and the design of a course website [14]. Their findings confirm that visual and verbal cues can shape perceptions about the diversity climate and reduce anticipated belonging and success. Fewer studies have demonstrated that these cues also affect real-world behaviors. One study found social advertising with images and messages to encourage women to take a STEM course to raise female enrollment [12]. Another study found a diversity statement on the enrollment page to raise enrollment among people with low educational attainment but not enrollment among women or people in developing countries [17]. Two more studies found the gender of people in lecture videos to influence women’s engagement inside of the course [13, 15]. These four studies offer causal evidence that certain visual and verbal cues can impact course participation for some sociodemographic groups. However, to achieve a more holistic understanding of major cues and their relation to who enrolls, we need a comprehensive and systematic analysis of which common course features do and do not predict sociodemographic variation in participation.
We conducted a large-scale field study to identify important features on the enrollment pages of 159 courses from 20 international institutions on two major online platforms. Online course enrollment pages have relatively standardized features across platforms. This enables us to investigate the following research question at large: Which features of course enrollment pages predict enrollment patterns in terms of age, gender, educational attainment, and socioeconomic status? We focus on these four social groups that have been identified in prior work as underrepresented in online learning: women (especially in STEM subjects), people with low educational attainment, people in developing countries, and older (mid- to late-career) learners [24–27]. We investigate sociodemographic variation along these four dimensions among 1.4 million enrollments across a wide range of courses and institutions. We study this variation as a function of different course features, because features can impact enrollments, but the causal arrow rarely points the other way.
The current research combines a pre-registered confirmatory study with an exploratory analysis and a replication study. First, using a taxonomy of psychologically inclusive design cues [12], we coded numerous features on the enrollment pages of all available courses on the Stanford Online platform. After examining the empirical distributions of these features, we pre-registered hypotheses about their relationship with specific enrollment patterns (e.g., Courses with female instructors have higher rates of female enrollment). We obtained official course enrollment numbers for each sociodemographic group under investigation to test our hypotheses and explore unexpected patterns. Finally, to replicate our findings in a new environment, we sampled over one hundred online courses offered by twenty international institutions on edX, a major online education platform. We coded their enrollment pages for features that were important in the first sample and tested which patterns replicate with official enrollment data. The findings of this research highlight course features that warrant close attention by policy-makers and researchers in efforts to broaden participation, as well as course features that warrant less attention in this context.
Hypotheses
We pre-registered theory-based hypotheses about which course features are likely to predict sociodemographic patterns in course enrollment (https://osf.io/z2g4p/). These hypotheses focus on patterns in female enrollment and enrollment from developing countries, because there is sufficient evidence to inform these hypotheses, unlike for patters in age and educational attainment. The hypotheses are organized into three tiers based on the strength of evidence in support of them: primary, secondary, and exploratory.
Our first primary hypothesis is grounded in official statistics documenting the continued underrepresentation of women and minorities in STEM fields [28]. We therefore expected lower female participation in STEM courses (H1). Grounded in a large body of work documenting the importance of academic role models [29–31], we expected instructor characteristics to influence enrollment patterns. In particular, female learners who notice a lack of female instructors on the course page may anticipate a reduced sense of belonging and decide not to enroll [32, 33] (H2). Likewise, people in developing countries who notice an absence of non-white course instructors may anticipate a reduced sense of belonging and decide not to enroll [8, 34]. We therefore expected to observe higher female participation in courses with more female instructors and higher participation from developing countries in courses with instructors that appear ethno-racially diverse (H3).
- H1 Courses in STEM subjects have lower rates of female enrollment.
- H2 Courses with (photos of) female instructors have higher rates of female enrollment.
- H3 Courses with (photos of) a majority of white instructors have lower rates of enrollment from developing countries.
Our secondary hypotheses are informed by research that has demonstrated a link between linguistic features of job advertisements and gender differences in the perceived appeal of those jobs [35, 36]. Specifically, this work found that job ads with more masculine than feminine wording caused women to think that more men worked in the occupation and lowered the job’s appeal to women. We therefore expected to observe lower female participation in courses with descriptions that have a more masculine tone (H4). Sociolinguistic research on gender differences has also found women to be more attuned to emotional language, especially negative sentiments, and to avoid negativity in their own language production [37, 38]. Course descriptions that highlight negative emotions (e.g., for courses covering war crimes, nuclear disasters) may thus reduce the appeal of the course to women. We therefore expected to observe lower female participation in courses with descriptions with a more negative sentiment (H5). Moreover, as a visual counterpart to the linguistic features, and as theorized in research on the global achievement gap in MOOCs [8], we expected lower participation from developing countries in courses that exhibit overt markers of prestige, such as pictures of august university campuses (H6).
- H4 Courses with descriptions that have more masculine tones have lower rates of female enrollment.
- H5 Courses with descriptions that have more negative sentiment analysis scores have lower rates of female enrollment.
- H6 Courses with prestige markers have lower rates of enrollment from developing countries.
Our exploratory hypotheses encode expectations with limited empirical support in the literature to date. Extending the analysis of how female participation may be related to linguistic features of course descriptions, we expected course descriptions invoking ‘joy’ to be construed as welcoming and encouraging female enrollment (H7), while descriptions invoking ‘disgust’ would be construed as unappealing and limit female enrollment (H8) [39, 40]. Next, we expected references to diversity on the course page to send a signal of inclusion to women and increase their enrollment (H9), as has been observed in prior studies [12, 41]. The remaining exploratory hypotheses concern common course properties: whether the course has prerequisites, quizzes or tests, and how many hours of weekly effort are required. Prior work has found that men are more likely to apply for a job when they meet only 60% of the stated qualifications, while women are more likely to apply when they meet all qualifications [42]. We therefore expect to observe lower female enrollment in courses with prerequisites (H10). Indications that a course contains quizzes or tests may raise identity-related threat and lower participation among women and people in developing countries (H11) [43, 44]. Finally, the estimated number of hours per week that a course requires is expected to be associated with enrollment from developing countries: a high number may discourage or intimidate people who face scarce resources, but it may also be perceived as a signal of high value [45]. We therefore expect that its association with participation can go in either direction (H12).
- H7 Courses with descriptions that express more joy have higher rates of female enrollment.
- H8 Courses with descriptions that express more disgust have lower rates of female enrollment.
- H9 Courses with references to diversity have higher rates of female enrollment.
- H10 Courses with prerequisites have lower female enrollment.
- H11 Courses with quizzes or tests have lower rates of (a) female enrollment, and (b) enrollment from people in less-developed countries.
- H12 Courses with higher weekly effort have higher (or lower) rates of enrollment from less-developed countries.
Methods
Course sample selection
We examined two independent samples of online courses from different platforms. The first sample (Stanford sample, n = 54 courses, 311,818 total enrollments) was drawn from courses offered on Stanford Online, a platform for free online courses, graduate and professional certificates, advanced degrees, and executive education programs offered by Stanford University. We considered all courses that were open for enrollment and offered in English, yielding a set of 54 courses with a wide range of subjects, enrollment sizes, and levels of difficulty. The second sample (edX sample, n = 105 courses, 1,090,168 total enrollments) was drawn from courses offered on edX.org, one of the largest platforms for massive open online courses, for-credit courses, and bachelor’s and master’s degree programs offered by 130 global partners (nonprofits, universities, companies, etc.) [46]. From the 2,600 courses offered on edX.org, we selected 105 courses from twenty institutions to approximate a representative sample. We restricted the list of potential courses to those offered by member institutions of the edX research data sharing program, because we could obtain official enrollment data for them. We selected a wide range of subjects from a variety of global institutions, oversampling courses from US institutions (71 out of 105), and including at least three courses from each of the major subject areas on the edX platform (Arts & Culture; Business & Management; Computer Science; Biology & Life Sciences; Energy & Earth Sciences; Engineering; Humanities; Social Sciences; etc.).
Course feature coding
The coding scheme was developed in iterations by closely examining the course enrollment pages in the Stanford sample for features that can be codified and that a potential learner sees before enrolling. Course enrollment pages on a given platform follow a standard template, as illustrated in the wireframe schematic in Fig 1. The specific layout and design varies between platforms, while the set of features remains mostly constant. Grounded in the taxonomy of psychologically inclusive design cues [12], we coded for visual content cues (e.g., photos of course instructors) and verbal content cues (e.g., presence of prerequisites, tone of text); we did not code for visual design cues because they are constant throughout a given platform, nor did we code for interaction design cues because there was no opportunity for interaction before enrolling.
A wireframe illustration of a course enrollment page highlighting key features that were coded.
Instructor gender and race were coded manually by the authors on the basis of instructor photos and names on the enrollment page. Gender was coded as -1 if all instructors were male, 0 if there were male and female instructors, and 1 if all instructors were female. Race was coded according to skin color for courses with instructor photos present as 0 if half or more instructors appeared to be white, and 1 if the majority of instructors were not white.
We used two online natural language processing tools to code linguistic features of text that appears on enrollment pages: IBM Watson’s Natural Language Understanding API for sentiment analysis (negative to positive; [-1,1]) and emotion detection (Sadness, Joy, Fear, Disgust, Anger; [0, 1]), as well as Textio Hire for an overall score (how inviting the text sounds to potential job applicants; [0, 1]; https://textio.com/) and gendered tone analysis (how much the tone appeals to masculine versus feminine applicants; [-3,3]; https://www.ibm.com/cloud/watson-natural-language-understanding). The IBM Watson tool is grounded in psycholinguistic research and widely used; it applies a stacked generalization-based ensemble method with features including n-grams, punctuation, greetings, and sentiment polarity to classify emotion categories. For standard emotion data sets such as ISEAR and SEMEVAL, IBM Watson’s emotional tone analysis achieves statistically better average performance than the best accuracy of state-of-the-art models: average F1 score of 41% and 68% compared to 37% and 63%, respectively (https://cloud.ibm.com/docs/tone-analyzer). The Textio Hire tool uses a proprietary algorithm trained on thousands of job descriptions and applicant information to predict general appeal and gender bias. We checked its face validity in the present context by sampling course page texts with high and low Textio scores, which confirmed a stark contrast in language: the masculine text included words like ‘business’, ‘companies’, ‘trade’, ‘leadership’, ‘top executives’, while the feminine text included words like ‘empathy’, ‘trust’, ‘happiness’, ‘emotional and social intelligence’.
The coding scheme for each sample is shown in Table 1; a detailed code book and data for the Stanford sample can be found in our pre-registration document (https://osf.io/z2g4p/). The coding scheme for the edX sample was based on the results for the Stanford sample. We coded features in the edX sample that showed a robust correlation (defined as p- value < 0.1 and r > 0.25) with at least one enrollment measure in the Stanford sample. We excluded linguistic features for the FAQ text as few courses in the edX sample presented a separate FAQ text. We additionally coded five exploratory features: three features that we pre-registered hypotheses about (Instructor Race, Intro in Course Title, and “About” Text Textio Tone) and two new features to codify institutional variation, since edX courses were offered by various international institutions (the institution’s 2019 QS World University ranking and the 2018 Human Development Index, or HDI, of the country where the institution is located; https://www.topuniversities.com/university-rankings/world-university-rankings/2019).
Measures of enrollment
We use four indicators of sociodemographic enrollment patterns that were established in prior research [17]: (1) the percentage of female learners in the course, (2) the percentage of learners who reside in countries with a United Nations Human Development Index (HDI) below 0.7 (officially classified as low-to-medium development), (3) the percentage of learners who do not have a college degree, and (4) the average age of learners. We obtained course-level aggregate enrollment data for each of the courses in both samples after we pre-registered our hypotheses. Gender, age, and education level was self-reported by learners during the enrollment process; HDI was determined using the country-level location from the IP address during enrollment.
Analytical approach
After coding the course pages in the Stanford sample, we examined the empirical distributions of all coded features to focus our attention on those with sufficient variance to find robust correlations with enrollment patterns. Our pre-registered hypotheses (https://osf.io/z2g4p/) were informed in part by the distributional characteristics of these variables, summarized in S1 Table. For the pre-registered analysis, we computed Pearson correlation coefficients and corresponding p-values using the cor.test() function in R. We used Pearson correlation coefficients, because they can be computed consistently for course features that are binary, ordinal, or continuous; they can be compared directly across outcomes and features, and their interpretation is intuitive to most readers. We first tested the correlates as specified in the hypotheses and then expanded our analysis to examine correlations between other coded variables and enrollment patterns. Finally, for course characteristics that show significant associations with enrollments across both samples, we estimate the overall strength of the association in terms of group averages. To this end, we fit a linear regression model predicting the measure of enrollment (e.g., percentage of female enrollment) with one course characteristic (e.g., a binary indicator for course prerequisites) and a mean-centered platform fixed effect to account for platform differences. Test statistics are computed using robust standard errors to account for potential heteroskedasticity.
Data and code availability
Raw data for both samples and analysis code for reproducing Figs 2–4 are available at https://osf.io/z2g4p/.
Histograms showing the distribution of each measure of enrollment for the 159 courses in the Stanford and edX sample. The sample mean (and standard deviation) for each measure are 38% (16%) female, 28% (10%) from developing countries, 22% (8.9%) not college educated, and 32 (3.2) average age.
Showing the average enrollment rate in STEM courses (red), non-STEM courses (blue), and all combined (black); error bars are 1 SEM.
Female enrollment (top panels) and enrollment from developing countries (bottom) varies with whether a course specifies prerequisites (left) and its expected amount of effort (right). Error bars show 1 SEM.
Results
Sociodemographic enrollment patterns vary substantially across the 159 courses, as shown in Fig 2. The average course enrollment is 38% female, 22% not college educated, 28% from developing countries (HDI < 0.7), and 32 years in age. The substantial spread in these distributions enables us to identify predictive course features: across all courses female enrollment ranges from 10% to 75%, developing country enrollment from 5% to 59%, not college educated enrollment from 4% to 45%, and average age from 24 to 40. Courses with higher female enrollment tended to enroll older students (Pearson correlation: r = 0.37, p < 0.001) and fewer from developing countries (r = −0.63, p < 0.001), but with similar college education levels (r = −0.05, p = 0.54). Courses with lower college educated enrollment tended to enroll younger students (r = −0.52, p < 0.001) and more from developing countries (r = 0.29, p < 0.001). Courses with higher developing country enrollment also enrolled younger students (r = −0.43, p < 0.001).
Results are presented in chronological order, beginning with confirmatory and exploratory findings for the Stanford sample (Table 2), then findings for the edX sample (Table 3), and finally a summary of findings that replicate across both samples (Table 4).
Findings in the Stanford sample
There is support for two of the three primary hypotheses in the Stanford sample: STEM courses have lower rates of female enrollment (H1) and courses with more female instructors have higher rates of female enrollment (H2). However, the hypothesized relationship between instructors’ ethno-racial diversity and enrollment from developing countries is not supported (H3). None of our secondary hypotheses concerning linguistic features and the presence of prestige markers are supported either (H4-6). Among the exploratory hypotheses (H7-12), we only find evidence that courses with prerequisites have lower rates of female enrollment (H10). Corresponding correlation coefficients with levels of statistical significance are presented in Table 2.
Beyond our hypotheses, we discover a number of unexpected significant associations between pre-registered course features and enrollment measures. STEM courses and courses with more male instructors have a higher rate of enrollment from developing countries. STEM courses, courses without prestige markers, and courses demanding more weekly effort enroll younger people. Courses demanding more weekly effort also have a lower rate of female enrollment. In subsequent post-hoc analyses of the remaining course features, we also find that certain linguistic features—sadness and appealing tone—significantly correlate with sociodemographic patterns in enrollment.
Findings in the edX sample
We conducted confirmatory analyses in the edX sample for six course features that exhibited a robust association (p <.05 and r >.25) with enrollment outcomes in the Stanford sample. We find strong evidence that these features are also predictive of enrollment patterns in this broader sample (Table 3). STEM courses again have lower rates of female enrollment (H1), a higher rate of enrollment from developing countries, and enroll younger people. Likewise, courses with more female instructors have a higher rate of female enrollment (H2). Courses with prerequisites have lower rates of female enrollment (H10), but higher rates of enrollment from developing countries and college-educated people. Courses demanding more effort have lower rates of female enrollment, but higher enrollment from developing countries (H12) and younger people. Courses with sad descriptions enroll more women in the edX sample, though in the Stanford sample, they enroll younger, less educated people. While the correlation between Textio Score for FAQs and female enrollment is robust in the Stanford sample, we are unable to replicate the finding because most courses in the edX sample do not have FAQs.
In addition to the confirmatory analyses, we explore correlations for three of the pre-registered features plus two institution-level features since the edX sample contains courses from 19 international institutions. We find that courses with (a majority of) non-white instructor(s) have higher enrollment among college-educated people. Courses with titles indicating an introductory level have higher rates of enrollment from younger people, those without a college degree, and those in developing countries. Courses with descriptions that use a feminine tone have higher rates of female enrollment (H4). Although the world ranking of the course institution does not predict sociodemographic enrollment patterns, its geographic location does. Specifically, courses offered by institutions in more developed countries have a higher rate of enrollment from males, people in developing countries, and older, college-educated people.
Consistent findings across samples
A number of course features stand out as significant predictors of sociodemographic enrollment patterns across both samples. They are summarized in Table 4.
Women account for only 29% of enrollments in STEM courses compared to 49% in non-STEM courses (H1). The percentage of women is higher in courses with more female instructors (H2): 32% all male instructors, 41% both male and female instructors, and 50% all female instructors. Fig 3 visualizes this remarkably linear trend, which appears for STEM and non-STEM courses alike. That is, even in courses covering non-STEM subjects, more women enroll if there are more female instructors.
More women enroll in courses without any prerequisites (45% without vs. 29% with; H10) and courses that demand less weekly effort: the average percentage of women is 44% in a course demanding 2.5 hours compared to 31% in a course demanding 7 hours. These associations are illustrated in Fig 4 (top row). STEM courses are twice as likely to have prerequisites than non-STEM courses (60% vs. 33%; χ2 = 10.1, p = 0.001) and demand 1.2 hours, or 29%, more weekly effort on average (t137 = 3.24, p = 0.001). Yet even within the subset of STEM courses, having prerequisites and demanding more weekly effort predicts lower female enrollment simultaneously in a regression model (prerequisites: β = −0.10, t73 = −3.06, p = 0.003; z-scored effort: β = −0.024, t73 = −2.30, p = 0.024). The pattern is the same for non-STEM courses (prerequisites: β = −0.089, t58 = −2.85, p = 0.006; z-scored effort: β = −0.045, t58 = −2.66, p = 0.010).
Overall, 57% of variation in the proportion of female enrollment can be explained by just four course characteristics: STEM subject, instructor gender, weekly effort, and prerequisites (multiple regression R2 = 0.566 with platform fixed effect; F6,132 = 28.7, p < 0.001). Despite correlations between these course characteristics, each one is a significant predictor of female enrollment in a combined ANOVA (each p < 0.001).
Enrollment from developing countries also varies based on course prerequisites and weekly effort (Fig 4, bottom row). This pattern is significant in the edX sample but not in the (smaller) Stanford sample, and therefore does not appear in Table 4. Nevertheless, across both samples, developing country enrollment is significantly higher in courses with prerequisites (31% with vs. 24% without; t156 = 4.33, p < 0.001) and in courses demanding more weekly effort (48% for 7hrs vs. 35% for 2.5hrs; t136 = 3.15, p = 0.002). Developing country enrollment is also higher in STEM courses (see Table 4), which are more likely to have prerequisites and demand more weekly effort. Repeating the analysis separately for STEM and non-STEM courses, we find that for non-STEM courses having prerequisites and demanding more weekly effort predicts higher developing country enrollment simultaneously in a regression model (prerequisites: β = 0.70, t58 = 3.35, p = 0.001; z-scored effort: β = 0.017, t58 = 2.01, p = 0.049). However, the associations are attenuated and not statistically significant among STEM courses (prerequisites: β = 0.017, t73 = 0.58, p = 0.56; z-scored effort: β = 0.015, t73 = 1.20, p = 0.24).
Overall, 24% of variation in the proportion of developing country enrollment can be explained by just three course characteristics: STEM subject, weekly effort, and prerequisites (multiple regression R2 = 0.238 with platform fixed effect; F4,134 = 10.4, p < 0.001). Despite correlations between these course characteristics, each one is a significant predictor of developing country enrollment in a combined ANOVA (each p < 0.025).
Finally, we find that STEM courses and courses demanding more weekly effort attract younger audiences on average (Table 4). The negative association between age and weekly effort is significant for STEM courses (β = −0.676, t74 = −2.43, p = 0.017), but attenuated for non-STEM courses (β = −0.444, t59 = −1.36, p = 0.179). Overall, 30% of variation in average enrollment age can be explained by just two course characteristics: STEM subject and weekly effort (multiple regression R2 = 0.303 with platform fixed effect; F4,134 = 10.4, p < 0.001; each p < 0.015).
Discussion
Efforts to increase the participation of historically underrepresented populations in educational programs, such as women in STEM fields, have been underway for decades [28]. Numerous studies have documented how concerns about anticipated belonging and a “chilly climate” can cause people to opt for different courses and academic pathways [16, 22, 23, 47]. Much prior work has studied how manipulating a particular feature in an educational environment, such as classroom decorations [22], affects students’ attitudes towards the environment. More recent work has extended this paradigm to investigate the behavioral consequences of specific course features in the field, such as how a diversity statement or social cues in course videos affect online course participation [13, 15, 17, 48]. These research efforts have produced causal evidence that visual and verbal cues can influence student attitudes and behavior, but they have skipped over answering a broader research question with significant implications: how are commonly used features of course pages related to course participation patterns?
To answer this question, we conducted a comprehensive and systematic investigation of how course features are related to sociodemographic variation in enrollment. We combined official enrollment records for 159 courses across 20 institutions from two major course platforms with expert coding of course features based on a taxonomy of psychologically inclusive design [12]. Unlike prior work in this area, we consider variation in four population characteristics—age, gender, educational attainment, national development context—to gain a holistic view of enrollment patterns. We also consider a much larger number of features simultaneously to distinguish ones that matter from those that do not. Our results confirm some theory-based hypotheses about important features but not others, and highlight the relative importance of currently underappreciated features. We find that 57% of variability in female enrollment across courses is explained by four basic course features: course subject, instructor gender, and signals of academic rigor. We now discuss how our behavioral findings contribute large-scale empirical evidence to psychological theories of identity and motivation, inspire new research directions, and provide practical recommendations for policy-makers.
The role of the course subject
The subject area of a course stands out as a key predictor of who enrolls. We find strong and robust evidence that STEM subjects enroll more men, younger people, and more people from developing countries. We can compare this result to national studies that have documented disparities in who participates and persists in STEM fields in the context of traditional educational institutions, such as in undergraduate and postgraduate programs [28]. The gender gap we find is consistent with prior expectation and evidence. Although the low barrier to entry for these free online courses has been expected to lower sociocultural hurdles to participation, we still find women to be underrepresented in STEM courses. Improving female enrollment in STEM courses has been the subject of some experimental research in online courses with mixed success [12, 13, 15, 17]. In light of persistent stereotypes about STEM and identity-based motivations for course selection [33, 49], our results suggest that how a course topic is labeled can have profound consequences for female enrollment. Whether presenting technical courses in a way that reduces their association with STEM could improve female participation warrants further investigation.
The appeal of STEM courses among younger generations and people in developing countries may reflect the increased demand for STEM education in developing economies including China, India, and Russia that are facing shortages of qualified STEM instructors [6, 50]. This might lead more students and young professionals in developing regions to seek out STEM learning opportunities in particular.
The role of instructor characteristics
Economists of education and social psychologists have argued that students benefit from exposure to in-group instructors, which is partly because they can act as role models to students and influence their judgements of anticipated belonging and success in the environment [29–31]. The present findings extend this insight by showing that female students are also more likely to enroll in courses with more female instructors, no matter if the course is about a STEM subject or not. Although it seems natural to focus on women when the average gender ratio in enrollments is skewed towards men, our correlational finding can also be interpreted to show that fewer men enroll in courses with more female instructors. Whether instructor gender influences either women’s or men’s or both their decisions to enroll warrants further examination. A study that manipulated the salience of instructor gender at the start of a STEM course found that women were exclusively impacted by the intervention [13], which suggests that a focus on female enrollment rates is warranted.
Instructor gender could matter primarily as a visual cue but also as a verbal cue when students infer gender by name. While all of the courses in the edX sample showed photos of instructors, 18.5% of the courses in the Stanford sample did not show instructor photos, but their gender was plausibly evident from their names. The strength of the association between instructor gender and female enrollment is similarly strong on course pages with and without instructor photos. The correlations presented in Table 2 account for courses with instructor photos, following our pre-registration. Of the ten courses without photos, seven have all male instructors and three both male and female instructors. Imputing these values yields similar correlations. Female: r = .515 (p < 0.001); Developing Country: r = −.382 (p = 0.004); Not College Educated: r = .078 (p = 0.57); Average Age: r = .078 (p = 0.58). Fig 3 shows estimates with instructor gender imputed. This finding that the correlation between instructor gender and female enrollment does not depend on the presence of photos suggests that even in the absence of a visual gender cue, people actively infer gender by name to use in their enrollment decision-making process. Policy-makers who seek to improve female participation in online courses should consider featuring more women in the instructional team on the course page to influence judgements of anticipated belonging. It may not be necessary for them to be course instructors; featuring female teaching assistants or instructional designers, which some courses already do, may also produce the desired outcome.
Prior work in higher education has argued that the benefits of identity-matched instructors also extend to underrepresented ethno-racial minorities [30, 31]. In the absence of information about ethno-racial identity for instructors or students, we instead focus on a dimension of national identity. Building on research in the context of hiring decisions that revealed patterns of discrimination against foreign-born job applicants [51], we expected a higher rate of enrollment from developing countries in courses with instructors of color. However, we did not find robust evidence that this course feature predicts any variation in who enrolls. This may be due to the coarse operationalization of instructors of color by classifying photos as white versus non-white, and of student identity by a national indicator of development.
The role of signals of academic rigor
A surprising finding is the importance of both course prerequisites and the expected weekly effort for female enrollment and enrollment from developing countries. These two course features can be interpreted as signals of rigor that shape judgements of anticipated success in the course, even though they are notably subjective. Unlike in a traditional college class, the number of hours someone spends on the learning materials will vary substantially with their prior knowledge and experience, motivation, and learning strategies. Likewise, course prerequisites are neither enforced nor presented in a standardized manner; it is left up to instructors to decide whether to list any prerequisites. The subjective nature of these features provides instructors and platform providers with a strategic lever considering these features’ strong association with female and international enrollment. How much this lever affects enrollment and whether it encourages underprepared students to enroll warrants careful investigation. This study focuses on participation rates, not success rates, but changes in course composition may affect success rates in the aggregate.
Based on prior evidence that women have a stronger preference to meet stated qualifications compared to men [42], we hypothesized that women would be less inclined to enroll in courses with prerequisites. The same argument can be applied to the stated amount of expected weekly effort. Women might give stronger consideration than men to whether they can actually fit the stated number of hours into their weekly schedule before deciding to enroll. Notably, the association between these signals of rigor and female enrollment are observed in STEM and non-STEM courses alike. To what extent lowering the stated number of hours and de-emphasizing course prerequisites to influence judgements of anticipated success can raise female enrollment (and/or lower male enrollment) warrants further research.
The presence of prerequisites and higher weekly effort is equally predictive of enrollment from developing countries. People may interpret them as signals of rigor and quality of instruction, especially in contexts like China, India, and Russia where qualified STEM instructors are in high demand [50]. Moreover, a higher stated amount of weekly effort and course titles indicating an introductory level are features that also attract younger people, who may have more time as well as workforce-related incentives to spend on learning new topics.
Another key signal of rigor is tied to the institution that offers a course. The most commonly used measure of an institution’s prestige and rigor is its world ranking. The edX sample ranges from institutions ranked worldwide in the top 20 to ones ranked below 200, and yet we do not find any association between rank and enrollment patterns. Instead, we find the expected associations with enrollment patterns when considering the level of national development for the institution’s location. Although most of the courses are offered by universities in highly developed countries, the sample includes institutions ranging from a Human Development Index of 0.72 to 0.92. The significant albeit modest associations we observe between this measure and all four enrollment patterns suggest that potential learners are attentive to which courses are offered by institutions in wealthy countries. In particular, wealthier institutions attract higher enrollment from developing countries but lower female enrollment, mirroring the pattern for other signals of rigor.
The role of linguistic style
Linguistic style of the text presented on course pages, especially the course description, was expected to matter for enrollment patterns. We encoded linguistic style using both sentiment and tone analysis. We find mixed evidence for their role in enrollment decisions. Our hypothesis that course descriptions written in a tone that is more appealing to women will encourage more women to enroll is supported in the edX sample but not the Stanford sample. This may be because there is more variation in tone in the edX sample, which includes courses from twenty different institutions. However, although variation in tone is actually lower in the Stanford sample, the same is true of the overall appeal score, which still significantly correlates with female enrollment in the Stanford sample but not in the edX sample. Findings of the sentiment analysis are mixed as well and course description sentiment is largely unrelated to enrollment patters, except for a mixed set of significant associations with the degree of sadness.
Although we used state-of-the-art linguistic models, this type of linguistic analysis is limited by the fact that these models tend to be proprietary and trained on datasets that do not resemble course descriptions. For platforms like edX with almost 3,000 unique courses, it may be feasible to train linguistic models using course descriptions to predict enrollment patterns and suggest edits to course staff for how to make descriptions appeal to different demographics. Insights from such a model could transfer to traditional college courses for which instructors write course descriptions with minimal guidance on how to make courses appeal to different groups of students.
Conclusion
This research offers strong evidence that participation from historically underrepresented groups in online courses is related to the way that courses are presented. Judgements about anticipated success and belonging in the learning environment are shaped by particular course features and inform people’s decisions to enroll or not. Much is at stake given that online courses already provide 110 million people worldwide with access to post-secondary education and professional development [52]. Course platforms and instructional teams that are looking for ways to increase enrollments without perpetuating existing participation gaps will find the specific recommendations of this work valuable. The contribution of this large-scale field study is to advance our understanding of the relationship between numerous common course features and four sociodemographic enrollment characteristics. The real-world behavioral insights into which features are in fact important for enrollment decisions and which ones are not should inform the direction of future research efforts. By virtue of the study’s scale and rigorous design, including pre-registration and replication, our correlational findings provide a strong basis for studies that test the causal effects of adapting course features that are identified as important.
The evidence supports a psychological account that early cues inform judgements of anticipated belonging and success in a learning environment. Specifically, course features related to gender identity such as the instructor gender and whether it is a STEM course are known to influence perceptions of anticipated belonging [22, 32]; we show that they are also strong predictors of female enrollment. Likewise, course features related to academic rigor such as the expected effort and prerequisites are known to influence perceptions of anticipated success [42]; we show that they are strong predictors of female and international enrollment as well. Given that these course features can act as deterrents to enrollment in courses that people can join for free without prior commitment, their consequences may be amplified in educational programs that require an upfront payment to enter the course [53]. We conclude that strategic changes to course features and cover pages promise to offer a simple and cost-effective approach for improving diversity and inclusion in online education.
Supporting information
S1 Table. Empirical distributions of primary, secondary, and exploratory features coded in the Stanford sample.
Mean , standard deviation σ, and five-number summary q.
https://doi.org/10.1371/journal.pone.0239766.s001
(PDF)
Acknowledgments
We thank Stanford University’s office of the Vice Provost for Teaching and Learning and edX’s Research Data Exchange program for making de-identified course enrollment data available to us for this study.
References
- 1.
Seaman JE, Allen IE, Seaman J. Grade Increase: Tracking Distance Education in the United States. Babson Survey Research Group. 2018.
- 2.
Ginder SA, Kelly-Reid JE, Mann FB. Enrollment and Employees in Postsecondary Institutions, Fall 2016; and Financial Statistics and Academic Libraries, Fiscal Year 2016: First Look (Provisional Data). NCES 2018-002. National Center for Education Statistics. 2017.
- 3.
Deming D, Goldin C, Katz L, Yuchtman N. Can online learning bend the cost curve of higher education. In: American Economic Review; 2014.
- 4. Waldrop MM. Online learning: Campus 2.0. Nature. 2013;495(7440):160–163.
- 5. Escueta M, Quan V, Nickow AJ, Oreopoulos P. Education Technology: An Evidence-Based Review. 2017; p. 102.
- 6. Chirikov I, Semenova T, Maloshonok N, Bettinger E, Kizilcec RF. Online education platforms scale college STEM instruction with equivalent learning outcomes at lower cost. Science Advances. 2020;6(15):eaay5324.
- 7. Hansen JD, Reich J. Democratizing education? Examining access and usage patterns in massive open online courses. Science (New York, NY). 2015;350(6265):1245–1248.
- 8. Kizilcec RF, Saltarelli AJ, Reich J, Cohen GL. Closing global achievement gaps in MOOCs. Science (New York, NY). 2017;355(6322):251–252.
- 9.
Kizilcec RF, Halawa S. Attrition and Achievement Gaps in Online Learning. In: Proceedings of the Second ACM Conference on Learning@Scale. ACM; 2015. p. 57–66.
- 10. Bettinger EP, Fox L, Loeb S, Taylor ES. Virtual classrooms: How online college courses affect student success. American Economic Review. 2017;107(9):2855–75.
- 11. Jaggars SS; AACU. Democratization of Education for Whom? Online Learning and Educational Equity. Diversity & Democracy. 2014;17.
- 12.
Kizilcec RF, Saltarelli AJ. Psychologically Inclusive Design. In: Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM; 2019.
- 13.
Kizilcec R, Saltarelli A, Bonfert-Taylor P, Goudzwaard M, Hamonic E, Sharrock R. Welcome to the Course: Early Social Cues Influence Women’s Persistence in Computer Science. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Honolulu, HI, USA: ACM; 2020.
- 14.
Metaxa-Kakavouli D, Wang K, Landay JA, Hancock J. Gender-Inclusive Design: Sense of Belonging and Bias in Web Interfaces. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM; 2018. p. 614.
- 15.
Brooks C, Gardner J, Chen K. How Gender Cues in Educational Video Impact Participation and Retention. In: Proceedings of the 13th Intl. Conference of the Learning Sciences; 2018. p. 1835–1842.
- 16. Cheryan S, Meltzoff AN, Kim S. Classrooms matter: The design of virtual classrooms influences gender disparities in computer science classes. Computers & Education. 2011;57(2):1825–1835.
- 17.
Kizilcec R, Saltarelli A. Can a diversity statement increase diversity in MOOCs? In: Proceedings of the ACM Conference on Learning at Scale. ACM; 2019.
- 18.
Chaturapruek S, Dee T, Johari R, Kizilcec R, Stevens M. How a data-driven course planning tool affects college students’ GPA: evidence from two field experiments. In: Proceedings of the ACM Conference on Learning at Scale. ACM; 2018.
- 19. Davies PG, Spencer SJ, Quinn DM, Gerhardstein R. Consuming Images: How Television Commercials that Elicit Stereotype Threat Can Restrain Women Academically and Professionally. Personality and Social Psychology Bulletin. 2002;28(12):1615–1628.
- 20. Davies PG, Spencer SJ, Steele CM. Clearing the air: identity safety moderates the effects of stereotype threat on women’s leadership aspirations. Journal of personality and social psychology. 2005;88(2):276–287.
- 21. Murphy MC, Steele CM, Gross JJ. Signaling Threat. Psychological science. 2007;18(10):879–885.
- 22. Cheryan S, Plaut VC, Davies PG, Steele CM. Ambient belonging: how stereotypical cues impact gender participation in computer science. Journal of personality and social psychology. 2009;97(6):1045–1060.
- 23. Master A, Cheryan S, Meltzoff AN, Master A, Cheryan S, Meltzoff AN. Computing Whether She Belongs: Stereotypes Undermine Girls’ Interest and Sense of Belonging in Computer Science. Journal of Educational Psychology. 2015;107(3).
- 24.
Chuang I, Ho A. HarvardX and MITx: Four years of open online courses–fall 2012-summer 2016. 2016;.
- 25. Dillahunt TR, Wang BZ, Teasley S. Democratizing higher education: Exploring MOOC use among those who cannot afford a formal education. The International Review of Research in Open and Distributed Learning. 2014;15(5).
- 26. Reich J, Ruipérez-Valiente JA. The MOOC pivot. Science. 2019;363(6423):130–131.
- 27. Hansen JD, Reich J. Democratizing education? Examining access and usage patterns in massive open online courses. Science. 2015;350(6265):1245–1248.
- 28.
National Science Foundation (NSF) and National Center for Science and Engineering Statistics (NCSES). Women, Minorities, and Persons With Disabilities in Science and Engineering; 2019.
- 29. Lockwood P, Kunda Z. Superstars and me: Predicting the impact of role models on the self. Journal of personality and social psychology. 1997;73(1):91.
- 30. Bettinger EP, Long BT. Do faculty serve as role models? The impact of instructor gender on female students. American Economic Review. 2005;95(2):152–157.
- 31. Price J. The effect of instructor race and gender on student persistence in STEM fields. Economics of Education Review. 2010;29(6):901–910.
- 32. Murphy MC, Steele CM, Gross JJ. Signaling threat: How situational cues affect women in math, science, and engineering settings. Psychological science. 2007;18(10):879–885.
- 33. Herrmann SD, Adelman RM, Bodford JE, Graudejus O, Okun MA, Kwan VS. The effects of a female role model on academic performance and persistence of women in STEM courses. Basic and Applied Social Psychology. 2016;38(5):258–268.
- 34.
Walton GM, Brady ST. The many questions of belonging. In: Elliot AJ, Dweck CS, Yeager DS, editors. Handbook of Competence and Motivation (2nd Edition): Theory and Application. New York, NY, USA: Guilford Press; 2017. p. 272–293.
- 35. Gaucher D, Friesen J, Kay AC. Evidence that gendered wording in job advertisements exists and sustains gender inequality. Journal of personality and social psychology. 2011;101(1):109.
- 36. Bem SL, Bem DJ. Does Sex-biased Job Advertising “Aid and Abet” Sex Discrimination? 1. Journal of Applied Social Psychology. 1973;3(1):6–18.
- 37. Meyers-Levy J, Loken B. Revisiting gender differences: What we know and what lies ahead. Journal of Consumer Psychology. 2015;25(1):129–149.
- 38. Sun B, Mao H, Yin C. Male and Female Users’ Differences in Online Technology Community Based on Text Mining. Frontiers in Psychology. 2020;11.
- 39. Haidt J. The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological review. 2001;108(4):814.
- 40. Plant EA, Hyde JS, Keltner D, Devine PG. The gender stereotyping of emotions. Psychology of Women Quarterly. 2000;24(1):81–92.
- 41. Apfelbaum EP, Sommers SR, Norton MI. Seeing race and seeming racist? Evaluating strategic colorblindness in social interaction. Journal of personality and social psychology. 2008;95(4):918.
- 42.
Mohr TS. Why Women Don’t Apply for Jobs Unless They’re 100% Qualified; 2014. Available from: https://hbr.org/2014/08/why-women-dont-apply-for-jobs-unless-theyre-100-qualified.
- 43. Steele CM, Spencer SJ, Aronson J. Contending with group image: The psychology of stereotype and identity threat. Advances in Experimental Social Psychology. 2002;34:379–440.
- 44.
Kizilcec RF, Davis GM, Cohen GL. Towards equal opportunities in MOOCs: affirmation reduces gender & social-class achievement gaps in China. In: Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. ACM; 2017. p. 121–130.
- 45.
Mullainathan S, Shafir E. Scarcity: Why having too little means so much. Macmillan; 2013.
- 46.
About edX; 2019. Available from: https://www.edx.org/about-us.
- 47. Walton GM, Logel C, Peach JM, Spencer SJ, Zanna MP. Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering. Journal of Educational Psychology. 2015;107(2):468–485.
- 48. Kizilcec RF, Bailenson JN, Gomez CJ. The Instructor’s Face in Video Instruction: Evidence From Two Large-Scale Field Studies. Journal of Educational Psychology. 2015;107(3):724–739.
- 49.
Oyserman D. Pathways to success through identity-based motivation. Oxford University Press, USA; 2015.
- 50.
Marginson S, Tytler R, Freeman B, Roberts K. STEM: country comparisons: international comparisons of science, technology, engineering and mathematics (STEM) education. Final report. 2013;.
- 51. Oreopoulos P. Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes. American Economic Journal: Economic Policy. 2011;3(4):148–71.
- 52.
Shah D. Online Degrees Slowdown: A Review of MOOC Stats and Trends in 2019; 2019. Available from: https://www.classcentral.com/report/moocs-stats-and-trends-2019/.
- 53. Heller DE. Student price response in higher education: An update to Leslie and Brinkman. The Journal of Higher Education. 1997;68(6):624–659.