2.1 Tinto’s theory of drop out

Tinto’s Theory of Drop Out describes a students intent to drop out based on the interplay of two quantities: 1) a student’s commitment to education, and 2) a student’s commitment to a specific institution. A student’s commitment to education is mediated by the initial state of a student entering the university and the dynamics that occur while the student attends university. These dynamics are dictated by a student’s participation and acceptance into the social and academic communities that are at a university. A student’s commitment to an institution is tempered by many factors such as the educational goals available at an institution (e.g., a technical university degree offerings versus a liberal arts university), family commitment to a university, and social acceptance at the university. While Tinto’s theory explicitly attempts to describe why students drop out from college, it is not uncommon that this theory is used to study student graduation from college given that graduation is an alternative outcome to dropping out [1, 9, 12]. In this paper we follow this trend to extend beyond the original goal of Tinto’s drop out model by using it as a framework to predict the time it takes a student to graduate. In this paper we focus on a student’s commitment to education as the framing for features that predict when a student will graduate if they do so.

It is common to characterize Tinto’s theory of dropout as a discrete time hazard problem [1, 8, 9, 12, 13]. Tinto’s theory posits that a student’s educational commitment changes over time as they work toward’s a degree and that effects, such as high school GPA, that may be strong in the beginning of a degree program are weak toward’s the end of a degree program. Discrete time hazard modeling provides a systematic method for examining the various effects on the probability to graduate over time [11]. To that end, discrete time hazard modeling has been used to examine the dynamic impact of various effects that Tinto predicts effect a student’s educational commitment. College GPA has been demonstrated to have a profound but diminishing time varying effect on graduation [1, 8, 14]. Financial aid and money spent on student services has also demonstrated a time varying (diminishing) effect on the probability to graduate [1, 14]. First generation student’s are considered high risk for drop out but this effect diminishes over time indicating that first generation students need specific resources other students may not [9]. Non-traditional enrollment factors such as delaying enrollment, working while enrolled, and stopping education for some period of time can all have a negative time varying effect on graduation [12]. In each case these studies showed a statistically significant time varying impact that supports the dynamic claims of Tinto’s theory.

A student’s preparation has long been known to impact a student’s ability to graduate. Preparation is assessed in many ways and can be represented as the experiences a student has in school and also their present innate ability to perform a specific function. Math preparation correlates with both performance in university and graduating [15, 16]. The same is true for physics and english preparation [17–19]. High school GPA and SAT scores typically account for some but not all of student success at university (GPA, etc.) [20–22]. Preparation for university often can be experienced differentially as well. Women in STEM (Science, Technology, Engineering, and Mathematics) degree programs report having less access to laboratory experiences in high school, are encouraged towards science by their father’s differently, and overall have a different preparation than their male counterparts [19, 23].

A student’s demographics can include the student’s gender, race, the financial support they can expect from their family, and if a student is the first in their family to attend college. Race has long been shown to be a factor in whether a student graduates or not. Black students are less likely to graduate than fellow white students when adjusted for socio-economic status and academic ability and they are more likely to have unwelcoming experiences in STEM programs while at university [1, 23, 24]. Female students are also likely to have unwelcoming experiences that cause them to switch from STEM programs [23]. However they have been shown to be equally likely to graduate or more likely to graduate in comparison to their male counterparts [23, 24]. The financial support both in terms of loans and scholarships and the socio-economic status of a student’s family have long been known to be a factor in university graduation with students who have more financial support typically being more likely to graduate [1, 24–27]. First generation students are less likely to participate in extra-curricular activities at university [28, 29] and are more likely to dropout even when adjusting for race, family income, gender, and preparation [9]. Ultimately, American students from different backgrounds often see different success rates at university due to the different experience they have at the university due to their race, gender, or socio-economic status.

2.2 Gradient boosting

In this paper we use a novel method known as “gradient boosting”. Gradient boosting can solve a large number of statistical modeling problems including logistic problems as found in this paper. Logistic problems are a group of statistical problems that assume that a sigmoidal function made from parameters θ and data x (i.e., ) approximates the probability P(Y|X). In plain language, the model uses input data X to determine how likely the outcome, Y, is to be true. In the context of this paper that would mean the input data X is used to predict whether a student will graduate in the following semester. In education research, the maximum likelihood method has commonly been used to find the solutions to logistic problems. Maximum likelihood solves the logistic regression problem by picking parameters θ that maximize the log likelihood function [30].

Since there is no closed form solution to this likelihood equation the solution is found iteratively via some optimization algorithm [30]. Maximum likelihood is the typical default solution for logistic problems in most statistical packages such as statsmodels [31] in python and glm in R [32].

Gradient boosting produces the solution to the logistic problem in a different way. The log-likelihood is maximized (or, in machine learning terms, the negative log-likelihood is minimized) in small steps: at each step, called boosting iteration, the gradient of the log-likelihood is computed to identify the best direction for the maximization, and it is then fed to a base learner, through which the model is updated. Different choices of base learners (in this paper we will use trees) provide different solutions, guaranteeing maximum flexibility [33]. An early stop, through a tuning parameter which controls the number of iterations, prevents overfitting and provide a better bias-variance trade-off [34]. If the base learner is weak enough in comparison to the signal-to-noise, it can be shown that, at least for a continuous response, boosted models outperform their unboosted versions in term of MSE [35]. Several modifications of the boosting algorithm have been proposed, including stochastic gradient boosting, that uses column and row subsampling to further avoid overfitting and even better deal with the bias-variance trade-off issue [36].

Gradient boosting has been demonstrated to produce better fit models in comparison to traditional methods across a number of domains both in binary classification and in hazard modeling [37–39]. Boosted logistic models have been demonstrated to be more effective at fitting data than models that use maximum likelihood estimation [40]. The structure of educational data can be highly complex often containing many sub groups that have different effects [41]. It is likely that gradient boosting can provide better parameter estimation for education research questions as well.

In this paper we use the Extreme Gradient Boosting (xgboost) algorithm [10]. Xgboost implements stochastic gradient boosting [42] with column and row-wise subsampling, regularization, decision tree base learners with a custom tree split finding algorithm, and an in-model data imputation system for missing data. The specifics of xgboost are explained in Section 4.

4.1 Discrete time hazard models

Discrete time hazard models are, essentially, logistic regression (or other classification) models calculated per time step for time changing data. They control for time to event predictions when the times to events are simultaneous or explicitly discrete and there is new data being collected over the course of the study period [11]. Traditional hazard analysis such as Cox regression is not capable of doing this type of analysis due to the duration being measured being a common value amongst many students [11]. This is due to the likelihood function of a continuous event duration model assumes independent and unique durations (i.e., the time it takes to graduate). When these durations are not unique, they can lead to overfitting. In this study the time unit used is a semester [1]. Thus the models attempt to predict whether a student will graduate in the immediate following semester. If a student drops out from the university, they are not included in following semester data set.

4.2 Logistic regression

The logistic regression equation used in this paper is as follows: (1) Where y(t) is the likelihood to graduate in the following semester and β_S and X_S are static features indicated by subtext S (Fig 1). The dynamic terms (indicated by subtext D) are calculated per semester t from semester 5 until semester 16 [1]. ϵ is then the irreducible random error. This is effectively an iteratively calculated multinomial logistic regression model [1]. The model is fit using the maximum likelihood estimation method. No regularization is used.

The logistic regression model is built using the statsmodels library in python [31].

4.3 Gradient boosted trees

Xgboost is an implementation of a stochastic gradient boosting machine [10, 42, 49] that can also be used to attempt to solve the logistic equation. Gradient boosted machines can be seen as models that are iteratively fit on the current residuals starting from the null model. The additive collection of these models produce the output. Xgboost uses decision trees as its base learners.

The gradient boosted model is thus: (2) Where m is the iteration index, h_m(X) is the previous iterations residual model and F_m(X) is the current iteration’s model fit to previous iteration’s residuals. The total number of iterations is set by M.

A gradient boosted machine can use any learner for the iterative procedure. In this paper we use the decision tree learner as implemented in Xgboost [10]. Decision trees are models that use a tree like structure to fit data and produce regressions and classifications. For each “leaf” node in a tree, a specific decision is made that separates data into two diverging paths. Each node represents a single variable. Categorical variables (e.g., gender) are simply split by the category. Continuous variables (e.g., semester GPA) are split by a decision boundary (e.g., GPA > 3.0). This boundary is typically determined through iteratively fitting the model to find the best boundary.

In Xgboost, each decision tree is trained from a randomly sampled set of rows and columns. Each tree is grown up to a maximum depth using a leaf growing algorithm that estimates whether an additional leaf will produce a better or worse tree [10]. Thus there is no separation between static and dynamic features as in the logistic regression model for each semester. All xgboost models are built using the xgboost library in python [10].

Xgboost has a built in algorithm to deal with missing data [10]. When missing, data is imputed for each tree by selecting the most common decision path for a node. Because a feature may appear in many trees in the model, the decision boundary can be very different per tree in comparison to an mean imputation scheme.

Xgboost models have hyperparameters that define how the model is to be constructed prior to the model being trained. These hyperparameters govern two classes of model design: 1) how the boosting functions, and 2) how the trees are grown and structured. In the case of boosting, the most common hyperparameter used is the number of total boosting learners. For the tree structure this includes the maximum depth a tree can grow to and the fraction of variable and row sub-selection. Typically these hyperparameters are determined by grid search when the total combination of hyperparameters is below 1000 combinations [50, 51].

Because the learners in the xgboost model are decision trees and not linear models, the xgboost model does not report typical coefficient values like in a traditional logistic regression. Instead they report feature “importances” known as “gain”. Each time a variable is used in a tree, the tree is built optimally by splitting in the optimum location. The increase in accuracy due to this split is the gain. The feature importance for a specific variable reported then is the average gain across all the instances that the variable is used in in the model. Xgboost uses a custom split finding algorithm that compares the utility of growing a tree to its maximum depth based on increase in accuracy [10].

4.4 Model evaluation

The goal of this study is to produce a predictive model of when student’s graduate. We use a number of techniques to verify the veracity of the model and increase its accuracy (Fig 2). These include: splitting the data into a training/testing sets, imputing missing data, and picking custom thresholds for the predicted probability of graduation. We also limit the over-estimation of a variable’s impact on when a student graduates by weighting explanation methods with the F₁-score [30]. The following section will explain these techniques in detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Model evaluation flowchart.

Data is split into testing and training sets and is evaluated for two separate models: logistic regression solved by maximum likelihood and gradient boosted trees (xgboost). Missing data for the xgboost model is not imputed from the mean and instead uses the built in imputation engine within xgboost [10].

https://doi.org/10.1371/journal.pone.0242334.g002

The data processing and model evaluation follows the procedure shown in the flowchart in Fig 2. Data for the discrete time hazard model models are:

1. queried from the pathways database [21]
2. Continuous values such as high school GPA and semester GPA are rescaled using the z-score. The means and standard deviations are determined by either the starting year cohort (high school GPA) or the timestamp for the enrolled semester (semester GPA). The median income is scaled using the logarithm.
3. Data is split into training (50%) and testing (50%) sets. Since the discrete time hazard models are evaluated per semester data is split by student and not by semester. Thus a student in training data in semester 5 is also in training data for subsequent semesters.
4. Missing data that is input into the logistic regression model is imputed from the means calculated during the rescaling step. Missing data for the xgboost model is not imputed prior to fitting the model since the xgboost algorithm has an imputation engine built in.
5. Each model is trained and evaluated using the F₁ score. In-sample F₁ scores are used to determine the best threshold for splitting the predicted probability distributions for classification (i.e., does the student gradute in the next semester).
6. Models using the selected threshold are then evaluated using out-of-sample test data to compute the F₁ score.

Several data handling procedures were used to prevent model overfitting, data leakage, and poor predictive performance due to unbalanced class issues. To evaluate the predictive capability of each model, data in this paper is separated into training and test sets for each semester that the discrete time hazard model model is applied. Separating data into partitioned training and testing sets is important to prevent a too optimistic evaluation of the model error [30]. Without having hold out data to evaluate the model performance, we have no way of knowing how a model will predict the outcome of data it has not seen before [30, 46, 52]. Using hold-out data is atypical in recommendations within education research for assessing predictive ability [53, 54] and is not used typically in modern papers using discrete time hazard analysis (e.g., [1]).

Data leakage could occur between two semesters if, for example, a student in semester 6 is in a training set and the same student is in a test set in semester 7 [46]. This is due to some of the features being cumulative thus they would carry forward information from the previous training period into the next test period. Thus, students are identified prior to model training as belonging to the training or testing data set. Students in the test data set are only given to the model to evaluate model predictive performance.

Students do not always have an entry in the database for their high school GPA or their math placement score. This can be due to a variety of reasons such as the high school GPA was not reported, the reported high school GPA was self reported and potentially untrustworthy, and the math placement score was not given to the student because they transferred math credits from another institution. To prevent errors associated with removing data [55], this data is imputed in one of two ways. For the logistic model data is imputed from the mean for the student’s cohort year. For the xgboost model data is imputed within the model. Xgboost will actively impute missing data within each tree learner based on the most likely branch for each decision node (see Section 4 and [10] for more details).

The predictive ability of models in this paper is estimated using the F₁-score and the recall [56]. The F₁-score is the harmonic mean of the precision and recall and is calculated with the following equation: (3) (4) (5)

The F₁ score is over the range 0 to 1 with 1 being perfect performance and 0 being random performance. Precision is defined as the ratio of true positives to the sum of true positives and false positives. That is, it is the ratio of true predicted graduations per semester to the sum of the true predicted graduations and those predicted to graduate but actually do not. Precision is over the range 0 to 1 with 1 being perfect performance and 0 being random performance. Recall is defined as the ratio of true positives to the sum of true positives and false negatives. That is, it is the ration of the number of students predicted to graduate to the total number of people who actually graduated per semester. Recall is over the range 0 to 1 with 1 being perfect performance and 0 being random performance.

We use the F₁-score over other statistics (such as the area under the reciever operating curve (AUC), [30]) because it balances the trade off between high precision (which includes falsely labeling students as graduated) and high recall (which includes falsely labeling students as not graduating). We use the Recall score in the cases of understanding model performance for sub-groups (such as under-represented minorities) because it gives a more direct interpretation of how accurately the model selects the graduating case.

Students are enrolled for many semesters however they only graduate in a single semester. Thus the majority class in each semester is that a student will not graduate in the following semester. This unbalanced class issue can lead to under-fitting of the model specifically such that models simply predict the majority class (in this case not graduating) far too often [57]. We attempted to use an over-sampling technique [58] to address this issue however this did not increase model performance. Instead we choose custom thresholds for the model probability distributions for predicting when a student graduated. The custom thresholds are picked using the F₁ scores calculated from the training data [59]. The best threshold is calculated via a grid-search between a threshold of 0 to 1 using increments of 0.01. This still, typically, does not increase F₁ scores.

Beyond predicting when a student will graduate we are interested in the factors that explain why a student was predicted to graduate (or not graduate). A typical logistic regression model using maximum likelihood produces coefficients that represent the magnitude and direction a particular feature given the other variables. For gradient boosted trees these values are calculated differently. Xgboost uses instead the average gain in prediction accuracy across all instances a feature is used in each tree learner. This is commonly called the “feature importance”. Additionally we can estimate how much a feature, depending on individual values, contributes to the predicted probability of graduating. In this case we use a method called “partial dependence” that was originally developed to use with gradient boosted models [49]. Both the xgboost model and the logistic regression model have a different level of confidence in the prediction per semester (Fig 3). Due to this variable confidence our confidence in feature importances and partial dependences varies as well. In order to account for this confidence we have weighted the feature importances and partial dependences with the out-of-sample F₁ score. This is explained below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Out-of-sample F₁ scores for all models per semester.

Xgboost clearly increases the precision and recall tradeoff for the bulk of semesters. Error bars are the bootstrapped standard deviation of the F₁ score.

https://doi.org/10.1371/journal.pone.0242334.g003

We estimate the effects of model features on prediction using the using the weighted mean of the feature importances. For all 11 semesters the feature importance is weighted by the overall F₁-score for the semester following this equation: (6)

This then allows for the cumulative weighted features in Fig 4 and the weighted features in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Average feature importance for all graduating semesters for the xgboost model.

Feature importances are weighted by the F₁ scores per semester. Enrollment factors and the cumulative average grade are more likely to predict when a student graduates than other factors.

https://doi.org/10.1371/journal.pone.0242334.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Xgboost feature importances for predicting if a student will graduate in the next semester.

Feature importances are weighted by the F₁ score calculated from test data. The row-wise sum of these features would produce Fig 4. By far the most important feature for graduating “on time” (within 8 semesters) is having enough credit hours. Student preparation is important for graduating “early” (<8 semesters). Students who change majors are more likely to graduate later and thus this feature becomes more important in later graduating semesters.

https://doi.org/10.1371/journal.pone.0242334.g005

We also estimate the effects of model features on prediction using the partial dependence [49]. A partial dependence plot represents the average model output for a single variable (S) across the entire feature space (C) in the context of all other features [49]. This means that for a specific feature S, the exact values of S are first, fixed for all rows in the data set for each unique value of S. Then the model is calculated per value. The average contribution of a specific unique value to the overall predicted probability of graduation is then considered the partial dependence. Partial dependence is then assumed to be a continuous function over the entire feature space of S. Partial dependence is estimated using the following equation as: (7)

The partial dependence is calculated for each semester and is weighted by the F₁-score for that semester (Figs 6 and 7) in a similar fashion to Eq 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The partial dependence per semester for the cumulative credit hours a student has obtained.

The stronger the partial dependence, the more contribution that value of cumulative credit hours has on the predicted probability that a student will graduate in the given semester. The partial dependence has been weighted by the per semester F₁ score. Having a total number of credit hours close to 120 is highly predictive of graduating if students graduate between 8 and 10 semesters of enrollment. Outside of this window the impact of the total number of credit hours diminishes. This is likely due to students having additional credit hours that do not count towards a degree such as when they change their major.

https://doi.org/10.1371/journal.pone.0242334.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The partial dependence per semester for the cumulative average GPA a student has obtained.

The stronger the partial dependence, the more contribution that the cumulative average GPA has on the predicted probability that a student will graduate in the given semester. The partial dependence has been weighted by the per semester F₁ score. Having a far above average GPA is a major contribution to graduating if a student graduates between semesters 9 and 10. This is likely due to these students never failing a course and having a high commitment to their chosen major.

https://doi.org/10.1371/journal.pone.0242334.g007

Partial dependence allows the researcher to see the direct contribution to the predicted probability of a given variable value. Because variables often have many values, they can give fluctuating contributions to the predicted probability. In linear models, it is assumed that variables produce linear contributions to the partial dependence [30, 49] and thus coefficients of a linear model represent unit increases [30, 60]. For nonlinear models such as gradient boosted trees, this assumption is relaxed and variables can produce nonlinear contributions to the predicted probability.

5.1 Gradient boosting

The models in this study attempt to predict whether a student will graduate or not during the observation period of being enrolled for 5-16 semesters. The xgboost model is generally more effective than the logistic regression model except in the last semesters (15-16) studied as assessed by the out-of-sample F₁-score (Fig 3). Xgboost is particularly more effective in highly imbalanced case of students graduating (approximately 5% per semester (Fig 8)) within ≤8 semesters (Fig 3). In the more balanced case (semesters 8-14 have approximately 20% of the students eligible to graduate do so (Fig 8)), Xgboost still performs better than the logistic regression model (Fig 3). In the final semesters (semesters 15-16) logistic regression outperforms xgboost slightly.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Students typically graduate within the window of 8 to 12 semesters.

The bars represent the number of students across all cohort years who graduate in the following semester. The green line represents the cumulative fraction of students who have graduated.

https://doi.org/10.1371/journal.pone.0242334.g008

Xgboost is more effective at correctly predicting the graduating semester of female students as assessed by the out-of-sample recall for all semesters except for semester 16 (Fig 9). Xgboost is also more effective at correctly predicting the graduating semester of under represented minority students for all semesters (Fig 9). Additionally, Xgboost is able to handle missing data cases better than the logistic regression model for all semesters (Fig 9). Due to the effectiveness of xgboost over logistic regression, the remaining results will focus on the xgboost model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Recall scores for sub-populations within the data.

The Xgboost model consistently labels women, under represented minorities, and students with missing data better than logistic regression. In Fig 3, there is a large disparity between the logistic regression model and the xgboost model for semesters 5-7. This may be due to the imputation method used with the logistic regression model and the correlation between graduating and missing data during those semesters. However, the gaps in model performance for women, minorities, and missing data in later semesters with no correlation are likely not due to imputation and demonstrate a strong preference for using the xgboost model over the logistic regression model.

https://doi.org/10.1371/journal.pone.0242334.g009

5.2 Effects on time-to-graduation

Tinto’s Theory of Drop Out posits two overarching quantities as governing student choice to drop out or continue to graduation: 1) educational goal commitment, and 2) institutional commitment. Students initially start university with these commitments. These commitments are then mediated by a student’s participating and integration into the academic system and the social system of a given institution. Generally speaking (Fig 10), graduating students take more credit hours per semester (N(grad) = 15.1 ± 0.01, N(no − grad) = 13.17 ± 0.05), enroll for more semesters (N(grad) = 10.57 ± 0.01, N(no − grad) = 8.79 ± 0.05), are more likely to be white or asian (N(grad) = 86.7%, N(no − grad) = 75.3%), and more likely to be female (N(grad) = 54.2%, N(no − grad) = 48.9%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. A comparison of a subset of features for students who graduate versus those who do not based on first year enrolled.

Since 1992 the student population has become more diverse racially, students have had increasing high school GPAs and math placement scores, and the time it takes students who graduate to graduate has been decreasing. Students who graduate typically take more credit hours than those who don’t, typically are better prepared as measured by high school GPA and math placement score, and are less diverse than the total university population.

https://doi.org/10.1371/journal.pone.0242334.g010

In this study, we characterize student’s initial commitments as being mediated via the following quantities: high school GPA, math placement score, the number of AP credit hours the student possesses, the median income of the high school the student attended, gender, and race. Preparation is about half as important in predicting when a student graduates in comparison to enrollment factors (Fig 4). For early graduating students (≤8 semesters), both math placement scores and high school GPAs were important in predicting that a student will graduate (Fig 5). While students are likely to have more AP credits on average if they graduate in 8-10 semesters (AP₈₋₁₀ = 4.06 ± 0.03, AP_other = 2.26 ± 0.01), having AP credits was not an important indicator for predicting graduation during any semester in comparison to other features (Figs 4 and 5). While white and asian students and female students are more likely to graduate (Fig 10), demographics are not major indicators for predicting when a student graduates for any semester (Figs 4 and 5). This is also true for the median incomes of the zipcode the students went to high school.

In this study academic integration is represented by a combination of features including the cumulative credit hours, cumulative mean grade; and the per-semester fraction above or below full-time credit hour, mean grade, non-major GPA. A student’s enrollment and performance are the most predictive features when predicting the semester when a student graduates (Figs 4 and 5). A student’s cumulative credit hours is most important to predicting whether a student graduates “on-time” (after 8 semesters) or not and is overall important for prediction. The cumulative credit hours is less important for prediction the further a student’s credits are away from 120 total credit hours (Fig 6). A student’s average grade is most important for predicting that students will graduate within 10 semesters and is overall important for predicting when a student will graduate (Figs 4 and 5). Having above average grades is important for predicting students who graduate in 9-10 semesters however this is less important for other semesters (Fig 7).

In this study we use per semester major GPA, major credit hours, cumulative number of majors, and whether a student changed major in that semester or not to represent their social integration into the departments and learning communities associated with their chosen degree program. Student’s performance within their major was not as an important indicator for predicting graduation in comparison to overall grade and cumulative credit hours (Fig 4). Changing a major has increased importance for predicting students who graduate later (Fig 5).

5.3 Gradient boosting

The models in this study attempt to predict whether a student will graduate or not during the observation period of being enrolled for 5-16 semesters. The xgboost model is generally more effective than the logistic regression model except in the last semesters (15-16) studied as assessed by the out-of-sample F₁-score (Fig 3). Xgboost is particularly more effective in highly imbalanced case of students graduating (approximately 5% per semester (Fig 8)) within ≤8 semesters (Fig 3). In the more balanced case (semesters 8-14 have approximately 20% of the students eligible to graduate do so (Fig 8)), Xgboost still performs better than the logistic regression model (Fig 3). In the final semesters (semesters 15-16) logistic regression outperforms xgboost slightly.

Xgboost is more effective at correctly predicting the graduating semester of female students as assessed by the out-of-sample recall for all semesters except for semester 16 (Fig 9). Xgboost is also more effective at correctly predicting the graduating semester of under represented minority students for all semesters (Fig 9). Additionally, Xgboost is able to handle missing data cases better than the logistic regression model for all semesters (Fig 9). Due to the effectiveness of xgboost over logistic regression, the remaining results will focus on the xgboost model.

6.1 Why does gradient boosting produce better fitted models than maximum likelihood estimation?

In almost every case, the xgboost model predicts graduation better than the logistic regression model. This can be for several reasons. First, the xgboost model fitting procedure is a slower iterative procedure (1000 iterations) than the maximum likelihood estimation (typically 5-6 iterations) that the logistic regression model uses. Thus, in each iteration the xgboost model can focus on the local neighborhood of feature details within the training data that the logistic regression model may miss [61].

Second, the xgboost model uses a custom imputation engine [10]. Whenever data is missing and an learner is using a feature with missing data, the missing data is imputed to be the most likely choice in the decision tree. Because there are many learners, they can account for different local patterns in the data. The data for the logistic regression model is imputed from the mean of the student’s starting year cohort. This mean imputation likely loses information that the xgboost model is able to attend to. It could be that a more sophisticated imputation model will increase the logistic regression model performance. Typically if data missingness correlates with the outcome variable, then data should be imputed to prevent bias due to the missing data [62]. In our case, data missingness for both high school GPA and math placement score correlates with early graduation rates (see S1 Fig). Thus the logistic regression model performance would likely increase for semester’s 5-7 with a more sophisticated imputation model. However this would not explain the substantial improvements xgboost makes over the logistic model for women and under represented minorities in later semesters.

Third, xgboost penalizes leafs within the tree learners that are fit on few examples from the training data [10]. Additionally, xgboost weights on class labels as well. This is especially useful in the highly unbalanced case of predicting when a student graduates. Because the weight of training data from students who do not graduate is tuned via a grid search, this prevents the xgboost model from overfitting on the majority class simply due to having more representative samples.

6.2 Effects on time-to-graduation

Tinto’s theory suggests that students have an initial level of intent to graduate from an institution upon entry [2, 4]. This intent is the combination of a students family background (e.g., socio-economic status), individual attributes (e.g., academic ability, race), and pre-college experiences (e.g., high school GPA). This intent is then tempered by the student’s at-college experience such as social integration [63], financial support [9], and academic performance.

6.3 Why does gradient boosting produce better fitted models than maximum likelihood estimation?

In almost every case, the xgboost model predicts graduation better than the logistic regression model. This can be for several reasons. First, the xgboost model fitting procedure is a slower iterative procedure (1000 iterations) than the maximum likelihood estimation (typically 5-6 iterations) that the logistic regression model uses. Thus, in each iteration the xgboost model can focus on the local neighborhood of feature details within the training data that the logistic regression model may miss [61].

Second, the xgboost model uses a custom imputation engine [10]. Whenever data is missing and an learner is using a feature with missing data, the missing data is imputed to be the most likely choice in the decision tree. Because there are many learners, they can account for different local patterns in the data. The data for the logistic regression model is imputed from the mean of the student’s starting year cohort. This mean imputation likely loses information that the xgboost model is able to attend to. It could be that a more sophisticated imputation model will increase the logistic regression model performance. Typically if data missingness correlates with the outcome variable, then data should be imputed to prevent bias due to the missing data [62]. In our case, data missingness for both high school GPA and math placement score correlates with early graduation rates (see S1 Fig). Thus the logistic regression model performance would likely increase for semester’s 5-7 with a more sophisticated imputation model. However this would not explain the substantial improvements xgboost makes over the logistic model for women and under represented minorities in later semesters.

Third, xgboost penalizes leafs within the tree learners that are fit on few examples from the training data [10]. Additionally, xgboost weights on class labels as well. This is especially useful in the highly unbalanced case of predicting when a student graduates. Because the weight of training data from students who do not graduate is tuned via a grid search, this prevents the xgboost model from overfitting on the majority class simply due to having more representative samples.

6.4 Limitations on this study

Given that a perfect F₁-score is 1 and in this paper we report F₁-scores around 0.5 at the maximum, it is likely that some of these effects such as financial aid or social integration have large effects on when a student graduates. In this paper we have no direct measurement of social integration. We have used proxy measurements, such as the credit hours a student has accumulated towards their currently registered major program, as data towards the student’s social integration. Ideally this would have increased fittedness in the models as these variables may represent some assessment of social integration. This proved not to be the case. The models still have middling fit values which is likely due to this study having no access to direct measurements of social integration. Social interaction data is difficult to come by without actively observing students through classroom observation, surveys, etc. Future work will leverage new methods created for using passively collected data to assess student social network status as a direct measurement of social integration [64]. Additionally, it is common in some studies, (e.g., [1]) to use individual heterogeneity models or “frailty” models to assess unmeasured contributions to graduation [65]. In our case, we attempted initially to use gamma distributed frailty terms [11, 66]. These proved to provide no increase in model performance thus were removed from subsequent analysis. Future work will consider what contributes to the unobserved heterogeneity (e.g., direct measurements of peer social engagement such as social network centrality).

Additionally, this paper has focused on a student’s educational commitment (per [4]) as opposed to their institutional commitment. Much of this is mediated by the fact that 88% of students in the study do in fact graduate, there may be a subset of students who leave simply due to the fact they become disillusioned with the institution and want to pursue a degree elsewhere. Thus there is likely other factors that are not observed that effects when a student a graduates. Many of these factors, such as life events such as family hardship, are necessarily hard to observe for an entire university population. There are also likely factors that are particular to this university that are not common or not impacting at other institutions. Future work will investigate the effect of student social network belonging and the amount of financial aid available to the students.