Fig 1.
Some possible shapes of the U-smile plot.
The U-smile plot of the BA or RB coefficients represents the effect of adding a new variable to a reference model. Its shape can indicate improved prediction for both classes (double smile – panel a) or either class with no changes in the other (smile + flat line, flat line + smile – panels b and c). It can also show worsened prediction for both classes (double frown - panel d) or either class with no changes in the other (frown + flat line, flat line + frown – panels e and f). A flat line of the entire U-smile plot indicates no change in prediction for both classes (double flat line – panel g), while a zigzag pattern shows improved prediction for one class and worsened prediction for the other (smile + frown, frown + smile – panels h and i).
Fig 2.
A step-by-step guide to constructing the BA-RB-I coefficients and the U-smile plot.
Subscripts 0 and 1 denote the non-event and event classes, respectively, superscripts + and – denote better and worse prediction of the new model as compared to the reference model, respectively, δ(ref) and δ are the residuals of the reference and new models, respectively, and n is the number of individuals in the group indicated by sub- and superscripts. BA coefficients: average absolute changes in prediction between new and reference models; RB coefficients: relative changes in prediction between new and reference models (relative to the prediction error of the reference model); I coefficients: proportions of the prediction changes in each class. Step 1. Prediction improvement-worsening (PIW) matrices present the four subclasses resulting from cross-tabulating prediction changes with the true outcome. Better prediction means that the new residuals were smaller than the reference ones, worse prediction means that they were larger than the reference ones. The left PIW matrix shows comparisons of residuals of the new and reference models. The right PIW matrix shows the number of individuals in each subclass. Step 2. The three-level approach of the U-smile method. At level 1, in each of the four subclasses, the magnitude of changes in the predicted probabilities (expressed by model residuals) is transformed into BA and RB coefficients, and the number of individuals is transformed into the I coefficients. At level 2, the net coefficients are determined for each class as differences of the subclass-specific coefficients of level 1, i.e., improvement coefficient less worsening coefficient. At level 3, the weighted overall BA-RB-I coefficients are calculated as weighted means of their respective net coefficients. Step 3. The U-smile plot of the BA-RB-I coefficients. The four subclasses are plotted on the x-axis in a specific order, and the values of the BA and RB coefficients are plotted on the y-axis. The point size is scaled according to the value of the respective I coefficient. The point colour and fill indicate the class and subclass, respectively: points of non-events are blue, points of events are red, points of subclasses with better prediction are solid-filled, and points of subclasses with worse prediction are lighter-filled.
Table 1.
The confusion matrix shows a cross-tabulation of the actual class with the model’s predicted class (based on the conventional probability threshold of 0.5).
Table 2.
Values of the evaluation measures for the reference model derived from the training and test datasets across imbalance ranging from 1% to 99% of the event class.
Fig 3.
Prediction error of the reference model in each class, quantified by the stratified Brier score (BS), across imbalance ranging from 1% to 99% of the event class.
Panel A shows results for the training and test datasets, while panel B shows the differences in stratified BS between test and training datasets. Points represent mean values from 1000 iterations. Smooth curves were fitted using the local polynomial regression fitting (LOESS) method.
Fig 4.
U-smile plots of the subclass-specific BA-RB-I coefficients across imbalance ranging from 1% to 99% of the event class, for four new models derived from the training dataset.
Two informative variables (ST depression and Str Rnd normal) and two non-informative variables (glucose and Rnd normal) were added to the reference model. The plotted coefficient values represent means from 1000 iterations. The I coefficient is the weighting factor for point size. BA coefficients: average absolute changes in prediction between new and reference models; RB coefficients: relative changes in prediction between new and reference models (relative to the reference prediction error); I coefficients: proportions of individuals with prediction changes in each class.
Fig 5.
U-smile plots of the subclass-specific BA-RB-I coefficients across imbalance ranging from 1% to 99% of the event class, for four new models derived from the test dataset.
Two informative variables (ST depression and Str Rnd normal) and two non-informative variables (glucose and Rnd normal) were added to the reference model. The plotted coefficient values represent means from 1000 iterations. The I coefficient is the weighting factor for point size. BA coefficients: average absolute changes in prediction between new and reference models; RB coefficients: relative changes in prediction between new and reference models (relative to the reference prediction error); I coefficients: proportions of individuals with prediction changes in each class.
Table 3.
Level 2: Values of the class-specific net BA-RB-I coefficients for models derived from the training dataset across imbalance ranging from 1% to 99% of the event class.
Table 4.
Level 3: Values of the weighted overall BA-RB-I coefficients and traditional performance measures for models derived from the training dataset across imbalance ranging from 1% to 99% of the event class.
Fig 6.
Trends in the BA-RB-I coefficients at the three levels of the U-smile method for four new models derived from the training dataset across imbalance ranging from 1% to 99% of the event class.
Two informative variables (ST depression and Str Rnd normal) and two non-informative variables (glucose and Rnd normal) were added to the reference model. Level 1 refers to the subclass-specific coefficients, level 2 to the class-specific net coefficients, and level 3 to the weighted overall coefficients. Smooth curves were fitted using the local polynomial regression fitting (LOESS) method. BA coefficients: average absolute changes in prediction between new and reference models; RB coefficients: relative changes in prediction between new and reference models (relative to the reference prediction error); I coefficients: proportions of individuals with prediction changes in each class.