Evaluating the three-level approach of the U-smile method for imbalanced binary classification
Fig 2
A step-by-step guide to constructing the BA-RB-I coefficients and the U-smile plot.
Subscripts 0 and 1 denote the non-event and event classes, respectively, superscripts + and – denote better and worse prediction of the new model as compared to the reference model, respectively, δ(ref) and δ are the residuals of the reference and new models, respectively, and n is the number of individuals in the group indicated by sub- and superscripts. BA coefficients: average absolute changes in prediction between new and reference models; RB coefficients: relative changes in prediction between new and reference models (relative to the prediction error of the reference model); I coefficients: proportions of the prediction changes in each class. Step 1. Prediction improvement-worsening (PIW) matrices present the four subclasses resulting from cross-tabulating prediction changes with the true outcome. Better prediction means that the new residuals were smaller than the reference ones, worse prediction means that they were larger than the reference ones. The left PIW matrix shows comparisons of residuals of the new and reference models. The right PIW matrix shows the number of individuals in each subclass. Step 2. The three-level approach of the U-smile method. At level 1, in each of the four subclasses, the magnitude of changes in the predicted probabilities (expressed by model residuals) is transformed into BA and RB coefficients, and the number of individuals is transformed into the I coefficients. At level 2, the net coefficients are determined for each class as differences of the subclass-specific coefficients of level 1, i.e., improvement coefficient less worsening coefficient. At level 3, the weighted overall BA-RB-I coefficients are calculated as weighted means of their respective net coefficients. Step 3. The U-smile plot of the BA-RB-I coefficients. The four subclasses are plotted on the x-axis in a specific order, and the values of the BA and RB coefficients are plotted on the y-axis. The point size is scaled according to the value of the respective I coefficient. The point colour and fill indicate the class and subclass, respectively: points of non-events are blue, points of events are red, points of subclasses with better prediction are solid-filled, and points of subclasses with worse prediction are lighter-filled.