Fig 1.
Steps to construct the BA and RB coefficients and the U-smile plot.
Step 1: Shorten or lengthen the model residuals (δ). The reference model (subscript (ref)) includes the set of reference predictors, and a candidate predictor is added to the reference model to create a new model. The superscript + denotes better prediction, i.e. the new model shortens the residuals, and superscript − denotes worse prediction, i.e. the new model lengthens the residuals. Step 2: A prediction improvement-worsening (PIW) matrix is a formal cross-tabulation of individuals into four subclasses based on changes in the residual length of the new versus reference model. Step 3: Compared to each other, the residual sums of squares (SS) of the new model and the reference model in each of the four subclasses. Step 4: The U-smile plot. The Y-axis shows the coefficients labelled Coeff—a general abbreviation which, depending on the type of coefficient presented, may be replaced by BA (the size of the absolute average change in residuals), RB (the size of the relative change in residuals), or I (the proportion of individuals with residuals change). The X-axis shows the division into four subclasses: means better prediction for the non-events (dark blue circle),
—worse prediction for the non-events (light blue circle),
—worse prediction for the events (light red circle), and
– better prediction for the events (dark red circle). The connected cilcles form a smile when the magnitude of the prediction improvement (the external dark blue and red cilcles in the plot) is greater than that of the prediction worsening (the inner light blue and red cilcles in the plot).
Table 1.
Range and interpretation of the BA, RB, and I coefficients in the subclasses and the net coefficients for the classes.
Fig 2.
(A) The distance between the cilcles representing the non-event outcome class (blue cilcles) and the event class (red cilcles) on the U-smile plot are the net effect size of prediction improvement offered by a new marker. (B) Examples of possible shapes of the U-smile plot. Prediction improvement: (a) for both outcome classes, (b) only for the non-events, (c) only for the events. Prediction worsening: (d) for both outcome classes, (e) only for the non-events, (f) only for the events. Cilcles lying at an approximately constant level translate into no prediction improvement or worsening compared to the reference model (g). A zigzag indicates prediction improvement in one outcome class and prediction worsening in the other (h and i). As the shape of the plot is more important, any grid or scale is an unnecessary burden of information.
Fig 3.
PIW plot with 4 subclasses stratification.
Cross-tabulating changes in predicted probabilities with outcome class (i.e. non-event and event) across the identity line yields four subclasses of individuals. Subscripts 0 and 1 denote the non-events and the events, respectively. In contrast, according to the identity line, the superscripts (+) denote changes in a favourable direction (shorter residuals and better prediction of the new model) and the superscripts (-) denote changes in an unfavourable direction (longer residuals and worse prediction of the new model). Dark blue points represent the non-events with better prediction, light blue points—the non-events with worse prediction, dark red points—the events with better prediction, and light red points—the events with worse prediction. (A) A complete prediction improvement-worsening (PIW) plot. (B) Residuals of the reference model (δ(ref)) and the new model (δ) of an exemplary point for each outcome subclass.
Table 2.
Parameters of the independent random variables generated from theoretical probability distributions for validating the U-smile method.
Data were generated to simulate a scenario when a non-informative predictor is added to the reference model (Random variables) and when an informative predictor is added to the reference model (Stratified random variables).
Fig 4.
The U-smile plots of the BA, RB and I coefficients for each new model derived from the training dataset under the independent scenario.
Table 3.
Values of the net BA, RB and I coefficients stratified by outcome class for 18 new models derived from the training and test datasets under the independent scenario.
The reference model was expanded by six real predictors from the Heart Disease dataset, 6 non-informative random variables (without stratification by outcome class), and 6 informative random variables (stratified by outcome class).
Table 4.
Comparisons of the reference model with each new model derived from the training dataset under the independent scenario.
The reference model was expanded with six real predictors from the Heart Disease dataset, six non-informative random variables (without stratification by outcome class), and six informative random variables (stratified by outcome class). Shown are the values of the AUC of all new models. ΔAUC shows the difference in AUC relative to the reference model. The AUC of the reference model is 0.758. The DeLong’s test for two correlated ROC curves was used to asses ΔAUC.
Fig 5.
The ROC curves of the reference and new models derived from the training dataset under the independent scenario.
Fig 6.
The U-smile plots of the BA, RB and I coefficients for each new model derived from the test dataset under the independent scenario.
Fig 7.
The prediction improvement-worsening (PIW) plots for each new model derived from the training dataset under the independent scenario.