Fig 1.
UpSet diagram for overlapping genes of six commercially available signatures [15].
Horizontal bars represent the number of genes in signature. Vertical bars indicate the number of overlapping genes. The inset graphic illustrates the interpretation for 3 sets: Connected dots indicate exclusive overlapping and an unconnected dot represents exclusive non-overlapping. For example, a single unconnected dot for MammaPrint means that 66 genes are contained only in this signature and are not shared in any combination with other signatures. Two connected dots for PAM50 and OncotypeDx denote that 6 genes are exclusively shared between these signatures. As can be seen, the overlapping is almost non-existing. In this study, we used the gene lists of EndoPredict and OncotypeDx for comparison, since they are the most prominent signatures.
Fig 2.
In the METABRIC cohort, we select 1262 estrogen-receptor positive, Her2-receptor negative patients who did not receive chemotherapy and who either died due to the disease or are still alive. This cohort is divided into training set and test set 1 with 883 (70%) and 379 (30%) patients, respectively. The training set is used to train the Sure Independence Screening algorithm (SIS). SIS selects 15 most important genes—including the Nottingham Prognostic Index (NPI)—that we call Hybrid signature. We select gene lists used by EndoPredict and OncotypeDx (not the corresponding recurrence or risk scores), and refer to them as EndoPredictGL and OncotypeDxGL. We also select the NPI scores alone. For comparison, 15 genes are selected at random (Random signature). To compare the Hybrid signature with a direct negative control, one gene in the Random signature is substituted with the NPI (NPI+Random). Using the training set, all six signatures are then subjected to Cox regressions. In the GSE96058 cohort, we select 1381 estrogen-receptor positive, Her2-receptor negative patients with overall survival who did not receive chemotherapy and who was younger than 80 years. Subsequently, we perform downsampling to ensure the same event-to-patients-at-risk ratio as in the training set and test set 1. We denote this downsampled set as test set 2. Predictions are made on the test set 1 and test set 2. Finally, we statistically compare the predictions and use Decision Trees to assess the survival analysis.
Table 1.
Cox proportional analysis of competing signatures in the METABRIC training set (n = 883).
The 1st column lists gene names used in each signature. NPI: Nottingham Prognostic Index, HR: Hazard Ratio, CI: Confidence Interval.
Fig 3.
Calibration plots for competing signatures.
Observed fraction of survivors (black line) is plotted against the predicted fraction of survivors (blue line) in the test set 1 (METABRIC, n = 379). A perfectly reliable model would show both lines lying on the diagonal (gray line). MAE is the mean absolute error. Q(0.9) is the 0.9 quantile of the MAE, indicating that 90% of errors lie within the interval [0, Q(0.9)].
Fig 4.
Time-dependent area under the curve (AUC) of competing signatures.
(top) Time-dependent AUC of competing signatures for patients in the test set 1 (METABRIC, n = 379). (bottom) Time-dependent AUC of competing signatures for patients in the test set 2 (GSE96058, n = 440). The insets show AUCs within the last 1.5 observation years. In the marginal plots the corresponding boxplots are shown.
Table 2.
Test of differences between IAUCs calculated on AUCs shown in Fig 4.
P-values from the Wilcoxon rank sum test for dependent samples. P-values are computed for the same points in time for the comparison IAUC1 > IAUC2, where IAUC1 denotes the IAUC in a row and IAUC2 denotes the IAUC in a column. For both data sets, rows are sorted w.r.t. the corresponding IAUC values in descending order (For the test set 1: Hybrid > OncotypeDxGL > NPI+Random > …).
Table 3.
Overall performance of competing signatures.
The Signature Skill Score (SSS) was computed using 100 random signatures that were generated additionally and did not contain Random and NPI+Random signatures.
Fig 5.
Survival curves of no chemo patients in the test set 1 (METABRIC) with respect to risk classifications for each signature.
Risk groups were identified by Decision Trees. For each signature, the algorithm found different numbers of risk groups indicated by Node 1, Node 2, etc. P-values were calculated from the two-sided logrank test.
Fig 6.
Survival curves of no chemo patients in the test set 2 (GSE96058) with respect to risk classifications for each signature.
Risk groups were identified by Decision Trees. For each signature, the algorithm found different numbers of risk groups indicated by Node 1, Node 2, etc. P-values were calculated from the two-sided logrank test.
Fig 7.
Survival probabilities for different combinations of groups with the best prognosis.
(top) Test set 1 (METABRIC, n = 379). (bottom) Test set 2 (GSE96058, n = 440). Patients only in the Node 1 risk group shown in Fig 5 and in Fig 6 are selected, i.e. risk groups with the best prognosis. We identify intersections: those patients who are shared between these Node 1 risk groups, and compute the corresponding survival probabilities. “Number of signatures in intersection” means the number of signatures sharing this particular intersection. For example, HEO is shared by 3 signatures, namely by the Hybrid (H), EndoPredictGL (E), and OncotypeDxGL (O) signatures. Survival probabilities are based on the Kaplan-Meier estimates for 10-year disease-free survival for the test set 1 and for the overall survival (provided in GSE96058) for the test set 2. Please note that each plot has different ranges of the number of patients on the x-axis.
Fig 8.
Predicted survival probability for four competing signatures.
(top) 100 patients selected at random in the test set 1 (METABRIC). (bottom) 100 patients selected at random in the test set 2 (GSE96058). A gray line represents a single patient on the x-axis. A perfect agreement between the signatures would show all points completely overlapping. The probabilities are based on the 10-year disease-free survival in the test set 1 and on the overall survival (provided in GSE96058) for the test set 2.