Figure 1.
Comparison of summary statistic results for the MaxAz and Joint sp model selection methods on the auditory oddball data.
Dotted horizontal lines indicate the p<0.01 significance levels. Cross-validated prediction accuracy (Az) results for each of 14 subjects under each of the model selection strategies are provided in A for the without motor network data and C for the whole brain data. For both model selection methods, prediction accuracy is significant at p<0.01 for all subjects. Reproducibility measure (mean probability of selection ) results for each of 14 subjects under each of the model selection strategies is provided in B for the without motor network data and D for the whole brain data. Here, a more drastic difference is noticeable between the two model selection strategies. While the Joint sp method is always above the p<0.01 line, the MaxAz method is significant at p<0.01 for only 7 (without motor network data) and 8 (whole brain data) of the 14 subjects.
Figure 2.
Comparison of reproducibility (mean absolute z-score ) for both model selection methods on the auditory oddball without motor network data A and auditory oddball whole brain data B.
Dotted horizontal lines indicate the p<0.01 significance levels. In both cases, reproducibility increases dramatically under Joint sp for many subjects.
Figure 3.
Reproducibility () vs prediction accuracy (Az) curves for two subjects: A Subject S4 (without motor network), and B Subject S5 (whole brain).
Reproducibility () vs prediction accuracy curves for two subjects: C Subject S4 (without motor network), and D Subject S5 (whole brain). Thick lines indicate the p<0.01 significance thresholds. In each of the figures, the black curve delineates the convex hull of the 1,100 classifiers. Those classifiers that fall in the interior are plotted in gray, while those that lie on the boundary are highlighted. Despite the general trend of a positive correlation between reproducibility and prediction accuracy measures, the MaxAz (red) and Joint sp (magenta) model selection strategies select very different classifiers. In particular, the Joint sp method appears to tradeoff a small reduction in prediction accuracy for a much larger improvement in reproducibility.
Table 1.
Number of significant voxels selected for each of 14 subjects.
Figure 4.
Group-level brain reproducibility maps evaluated on the auditory oddball without motor network data (MNI coordinates: (0,−44,−8), R-L orientation).
Reproducibility was evaluated at the voxel level by testing each voxel's probability of selection or absolute z-score statistic against a null distribution generated by a permutation test. Subject-specific significance masks were created by thresholding at FDR α = 0.05. After transforming to MNI space, masks were summed so that the value at each voxel equals the number of subjects that declare it to be significant. This group mask was then spatially clustered and each cluster reports the total number of subjects that contributed to it. (a) Clusters from the selection probability statistic on the without motor network data; (b) Clusters from the absolute z-score statistic on the without motor network data. Associated regions are listed in Table 2.
Figure 5.
Group-level brain reproducibility maps evaluated on the auditory oddball whole brain data (MNI coordinates: (2,−20,−10), R-L orientation).
Reproducibility was evaluated at the voxel level by testing each voxel's probability of selection or absolute z-score statistic against a null distribution generated by a permutation test. Subject-specific significance masks were created by thresholding at FDR α = 0.05. After transforming to MNI space, masks were summed so that the value at each voxel equals the number of subjects that declare it to be significant. This group mask was then spatially clustered and each cluster reports the total number of subjects that contributed to it. (a) Clusters from the selection probability statistic on the whole brain data; (b) Clusters from the absolute z-score statistic on the whole brain data. The absolute z-score method appears to select a more focal subset. Associated regions are listed in Table 3.
Table 2.
Group-level clusters of significant voxels on the auditory oddball without motor network data.
Table 3.
Group-level clusters of significant voxels on the auditory oddball whole brain data.