Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Feature and decision-level fusion for schizophrenia detection based on resting-state fMRI data

  • Ali H. Algumaei ,

    Contributed equally to this work with: Ali H. Algumaei, Rami F. Algunaid

    Roles Formal analysis, Investigation, Methodology, Software, Writing – original draft

    Affiliation Department of Biomedical Engineering and Systems, Faculty of Engineering, Cairo University, Giza, Egypt

  • Rami F. Algunaid ,

    Contributed equally to this work with: Ali H. Algumaei, Rami F. Algunaid

    Roles Formal analysis, Methodology, Software

    Affiliation Department of Biomedical Engineering and Systems, Faculty of Engineering, Cairo University, Giza, Egypt

  • Muhammad A. Rushdi,

    Roles Supervision, Writing – original draft, Writing – review & editing

    Affiliation Department of Biomedical Engineering and Systems, Faculty of Engineering, Cairo University, Giza, Egypt

  • Inas A. Yassine

    Roles Methodology, Supervision, Writing – review & editing

    iyassine@eng.cu.edu.eg

    Affiliation Department of Biomedical Engineering and Systems, Faculty of Engineering, Cairo University, Giza, Egypt

Abstract

Mental disorders, especially schizophrenia, still pose a great challenge for diagnosis in early stages. Recently, computer-aided diagnosis techniques based on resting-state functional magnetic resonance imaging (Rs-fMRI) have been developed to tackle this challenge. In this work, we investigate different decision-level and feature-level fusion schemes for discriminating between schizophrenic and normal subjects. Four types of fMRI features are investigated, namely the regional homogeneity, voxel-mirrored homotopic connectivity, fractional amplitude of low-frequency fluctuations and amplitude of low-frequency fluctuations. Data denoising and preprocessing were first applied, followed by the feature extraction module. Four different feature selection algorithms were applied, and the best discriminative features were selected using the algorithm of feature selection via concave minimization (FSV). Support vector machine classifiers were trained and tested on the COBRE dataset formed of 70 schizophrenic subjects and 70 healthy subjects. The decision-level fusion method outperformed the single-feature-type approaches and achieved a 97.85% accuracy, a 98.33% sensitivity, a 96.83% specificity. Moreover, feature-fusion scheme resulted in a 98.57% accuracy, a 99.71% sensitivity, a 97.66% specificity, and an area under the ROC curve of 0.9984. In general, decision-level and feature-level fusion schemes boosted the performance of schizophrenia detectors based on fMRI features.

1 Introduction

Early diagnosis of mental disorders is considered a challenging task. Schizophrenia is one of these chronic mental disorders that typically appear in late adolescence or early adulthood and affect about 1% of the population around the world [14]. This disorder is characterized by hallucinations, delusions and negative symptoms such as social withdrawal, self neglect, etc. [5].

Within the last decade, most studies have shown that machine learning and pattern classification techniques are useful for finding potential biomarkers for schizophrenia based on different types of discriminative features: structural features [618], functional features [1925] or combination of functional and structural features [12, 2627].

Structural studies reveal the anatomical changes of the brain such as expansion of the lateral ventricles [28], decreased volumes of medial temporal structures such as the amygdala, the parahippocampal gyrus, the hippocampus [2931], the prefrontal cortex [32, 33], the superior temporal gyrus [30], and the inferior parietal lobule [34, 35]. Furthermore, the abnormal enlargement of the left hemisphere compared to the right hemisphere can be considered as a sign of schizophrenia in male patients [36, 37]. Moreover, other studies revealed structural correlations between temporal lobe and prefrontal brain volumes [38, 39], in addition to some disturbances of functional brain connectivity between temporal and frontal lobes. These changes confirm the disconnectivity hypothesis [40, 41], which explains the widespread problems of the functional connectivity in schizophrenia.

Resting-state functional magnetic resonance imaging (Rs-fMRI) is considered as one of the most promising tools in the diagnosis of different mental disorders specially schizophrenia, as it allows exploring the brain functions at rest [24, 42]. Recently, most studies have examined the automatic diagnosis of brain disorders using Rs-fMRI [4351].

Slow activity fluctuations, measured by the blood oxygenation level dependent (BOLD) signal in Rs-fMRI, allow defining the resting-state networks and obtaining the correlated activity between the various brain regions. The functional connectivity [52, 53] maps can be created through calculating the correlation measures of these fluctuations. These maps may be used as biomarkers or distinctive features for mental or individual variations. Tang et al. [54] reached an accuracy of 93.2% using 22 schizophrenic patients and 22 healthy subjects based on functional connectivity. Yu et al. [55] achieved a 62% accuracy for a system trained and tested on the functional connectivity measures of 24 schizophrenic patients, 25 healthy siblings and 22 healthy subjects. The major drawback of the previously stated studies is the employment of the small datasets. Guo et al. [56] achieved an accuracy of 75% using a dataset formed of 46 unaffected siblings of schizophrenic patients and 50 healthy subjects based only on the fractional amplitude of low frequency fluctuations (fALFF). While this study enjoys a relatively large dataset, it still suffers from the limited investigation of suitable features for schizophrenia diagnosis and the relatively low reported accuracy. Chyzhyk et al. [57] employed a classification system based on an Rs-fMRI dataset provided by the Center of Biomedical Research Excellence (COBRE) in mental illness and brain function. Experiments were conducted with four feature types, namely: regional homogeneity (ReHo), voxel-mirrored homotopic connectivity (VMHC), fractional amplitude of low-frequency fluctuations (fALFF) and amplitude of low-frequency fluctuations (ALFF), where the reported accuracy reached 91% using the VMHC features.

Recently, data fusion techniques have been explored for improving the performance of detection and classification systems in numerous applications [58, 59]. Fusion can be generally performed on different levels, especially the feature and decision levels. Such fusion schemes have been particularly applied in biomedical signal analysis [60], and multiple medical imaging modalities [61]. In particular, fusion of brain imaging data has been consistently improving the performance of automated systems for mental illness detection [62].

In this paper, we investigated a schizophrenia detection framework based on the individual ReHo, VMHC, fALFF and ALFF feature types. The resultant feature sets were optimized using different feature selection techniques. For this framework, we explored decision-level fusion between the outputs of classifiers trained on single-feature types. We also investigated feature-level fusion schemes in which pairs, triples, and quadruples of feature types are combined and used for classifier training. In the first single-feature-type scheme, each of the four feature types is exploited independently. The second scheme applies majority-vote fusion on the decision level to three classifiers trained on the ReHo, VMHC, and fALFF features, respectively. The third scheme explores feature-level fusion through combining different types of features before feature selection. The different schemes are shown in Fig 1, where the colored arrows represent the flow between the different modules forming the different schemes. The red arrows trace the flow of the single-feature-type schemes, while the blue arrows are associated with the feature-level fusion schemes. For the decision-level fusion schemes, the single-feature-type schemes are combined through the orange arrows to reach the final decisions.

thumbnail
Fig 1. A general block diagram of the proposed schizophrenia detection schemes.

The single-feature-type schemes follow the red arrows. The feature-fusion schemes are indicated by the blue arrows. For the decision-level fusion schemes, the orange arrows trace the path from the single-feature-type schemes to the final fused decisions.

https://doi.org/10.1371/journal.pone.0265300.g001

The rest of the paper is organized as follows. Section II describes the whole system modules including dataset description, preprocessing, resting-state functional activity measures, feature selection, and pattern classification. Section III reports and discusses the experimental results. Section IV concludes the paper and gives suggestions for future work.

2 Materials and methods

2.1 Dataset description

We employed a dataset created for studying the neural mechanisms of schizophrenia. This dataset was collected by the Center for Biomedical Research Excellence (COBRE) http://fcon1000.projects.nitrc.org/indi/retro/cobre.html through the Mind Research Network for Neurodiagnostic Discovery (MRN) at the University of New Mexico. The dataset was collected in accordance with the recommendations of the Declaration of Helsinki. The COBRE data acquisition protocol was approved by the Institutional Review Boards of all of the participating institutions. Informed written consent was obtained from all participants at each site. The dataset includes raw functional and anatomical MR data for 140 subjects distributed equally for normal and schizophrenic patients. Diagnostic information was collected using the structured clinical interview utilized for DSM disorders (SCID). A multi-echo MPRAGE (MEMPR) sequence was utilized for anatomical imaging with the following parameters: TR/TE/TI = 2530/[1.64, 3.5, 5.36, 7.22, 9.08]/900 ms, flip angle = 7, slab thickness = 176 mm, FOV = 256 × 256 mm, data matrix = 256 × 256 × 176, number of echoes = 5, total scan time = 6 min, voxel size = 1 × 1 × 1 mm, pixel bandwidth = 650 Hz. With 5 echoes, the TI, TR and time to encrypt partitions for MEMPR are similar to those of conventional MPRAGE, and lead to similar GM/WM/CSF contrast. Data for Rs-fMRI was collected using echo planar imaging (EPI) with ramp sampling correction using the intercomissural line (AC-PC) as a reference (TR: 2 s, TE: 29 ms, matrix size: 64 × 64, 32 slices, voxel size: 3 × 3 × 4mm3). Rs-fMRI, anatomical MRI, and phenotypic data are recorded for every subject. A brief summary of the demographic data found in the COBRE schizophrenia dataset is shown in Table 1.

thumbnail
Table 1. Summary of participant demographics in the COBRE schizophrenia dataset.

https://doi.org/10.1371/journal.pone.0265300.t001

2.2 Data preprocessing

All the preprocessing steps were executed in MATLAB using the data processing & analysis for brain imaging (DPABI) software tool [63]. For each participant, slice time correction was applied for interleaved acquisition. Head motion correction based on Friston’s 24-parameter motion model [64] was performed. Co-registration of structural and functional images in order to map the functional information to the anatomical space was executed. Then, the images were spatially normalized to the Montreal Neurological Institute (MNI) standard using the DARTEL template [65] and resampled to 3 × 3 × 3 mm3. The generated images were spatially smoothed with a 4-mm full-width half-maximum (FWHM) Gaussian kernel. Moreover, the images were linearly detrended and temporally filtered by a bandpass filter (0.01–0.1 Hz) to reduce low-frequency drifts and remove physiological high-frequency noise [66].

2.3 Resting-state functional activity measures

We explain here the functional activity measures calculated over the Rs-fMRI dataset.

2.3.1 ALFF and fALFF features.

The ALFF [67] and fALFF [68] features measure the magnitude of low frequency fluctuations (LFFs) of the BOLD signal. The ALFF features are calculated as the average power spectrum, obtained across the range (0.01—0.1 Hz) for each voxel. The fALFF features represent the ratio of the signal power of the low-frequency range (0.01 to 0.1 Hz) to the power associated with the total detectable frequency range (0 to 0.25 Hz) [67].

2.3.2 VMHC features.

This feature set measures the brain functional homotopy through a voxel-wise measure of connectivity among the brain hemispheres, under the assumption of synchrony in spontaneous brain activity among the homotopic regions for each hemisphere. An approximation of the homotopic connectivity is computed between an individual voxel in one of the brain hemispheres and its symmetric counterpart in the other hemisphere of the brain, assuming morphology regularity between them. This connectivity is calculated based on the Pearson correlation coefficient between voxel pairs across the hemispheres. Then, the Pearson correlation coefficient value is converted into a Fisher z-transformed value representing the VMHC activity measures [69].

2.3.3 ReHo features.

This feature set measures the similarity between the time series of a particular voxel and those of its nearest neighbors [70]. To compute the similarities inside a voxel cluster, Kendall’s coefficient of concordance (KCC) is applied. Here, each cluster contains 27 neighboring voxels. For each subject, a voxel-based map is built by standardizing and smoothing the KCC values based on a 4-mm FWHM Gaussian kernel [71].

For each of the above-mentioned four feature types, 271633 features were originally extracted.

2.4 Feature selection

To improve the classification performance, we examined 15 different feature selection measures [72]. Each feature selection algorithm provides a list of features ranked by the feature strength or discriminability from the most discriminating feature to the least discriminating one. Then, we followed a sequential forward selection approach. That is, we use the most significant feature, followed by the most two significant features, where the system has been trained and the performance is evaluated using the validation dataset for each case. This process is continued where the significant feature subset is enlarged by adding one feature at a time. This process was employed for the different feature selection algorithms. We selected the following four top-performing feature selection algorithms: feature selection via concave minimization (FSV) [73], L0-norm [74], Relief [75] and Wilcoxon sum-rank test [76].

2.5 Decision and feature fusion

Several schemes for decision-level and feature-level fusion are investigated to improve the schizophrenia detection performance. For decision-level fusion, we applied majority voting on three SVM classifiers which are based on the ReHo, VMHC, and fALFF feature types, respectively. Feature-level fusion combines different fMRI feature types to exploit the strengths of each type.

2.6 Support vector machine (SVM) classification

A support vector machine (SVM) is a linear classifier that learns the best hyperplane that has the maximum possible distance to the closest data point in the training set belonging to any class using the support vectors. Thus, a SVM is typically more robust compared to other classifiers. The SVM effectiveness is enhanced mainly through the employment of the kernel trick in order to handle data nonlinearity in the feature space. This trick basically transforms the data points into a higher-dimensional feature space in order to increase the linear separability between the data points. The most commonly used kernels include linear, quadratic, and radial basis function (RBF) kernels [77].

After feature selection and fusion, supervised machine learning procedures were used to discriminate the schizophrenic patients from healthy subjects. The COBRE dataset, employed in this work, was divided into training, validation and testing subsets. Training and testing of the classifiers were performed using the LIBSVM http://www.csie.ntu.edu.tw/cjlin/libsvm/ library. SVM, with a linear kernel, was employed for the classification task for all proposed schemes, with soft margin C = 10. Five repetitions of nested loop 10-fold cross-validation were performed, in this study, in order to demonstrate the robustness of the system, where the inner 10-fold cross validation was employed only in the feature selection task, i.e. selecting the optimum number of features. It is worth noting that the inner 10-fold cross-validation was employed only in selecting the optimum number of features, while the SVM kernel and parameters were selected by trial and error.

For each fold in the outer loop, one-tenth of the data is randomly selected for the testing and performance evaluation, while the rest is employed in training and validation in the inner 10-fold cross-validation loop. In this inner loop, one-tenth of the remaining samples is randomly selected for the validation and optimization of the feature selection process, while the rest is employed for classifier training. Accuracy, specificity and sensitivity are used to evaluate the classifier performance, select the hyper parameters and verify the system robustness. The overall system performance was evaluated using the accuracy, sensitivity and specificity, which are computed as follows: (1) (2) (3) where the true positive (TP) is the number of correctly classified schizophrenia patients, the false positive (FP) is the number of healthy subjects incorrectly classified as schizophrenia patients, the true negative (TN) is the number of correctly classified healthy subjects, and the false negative (FN) is the number of schizophrenia patients incorrectly classified as healthy subjects. Moreover, the 5x2 cross-validation statistical test was employed to measure the statistical significance of the difference in accuracy between the classifier based on the ReHo activity measure and the classifiers based on other activity measures [78]. The test is carried out as follows. Let A be the ReHo-based classifier and B be the classifier based on another activity measure. The null hypothesis is that the ReHo-based classifier A has the same accuracy as the other classifier B. The alternative hypothesis is that the two classifiers have different accuracies. For each classifier, five repetitions of 2-fold cross-validation are made. Then, for each pair of classifiers, the differences in accuracy are used to compute the following t-statistic: (4) where

  • is the difference of the classifier scores for the first fold of the first iteration,
  • is the estimated variance of the score difference for the ith iteration (This variance is computed as ),
  • is the difference of the classifier scores for the jth fold of the ith iteration,
  • is the mean score for the ith repetition over the two associated folds.

The t-statistic is assumed to follow a t-distribution with 5 degrees of freedom. We assume a significance level of 0.05. The corresponding threshold is t* = 2.57. For any two classifiers, the null hypothesis is rejected (i.e. the difference in accuracy for the two classifiers is statistically significance) if |t| > t*. Thus, an absolute t-statistic larger than t* indicates that the null hypothesis can be rejected and hence that the ReHo-based classifier accuracy is indeed different from the accuracy of the other classifier.

3 Results

3.1 Classification outcomes

We experimented with 15 feature selection algorithms and reported the top four best performing ones as shown in Table 2.

thumbnail
Table 2. Classification accuracy (%) with feature selection algorithms and feature types.

https://doi.org/10.1371/journal.pone.0265300.t002

Our experimental results show that the FSV method gives the best performance for the COBRE http://fcon1000.projects.nitrc.org/indi/retro/cobre.html dataset. The FSV algorithm lists the features according to their discriminability. In our experiments, we used the best single feature to train and test a SVM classifier and obtain the corresponding average validation accuracy over 10 folds. Then, we used the best two features to train and test a SVM classifier and obtain the corresponding average validation accuracy. At each stage of the experiments, the number of used features was increased until all of the features were used for training and validation. The best number of features corresponding to the classifier with the highest average validation accuracy was selected and employed for testing in the outer loop.

We found 95% confidence intervals for the classifier performance using the Wilson score interval method [79, 80]. Among the four types of activity measures (i.e. ALFF, fALFF, ReHo and VMHC), the best average test accuracy of 94.57% is achieved by using the ReHO activity measure with 83 discriminative features as shown in Table 3.

thumbnail
Table 3. Schizophrenia detection results with single-feature-type classifiers and decision-level classifiers.

Feature selection was applied to each of the four single feature types (ALFF, fALFF, ReHo and VMHC). The decision-level classifier used all feature types except for the ALFF one, which has the worst performance. For comparison, the last column shows the corresponding accuracies obtained by Chyzhyk et al. [57].

https://doi.org/10.1371/journal.pone.0265300.t003

We investigated the fusion of decisions and features based on Rs-fMRI activity in order to improve the classification performance. The decision-fusion scheme, shown in Fig 2, computes the ReHo, VMHC, and fALFF features, classifies each feature type using a SVM, and fuses the three decisions. Then, a majority vote is carried among the three classifiers to get the final decision. Table 3 shows the results of the decision-level fusion scheme of Fig 2. Table 3 shows the sensitivity, specificity, and accuracy measures on the test set for each feature type. Our accuracy based on fALFF (92.71%), in Table 3, is significantly better than the 75% accuracy reported by Guo et al. [56]. Moreover, we achieved better results than those obtained by Chyzhyk et al. [57] based on the 4 features types as shown in Table 3. Furthermore, we investigated feature-level fusion schemes. Table 4 shows schizophrenia detection results with different pairwise, triple, and quadruple combinations of the ALFF, fALFF, ReHo and VMHC feature types. Feature selection was applied after combining the features, and the number of features associated with the minimum validation error was selected for each combination. The pairwise combination of the ALFF and fALFF feature types resulted in the best accuracy (97.71%), specificity (97.80%), and sensitivity 98.80%. The ALFF, fALFF, and VMHC combination achieves the best performance among all triple combinations. As well, feature-level fusion of the four feature types leads clearly to the best overall performance metrics with an accuracy of 98.71% and a sensitivity of 99.71%.

thumbnail
Fig 2. Average accuracy calculation for the decision-level fusion scheme.

https://doi.org/10.1371/journal.pone.0265300.g002

thumbnail
Table 4. Schizophrenia detection results with feature-level fusion of different pairwise, triple, and quadruple combinations of single feature types.

https://doi.org/10.1371/journal.pone.0265300.t004

3.2 Statistical significance testing

In order to test the statistical significance of the performance differences between the classifiers listed in Tables 3 and 4, we computed t-statistics based on Eq (4). The classifier based on the ReHo activity measure was employed as the reference algorithm since it provided the highest accuracy in the case of single-feature-type classifiers. The resulting p-values for all significance tests associated with single-feature-type classifiers and the decision-level classifier are listed in Table 5. On the one hand, the statistical results show that the difference in accuracy between the ReHo-based and VMHC-based classifiers is not statistically significant. On the other hand, the differences in accuracy between the ReHo-based classifier and the fALFF-based classifier, the ALFF-based classifier, and the decision-level fusion classifier are statistically significant. For the feature fusion classifiers, the p-values listed in Table 6 show statistically significant improvements for all classifiers (except for two classifiers: the ALFF-ReHo-VMHC and fALFF-ReHo-VMHC classifiers) over the ReHo-based classifier.

thumbnail
Table 5. P-values for test of significance of the outcomes of the classifiers based on single-feature-type classifiers and decision-level classifier.

Statistical comparison is made with respect to the ReHo-based classifier.

https://doi.org/10.1371/journal.pone.0265300.t005

thumbnail
Table 6. P-values for test of significance of the outcomes of the classifiers based on feature-level fusion of different pairwise, triple, and quadruple combinations of single feature types.

Statistical comparison is made with respect to the ReHo-based classifier.

https://doi.org/10.1371/journal.pone.0265300.t006

3.3 ROC analysis

Receiver operating characteristic (ROC) curves were generated for each of the 10 folds of each experiment. An average ROC curve can be obtained by projecting curves from two-dimensional space onto a single dimension and averaging them traditionally. However, this projection raises questions of appropriateness and conservation of the characteristics of interest. The vertical averaging [81] is employed to plot the ROC curvse, where the FP rates are fixed and the corresponding TP are averaged.

We reported the results of vertical averaging method which achieves the best performance in comparison with the threshold averaging method. Fig 3(a)–3(c) shows the results based on vertical averaging for single, pairwise, triple and quadruple combinations of features, respectively.

thumbnail
Fig 3.

ROC curves for schizophrenia classification using different feature combinations and ROC vertical averaging where the numbers in brackets denote the area under the curve (AUC): (a) Single feature types. (b) Pairwise feature type combinations. (c) Triple and quadruple combinations of features.

https://doi.org/10.1371/journal.pone.0265300.g003

3.4 Discriminative feature mapping

The discriminative ALFF, fALFF, ReHo and VMHC connectivity maps were constructed using a two-sample t-test to statistically verify the significance of the difference between healthy subjects and schizophrenic patients. The significance level was set at the corrected p < 0.05 for multiple testing using the false discovery rate (FDR) method [82] (min z > 2.3, cluster significance: p < 0.05). The most discriminative features for classification are shown in Table 7. Moreover, these regions are highlighted using BrainNet Viewer [83] in Fig 4. The top features are listed in descending order of their weights. The highly activated regions include the lingual gyrus (LING), Precuneus (PCUN), anterior cingulate and paracingulate gyri (ACG), parahippocampal gyrus (PHG), precental gyrus (PreCG), cerebelum_6 (CRBL6), temporal pole: middle temporal gyrus (TPOmid), vermis_10 (VER10), middle frontal gyrus, orbital part (ORBmid), cerebelum_Crus2 (CERcr2), superior parietal gyrus (SPG), anterior cingulate and paracingulate gyri (ACG), cerebelum_7 (CER7) and postcentral gyrus (PoCG).

thumbnail
Fig 4. Localization of the areas in the AAL atlas.

Circles represent AAL nodes. The blue circles represent the areas with higher discriminability between healthy and schizophrenia groups and the green circles represent unaffected areas.

https://doi.org/10.1371/journal.pone.0265300.g004

thumbnail
Table 7. The most discriminative measures and corresponding AAL regions.

https://doi.org/10.1371/journal.pone.0265300.t007

In comparison to healthy controls, the schizophrenic patients showed significant ReHo increases in the right LING and the right PCUN. Also, ALFF increases in the right PHG and the left PreCG, while fALFF increases in VER10 and the left ORBmid. As will, VMHC increases in the left PHG and the left ACG. Schizophrenic patients showed significant ReHo decreases in the left PCUN and the right ACG. Also, ALFF decreases in the right CERcr2 and the right TPOmid, while fALFF decreases in the right CERcr2 and the right SPG. In addition, VMHC decreases in the right CER7 and the left PoCG.

3.5 Robustness to noise

We performed some additional experiments to investigate the robustness of the proposed method with different feature combinations. In particular, we added Rician noise with two different levels of σ = 1 and σ = 2 to the test data. The performance outcomes under these noise conditions are summarized in Tables 8 and 9 for the single-feature-type classifiers and fused-feature classifiers, respectively. Moreover, the performance outcomes under the noise conditions are visualized for the best classifiers with single, pairwise, triple, and quadruple feature combinations as well as decision-fusion classifier in Fig 5.

thumbnail
Fig 5. Schizophrenia detection performance outcomes under the Rician noise conditions for the best classifiers with single, pairwise, triple, and quadruple feature combinations as well as the decision-fusion classifier.

https://doi.org/10.1371/journal.pone.0265300.g005

thumbnail
Table 8. Effects of Rician noise on schizophrenia detection performance with single-feature-type and decision-level fusion schemes for fMRI data contaminated with Rician noise levels of σ = 1 and σ = 2.

https://doi.org/10.1371/journal.pone.0265300.t008

thumbnail
Table 9. Effects of Rician noise on schizophrenia detection performance with feature-level fusion of different pairwise, triple, and quadruple combinations of single feature types for fMRI data contaminated with Rician noise levels of σ = 1 and σ = 2.

https://doi.org/10.1371/journal.pone.0265300.t009

For the best single-feature-type classifier, namely the ReHo-based classifier, the 95.57% detection accuracy dropped by 2.11% and 6.11% with the two noise levels, respectively. For the decision-level fusion classifier, the 97.85% detection accuracy dropped by 2.07% and 9.31% with the two noise levels, respectively. The best pairwise-feature-type classifier, i.e. the classifier based on the ALFF and fALFF features, the 97.71% detection accuracy dropped by 1.85% and 10% with the two noise levels, respectively. For the best triple-feature-type classifier, that is the one based on the ALFF-fALFF-VMHC feature combination, the 97.85% detection accuracy dropped by 2.13% and 9.92% with the two noise levels, respectively. Finally, for the classifier with the quadruple feature combination, the 98.71% detection accuracy dropped by 2.50% and 8% with the two noise levels, respectively. These results show that all classifiers essentially experience similar degradation rates under the Rician noise conditions. While the degradation rate is small for a noise level of σ = 1, this rate becomes understandably severe for the high noise level of σ = 2. Nevertheless, the classifier with the quadruple combination shows better robustness with the the highest accuracy under the noise-free condition and the smallest performance drop (8%) under the high noise condition σ = 2. Similar degradation patterns can be observed for the other performance metrics.

4 Discussion

In this paper, we investigated different fusion schemes of resting-state functional activity measures for discriminating between schizophrenic and healthy subjects. The improvements in classification performance can be ascribed to the choice of the feature selection and fusion approaches. For the decision fusion scheme in Table 3, the results outperformed those of the single-feature-type schemes. This shows that the fusion of the decisions of weak classifiers leads to more accurate classification performance [84]. Further, the feature fusion schemes in Table 4 show even better performance enhancements. This additional improvement can be ascribed to the fact that the feature-fusion schemes combine and optimize the selection of features, while the decision-level fusion scheme classifies samples by merely conducting a vote among only three single-feature-type classifiers.

We can understand the improvements obtained by different variants of the feature fusion classifiers by looking at the contributions of the single feature types to each of these classifiers. Specifically, Fig 6 shows the selected percentages of each of the ALFF, fALFF, ReHo, and VMHC feature types for optimizing the performance of the classifiers with pairwise, triple, and quadruple feature combinations. Obviously, each combination has different selections of individual feature types. For example, for pairwise combinations, the ALFF feature type is dominated by the other features; the fALFF type is dominated by the ReHo and VMHC ones; and the ReHo type is slightly dominated by the VMHC one. For triple combinations, the ALFF type consistently shows zero or marginal contributions, while the fALFF features are generally dominated by the ReHo and VMHC features. For the quadruple feature combination associated with the best performance, the same pattern is observed where the ALFF type has no contribution, while the VMHC type has the highest contribution followed by the fALFF and ReHo types.

thumbnail
Fig 6. Contribution percentage of each of the ALFF, fALFF, ReHo, VMHC feature types in the performance of the classifiers with pairwise, triple, and quadruple feature combinations.

https://doi.org/10.1371/journal.pone.0265300.g006

To put our results in context with other relevant studies, we examined the schizophrenia detection results in some of these studies. Firstly, the high performance metrics of our method agree with the high accuracies reported by other studies on the COBRE dataset. For example, Qureshi et al. [85] achieved an accuracy of 100% on the COBRE dataset with 10-fold cross-validation scheme and extreme learning machines (ELM). Also, Chyzhyk et al. [86] attained a 100% accuracy in the classification of healthy subjects and schizophrenic patients with and without auditory hallucinations. Juneja et al. [87] obtained a 98% accuracy on a multisite dataset from the Function Biomedical Informatics Research Network (FBIRN). Other schizophrenia detection methods achieved lower accuracies ranging from 62% to 94% for different variations of the schizophrenia classes, the image modalities and the collected datasets [88].

Furthermore, the performance metrics of the schizophrenia detection systems rank differently compared to other brain diseases [88]. On the one hand, the inter-quartile range of the reported overall accuracies for schizophrenia (76%-92%) is generally higher than those associated with stroke (67%-87%), ADHD (67%-76%) and depression (77%-86%). On the other hand, the inter-quartile range for schizophrenia is generally lower than those of the Alzheimer’s disease (AD) (86%-97%) and Parkinson’s disease (PD) (87%-98%).

The vertical averaging method was employed to plot the ROC curves as shown in Fig 3. We noticed that the AUC values of the combinations of features are slightly better than those based on single feature types.

The significant regions that are shown in Table 7 are in agreement with the previous findings. Studies based on ReHo features reported an increase in ReHo values in the right LING, the right PCUN and a decrease in ReHo values in the left PCUN and the right ACG [89, 90]. ALFF-based studies reported significant ALFF increases in the right PHG and decreased ALFF values in the right PHG and the left PreCG [91]. Also, fALFF-based studies demonstrated significant fALFF increases in VER10 and the left ORBmid and decreased fALFF values in the right CERcr2 and the right SPG [91, 92]. Finally, VMHC-based studies reported significant VMHC increases in the left PHG and the left ACG and decreased VMHC values in the right CER7 and the left PoCG [93]. As shown in Table 3, the standard deviation of the accuracy results for the outer 10-fold cross-validation scheme is considered small, and this demonstrates the low bias and high robustness of the proposed system for schizophrenia diagnosis. Moreover, the extracted features can be employed as biological markers that may help identify subjects at increased risk of disease development, and hence improve disease prognosis. Furthermore, the localization of the affected regions, in Table 7, can be employed in further research to localize and understand how the different regions are affected and changed through the progression of the disease.

5 Conclusion

In this paper, we introduced different feature-level and decision-level fusion schemes for discriminating between schizophrenic and healthy subjects and identifying schizophrenia-affected brain regions using whole-brain Rs-fMRI analysis. The highest average test accuracy of 98.71% was obtained with feature fusion of the four types of features considered in this paper. In summary, our work employs optimized feature selection algorithms, explores different fusion schemes, and exploits a large-scale dataset for schizophrenia detection. For future work, we seek to address this detection problem with deep learning and graph-theoretic techniques for two main reasons. Firstly, in practical applications for schizophrenia detection, the actual test data may be contaminated by noise and artifacts. Under these conditions, the handcrafted features may be not quite robust. This is why other graph-theoretic or deep learning methods with better noise robustness measures should be sought. Secondly, although the employed COBRE dataset is considered big compared to the other available datasets, a larger dataset is still needed with few hundreds of MRI volumes, acquired from different centers, with different MRI machines, and at different specifications. With these large real-world data variations, our current handcrafted-feature method may not achieve the same performance outcomes, and hence deep learning architectures would be needed to effectively capture the increasing data complexity.

References

  1. 1. Bhugra D. The global prevalence of schizophrenia. PLoS Med. 2005;2(5):e151. pmid:15916460
  2. 2. Insel TR. Rethinking schizophrenia. Nature. 2010;468(7321):187–193. pmid:21068826
  3. 3. Ripke S, O’Dushlaine C, Chambert K, Moran JL, Kähler AK, Akterin S, et al. Genome-wide association analysis identifies 13 new risk loci for schizophrenia. Nature Genetics. 2013;45(10):1150–1159. pmid:23974872
  4. 4. Tandon R, Keshavan MS, Nasrallah HA. Schizophrenia,“just the facts” what we know in 2008. 2. Epidemiology and etiology. Schizophrenia Research. 2008;102(1):1–18.
  5. 5. Picchioni MM, Murray RM. Schizophrenia. BMJ. 2007;335(7610):91–95. pmid:17626963
  6. 6. Nakamura K, Kawasaki Y, Suzuki M, Hagino H, Kurokawa K, Takahashi T, et al. Multiple structural brain measures obtained by three-dimensional magnetic resonance imaging to distinguish between schizophrenia patients and normal subjects. Schizophrenia Bulletin. 2004;30(2):393. pmid:15279055
  7. 7. Pardo PJ, Georgopoulos AP, Kenny JT, Stuve TA, Findling RL, Schulz SC. Classification of adolescent psychotic disorders using linear discriminant analysis. Schizophrenia Research. 2006;87(1):297–306. pmid:16797923
  8. 8. Davatzikos C, Shen D, Gur RC, Wu X, Liu D, Fan Y, et al. Whole-brain morphometric study of schizophrenia revealing a spatially complex set of focal abnormalities. Archives of General Psychiatry. 2005;62(11):1218–1227. pmid:16275809
  9. 9. Kawasaki Y, Suzuki M, Kherif F, Takahashi T, Zhou SY, Nakamura K, et al. Multivariate voxel-based morphometry successfully differentiates schizophrenia patients from healthy controls. Neuroimage. 2007;34(1):235–242. pmid:17045492
  10. 10. Yoon U, Lee JM, Im K, Shin YW, Cho BH, Kim IY, et al. Pattern classification using principal components of cortical thickness and its discriminative pattern in schizophrenia. Neuroimage. 2007;34(4):1405–1415. pmid:17188902
  11. 11. Fan Y, Shen D, Davatzikos C. Classification of Structural images via High-Dimensional Image Warping, Robust Feature Extraction, and SVM. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2005. p. 1–8.
  12. 12. Fan Y, Shen D, Gur RC, Gur RE, Davatzikos C. COMPARE: classification of morphological patterns using adaptive regional elements. IEEE Transactions on Biomedical Engineering. 2007;26(1):93–105. pmid:17243588
  13. 13. Caprihan A, Pearlson GD, Calhoun VD. Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements. Neuroimage. 2008;42(2):675–682. pmid:18571937
  14. 14. Sun D, van Erp TG, Thompson PM, Bearden CE, Daley M, Kushan L, et al. Elucidating a magnetic resonance imaging-based neuroanatomic biomarker for psychosis: classification analysis using probabilistic brain atlas and machine learning algorithms. Biological Psychiatry. 2009;66(11):1055–1060. pmid:19729150
  15. 15. Takayanagi Y, Kawasaki Y, Nakamura K, Takahashi T, Orikabe L, Toyoda E, et al. Differentiation of first-episode schizophrenia patients from healthy controls using ROI-based multiple structural brain variables. Progress in Neuro-Psychopharmacology and Biological Psychiatry. 2010;34(1):10–17. pmid:19751790
  16. 16. Takayanagi Y, Takahashi T, Orikabe L, Mozue Y, Kawasaki Y, Nakamura K, et al. Classification of first-episode schizophrenia patients and healthy subjects by automated MRI measures of regional brain volume and cortical thickness. PloS One. 2011;6(6):e21047. pmid:21712987
  17. 17. Ardekani BA, Tabesh A, Sevy S, Robinson DG, Bilder RM, Szeszko PR. Diffusion tensor imaging reliably differentiates patients with schizophrenia from healthy volunteers. Human Brain Mapping. 2011;32(1):1–9. pmid:20205252
  18. 18. Gutiérrez-Gómez L, Vohryzek J, Chiêm B, Baumann PS, Conus P, Do Cuenod K, et al. Stable biomarker identification for predicting schizophrenia in the human connectome. NeuroImage: Clinical. 2020;27:102316. pmid:32623137
  19. 19. Georgopoulos AP, Karageorgiou E, Leuthold AC, Lewis SM, Lynch JK, Alonso AA, et al. Synchronous neural interactions assessed by magnetoencephalography: a functional biomarker for brain disorders. Journal of Neural Engineering. 2007;4(4):349. pmid:18057502
  20. 20. Demirci O, Clark VP, Calhoun VD. A projection pursuit algorithm to classify individuals using fMRI data: Application to schizophrenia. NeuroImage. 2008;39(4):1774–1782. pmid:18396487
  21. 21. Calhoun VD, Maciejewski PK, Pearlson GD, Kiehl KA. Temporal lobe and default hemodynamic brain modes discriminate between schizophrenia and bipolar disorder. Human Brain Mapping. 2008;29(11):1265–1275. pmid:17894392
  22. 22. Michael AM, Calhoun VD, Andreasen NC, Baum SA. A method to Classify Schizophrenia Using Inter-task Spatial Correlations of Functional Brain Images. In: Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE. IEEE; 2008. p. 5510–5513.
  23. 23. Arribas JI, Calhoun VD, Adali T. Automatic Bayesian classification of healthy controls, bipolar disorder, and schizophrenia using intrinsic connectivity maps from FMRI data. IEEE Transactions on Biomedical Engineering. 2010;57(12):2850–2860. pmid:20876002
  24. 24. Shen H, Wang L, Liu Y, Hu D. Discriminative analysis of resting-state functional connectivity patterns of schizophrenia using low dimensional embedding of fMRI. Neuroimage. 2010;49(4):3110–3121. pmid:19931396
  25. 25. Castro E, Martínez-Ramón M, Pearlson G, Sui J, Calhoun VD. Characterization of groups using composite kernels and multi-source fMRI analysis data: application to schizophrenia. Neuroimage. 2011;58(2):526–536. pmid:21723948
  26. 26. Ford J, Shen L, Makedon F, Flashman LA, Saykin AJ. A combined structural-functional classification of schizophrenia using hippocampal volume plus fMRI activation. In: Engineering in Medicine and Biology, 2002. 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES Conference, 2002. Proceedings of the Second Joint. vol. 1. IEEE; 2002. p. 48–49.
  27. 27. Kim DI, Sui J, Rachakonda S, White T, Manoach DS, Clark VP, et al. Identification of imaging biomarkers in schizophrenia: a coefficient-constrained independent component analysis of the mind multi-site schizophrenia study. Neuroinformatics. 2010;8(4):213–229. pmid:20607449
  28. 28. Shenton ME, Dickey CC, Frumin M, McCarley RW. A review of MRI findings in schizophrenia. Schizophrenia Research. 2001;49(1):1–52. pmid:11343862
  29. 29. Barta PE, Pearlson GD, Brill LB, Royall R, McGilchrist IK, Pulver AE, et al. Planum temporale asymmetry reversal in schizophrenia: replication and relationship to gray matter abnormalities. American Journal of Psychiatry. 1997;154(5):661–667. pmid:9137122
  30. 30. Holinger DP, Shenton ME, Wible CG, Donnino R, Kikinis R, Jolesz FA, et al. Superior temporal gyrus volume abnormalities and thought disorder in left-handed schizophrenic men. American Journal of Psychiatry. 1999;156(11):1730–1735. pmid:10553736
  31. 31. Shenton ME, Kikinis R, Jolesz FA, Pollak SD, LeMay M, Wible CG, et al. Abnormalities of the left temporal lobe and thought disorder in schizophrenia: a quantitative magnetic resonance imaging study. New England Journal of Medicine. 1992;327(9):604–612. pmid:1640954
  32. 32. Gur RE, Cowell PE, Latshaw A, Turetsky BI, Grossman RI, Arnold SE, et al. Reduced dorsal and orbital prefrontal gray matter volumes in schizophrenia. Archives of General Psychiatry. 2000;57(8):761–768. pmid:10920464
  33. 33. Szeszko PR, Bilder RM, Lencz T, Pollack S, Alvir JMJ, Ashtari M, et al. Investigation of frontal lobe subregions in first-episode schizophrenia. Psychiatry Research: Neuroimaging. 1999;90(1):1–15. pmid:10320207
  34. 34. Goldstein JM, Goodman JM, Seidman LJ, Kennedy DN, Makris N, Lee H, et al. Cortical abnormalities in schizophrenia identified by structural magnetic resonance imaging. Archives of General Psychiatry. 1999;56(6):537–547. pmid:10359468
  35. 35. Peng LW, Lee S, Federman EB, Chase GA, Barta PE, Pearlson GD. Decreased regional cortical gray matter volume in schizophrenia. Am J Psychiatry. 1994;151(6):843.
  36. 36. Frederikse M, Lu A, Aylward E, Barta P, Sharma T, Pearlson G. Sex differences in inferior parietal lobule volume in schizophrenia. Archives of General Psychiatry. 2000;157(3):422–427.
  37. 37. Niznikiewicz M, Donnino R, McCarley RW, Nestor PG, Iosifescu DV, O’Donnell B, et al. Abnormal angular gyrus asymmetry in schizophrenia. Archives of General Psychiatry. 2000;157(3):428–437.
  38. 38. Breier A, Buchanan RW, Elkashef A, Munson RC, Kirkpatrick B, Gellad F. Brain morphology and schizophrenia: a magnetic resonance imaging study of limbic, prefrontal cortex, and caudate structures. Archives of General Psychiatry. 1992;49(12):921–926. pmid:1449382
  39. 39. Wible CG, Shenton ME, Hokama H, Kikinis R, Jolesz FA, Metcalf D, et al. Prefrontal cortex and schizophrenia: a quantitative magnetic resonance imaging study. Archives of General Psychiatry. 1995;52(4):279–288. pmid:7702444
  40. 40. Friston KJ. The disconnection hypothesis. Schizophrenia Research. 1998;30(2):115–125. pmid:9549774
  41. 41. McGuire P, Frith C. Disordered functional connectivity in schizophrenia. Psychological Medicine. 1996;26(04):663–667. pmid:8817700
  42. 42. Du W, Calhoun VD, Li H, Ma S, Eichele T, Kiehl KA, et al. High classification accuracy for schizophrenia with rest and task fMRI data. Frontiers in Human Neuroscience. 2012;6(145). pmid:22675292
  43. 43. Brier MR, Thomas JB, Snyder AZ, Benzinger TL, Zhang D, Raichle ME, et al. Loss of intranetwork and internetwork resting state functional connections with Alzheimer’s disease progression. The Journal of Neuroscience. 2012;32(26):8890–8899. pmid:22745490
  44. 44. Liu F, Guo W, Fouche JP, Wang Y, Wang W, Ding J, et al. Multivariate classification of social anxiety disorder using whole brain functional connectivity. Brain Structure and Function. 2015;220(1):101–115. pmid:24072164
  45. 45. Tang Y, Jiang W, Liao J, Wang W, Luo A. Identifying individuals with antisocial personality disorder using resting-state FMRI. PloS One. 2013;8(4):e60652. pmid:23593272
  46. 46. Hoekzema E, Carmona S, Ramos-Quiroga JA, Richarte Fernández V, Bosch R, Soliva JC, et al. An independent components and functional connectivity analysis of resting state fMRI data points to neural network dysregulation in adult ADHD. Human Brain Mapping. 2014;35(4):1261–1272. pmid:23417778
  47. 47. Zeng LL, Shen H, Liu L, Hu D. Unsupervised classification of major depression using functional connectivity MRI. Human Brain Mapping. 2014;35(4):1630–1641. pmid:23616377
  48. 48. Algunaid RF, Algumaei AH, Rushdi MA, Yassine IA. Schizophrenic patient identification using graph-theoretic features of resting-state fMRI data. Biomedical Signal Processing and Control. 2018;43:289–299.
  49. 49. Yang B, Chen Y, Shao QM, Yu R, Li WB, Guo GQ, et al. Schizophrenia Classification Using fMRI Data Based on a Multiple Feature Image Capsule Network Ensemble. IEEE Access. 2019;7:109956–109968.
  50. 50. Yan W, Calhoun V, Song M, Cui Y, Yan H, Liu S, et al. Discriminating schizophrenia using recurrent neural network applied on time courses of multi-site FMRI data. EBioMedicine. 2019;47:543–552. pmid:31420302
  51. 51. Yu Q, Allen E A, Sui J, Arbabshirani M R, Pearlson G, Calhoun V D. Brain connectivity networks in schizophrenia underlying resting state functional magnetic resonance imaging. Current topics in medicinal chemistry. 2012;12(21):2415–2425. pmid:23279180
  52. 52. Silfverhuth MJ, Hintsala H, Kortelainen J, Seppänen T. Experimental comparison of connectivity measures with simulated EEG signals. Medical & Biological Engineering & Computing. 2012;50(7):683–688. pmid:22614134
  53. 53. Blinowska KJ. Review of the methods of determination of directed connectivity from multichannel data. Medical & Biological Engineering & Computing. 2011;49(5):521–529. pmid:21298355
  54. 54. Tang Y, Wang L, Cao F, Tan L. Identify schizophrenia using resting-state functional connectivity: an exploratory research and analysis. Biomedical Engineering Online. 2012;11(1):1. pmid:22898249
  55. 55. Yu Y, Shen H, Zhang H, Zeng LL, Xue Z, Hu D. Functional connectivity-based signatures of schizophrenia revealed by multiclass pattern analysis of resting-state fMRI from schizophrenic patients and their healthy siblings. Biomedical Engineering Online. 2013;12(1):1. pmid:23390976
  56. 56. Guo W, Su Q, Yao D, Jiang J, Zhang J, Zhang Z, et al. Decreased regional activity of default-mode network in unaffected siblings of schizophrenia patients at rest. European Neuropsychopharmacology. 2014;24(4):545–552. pmid:24491950
  57. 57. Chyzhyk D, Savio A, Graña M. Computer aided diagnosis of schizophrenia on resting state fMRI data by ensembles of ELM. Neural Networks. 2015;68:23–33. pmid:25965771
  58. 58. Castanedo F. A review of data fusion techniques. The Scientific World Journal. 2013;2013. pmid:24288502
  59. 59. Khaleghi B, Khamis A, Karray FO, Razavi SN. Multisensor data fusion: A review of the state-of-the-art. Information fusion. 2013;14(1):28–44.
  60. 60. Kumari P, Vaish A. Feature-level fusion of mental task’s brain signal for an efficient identification system. Neural Computing and Applications. 2016;27(3):659–669.
  61. 61. James A, Dasarathy BV. In: A Review of Feature and Data Fusion with Medical Images; 2015. p. 491–507.
  62. 62. Calhoun VD, Sui J. Multimodal fusion of brain imaging data: a key to finding the missing link (s) in complex mental illness. Biological psychiatry: cognitive neuroscience and neuroimaging. 2016;1(3):230–244.
  63. 63. Yan CG, Wang XD, Zuo XN, Zang YF. DPABI: Data Processing & Analysis for (Resting-State) Brain Imaging. Neuroinformatics. 2016; p. 1–13.
  64. 64. Friston KJ, Williams S, Howard R, Frackowiak RS, Turner R. Movement-related effects in fMRI time-series. Magnetic Resonance in Medicine. 1996;35(3):346–355. pmid:8699946
  65. 65. Ashburner J. A fast diffeomorphic image registration algorithm. Neuroimage. 2007;38(1):95–113. pmid:17761438
  66. 66. Jo HJ, Gotts SJ, Reynolds RC, Bandettini PA, Martin A, Cox RW, et al. Effective preprocessing procedures virtually eliminate distance-dependent motion artifacts in resting state FMRI. Journal of Applied Mathematics. 2013;2013. pmid:24415902
  67. 67. Yu-Feng Z, Yong H, Chao-Zhe Z, Qing-Jiu C, Man-Qiu S, Meng L, et al. Altered baseline brain activity in children with ADHD revealed by resting-state functional MRI. Brain and Development. 2007;29(2):83–91.
  68. 68. Zou QH, Zhu CZ, Yang Y, Zuo XN, Long XY, Cao QJ, et al. An improved approach to detection of amplitude of low-frequency fluctuation (ALFF) for resting-state fMRI: fractional ALFF. Journal of Neuroscience Methods. 2008;172(1):137–141. pmid:18501969
  69. 69. Zuo XN, Kelly C, Di Martino A, Mennes M, Margulies DS, Bangaru S, et al. Growing together and growing apart: regional and sex differences in the lifespan developmental trajectories of functional homotopy. The Journal of Neuroscience. 2010;30(45):15034–15043. pmid:21068309
  70. 70. Zang Y, Jiang T, Lu Y, He Y, Tian L. Regional homogeneity approach to fMRI data analysis. Neuroimage. 2004;22(1):394–400. pmid:15110032
  71. 71. Maurice K, Dickinson GJ. Rank correlation methods. London: Edward Arnold. 1990;.
  72. 72. Roffo G. Feature Selection Library (MATLAB Toolbox). ArXiv Preprint ArXiv:160701327. 2016;.
  73. 73. Bradley PS, Mangasarian OL. Feature Selection via Concave Minimization and Support Vector Machines. In: ICML. vol. 98; 1998. p. 82–90.
  74. 74. Guyon I, Weston J, Barnhill S, Vapnik V. Gene selection for cancer classification using support vector machines. Machine Learning. 2002;46(1-3):389–422.
  75. 75. Urbanowicz RJ, Meeker M, La Cava W, Olson RS, Moore JH. Relief-based feature selection: Introduction and review. Journal of biomedical informatics. 2018;85:189–203. pmid:30031057
  76. 76. Haury AC, Gestraud P, Vert JP. The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures. PloS One. 2011;6(12):e28210. pmid:22205940
  77. 77. Hsu CW, Lin CJ. A comparison of methods for multiclass support vector machines. IEEE transactions on Neural Networks. 2002;13(2):415–425. pmid:18244442
  78. 78. Alpaydm E. Combined 5× 2 cv F test for comparing supervised classification learning algorithms. Neural computation. 1999;11(8):1885–1892.
  79. 79. Wilson EB. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association. 1927;22(158):209–212.
  80. 80. Agresti A, Coull BA. Approximate is Better than Exact for Interval Estimation of Binomial Proportions. The American Statistician. 1998;52(2):119–126.
  81. 81. Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters. 2006;27(8):861–874.
  82. 82. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B (Methodological). 1995; p. 289–300.
  83. 83. Xia M, Wang J, He Y. BrainNet Viewer: a network visualization tool for human brain connectomics. PloS One. 2013;8(7):e68910. pmid:23861951
  84. 84. Mangai UG, Samanta S, Das S, Chowdhury PR. A survey of decision fusion and feature fusion strategies for pattern classification. IETE Technical review. 2010;27(4):293–307.
  85. 85. Qureshi MNI, Oh J, Cho D, Jo HJ, Lee B. Multimodal discrimination of schizophrenia using hybrid weighted feature concatenation of brain functional connectivity and anatomical features with an extreme learning machine. Frontiers in neuroinformatics. 2017;11:59. pmid:28943848
  86. 86. Chyzhyk D, Grana M, Öngür D, Shinn AK. Discrimination of schizophrenia auditory hallucinators by machine learning of resting-state functional MRI. International Journal of Neural Systems. 2015;25(03):1550007. pmid:25753600
  87. 87. Juneja A, Rana B, Agrawal R. A novel fuzzy rough selection of non-linearly extracted features for schizophrenia diagnosis using fMRI. Computer methods and programs in biomedicine. 2018;155:139–152. pmid:29512494
  88. 88. Sakai K, Yamada K. Machine learning studies on major brain diseases: 5-year trends of 2014–2018. Japanese journal of radiology. 2019;37(1):34–72. pmid:30498877
  89. 89. Wang S, Wang G, Lv H, Wu R, Zhao J, Guo W. Abnormal regional homogeneity as potential imaging biomarker for psychosis risk syndrome: a resting-state fMRI study and support vector machine analysis. Scientific Reports. 2016;6:27619. pmid:27272341
  90. 90. Liu H, Liu Z, Liang M, Hao Y, Tan L, Kuang F, et al. Decreased regional homogeneity in schizophrenia: a resting state functional magnetic resonance imaging study. Neuroreport. 2006;17(1):19–22. pmid:16361943
  91. 91. Hoptman MJ, Zuo XN, Butler PD, Javitt DC, D’Angelo D, Mauro CJ, et al. Amplitude of low-frequency oscillations in schizophrenia: a resting state fMRI study. Schizophrenia research. 2010;117(1):13–20. pmid:19854028
  92. 92. Yu R, Chien YL, Wang HLS, Liu CM, Liu CC, Hwang TJ, et al. Frequency-specific alternations in the amplitude of low-frequency fluctuations in schizophrenia. Human brain mapping. 2014;35(2):627–637. pmid:23125131
  93. 93. Guo W, Liu F, Chen J, Wu R, Li L, Zhang Z, et al. Treatment effects of olanzapine on homotopic connectivity in drug-free schizophrenia at rest. The World Journal of Biological Psychiatry. 2017; p. 1–9. pmid:28649941