Statistics

Two different statistical analyses were performed, and both showed no significant difference in the NIRS response between yes/no questions. The first analysis was different than the one performed by Chaudhary and colleagues and is presented in S1 Text in more detail. For the second analysis, I point out methodological flaws in the statistical analysis of Chaudhary and colleagues and show results with corrected methods.

As the statistical analysis by Chaudhary and colleagues was not described with sufficient detail to exactly reproduce it, Chaudhary and colleagues responded in the first review round to this comment that they averaged the data first over all trials and then over all sessions and performed a t test on those averages. The problem with this kind of analysis is that the variance over trials/sessions is removed by the averaging, and only the variance over the channels is retained. Performing a statistical test will then compare the mean of yes-trials with the mean of no-trials while considering the variance over all channels. As the channels are highly correlated (not independent), the variance is very low and will lead to the wrong result, that the difference is significant. The statement that this kind of analysis is not correct can be tested by using a permutation test. Using a random permutation of the trials, instead of a separation by yes/no, should show no significant difference if the statistical analysis is correct. But for the method used by Chaudhary and colleagues, the results are significant in each of 10 random permutations (see figure A1 in S1 Text), which demonstrates that the used method is not correct.

This does not mean that averages should not be used for statistical analysis, but that the order of averaging matters. If the data are first averaged over channels and then over trials, the variance over sessions is retained. But in this case, there is no significant difference between yes/no questions (p > 0.05; t test, not corrected for multiple comparisons) for any patient. Fig 1 shows the results of an analysis of data from patient B with the two different orders of averaging. Although the mean response is the same for both analyses, the variance and the results of a statistical test are different.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Comparison of statistical methods on data of patient B.

Both plots show the average response to yes/no questions as well as the standard deviation. (A) Method used by Chaudhary and colleagues, first averaging over trials, then over sessions, and then over channels. For the t test, the variance over channels is retained and used. Time points with significant difference (p < 0.0005) are marked gray. (B) Same method but with a correct order of averaging, first averaging over channels, then over trials, and then over sessions. For the t test, the variance over sessions is retained and used. No time points show a significant difference (p > 0.05).

https://doi.org/10.1371/journal.pbio.2004750.g001

Regarding the averaging over channels, it should also be mentioned that averaging across channels is generally not recommended. Although it is not incorrect, averaging channels can reduce or even cancel out an effect in the data if the signal of interest is highly localized or shows different patterns across different brain areas. For this reason, an analysis treating each channel independently (as in S1 Text) is recommended.

Offline classification

The offline classification of the data was also reproduced. As some details of this analysis were not given by Chaudhary and colleagues (e.g., hyperparameter), it was not possible to reproduce the results with exactly the same method, but a similar approach was used. A support vector machine (SVM) with a linear kernel was applied to the relative change in HbO. For each day of each patient, a 10-fold cross-validation was performed in which the data were randomly divided into 10 blocks, and 9 blocks were used for training the classifier and tested on the remaining block. This process was repeated 10 times so that each block was used for testing once. For training the classifier, the training data were balanced by randomly removing trials of the majority class from the training data. The optimal hyperparameter for the SVM was estimated by performing a grid search with a 10-fold cross-validation on the training data. When using the preprocessed data as input for classification, an average accuracy of 49.4% was obtained. The performance was not significantly different [10] than chance level (p > 0.05, not corrected for multiple comparisons) for all of the 42 days. More detailed results can be found in S1 Text.