^{1}

^{2}

^{*}

Conceived and designed the experiments: JUA. Performed the experiments: JUA ACR. Analyzed the data: ACR JUA. Contributed reagents/materials/analysis tools: ACR JUA. Wrote the paper: ACR JUA.

The authors have declared that no competing interests exist.

Recently, an exact binomial test called SGoF (Sequential Goodness-of-Fit) has been introduced as a new method for handling high dimensional testing problems. SGoF looks for statistical significance when comparing the amount of null hypotheses individually rejected at level γ = 0.05 with the expected amount under the intersection null, and then proceeds to declare a number of effects accordingly. SGoF detects an increasing proportion of true effects with the number of tests, unlike other methods for which the opposite is true. It is worth mentioning that the choice γ = 0.05 is not essential to the SGoF procedure, and more power may be reached at other values of γ depending on the situation. In this paper we enhance the possibilities of SGoF by letting the γ vary on the whole interval (0,1). In this way, we introduce the ‘SGoFicance Trace’ (from SGoF's significance trace), a graphical complement to SGoF which can help to make decisions in multiple-testing problems. A script has been written for the computation in R of the SGoFicance Trace. This script is available from the web site

Multiple-testing problems have received much attention since the advent of the ‘-omic’ technologies: genomics, transcriptomics, proteomics, etc. They usually involve the simultaneous testing of thousands of hypotheses, or nulls, producing as a result a number of significant

How many effects should be declared?

Which FDR is reasonable to assume for the discoveries in a given experimental setup?

Recently, Carvajal-Rodríguez et al. _{1}<_{2}<…<_{S} to be the sorted _{i}≤γ }. Under the intersection null, _{α}(γ), where _{α}(γ) is the 100(1-α)% percentile of the Binomial(_{α}(γ)+1 smallest

The SGoF can be interpreted/described as a sequential algorithm that in a step-wise mode decides if a candidate

One of the main advantages of SGoF is that it exhibits an increasing power with the number of tests. As mentioned, this is not true for other multiple-testing corrections (including those controlling the FDR rather than the FWER), which, under some settings, can hardly find even one true effect in high dimensions _{α}(γ)+1) leads to an estimated FDR, the new method informs in an indirect way about ‘which FDR is reasonable to assume’ for a given data set, answering the former questions (1) and (2) at the same time. As stated before, this information is not provided by other existing methods, for which the choice of the FDR parameter must be subjective.

The original implementation of SGoF concentrates in the case γ = α ( = 0.05). Obviously, the choice γ = α, while being intuitive, is not essential for the SGoF procedure. Therefore, it could be interesting to look at SGoF results when γ moves away from α. The only caution that should be taken is that

We have simulated the simultaneous testing of

In

The dashed lines correspond to the BH method at nominal FDR of 0.05. The number of hypotheses is 1,000 and there is an effect of 10%. Averages were computed through 1,000 Monte Carlo trials.

Other choices of

SGoF(0.01) | SGoF(0.05) | SGoF(0.09) | SGoF(0.13) | SGoF(0.17) | BH | ||

10% |
Power | 0.2786 | 0.3815 | 0.4004 | 0.3984 | 0.3857 | 0.1202 |

FDR | 0.1327 | 0.2004 | 0.2164 | 0.2145 | 0.2086 | 0.0516 | |

10% |
Power | 0.1202 | 0.1958 | 0.2198 | 0.2237 | 0.2242 | 0.0211 |

FDR | 0.2160 | 0.2963 | 0.3203 | 0.3224 | 0.3228 | 0.0503 | |

10% |
Power | 0.0259 | 0.0529 | 0.0691 | 0.0757 | 0.0801 | 0.0028 |

FDR | 0.2932 | 0.3779 | 0.4087 | 0.4140 | 0.4222 | 0.0472 | |

30% |
Power | 0.3314 | 0.4998 | 0.5402 | 0.5499 | 0.5481 | 0.3340 |

FDR | 0.0504 | 0.0977 | 0.1117 | 0.1157 | 0.1152 | 0.0507 | |

30% |
Power | 0.1642 | 0.3042 | 0.3556 | 0.3767 | 0.3871 | 0.0824 |

FDR | 0.0878 | 0.1444 | 0.1682 | 0.1785 | 0.1830 | 0.0496 | |

30% |
Power | 0.0568 | 0.1312 | 0.1688 | 0.1887 | 0.2019 | 0.0067 |

FDR | 0.1683 | 0.2323 | 0.2583 | 0.2713 | 0.2795 | 0.0505 |

The SGoFicance Trace is a graphical device constructed from the SGoF multitest. Let SGoF(γ) denote the SGoF algorithm described in the _{i}'s vs. γ.

(A): SGoF's log-significance plot; (B): SGoF's number of effects; (C): SGoF's FDR; (D): SGoF's threshold for

The first plot (_{i}'s pertaining to the non-true nulls would tend to be located close to zero, a monotone increasing shape is expected in this plot. An horizontal dashed line at point log(α) was incorporated to the plot for completeness. Here, α represents the FWER that is controlled by SGoF(γ) in a weak sense. The default value for α was set at 0.05.

For each γ, the number of effects declared by SGoF(γ) at FWER α is _{α}(γ) = _{α}(γ)+1. Unlike _{α}(γ) values against γ, for the particular choice α = 0.05. In a typical multiple-testing problem, an inverted U-shape will be found in this plot, meaning that the largest number of effects corresponds to intermediate values of the γ parameter. Notice that, while _{α}(γ) also increases as γ gets larger.

A commonly used measure of performance in multiple-testing problems is the FDR. Hence, it is interesting to evaluate the FDR of SGoF(γ) for each value of γ. In practice, the FDR is unknown, but some estimation methods can be used to find it. The FDR estimation procedure starts from some preliminary assessment of the proportion of true nulls, π_{0}. Different methods have been proposed in the literature to do so. We have implemented the method proposed by Dalmasso et al. _{0} by the average of the -log(1 - _{i})'s. Denoting this quantity by _{0}, the estimated FDR of SGoF(γ) was just eFDR_{α}(γ) = _{α}(γ)*_{0}/_{α}(γ), where _{α}(γ) stands for the threshold of the _{α}(γ) effects, that is, _{α}(γ) is such that _{α}(γ) = #{_{i}≤_{α}(γ)}.

As for the plot of the SGoF's number of effects, the FDR plot (_{α}(γ) attains its largest value. Recall that, unlike the FDR-based methods, SGoF is not constructed to respect a given fixed proportion of false discoveries. For this reason, this plot is informative about the ‘reasonable amount of FDR’ that could be faced in a given situation. Again, the nominal FWER α was set to 0.05.

Finally, the SGoFicance Trace provides a plot showing the threshold values _{α}(γ)

To resume the above explanations, the SGoFicance Trace at a FWER of 5% corresponding to the randomly chosen Monte Carlo trial #101 (from the simulation study described in the first section) is shown in _{α}(γ = 0.13) such as that 70 = #{_{i}≤_{α}(γ)} (

As an illustrative example, we took the microarray study of hereditary breast cancer by Hedenfalk et al.

We have computed the SGoFicance Trace at a FWER of 5% for this data set and the result is displayed in

(A): SGoF's log-significance plot; (B): SGoF's number of effects; (C): SGoF's FDR; (D): SGoF's threshold for

In the case of choosing the initially suggested γ value of 0.26, the threshold _{0}: E[_{α}(γ) = min(_{α}(γ)+1,

The SGoFicance Trace is also useful when the aim is to keep up a given proportion of false positives. In this regard, _{0} = 0.67 rather than our 0.72).

It could be asked what happens to Hedenfalk data's SGoFicance Trace when applied at a different FWER level, more or less conservative than 5%, as for instance 0.01% and 15% respectively. An interesting finding is that the maximum FDR in these two quite extreme cases was never above the maximum FDR of 20% revealed by

Some general guidelines can be given for the use of the SGoFicance tool. We can distinguish between two extreme cases. First, the researcher is specially interested in the detection of effects while having information on the corresponding FDR. This can be the case of any exploratory study at the genome or proteome-wide level comparing for example two species. In this case the panel B of the SGoFicance should guide the decision. This panel will immediately tell the researcher the maximum number of true effects that can be detected under SGoF and the FDR that should be assumed in doing so. Moreover, it will inform the researcher about the maximum FDR that should be assumed because a higher one will not translate in more power since SGoF will be unable to find statistical significance for a larger number of true effects. On the other hand, we can think in a second general case where the researcher is interested in minimizing the FDR, as in an association study from which, with limited economic resources, we are going to isolate the detected genes or proteins. In this case the panel C should be the first to be consulted to set the desired FDR and afterwards exploring the corresponding statistically significant number of effects and

As a conclusion, the SGoFicance tool aims to provide solution to the extreme situations above mentioned besides the whole range of intermediate cases where equilibrium between FDR control and power is desired.

The algorithms necessary for the computation of the SGoFicance Trace were implemented in a script programmed in the R language

The R script for the computation of the SGoFicance is available from the site

We thank Monica Martínez-Fernández, Ana M^{a} Rodríguez and E. Rolán-Álvarez for comments on the manuscript.