Wilcoxon-Mann-Whitney (WMW) statistic.

Definition 2 (Mann-Whitney U-Statistic) For 1 ≤ ℓ ≤ d, we let with

Proposition 1 (cf. Theorem 2 (iii*) of [19]) Assume that n_i/N → τ_i ∈ (0,1) for N → ∞, i ∈ {A,B}. Then, under H₀, it holds that (2) where Σ = (σ_ℓr)_{1 ≤ ℓ,r ≤ d} with entries where 1 ≤ j ≤ n_A and 1 ≤ k ≠ k′ ≤ n_B. In Eq (2) and throughout, $\stackrel{𝓓}{\to }$ denotes convergence in distribution, and 𝓝_d(μ,Σ) stands for the d-variate normal distribution with mean μ and covariance matrix Σ.

Corollary 1 (Theorem 9.1 in [19]) Let $\widehat{\Sigma }$ be a consistent estimator of Σ. Assuming that det(Σ) > 0 it holds that is under H₀ asymptotically χ²-distributed with d degrees of freedom as N → ∞.

Empirical relative effects.

The empirical counterpart of the vector p_AB of relative effects is denoted by ${\widehat{\mathbf{\text{p}}}}_{AB}={({\widehat{p}}_{AB}^{(1)},\dots ,{\widehat{p}}_{AB}^{(d)})}^{\top }$ with ${\widehat{p}}_{AB}^{(\ell )}=\int {\widehat{F}}_A^{(\ell )}d{\widehat{F}}_B^{(\ell )}$, 1 ≤ ℓ ≤ d, where ${\widehat{F}}_i^{(\ell )}$, given by ${\widehat{F}}_i^{(\ell )}(x)=n_i^{-1}{\sum }_{k=1}^{n_i}\frac{1}{2}({𝕀}_{(-\infty ,x]}(X_{ik}^{(\ell )})+{𝕀}_{(-\infty ,x)}(X_{ik}^{(\ell )}))$, denotes the normalized version of the empirical cdf in group i ∈ {A,B} pertaining to coordinate ℓ. Notice that ${\widehat{p}}_{AB}^{(\ell )}=1-U^{(\ell )}$ almost surely for all 1 ≤ ℓ ≤ d, where U^(ℓ) is as in Definition 2, under the assumption of absolutely continuous distributions (that there is zero probability for ties).

Proposition 2 (Theorem 3.3 in [20]) Let V_N ∈ ℝ^d×d denote the matrix with entries with the transformed random variables $Y_{Ak}^{(\ell )}=F_B^{(\ell )}(X_{Ak}^{(\ell )})$ and $Y_{Bk}^{(\ell )}=F_A^{(\ell )}(X_{Bk}^{(\ell )})$. Assuming that V_N converges to a positive definite covariance matrix V as N → ∞, it holds that

Furthermore, in (4.6) of [20] a consistent estimator ${\widehat{V}}_N$ defined via the ranks of the observations has been provided.

Corollary 2 Making use of Proposition 2 and a Studentization by ${\widehat{V}}_N$, it follows by Slutsky’s lemma in analogy to Corollary 1 that, under $H_0^{\prime }:{\mathbf{p}}_{AB}={\mathbf{1}}_d/2$, the statistic is asymptotically ${\chi }_d^2$-distributed as N → ∞.

Closure principle.

A (non-randomized) multiple testing procedure φ = (φ₁,…,φ_d)^⊤ for testing ℋ or ℋ′, respectively, is a vector of measurable mappings (individual tests) from the sample space into {0,1}^d. In this, the event {φ_ℓ = 1} means rejection of the ℓ-th null hypothesis H_ℓ or $H_{\ell }^{\prime }$, respectively. For given distributions P and Q, the FWER of φ is defined as the probability under (P,Q) of at least one type I error, i. e., where I₀(P,Q) ⊆ {1,…,d} denotes the index set of true null hypotheses in ℋ or ℋ′, respectively. The multiple test φ is said to control the FWER strongly at a given level α ∈ (0,1), if FWER_(P,Q)(φ) ≤ α for all possible pairs (P,Q).

One construction principle for FWER-controlling multiple tests is the closed test principle according to [21]. The general idea behind this method is to add to the system of hypotheses of interest all their intersections H_S or $H_S^{\prime }$, respectively, where S ∈ 2^{1,…,d}. Even if these intersection hypotheses are not of scientific interest, they are tested auxiliarly in order to provide a multiplicity correction. Namely, a closed test procedure tests every such intersection hypothesis at full level α by an arbitrarily chosen level α test φ_S or ${\phi }_S^{\prime }$, respectively. The adjustment for multiplicity is then performed via the decision rule that only those coordinate-specific hypotheses H_ℓ or $H_{\ell }^{\prime }$, respectively, are rejected for which all intersection hypotheses H_S ($H_S^{\prime }$) with ℓ ∈ S have been rejected by φ_S (${\phi }_S^{\prime }$). Thus, the price to pay for the multiplicity of the problem is that one has to perform 2^d tests. A concise description of this principle can for instance be found in Section 3.3 of [9].

Remark 1 Application of the closed test principle is particularly convenient in our context by noticing that the assertions of Propositions 1 and 2 and their corollaries remain valid if the respective full d-dimensional vector of test statistics is replaced by a subvector which only contains the indices in the subset S to which φ_S or ${\phi }_S^{\prime }$, respectively, refer. In the corollaries, only the degrees of freedom of the asymptotic χ²-distributions have to be changed from d to ∣S∣.

Resampling-based approach.

The results from the previous sections can also be used to construct asymptotically pivotal statistics for usage in a resampling approach. This strategy is assumed to keep α more accurately for finite N than the asymptotic methods resulting from Corollaries 1 and 2. In [22], multivariate multiple permutation tests have been developed for more restrictive families of hypotheses than ℋ or ℋ′, namely, for families where differences of coordinate-specific functionals of P and Q, respectively, are of interest. In contrast, the relative effect depends both on P and on Q. In Theorem 1 we adapt the theory derived in [22] to the case that multivariate relative effects are of interest. Thereby, we obtain an asymptotically FWER-controlling resampling procedure based on the statistic W_N or $W_N^U$, respectively.

To this end, let π denote an arbitrary but fixed permutation of the set {1,…,N} and let ${\mathbf{\text{X}}}^{\pi }=({\mathbf{\text{X}}}_{A1}^{\pi },\dots ,{\mathbf{\text{X}}}_{A{n}_A}^{\pi },$ ${\mathbf{\text{X}}}_{B1}^{\pi },\dots ,{\mathbf{\text{X}}}_{B{n}_B}^{\pi })$ be the matrix containing the permuted observation vectors from X = (X_A1,…,X_AnA, X_B1,…,X_BnB). We make the convention that the first n_A columns of X and X^π correspond to group A and the remaining n_B columns to group B. Denote by τ = τ(π,n_A,n_B) the fraction of observations from group B within the first n_A columns of X^π, and let ${\mathbf{\text{p}}}_{AB}^{\pi }={\mathbf{\text{p}}}_{A^{\prime }{B}^{\prime }}$, where Analogously, let ${\widehat{\mathbf{\text{p}}}}_{AB}^{\pi }={\widehat{\mathbf{\text{p}}}}_{A^{\prime }{B}^{\prime }}({\mathbf{\text{X}}}^{\pi })$ denote the estimator of the vector of relative effects based on the permuted data set X^π. A simple calculation yields that ${\mathbf{\text{p}}}_{AB}^{\pi }=\tau (1+n_A/n_B){\mathbf{\text{1}}}_d/2+[(1-\tau (1+n_A/n_B)]{\mathbf{\text{p}}}_{AB}$. Finally, let ${\mathfrak{\text{p}}}_{AB}^{\pi }=\tau (1+n_A/n_B){\mathbf{\text{1}}}_d/2+[(1-\tau (1+n_A/n_B)]{\widehat{\mathbf{\text{p}}}}_{AB}$.

Theorem 1 Under the general setup from above, assume that the sample sizes n_A and n_B fulfill the regularity assumptions given in Lemma 5.3 of [23] as N → ∞. Define the statistic (3) where ${\widehat{V}}_N^{\pi }$ denotes the estimator from (4.6) in [20] applied to X^π. Then, the permutation distribution of $W_N^{\pi }$ (i. e., its discrete distribution induced by letting π be uniformly distributed on all N! possible permutations of the set {1,…,N}, while keeping the data X fixed), the cdf of which we denote by ${\widehat{R}}_N^W$, satisfies A result analogous to Theorem 1 can be obtained for the statistic $W_N^U$. Based on them, an asymptotic null distribution for W_N or $W_N^U$, respectively, is given by its permutation distribution. This permutation distribution, in connection with Remark 1, can be used instead of ${\chi }_{\mid S\mid }^2$ in order to calibrate each test ϕ_S or ${\varphi }_S^{\prime }$, respectively, for type I error control at level α.

Proof of Theorem 1.

We approximate the conditional distribution (given X) of X^π by an asymptotically equivalent unconditional two-groups model. To this end, denote by Z = (Z₁,…,Z_N) a random matrix, the columns of which are stochastically independent such that the first n_A columns are distributed as P′ and the remaining n_B columns are distributed as Q′. Following the argument of Theorem 3.5 in [20] the statistic T_N(Z) has asymptotically a centered d-variate normal distribution with some covariance matrix which is non-degenerate for eventually all large N under our general assumptions. Also, we note that both ${\widehat{\mathbf{\text{p}}}}_{AB}^{\pi }$ and ${\mathfrak{\text{p}}}_{AB}^{\pi }$ consistently estimate ${\mathbf{\text{p}}}_{AB}^{\pi }$. Applying the reasoning of Lemma 5.3 in [23], together with the continuous mapping theorem and Slutsky’s lemma, completes the proof.

Remark 2 In dimension d = 1, a similar Studentized permutation approach has been discussed in [24]; see also [25].

Identification of differentially methylated CpG loci.

The UK Ovarian Cancer Population Study (see [26]) aimed at detecting differentially methylated loci between ovarian cancer cases and healthy controls (GEO accession number GSE19711). To this end, 274 healthy controls were compared with 131 untreated, confirmed ovarian cancer cases. Upon rigid quality control, 264 controls and 124 cases remained in the study. When applying our method, we randomly assigned 176 and 84 controls and cases, respectively, to the screening sub-sample of a two-stage selection approach (cf. [27] and references therein). We applied the univariate two-sample Wilcoxon test at each locus on the screening sample and ranked the resulting p-values in ascending order. The remaining 88 and 40 control and case subjects, respectively, were used for the confirmatory analysis (second step). The ten top-ranked loci from the screening stage were tested for a relative effect unequal 1/2 based on asymptotic critical values from the limit distribution (χ²) and permutation-based critical values (Perm) on the confirmatory group. In Table 6, the results are presented as multiplicity-adjusted p-values. For locus 1 ≤ ℓ ≤ 10, the multiplicity-adjusted p-value denotes the smallest FWER level such that $H_{\ell }^{\prime }$ is rejected by the respective multiple test procedure. With both methods, all ten candidate CpG sites have a multiplicity-adjusted p-value below 5%, an FWER level which is often chosen in practice.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Results for the first real data example.

https://doi.org/10.1371/journal.pone.0125587.t006

Among the ten loci displayed in Table 6, there are two which are associated with the FUT7 gene. In turn, the FUT7 gene encodes the Alpha-(1,3)-fucosyltransferase, see [28]. This enzyme plays a role in connection with the surfaces of granulocytes, monocytes and natural killer cells.

Association of immune cell counts with cancer.

As mentioned in Section “Basic Model” the discussed rank-based methods can be applied under almost no assumptions due to their nonparametric nature. Furthermore, our approach implicitly adapts to the dependency structure in the data via the permutation approach. Hence, it is especially well-suited for situations with highly dependent coordinates, for example resulting from the consideration of derived parameters.

Such a situation was present in [29]. In their study, a set of three pre-identified gene regions was considered. These regions have been shown to be associated with particular cell types. Namely, demethylated Foxp3 is associated with regulatory T-cells (Tregs), CD3 with all T-cells, and GAPDH with all leukocytes. From this, three immune relevant parameters were derived: the number of Tregs, the total number of T-cells (tTL) and the cellular ratio of immune tolerance (immunoCRIT). As the Tregs constitute a subclass of the tTL and the immunoCRIT is the ratio of the two other values, these three parameters are highly dependent. Nonetheless each parameter is immune relevant in its own right.

We assessed the association of the three parameters with a disease indicator for cancer, with cancerogenesis, and with cancer progression. In this context, the evaluation of the individual roles of the parameters had to be investigated. This is because cancer tolerance may be either driven by the immunoCRIT or by its individual components, i. e., the shear amount of Tregs or all T-cells. In addition, it is important to understand, even if the most important part is the immunoCRIT, which of the components drives the change during cancerogenesis. The results are presented in Table 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Results for the second real data example.

https://doi.org/10.1371/journal.pone.0125587.t007

Our data indicate a statistically significant role of all three parameters with respect to all three endpoints, with the exception that the Treg parameter is not significantly associated with cancer progression. Thus, our multiple permutation test confirms the notion that manifestation of cancer is strongly associated with a shift in immune tolerance as monitored by Tregs, overall T-cells and the immunoCRIT. Notably, the change of the overall immunological tolerance from healthy towards cancer tissue is driven by both the number of Tregs and the overall number of T-cells. However, once a tumor is established the continuing increase of immunoCRIT-mediated tolerance along with higher tumor stages is mainly caused by a diminished overall T-cell number and not by Treg increase. Hence, while there is an undoubted dependency among these parameters, the biological mechanisms of cancer development allow for a detachment of these parameters such that individual changes of one of the parameters can be observed and statistically evaluated.