Introduction

The relationship between social choice aggregation rules and non-parametric statistical analysis is well-established [see, e.g., 1, 2, 3, 4, 5, 6, 7, 8, 9]. In a related literature, fundamental relationships between discrete choice and non-parametric statistical analysis have been developed [see 10, 11, 12, 13, 14, 15, 16, 17]. While violations of social choice principles can inform us as to the possible paradoxes present in non-parametric tests, the mapping is imperfect due to the issue of statistical significance. Consider a comoparison of three groups, where the term “group” refers to sampled elements of the same population. If, for example, a violation of transitivity occurs in a set of raw, ordinal data when comparing three groups pairwise, this does not imply that any sort of paradox is evident in the results of a corresponding set of pairwise significance tests. This distinction is studied prominently in Bargagliotti and Greenwell [9].

In a seminal work, Haunsperger and Saari [4] find that “…two or more data sets each may individually support a certain ordering of the samples under [the] Kruskal-Wallis [test], yet their union, or aggregate, yields a different outcome” [18, p. 261]. In other words, the Kruskal-Wallis (KW) test [19] is not necessarily consistent upon aggregation of data. The works of Haunsperger and Saari [4] and later of Haunsperger [18] provide characterizations as to aggregation paradoxes in non-parametric statistics. These works extend a prominent literature concerning the classic Yule-Simpson (association or statistical aggregation) Paradox [see 20 or 21 and, e.g., 22, 23, 24, 25, 26, 27, 28, or 29 for later applications] to the non-parametric realm. The Yule-Simpson Paradox has far-reaching implications within statistics and econometrics. Wardrop [25] demonstrates that the Paradox has an important bearing upon the existence (non-existence) of the hot-hand fallacy and related law of small numbers. Specifically, Wardrop notes that researchers studying whether a hot hand fallacy exists should analyze data at the level of aggregation that (potentially fallacy-prone) fans view the data [see, e.g., Miller and Sanjurjo [30] for recent, related work on the the hot hand fallacy].

Bickel et al. [31] find evidence that the Paradox can influence evaluation of gender bias in graduate admissions data. Specifically, statistically significant results at the program level need not hold when evaluating graduate admissions at the university level. Albers [32] finds a related result in evaluating gender bias in research funding. In light of the Yule-Simpson Paradox, these studies establish the importance of analyzing data at the appropriate level of aggregation, whatever it may be. Charig et al. [33] examine treatment effects for small and large kidney stones, both separately and pooled. Consistent with the Paradox, they find evidence that treatment efficacy reverses from separate to pooled tests.

Further developing the Paradox for non-parametric settings, Haunsperger [18] discovers a computable criterion by which the consistency (upon aggregation) of an ordinal data set (when KW-tested) can be characterized. Bargagliotti [34] finds that Bhapkar’s V test [35] and the Wilcoxon-Mann-Whitney (WMW) test [Wilcoxon 1945, Mann and Whitney 1947]—two other central non-parametric tests—are also not generally consistent upon aggregation. Hao and Houser [38] state, “Both robust and simple to implement, [the WMW test] has gained exceptional popularity among empirical scientists, even social scientists. For example, in 2009 almost half of papers in Experimental Economics used the WMW test. Hao and Houser [38] demonstrate that adaptive procedures for the WMW test can lead to “substantial improvements in the ability to detect differences in locations” [p. 1940]. Specifically, Gastwirth [39] and Hogg et al. [40] develop adaptive ranks tests with such a quality. In some respects, the Yule-Simpson Paradox relates to the power and sensitivity of given statistical tests, and how these qualities (may) relate to sample size. The Paradox also relates to a broader literature on information aggregation and paradoxes in which more information is not always revealed to result in better (expected) outcomes [see, e.g., Kaufmann and Weber [41], Koessler et al. [42], Bennouri et al. [43], Axelrod et al. [44], Hanson et al. [45]]. Of course, information aggregation paradoxes may be symptomatic of the decision-maker, of the information aggregation mechanism, or of both. Bennouri et al. [43] demonstrate that accepted information aggregation mechanisms can vary substantially in terms of effectiveness (i.e., in supporting optimal decisions).

The KW test [see, e.g., 46; 47; 48 for more recent descriptions and generalizations of the test] represents a non-parametric statistical analog of the n(≥2) group rank sum aggregation rule in social choice theory. The WMW test represents a related two-sample rank sum test. The Wilcoxon rank sum test and the Mann-Whitney U test are nominally distinct (rank sum) tests; their respective test statistics are linearly dependent upon one another such that the two tests lead to the same end (in terms of significance results). Bhapkar’s V test, on the other hand, compares categorically-assigned data across groups. That these non-parametric tests fail to attain general consistency upon aggregation is both surprising and problematic. Philosophically, these results are akin to a social aggregation rule that takes as inputs two Yes votes (e.g., from an election governed by a Yes-No voting rule), aggregates them, and returns No as the social choice (outcome). We might rather expect two such primitive data sets to strengthen the case for the original result upon aggregation. That such an expectation is not met for this set of tests suggests not only that aggregated statistical rankings are potentially arbitrary but that the primitive rankings—or any statistical rankings from the test—are themselves potentially arbitrary. That is, even a primitive data set is an aggregation of data subsets from the primitive set’s power set.

While Bargagliotti [34] shows a necessary condition (of the data) to ensure consistent (upon aggregation) results for the KW and Bhapkar’s V tests, respectively, the literature has not determined whether inconsistency upon aggregation is a general paradox among non-parametric statistical tests. Both the title of Haunsperger and Saari (“The Lack of Consistency for Statistical Decision Procedures”) and that of Haunsperger (“Aggregated Statistical Rankings are Arbitrary”) certainly leave open the intriguing possibility of a general result. Selvitella [49] further considers the “ubiquity” of the Yule-Simpson Paradox. Based upon early results within the literature, Haunsperger [18] writes, “Haunsperger and Saari [4] show that this Simpson-like paradox occurs with many, if not most, statistical decision procedures. Because of the results [therein], we must expect KW to be the non-parametric system most immune to such difficulties” [p. 263]. We consider, then, whether general consistency under aggregation is an impossibility among non-parametric tests. Alternative tests, such as the sign test for matched pairs, apply distinct aggregation rules to non-parametric data in order to assign aggregated statistical rankings of groups. Herein, we examine whether the sign test for matched pairs (heretofore, “sign test”) is generally consistent upon aggregation, where the sign test represents another central non-parametric test with a social choice aggregation rule analog. It is important to note that the McNemar [50] test is closely related to the sign test for matched pairs. Specifically, the latter test represents an exact test of the former.

The sign test uses a binomial(n, p = 0.5) test statistic to pairwise rank two groups from inter-group (inter-sample) comparisons of matched-pair elements. The sign test statistic is calculated in a manner that is procedurally-equivalent to the two-candidate case of Borda rule aggregation and to the two-candidate case of majority rule aggregation (the latter two of which are, then, themselves equivalent to one another in the two-group case). That is to say, each paired-element comparison of the sign test yields a mutually exclusive “vote” in favor of one of the two groups. This is equivalent to two-candidate Borda or two-candidate majority rule voting, whereby each voter makes a binary preference comparison of two candidates and (assuming voting is sincere) casts a mutually exclusive vote in favor of her preferred candidate. The two candidates are then ranked according to majority rule type aggregation.

In Section 2 of this paper, we construct a theoretical framework by which to consider whether the sign test is consistent upon aggregation. In Section 3, we utilize this framework to explore the properties of aggregated statistical rankings from the sign test. We ultimately find that statistical rankings from the sign test are consistent upon aggregation, and we present a general proof of the result. That is, we utilize the cumulative distribution function (cdf) of a binomial(n, p = 0.5) random variable to show generally that the aggregation of two or more data sets—each rendering a sign test result of the same quality—can only strengthen the original result (i.e., can only decrease the p-value of the sign test statistic). This result is shown for any number of aggregated (primitive) data sets. We prove an additional result as to the nature of sign test consistency. Namely, the sign test p-value exhibits a greater proportional decrease (in aggregation) as the original (primitive data sign test) p-value itself becomes smaller. In aggregation, then, we expect a greater proportional p-value decline given significant sign test results for the constituent, primitive data (than for insignificant sign test results). This represents something of a strong form consistency result, whereby data aggregation reinforces significant sign test results to a greater (proportional) degree. Lastly, we construct and define a generalized notion of aggregation to consider the aggregation of data sets of different sample sizes. We find preliminary evidence that the sign test is also consistent upon generalized aggregation. Analysis of the generalized form is of great practical importance, as data sets of different sample sizes are often aggregated (e.g., in the case of many longitudinal and unbalanced panel data sets).

The matched pairs nature of the sign test is crucial to the result of consistency upon aggregation. In the aggregation of ordinal or numerical data with pre-assigned matched pairs, data are not aggregated into an overall outcome sequence but, rather, into a larger set of matched pairs. As such, sign test aggregation is reinforcing or consistent rather than potentially inconsistent. We find that it is possible for a non-parametric test to exhibit consistency.

In Section 4, we develop a generic application in which we present a set of ordinal data for two distinct groups. Elements of the data can be paired across group (based, e.g., on some characteristic such as “twin-ness” in a control-treatment study) and analyzed via the sign test (for matched pairs). Alternatively, elements for the respective groups may remain unpaired, and elements can then be aggregated into an outcome or rank-sequence such that overall group rankings are generated from a WMW rank sum test. Within the application, we demonstrate a case of inconsistency upon aggregation for the WMW test. We do so to show readers possible characteristics of an ordinal data that is susceptible to inconsistency. For the same ordinal data, we demonstrate an example of the (proved, general) consistency upon aggregation of the sign test. As such, the procedural differences between the two tests are also highlighted (with respect to the consistency property). For the application, we aggregate sets of ordinal data twice, thrice, and four times. It is shown that the sign test p-value shrinks monotonically in the number of data sets aggregated. This follows from the general results of Section 3. For the same ordinal data, we observe a case in which the WMW test p-value rises monotonically in the number of data sets aggregated (as was shown to be possible by Bargagliotti [34]).

Section 5 identifies and develops an empirical application from the sport of high school team cross country running, a sport that uses rank sum scoring to map from an individual outcome sequence (of individual finishing positions) to team scores. The empirical application uncovers evidence of rank sum “inconsistency” and sign test consistency for high school team cross country meet data. Section 6 concludes.

2 Theoretical set-up and preliminary Lemmas

3 Main results

Theorem 1 Consistency upon Aggregation: Let X_n be a Binomial random variable with parameters n and p = 1/2. Then, with N = rn, the inequality in (1) holds for all integers k ≥ n/2, and integers r ≥ 2.

Proof. In terms of the coefficients introduced in (2) and (3), and where N = rn. The inequality in (1) can be expressed as (9)

We have seen in Lemma 2 that d_k strictly increases as k decreases toward n/2. Thus, the maximum value for the ratio of the two binomial probabilities in (9) is attained with k = (n + 1)/2 for odd n and k = n/2 for even n. We now show that this maximum value is less than 1 for both odd and even n.

When n is odd and n = 2m − 1, we take k = m = (n + 1)/2 in (9). By the symmetry of the binomial probabilities, P(X_n ≥ m) = 1/2.

Now N(= rn) may be even or odd. When N is even, r(≥2) is also even, and rn/2 is the median of X_N. Further, P(X_N ≥ rn/2)>1/2 and P(X_N ≥ rn/2 + 1)<1/2. Thus, where the first inequality holds since r/2 ≥ 1. When N(= rn) is odd, r(≥3) is also odd, and P(X_N ≥ (rn + 1)/2) = 1/2. Thus, where the strict inequality holds since (rn + r)/2 and (rn + 1)/2 are both integers and the difference (r − 1)/2 ≥ 1. Thus, the strict inequality in (9) holds for all k ≥ n/2 when n is odd.

When n is even and n = 2m, we take k = n/2 = m, and note that N is always even. By the symmetry of the binomial probabilities,

Thus, the strict inequality in (9) holds iff (10)

We will now establish (10). Note that X_N has the same distribution as X_n + Y where Y is Binomial with parameters N − n and p = 1/2, and X_n and Y are independent.

Since is largest when i = n/2 = m, so is P(X_n = i). Hence, which establishes (10). That is, (9) holds for all k ≥ n/2 when n is even as well.

A similar result on the lower tail probabilities can be established by symmetry. As such, we have shown that any significant sign test result is consistent upon n-aggregation. This result has important implications for classic results (e.g., by [31] and [33]). If we apply the sign test to the underlying data of these studies, while normalizing the sample size to equality across group and across treatment, it is not possible for an aggregation paradox to occur. Later in the paper, we will consider whether the sign test is susceptible to instances of aggregation paradox given unequal sample sizes across group. We can refine Theorem 1 by observing that we have proved a great deal more. In fact, we have established the following.

Theorem 2 Strong Form Consistency upon Aggregation: Let X_N be a Binomial random variable with parameters N = rn and p = 1/2 where n ≥ 1 and r ≥ 2 is an integer. Then, as the integer k decreases in [n/2, n], P(X_N ≥ rk)/P(X_n ≥ k) monotonically increases with a strict upper bound of 1.

That the ratio is bounded by 1 follows from Theorem 1. Theorem 2 tells us that the n-aggregated data associated p-value rises toward the primitive data associated p-value as the primitive data p-value rises. In other words, n-aggregation creates a greater proportional decrease in the sign test p-value as the original (primitive) p-value itself becomes smaller. In aggregation, then, we expect a greater proportional p-value decline given significant sign test results for the constituent, primitive data (than for insignificant sign test results). This represents something of a strong form consistency result, whereby data aggregation reinforces significant sign test results to a greater (proportional) degree. A more general result of the following form is highly desirable.

Theorem 3 (Conjecture) Let X_N denote a Binomial random variable with parameters N and p = 1/2. Then for positive integers n₁ < n₂ and constant c ∈ [1/2, 1] such that cn₁ and cn₂ are integers, the following inequality holds. (11)

We are able to establish this claim for the boundary values of c (that is, c = 1/2 or 1) using the arguments presented earlier, and conjecture that the result is true for other c values in [1/2, 1]. Limited computational work supports this conjecture. Further, a large-sample approximation, described below, leads us to believe in the conjecture, especially when n₁ and n₂ are large and far apart.

From central limit theorem, as n → ∞, we know that . Further, the binomial distribution, being already unimodal and symmetric (as p = 1/2), the convergence is fast. So, if n is large we have the following approximation: (12) where Φ is the cdf of a standard normal random variable. This “close” approximation to the tail probability strictly decreases as n increases as long as c > 1/2 and hence we believe (11) holds for large n₁ < n₂.

Remark 2. When c = 1/2 and n₁ and n₂ are odd integers, the strict inequality claimed in (11) does not hold and each of the probabilities equals 1/2 as for an odd integer n,

Also, it follows from (12) that, irrespective of the odd or even nature of n, the limiting value of this probability is 1/2.

4 Application 1: Demonstrating aggregation of statistical rankings for Sign and WMW tests

4.2 Tests of aggregated, ordinal data

Now, let us replicate the (ordinality of the) primitive data once, combine the primitive data and its ordinal replicate, and re-test the aggregated data. One should note that an ordinally-replicated data set may be quite different numerically from the primitive (source) data from which it arises such that two ordinally-identical data sets can combine to form various, feasible aggregated sequences. The aggregated outcome sequence consists of 36 rank-ordered elements. One possible aggregated outcome sequence is: where is ordinally equivalent to F_AB. In this aggregated sequence, we have simply spliced the replicate sequence, , into F_AB such that is strung (consecutively) between the penultimate and ultimate elements of the original sequence. The implication of this aggregated outcome sequence is that—for the most part—the numerical values of the original data outrank those of the ordinal-replicate data (despite the identical ordinal quality of the two data sets). From , we find that

The upper-tail WMW rank sum p-value for this sample difference, , is equal to 0.0438. At the α = 0.01 significance level, then, we conclude from a WMW test of the aggregated data that A∼_{α = 0.01} B (i.e., that A and B are rank-indistinguishable at the α = 0.01 significance level), where “∼_{α = 0.01}” reads “is rank-indistinguishable to at the α = 0.01 significance level.”

We now consider a sign test for matched pairs upon the same aggregated data. Specifically, we combine M_AB and its replicate into the same data set to obtain the set ,

As M_AB is an unordered set of matched pairs (i.e., the set of elements is unordered, whereas each element is itself an ordered pair), the aggregation of M_AB and its ordinal replicate, , is also an unordered set of matched pairs. As such, features the same matched pair ordinal comparisons as does M_AB, while representing each such pair twice. Matches for the sign test are pre-assigned at the primitive data level according to the (also pre-assigned) matching criterion (e.g., “twinness”). Therefore, matched pairings are preserved in aggregation. This can be said in a more straightforward manner: The identity of one’s twin (match), once assigned, is invariant to the level of data aggregation. Matched pairing preservation in data aggregation is an important property that causes the sign test to be distinct from independent sample non-parametric tests. As the profile of matched pairings is itself a feature of the primitive data in the case of the sign test (in much the same way that the primitive data outcome sequence is a feature of the primitive data in the case of a WMW test), one would not be strictly aggregating the primitive data if one were to permute the matched pairings in aggregation. Rather, one would thereby change a feature of the primitive data at the same time (in the same manner that a re-ordering of primitive data would go beyond aggregation in the case of a WMW test). Given this feature of the sign test, the matched pair elements of M_AB and , respectively, are not rank-compared with one another (across primitive group) within a sign test analysis of .

Of the eighteen matched pairs in the once ordinally-replicated data, an element of A outranks the corresponding element of B for sixteen matched pairs. Therefore, we have the followingsign-test statistic for the aggregated data:

Then, the upper-tailed sign test for matched pairs p-value for this test statistic value, is equal to 0.000484. Then, we conclude from a sign test for matched pairs of the aggregated data that A ≻_{α = 0.01} B. For the once ordinally-replicated data, the sign test retains significance at the α = 0.01 level. Consistent with our general results, the sign test p-value diminishes with data aggregation in this case. On the other hand, the WMW test loses significance at the α = 0.01 level when applied to the once ordinally-replicated data.

To consider the result of Theorem 2, we alter M_AB as follows.

We also alter as follows.

Aggregating and , we obtain .

Given and , we have the following values for the sign test statistic:

From our previous analysis of M_AB and , we have that:

With respect to Theorem 2, then, we have that

We also have that

This result illustrates the (general) property shown in Theorem 2. Namely, the sign test p-value exhibits a greater proportional decrease when replicated as the p-value for the sign test of the primitive data decreases (i.e., for more significant sign test results of the primitive data). Not only does the sign test exhibit consistency upon aggregation. Sign test results are more strongly reinforced in aggregation—in terms of proportional p-value decrease—as the original sign test p-value (of the primitive data) decreases.

Returning to F_AB, M_AB, , and , we iterate this aggregation to combine three and then four instances of the primitive ordinal data as follows.

The case of 3 Ordinal Replicates Aggregated.

We now take where is ordinally equivalent to F_AB and therefore to , and

For this case, the WMW test p-value is 0.122, and the sign test p-value is 0.000027. The WMW test p-value has risen once again in level of aggregation, whereas the sign test p-value has once again fallen.

The case of 4 Ordinal Replicates Aggregated.

Now take where is ordinally equivalent to F_AB, , and therefore to , and

For this case, the WMW test p-value is 0.209, and the sign test p-value is less than 0.00001. The WMW test p-value has risen once again in level of aggregation, whereas the sign test p-value has once again fallen. The overall significance testing results for the primitive and aggregated data are represented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Application 1 statistical test p-values.

https://doi.org/10.1371/journal.pone.0228627.t001

Why does the WMW test display inconsistency upon aggregation?

Boudreau et al. [52] explore the social choice properties of rank sum scoring. They find that rank sum scoring approaches may invite various, related (social choice) paradoxes when a group with a superior rank sum “falls behind” (i.e., has fewer elements represented) at some point in the outcome sequence. In the application, group A has a superior rank sum in the primitive outcome sequence. However, group A also possesses the weakest overall element in the primitive outcome sequence. This makes group A’s significant rank superiority in the primitive outcome sequence vulnerable upon aggregation. In the aggregated data, if sufficiently many elements position after element b₉ but before element a₉ (as did occur), this particular interposition disproportionately inflates group A’s rank sum score. Such an outcome is not possible for matched pairs ordinal data aggregation. In that case, pairs remain assigned such that the primitive ordinal data and its replicate(s) reinforce one another.

5 (Empirical) Application 2: The aggregation paradox in ranking cross-country running team performance

As in Hammond [51] and Boudreau et al. [52], we use the setting of team cross country races to understand aggregation properties of ordinal data. In a cross country running meet, rank sum scoring is used as the official scoring methodology to assign team rankings. Boudreau et al. [52] describe the scoring methodology in detail:

In cross country running, for example, teams are compared on the performances of individual runners in a race; the outcome sequence is the overall ranking as to how runners finish. Rank sum scoring means that each group (team) receives a number of points equal to an element-wise ranking in the outcome sequence (1 for 1st place, 2 for 2nd, and so on), and comparison of groups is based on the sum of their element scores: groups with lower scores rank above those with higher scores [p. 220].

In the present application, we have identified two high school cross country programs for comparison: Carmel High School (Carmel, Indiana) and Fishers High School (Fishers, Indiana). We consider these two schools because their respective Varsity (Boys) teams competed against one another during the 2016 season, as did their respective Junior Varsity (Boys) teams. The two meets—the Varsity Indiana High School Athletic Association Noblesville Sectional meet and the Junior Varsity Hamilton County meet—occurred on the same running course (White River Elementary School Course; Noblesville, Indiana) such that the aggregation of the two data sets is valid (i.e., runners across meets faced the same basic performance elements). Moreover, Carmel’s Varsity and Junior Varsity teams won each respective meet by the very same convincing margin when rank sum scored against Fishers without the consideration of third-party teams (as in the WMW test). That is to say, respective pairwise rank sum scores were the same for each school in each race. Lastly, this particular setting was chosen because both the Varsity and Junior Varsity meets have been publicly recorded on Athletic.net (from which we obtained the data). It is typically difficult to find publicly-available Junior Varsity meet results. The results of these two cross country meets can be found at https://www.athletic.net/CrossCountry/meet/119851/results/481308 and at https://www.athletic.net/CrossCountry/Results/Meet.aspx?Meet=119845&show=all respectively.

This setting provides something of a natural experiment by which we can compare the two Cross Country programs at different levels of aggregation (i.e., a comparison of Varsity teams, a comparison of Junior Varsity teams, and an aggregated comparison of the two programs across the levels of competition) under the same basic competitive conditions. From the finishing time results, we obtain the observed primitive outcome sequence for the Varsity meet as where F_V(C, S) symbolizes the observed Varsity race outcome sequence for our two Varsity teams, C = {c₁, c₂, c₃, c₄, c₅, c₆, c₇} is a set representing the 7-element (7-runner) Carmel Varsity team, and S = {s₁, s₂, s₃, s₄, s₅, s₆, s₇} is a set representing the 7-element (7-runner) Fishers Varsity team. The resulting rank sum scores are and the upper-tail WMW rank sum p-value for this sample difference, , is equal to 0.019. In a WMW rank sum test of the Varsity data, then, we conclude that C≻_{α = 0.025} S. At the α = 0.025 significance level or any higher α-level, that is, we conclude that Carmel’s 2016 Varsity team was faster than Fisher’s 2016 Varsity team.

For the Junior Varsity meets, the observed race outcome sequence is given by where symbolizes the observed race outcome sequence for our two Junior Varsity teams in the Junior Varsity race, is a set representing the 7-element (7-runner) Carmel Junior Varsity team, and is a set representing the 7-element (7-runner) Fishers Junior Varsity team. This leads to rank sums

We see that the rank sum scores for the two teams are the same for the Varsity and Junior Varsity meets. Then, the upper-tail WMW rank sum p-value for the Junior Varsity sample difference, , is also equal to 0.019. In a WMW rank sum test of the Junior Varsity data, then, we conclude that C≻_{α = 0.025} S. At the α = 0.025 significance level or at any higher α-level, that is, we conclude that Carmel’s 2016 Junior Varsity team was significantly higher in quality than Fisher’s 2016 Junior Varsity team.

One might ponder the utility of ranking teams at the Junior Varsity level. Junior Varsity competition often provides a leading indicator of a program’s future strength. Further, the quality of a Junior Varsity team can indicate program depth and robustness against injury. We might therefore expect a strong overall program to be strong at each level.

Using finishing times, we pool the two outcome sequences to create the aggregated sequence and obtain

As with its constituent outcome sequences, represents the true (observed) outcome sequence when the two meets (races) are pooled and runners are sequenced in ascending order of finishing time. Given the observed primitive race sequences, the pooled (aggregated) outcome sequence is simply the 14 Varsity elements (runners) in their primitive order followed by the 14 Junior Varsity elements. In other words, the first element in outcome sequence F_JV follows the final element in outcome sequence F_V. Each meet represents a distinct level of competition by which these two cross country programs are ranked against one another.

Given that and , the WMW rank sum p-value for this sample difference, , is equal to 0.069. In a WMW rank sum test of the aggregated data, then, we conclude that (C∼_{α = 0.025} S), where “∼_{α = 0.025}” reads “ranks indifferently to at the α = 0.025 significance level.” In fact, we also find that (C∼_{α = 0.05} S). At the α = 0.05 significance level or any smaller α-level, that is, we conclude from the aggregated data that Carmel’s 2016 Boys Cross Country program was not significantly different from that of Fishers. We obtain this result despite finding significant evidence that each of Carmel’s Boys Cross Country (primitive or constituent) teams (i.e., the Varsity and Junior Varsity teams) separately ranks significantly higher than its respective counterpart team at Fishers High School. In other words, we find an empirical example of an aggregation paradox when comparing these two Cross Country programs. This aggregation paradox is fairly extreme. Not only does the test of aggregated data lose significance at the α = 0.025 level but also at the α = 0.05 level. Further, the p-value for the aggregated data is more than the sum of the p-values for the two constituent data sets!

Of course, there are alternatives in sport to rank sum scoring. For example, team tennis matches typically employ a methodology that is consistent with the sign test methodology. For a dual team tennis match, each team assigns an intra-squad rank to each player or set of players, and each player for a given team is match paired to play against the player on the opposing squad who possesses the same intra-squad rank. Matches then take place and a team winner is assigned based on majority rule aggregation of individual match results. We can apply such a scoring rule to the case of team cross country. More specifically, we can apply this sign test type team tennis scoring to the application data from the two high school cross country meets considered.

In sign test type team tennis scoring, each Carmel runner defeated his assigned pair on the Fishers team (for each of the two competition levels). As such,

Then, we have that . By aggregating the two data sets, we obtain and

Then, we have that . The sign test p-value diminishes in level of aggregation for the empirical application. Whereas the results of the WMW test are inconsistent upon aggregation in this empirical application, we again observe an instance of the general consistency upon aggregation quality of the sign test.

6 Application 3: Aggregation properties of the binomial test for equality of dependent proportions

McNemar’s test for the equality of proportions of matched pairs considers the discordant pairs with responses (Yes, No) and (No, Yes). Let n be the total number of discordant pairs, and X be the number of discordant pairs with response (Yes, No). Then the test statistic is given by , and the null hypothesis is rejected if T² is too large (see for example, Rosner ([53], p. 375)). Under the null hypothesis, conditioned on n X has a Binomial distribution with parameters n and p = 1/2. When n is small (say under 20) the exact Binomial distribution of X is used ([53], p. 377) to compute the p value and the null hypothesis of equality of proportions is rejected when X is too small or large. When n is large, chi-square approximation with 1 degree of freedom is used and the upper tail probability associated with observed T² is used as the p value. Thus, when number of discordant pairs in two data sets are equal, say n each, we can use Theorem 2 to establish strong form consistency upon aggregation of the test. When the number of discordant pairs vary, at least when the n_i are large, (11) can be used to establish consistency upon aggregation.

7 Application 4: Computation of Yule-Simpson Paradox for a case of rank sum scoring

Here, we consider a case of rank sum scoring in which two groups, each with two elements, are compared. The outcome of the comparison is a 4-element sequence (e.g., a, b, b, a). There are 4!/(2!2!) = 6 such sequences. For each sequence, we replicate that same sequence and pool the two ordinally-identical data sets to create every possible pooled sequence of eight elements for the two groups that preserves the within-sample ordering for each original sequence. For example a, b, b, a can be replicated and pooled with its replicate, a′, b′, b′, a′ to create a′, b′, b′, a′, a, b, b, a or, alternatively, a, b, b, a′, a, b′, b′, a′. For each of the six sequences in this case, there are 70 possible poolings. This can be easily verified via a decision tree. Essentially, there are up to 5 “bins” in which to place elements of the replicate data set into the original data set, but the number of bins for a given element is constrained because one cannot re-order the original, constituent sequences. A decision tree shows that there are possible poolings for each original sequence. Across all 6 original sequences, there are 70 ⋅ 6 = 420 possible poolings. We find that 72 of these poolings (17.14 percent) result in an instance of Yule-Simpson Paradox of one of the following type: R_c(A)≤R_c(B) but R_p(A)>R_p(B) or R_c(A)≥R_c(B) but R_p(A)<R_p(B). That is, one group has at least as low a rank sum score as the other in the constituent data set, but this is not the case in the pooled data set. This result suggests that instances of Yule-Simpson Paradox are fairly regularly occurring for small sample cases. Qualitatively, the result is in line with previous small sample results on Yule-Simpson Paradox incidence for other tests (see [28]; [54]).

8 Conclusion

In this work, we have used social choice theory and non-parametric statistical theory to examine the Yule-Simpson Aggregation Paradox as it applies to non-parametric statistical tests. As discussed, the Paradox relates to a broader literature concerning the effectiveness of information aggregation mechanisms. Herein, we have shown that the sign test for matched pairs exhibits general consistency upon aggregation. This is the first non-parametric test for which this property has been demonstrated generally, while several non-parametric tests have been shown in the prior literature to not possess this property. This result illustrates that there exists a non-parametric statistical test for which consistency upon aggregation is possible. In the present work, moreover, we find paired matching of ordinal data (across groups) to be important toward the result of general consistency. We prove an additional result as to the nature of sign test consistency. Namely, the sign test p-value exhibits a greater proportional decrease in aggregation as the original (primitive data sign test) p-value itself becomes smaller. In aggregation, then, we expect a greater proportional p-value decline given significant sign test results for the constituent, primitive data (than for insignificant sign test results). This represents something of a strong form consistency result, whereby data aggregation reinforces significant sign test results to a greater (proportional) degree.

Incorporating a generalized form of data aggregation, we also generate preliminary evidence that the sign test possesses a generalized form of consistency upon aggregation, in which the primitive data sets have varying sample sizes. For tractability, n-aggregation has been incorporated as a standard approach in examining the Yule-Simpson Paradox for ordinal data. However, analysis of the generalized, n_i − aggregation form is an important consideration. Often, data of different sample sizes are pooled (e.g., in the case of many longitudinal and unbalanced panel data sets). While we believe that the result holds for the n_i − aggregation problem as well, our proof is incomplete; we can formally claim the consistency only for large samples (see Theorem/Conjecture 3). With the n − aggregation form, we proved the consistency for any n by directly comparing P(X_n ≥ k) and P(X_N ≥ rk) where N = rn. This approach does not seem to work when the n_i are unequal. We focused on the p = 1/2 case, as our interest was on the p − value, or on the property of the test under the typical null hypothesis. If one is interested in the power properties, we need similar results for p ≠ 1/2. Using the normal approximation, we can conclude that the inequality stated in (11) holds for c > p when n₁ and n₂ are large and sufficiently apart.

We further incorporate a generic application that demonstrates tests of the same (aggregated) data by the sign test and by the WMW rank sum test, respectively. In the example, the WMW test results exhibit inconsistency upon aggregation. Sign test results demonstrate (the predicted) consistency upon aggregation, however. In the application, we further verify that the sign test results exhibit “strong form consistency” in aggregation.

An empirical application obtained from Indiana Boys High School Cross Country running data demonstrates a real-world application exhibiting both rank sum inconsistency and sign test consistency. Future work might evaluate alternative matched pairs style non-parametric tests to determine if there is a family of such tests possessing consistency upon aggregation. Indeed, the property is of central importance. As all data sets can be viewed as a potential aggregation of primitive data sets (i.e., from the power set of the aggregated set of data), consistency tells us generally whether a given statistical test evaluates data in an unambiguous manner.