Background

Sex is a biological attribute that can strongly impact organisms’ traits, with differences between males and females being central to questions in the biological sciences [1,2]. In contrast, biomedical research has primarily focused on male subjects [3], posing a danger to female health [4,5]. Aware of these issues, the US National Institutes of Health and other health agencies have demanded using multiple sexes in animal studies when possible [6]. As a consequence, the number of biological and biomedical studies using both female and male animals as research subjects has increased in the last decade [7], leading to the accumulation of data that can be used to synthesize and quantify sex differences across biological domains.

Realizing the accumulation of sex-specific data, many perspective pieces have encouraged researchers to investigate sex differences more carefully [8–10]. Yet, some of these pieces, and most of the biological literature, focus exclusively on mean differences between males and females. A fixation on mean differences has been present for a long time in science because researchers tend to focus on dimorphism in trait averages [11], lack sufficiently powerful data, or have limited statistical tools available (or difficulty using them). Yet, measures such as variance, correlation, skewness, and kurtosis can be critical to understanding sex differences. For example, certain traits in mice may exhibit no disparity in average values between sexes, but substantial differences emerge in terms of variability [12,13]. These differences could be more easily assessed because of an effect size statistic that measures differences in variability between two groups (proposed by [14]), illustrating how novel statistical tools can expand possible research questions and provide new scientific insights, such as identifying sex differences in trait selection or canalization.

Beyond variability, the relative shape of trait distributions to the normal distribution (measured by skewness and kurtosis, i.e., asymmetry of a distribution and heaviness of a distribution’s tails, respectively; Fig 1A and 1B) can also be crucial to understanding ecological and evolutionary processes and patterns [15–19], as well as improving medical diagnostics [20,21]. For instance, skewness can bias heritability estimates because evolutionary biologists assume that phenotypic components (genetic and environmental) are normally distributed [18]. Furthermore, kurtosis can be used to understand community assembly processes [16]. Besides the shape of trait distributions, evolutionary biologists and quantitative geneticists can quantify correlation matrices to understand trait plasticity and evolvability [22–24], which could then be used for group comparisons (as in [25]; Fig 1C). Although location-scale-shape models [26–28] may be used to explore between-group differences (e.g., males and females) in skewness, kurtosis, or within-group correlations, there are no effect size statistics that can easily measure such differences (but see also [29]).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Simulated trait distributions for two groups with different shapes (A: distinct skewness, B: distinct kurtosis), and different correlations between two traits for two groups (C).

The data and code needed to generate this Figure can be found in https://zenodo.org/records/18386956.

https://doi.org/10.1371/journal.pbio.3003653.g001

Here, we propose three new effect size statistics to evaluate between-group differences in skewness (Δsk), kurtosis (Δku), and correlation (ΔZr), key moments of a distribution that are usually unexplored. These effect size statistics will be valuable to explore sex differences but can also be applied in other fields of study and used to compare differences between any two groups of interest. Meta-analyses using these new effect sizes will create multiple avenues for novel biological enquiries. The present moment is particularly conducive for analyses using these new effect sizes because the individual-level data (e.g., individual participant data [30,31]) required for their calculation are increasingly available from new technological advances that allow faster data collection and sharing (e.g., automated phenotyping).

Difference in skewness and kurtosis

The mean and variance represent the first and second moments of a distribution, respectively. However, the third and fourth moments of a distribution (i.e., skewness and kurtosis, respectively) can also be valuable as they characterize the distribution’s shape. More specifically, skewness reflects the distribution’s asymmetry around its mean. While positive skewness indicates an elongated right tail with an excess of high values, negative skewness suggests an elongated left tail with an excess of low values. This asymmetry can influence the interpretation of means and variation, as the mean tends to be larger than the median in positively skewed distributions, while the mean tends to be smaller than the median in negatively skewed distributions. Note that a perfectly normal distribution is symmetric (i.e., skewness = 0), where the mean is equal to the median. Sample skewness (sk) [32] can be expressed as:

(1)

where x_i is a raw data value, x̄ is the sample mean, and n is the sample size. Skewness sampling variance (s²_sk) [32] can then be expressed as:

(2)

On the other hand, kurtosis measures tail heaviness: high kurtosis distributions have heavier tails (i.e., proportionally more extreme values than central values), whereas low kurtosis distributions have lighter tails. For comparison, a normal distribution is expected to have kurtosis = 3. Sample excess kurtosis (ku) [32] can be expressed as:

(3)

with sampling variance (s²_ku) [32] as:

(4)

Evaluating skewness and kurtosis provides valuable insights into a variable distribution, which is crucial for interpreting means, assessing variability, and making informed decisions in statistical analyses. Although meta-analyses can use skewness (Eq 1) and kurtosis (Eq 3) to investigate single variables, effect size statistics that compare these metrics between two groups are lacking. Thus, we propose the difference between two groups in skewness (Δsk), expressed as:

(5)

and its sampling variance (s²_Δsk) as:

(6)

where ρ_sk represents the sampling correlation in skewness between the two groups (zero if assumed to be independent). Similarly, we propose the difference between two groups in kurtosis (Δku), expressed as:

(7)

and its sampling variance (s²_Δku) as:

(8)

where ρ_ku represents the sampling correlation in kurtosis between the two groups (zero if assumed to be independent).

However, we note that Eqs 2 and 4 assume normality for sampling variances. When the underlying distributions are skewed or heavy-tailed, sampling error variances for skewness and kurtosis (Eqs 2 and 4) and, by extension, for their between-group contrasts (Eqs 5–8), can misestimate uncertainty. To assess robustness and to provide distribution-free alternatives, we complemented the analytic formulas with resampling-based estimators computed within each group and summed for the difference (i.e., jackknife [33]; see our simulation study below).

Difference in correlation

Numerous meta-analyses estimate the correlation between two variables [34,35]. To do so, researchers use the effect size statistic Zr [36], which can be expressed as:

(9)

and its sampling variance (s²_Zr) [36] as:

(10)

where r is Pearson’s correlation coefficient between two variables and n is the sample size.

Although Zr alone remains extremely useful to test correlational hypotheses, researchers from all fields would benefit from being able to compare Zr values between two groups. Although Cohen [37] proposed the difference between two groups in Zr as q, he did not provide an equation to calculate its sampling variance. Consequently, this effect size statistic has not been used despite its potential. We therefore propose the difference between two groups in Zr with a new name (ΔZr), as:

(11)

and its sampling variance (s²_ΔZr) as:

(12)

where ρ_Zr represents the sampling correlation in Fisher’s Zr between the two groups (zero if assumed to be independent).

Simulation study

We conducted Monte-Carlo simulations to evaluate bias and variance estimation for our new effect sizes Δsk, Δku, and ΔZr. For Δsk and Δku, we simulated independent samples for two groups from Pearson distributions with known moments using the rpearson function from the R package PearsonDS v. 1.3.2 [38]. We conducted two simulations: (1) by changing skewness between groups that involved moderate departures from normality in which group-specific skewness from sk ∈ {−1, −0.5, 0, 0.5, 1} and kurtosis was fixed at 3; (2) by holding skewness constant (sk = 0) while manipulating kurtosis from ku ∈ {2.5, 3, 4, 5, 6}. In all cases, we simulated scenarios where: (i) the variance between each group was the same ( =  = 1) or different ( versus ); (ii) the mean between the two groups was the same (u₂ = u₁ = 0) or different (u₂ = 5, u₁ = 0). For simplicity, we assumed equal sample sizes between groups with sample size varying from n ∈ {10, 20, …, 100, 150, 500}. We created all unique combinations of the above scenarios resulting in 1,200 independent scenarios (when considering each of the 100 scenarios at each sample size). We estimated Δsk and Δku for each scenario using formulas for within-group sample skewness with small-sample correction (Eq 1) and excess kurtosis with small-sample correction (Eq 3) to estimate point estimates. To estimate associated sampling variance for Δsk and Δku we used the analytical variance estimators derived here (Eqs 2 and 4) and an associated re-sampling (jackknife) approach to compute group sampling variances separately followed by pooling. Importantly, our simulations assume no correlation between groups.

For ΔZr simulations, we simulated two groups each containing two variables with known correlations within each group. For ΔZr, we drew bivariate normal data with target within-group correlations r ∈ {−0.8, −0.4, −0.2, 0, 0.2, 0.4, 0.6, 0.8} using the mvnorm function from the package MASS v. 7.3.61 [39]. Marginals were standard normal and group sizes varied from n ∈ {10, 20, …, 100, 150, 500}. We created all unique combinations of scenarios resulting in 768 unique scenarios. We estimated ΔZr using Fisher’s Z transformation Zr and calculating ΔZr as the difference of Zr across groups (Eqs 9–11). Sampling variance for ΔZr used Eq 10 and a jackknife approach. Again, we assumed no correlation between our groups.

Note that our simulations did not explore differences in sample size between groups. However, many groups being compared in meta-analyses have the same or very similar sample size. Additionally, simulations often show relatively small impacts of unbalanced sample sizes [40,41], which is why we originally did not vary sample size between groups in our simulations.

We resampled 2,500 times for each scenario across all simulations. Performance metrics were (a) bias of the point estimator, (b) relative bias of the sampling-variance estimator, (c) coverage (95%), and (d) Monte-Carlo standard errors (MCSEs). See Supporting information for full formulas. We also evaluated the performance of these effects for meta-analysis (see details in Sections 8.4 and 9.4 of the Supporting information).

Simulation results

In all cases, we found the Monte Carlo standard error (MCSEs) to be low for all our performance metrics (range of MCSEs for Δsk: 0 to 0.01; Δku: 0 to 0.624; ΔZr: 0 to 0.004). Δsk, Δku, and ΔZr point estimators exhibited small sample bias with less than 20–30 samples, except for Δku, which showed this bias below n < 50–60, indicating effect sizes involving kurtosis are more challenging to estimate (S1 and S2 Figs). Differences in the mean and variance between groups did not differentially affect bias (S3 Fig). Regardless, small sample biases were moderate, and there was rarely a consistent over- or under-estimation in point estimates across the scenarios evaluated (S1 Fig). Bias-corrected jackknife estimates reduced the small-sample bias relative to analytical bias corrected-moment estimators (mean square bias, jackknife, and analytical, for Δsk: 1.109, 3.375; Δku 477.71, 891.659; ΔZr 0.029, 0.214).

In contrast to point estimators, the effectiveness of sampling variance estimators for Δsk, Δku, and ΔZr varied. Analytical sampling variance formulas for Δsk and Δku were consistently biased (S4 Fig). Jackknife resampling when combined with analytical point estimates (Fig 2) performed the best. Under these conditions, estimators performed well when n > 50. In contrast, the performance of sampling variance estimators for ΔZr was best when using the analytical formulas for both the point estimator and its associated sampling variance (Fig 2).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Bias in Δsk, Δku, and ΔZr effect estimates (A, D, G), relative bias in their sampling variance using jackknife-based approximation (B, E, H), and coverage of effect estimates (C, F, I) across simulations where samples ranged in group sample sizes between n ∈ {10, 20, …, 100, 150, 500}.

A total of 100 simulated scenarios were assessed for Δsk and Δku whereas 64 simulated scenarios were assessed for ΔZr. We ran 2,500 simulations for each scenario. For simplicity, we only present results from our recommended point estimators and sampling variance estimators using jackknife. See supplementary material for full simulation results. The data and code needed to generate this Figure can be found in https://zenodo.org/records/18386956.

https://doi.org/10.1371/journal.pbio.3003653.g002

Coverage was close to nominal (95%) for Δsk and ΔZr across sample sizes (Fig 2C and 2I). Coverage for Δku, however, was poor across many simulated scenarios (Fig 2F). Increased sample size did not improve coverage. Poor coverage was the result of skewed sampling distributions from Jackknife approaches (S5 and S6 Figs). At small sample sizes, Δku was estimated poorly when true Δku was high, leading to non-skewed distributions with good coverage. In contrast, large sample sizes improved point estimation of Δku when differences existed, but the sampling distribution became highly skewed leading to poor coverage (S5 and S6 Figs). These problems stem from the fact that the standard error formula for kurtosis assumes normality (see [42]).

Considering these simulation results, we suggest pairing the formula-based point estimators for skewness (Eq 1) and kurtosis (Eq 3) with jackknife standard errors for Δsk and Δku. For ΔZr, the standard analytic variance is recommended (Eqs 9–12). This choice balances efficiency under normality with robustness to realistic deviations from it and aligns with our broader guidance to avoid very small group sizes for these statistics. Given the challenges in estimating Δku, and the poor properties of its sampling variance [42], we recommend weighted meta-analytic models using sample size instead of sampling variance (see Supporting information and [41]).

Worked examples: Sex differences in mice

To illustrate the application of our proposed effect size statistics, we used data compiled by the International Mouse Phenotyping Consortium (IMPC, version 18.0; [43]; http://www.mousephenotype.org/). We examined differences between male and female mice in two pairs of traits from distinct functional domains: morphology (fat mass and heart weight) and physiology (glucose and total cholesterol). We selected these traits because they are widely understood traits, even by non-specialists, and had a large sample size (more than 10,000 individuals measured). More specifically, we assessed differences between the sexes in mean (using the natural logarithm of the response ratio [44], hereby lnRR), variability (using the natural logarithm of the variance ratio [14], hereby lnVR), skewness (using Δsk), and kurtosis (using Δku) for each trait, as well as in the difference in correlation for each trait pair (using ΔZr). The IMPC dataset contains data from multiple phenotyping centers and mice strains, so we selected the ones with the most data points for our analyses here, computing the aforementioned effect size statistics separately for each one of them.

We performed a meta-analysis for each effect size statistic to obtain a mean effect size for each trait (or pair of traits, in the case of ΔZr), using “effect size ID,” “phenotyping center,” and “mice strain” as random factors in meta-analytical models (due to substantial heterogeneity, Table 1). In the case of Δku, we fitted a weighted meta-analytic model using sample size instead of sampling variance (see previous sections and [41]). In all these analyses, positive effect sizes denoted a greater estimate (mean, variability, skewness, kurtosis, or correlation) for males than females. We conducted all statistical analyses in the software R 4.5.1 [45]. We used the functions moment_effects and cor_diff, which have been incorporated into the package orchaRd v. 2.1.3 [46], to compute Δsk, Δku, and ΔZr. We fitted meta-analytical models using the rma.mv function from the package metafor v. 4.8-0 [47]. All methodological details and additional information can be found in our tutorial, at https://pietropollo.github.io/new_effect_size_statistics/.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Heterogeneity estimates (I²) for each meta-analytical model fitted in our study.

https://doi.org/10.1371/journal.pbio.3003653.t001

We found that males, on average, had greater fat mass and heart weight than females regardless of phenotyping center and mice strain (Fig 3A, 3B, 3F, and 3G). The variability among individuals regarding these traits was also greater for males than for females, except for fat mass from one specific phenotyping center and mice strain (Fig 3C). By contrast, females had a similar skewness in fat mass and heart weight compared with males (Fig 3D and 3I). However, Δsk values for fat mass and heart weight varied across phenotyping centers and mice strains, with negative and positive values present (Fig 3D and 3I). Sex differences in kurtosis for fat mass and heart weight followed a very similar pattern to the one described for skewness: Δku values overlapping zero with some variation across individual effect sizes (Fig 3E and 3J). Moreover, the correlation between fat mass and heart weight was, on average, greater for females than males (Fig 4A and 4B). However, this difference in correlation was absent for some phenotyping centers and mice strains (Fig 4A and 4B).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Examples of morphological sex differences in mice (fat mass, A–E; heart weight, F–J) for various phenotype centers (each with a different color in panels B–E and G–J) and mice strains (each with a different shape in panels B–E and G–J), with the bottom estimate in panels B–E and G–J (turquoise diamond) representing the mean effect size.

A and F show distributions of these traits (scaled by subtracting the mean from each value and then dividing the result by the standard deviation) for males (black with dashed borders) and females (white with solid borders), with the sample size of females and males shown as Nf and Nm, respectively. Panels B–E and G–J show effect sizes (lnRR: natural logarithm of the response ratio; VR: variance ratio; Δsk: difference in skewness; Δku: difference in kurtosis), with their respective point estimate and 95% confidence interval stamped. The data and code needed to generate this Figure can be found in https://zenodo.org/records/18386956.

https://doi.org/10.1371/journal.pbio.3003653.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Relationship between fat mass and heart weight (A, B) and glucose and total cholesterol (C, D) in mice.

Panels A and C show these relationships (with variables scaled by subtracting the mean from each value and then dividing the result by the standard deviation) separately for males (dashed line) and females (solid line), each subpanel representing a different phenotyping center and/or mice strain, with the sample size of females and males shown as Nf and Nm, respectively. Panels B and D then show differences in correlation (ΔZr) between males and females (point estimate and 95% confidence interval stamped), where each color represents a distinct phenotype center and each shape represents a distinct mice strain, with the bottom estimate in each panel (turquoise diamond) representing the mean effect size. Note that panels A and C contain individual data points, which may appear as background shading in cases with large sample sizes. The data and code needed to generate this Figure can be found in https://zenodo.org/records/18386956.

https://doi.org/10.1371/journal.pbio.3003653.g004

We also found that male and female mice were, on average, similar in terms of blood glucose levels (Fig 5A and 5B), although males had higher total cholesterol than females (Fig 5F and 5G). We observed the same pattern regarding the variability of these traits: on average, the sexes were similarly variable in glucose (Fig 5C), but the variability of total cholesterol was greater in males than in females (Fig 5H). Contrasting with morphological traits, sex differences in skewness and kurtosis were mostly absent (Fig 5D, 5E, 5I, and 5J). Lastly, males and females showed a similar relationship between glucose and total cholesterol, albeit this relationship was stronger for males than for females in some instances (Fig 4C and 4D).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Examples of physiological sex differences in mice (glucose, A–E; total cholesterol, F–J) for various phenotype centers (each with a different color in panels B–E and G–J) and mice strains (each with a different shape in panels B–E and G–J), with the bottom estimate in panels B–E and G–J (turquoise diamond) representing the mean effect size.

A and F show distributions of these traits (scaled by subtracting the mean from each value and then dividing the result by the standard deviation) for males (black with dashed borders) and females (white with solid borders), with the sample size of females and males shown as Nf and Nm, respectively. Panels B–E and G–J show effect sizes (lnRR: natural logarithm of the response ratio; VR: variance ratio; Δsk: difference in skewness; Δku: difference in kurtosis), with their respective point estimate and 95% confidence interval stamped. The data and code needed to generate this Figure can be found in https://zenodo.org/records/18386956.

https://doi.org/10.1371/journal.pbio.3003653.g005

Our findings that females have, on average, lower (Fig 3B and 3G), less variable (Fig 3C and 3H), but similar skewness (Fig 3D and 3I) and extreme values (kurtosis; Fig 3E and 3I) of fat mass and heart weight compared with males may contribute to sex-related differences in the development of diseases associated with these traits and their biomarkers (e.g., QTc interval length [48]). Moreover, a stronger relationship between fat mass and heart weight in females than in males (Fig 4B) may represent a greater risk of cardiohypertrophy arising from obesity in the former compared with the latter [49]. Meanwhile, absent or less pronounced sex differences in glucose and total cholesterol (Fig 4) may suggest other sources of variation may contribute to sex differences in the symptomology of diseases associated with these measurements (e.g., [50–52]). Characterizing sex differences in biological traits, as we have done here, can provide new perspectives on evolutionary, ecological, and medical patterns, possibly improving healthcare and environmental interventions.

Limitations

Despite the enormous potential of the effect size statistics we proposed here, they are not free of limitations. For instance, skewness and kurtosis (and therefore the difference in these estimates between two groups; i.e., Δsk and Δku, respectively) are more likely to become extreme with small sample sizes and with variables with few unique values, either because the variable is discrete or because it is naturally constant (e.g., number of vertebrae in mice). We thus recommend that researchers only compute Δsk and Δku for continuous variables with a minimum sample size of 50 for each group (as shown in our simulations). Importantly, we found that Δku variance estimates can be biased in many situations, highlighting that exploring Δku should be a priority for future work. Because of this issue, meta-analyzing Δku requires sample size-based weights instead of the standard sampling variance (see supplementary material and [41]). Lastly, although Δsk, Δku, and ΔZr can be calculated, respectively, from reported skewness, kurtosis, or within-group correlations for different samples, empirical studies rarely report these estimates. Therefore, calculating these effect sizes will probably require raw data, which, fortunately, are now becoming more readily available.

Future opportunities

The effect size statistics proposed in the present study can be useful across the life sciences, social sciences, and medicine. This is because skewness and kurtosis, and consequently differences between any two or more groups in these estimates (i.e., Δsk and Δku), may help researchers to understand epidemiological trends [53], genetic patterns relevant to medical diagnosis [20,21], disruptive selection on quantitative traits [54], body size patterns across individuals [55] and species [56], reproductive patterns [57], regime shifts in ecosystems [58], heritability [18], community assembly processes [16], and possibly many other topics. Meanwhile, comparisons regarding correlations have been used to explore memory processing during sleep [59], physiological patterns in patients with certain medical conditions [60], and selection patterns [22–24], to name a few. Because ΔZr can be used in virtually any comparison between two groups of correlational data, the opportunities for its use are endless. Most importantly, Δsk, Δku, and ΔZr are unitless measures, so they can be meta-analyzed to uncover patterns between two groups (e.g., males and females). Moreover, the growing availability of raw data and big data approaches, facilitated by technological advances, makes these effect size statistics particularly valuable for modern research.

Supporting information

Acknowledgments

We thank Yefeng Yang for his contribution in the early stage of this study.

Declaration of AI use: The authors declare that they occasionally used GPT-4-turbo (OpenAI) to improve the clarity and readability of this work. After using these tools, the authors reviewed and edited the content as needed and took full responsibility for the content of the publication.