2.1 Population-based association studies

The variants at a genotyped locus will usually not themselves have causal effects on the trait, but will instead be in cis-LD with—and thus “tag”—causal variants at nearby loci. Thus, we typically think of the association at a focal genotyped locus as reflecting the direct contributions of a relatively small number of tightly linked loci, L_local, found within tens or perhaps hundreds of kb from the focal locus [46]. For concreteness, we assume some fixed L_local in our analyses. Informally, we intend “local” to capture only those loci that are within a sufficiently short recombination distance of the focal locus that the timescale of recombination between these loci is longer than that by which other processes, such as population structure, assortative mating, and selection, generate LD between more distant loci. We recognize that this definition is somewhat arbitrary, that in practice researchers seldom have a predefined number of “local” SNPs in mind, and that others might prefer alternative definitions (such as the entire chromosome of the focal locus) or to dispense with the idea of “local” altogether. However, we believe our definition is close to what researchers implicitly have in mind when they think of a genotyped locus as “tagging” causal loci. Moreover, many of the processes that we study below generate long-range LD not just within chromosomes but also across different chromosomes; to extend the definition of “tagging” to the entire chromosome of the focal locus, but not to the entire genome, is then also an arbitrary choice.

Under the additive model, the standard interpretation is that a population association study performed at a focal genotyped locus λ provides an estimate of the quantity (2) where p_λ is the frequency of the focal allele at λ, and D_λl is the degree of cis-LD between the focal allele at λ and a causal allele at a nearby locus l∈L_local. It is reasonable to think of this quantity as the “direct effect” tagged by the focal variant at the genotyped locus λ: in the absence of confounding, it can be interpreted as the average phenotypic effect of randomly choosing a non-focal allele in the population and swapping it for a focal allele, where in this hypothetical swap, the causal alleles near the locus are included (though see ref. [44] for how G×E and G×G interactions can complicate such causal interpretations).

We shall interpret systematic deviations of effect-size estimates from Eq (2) as “biased.” This is because our focus is on the estimation of direct effects at individual loci rather than on other applications such as trait prediction. For the same reason, and following standard terminology in statistics, we refer to the causes of these biases as “confounds,” since they are a form of omitted variable bias in the marginal regression at the study locus λ. We recognize that these definitions of “bias” and “confounding” might be less appropriate in some other settings.

Effect-size estimation in a population GWAS is complicated by the presence of environmental and genetic stratification. Under the model in Eq (1), if we perform a standard population association study at locus λ, the estimated effect of the focal allele on the trait Y is (3) where, of the cis- and trans-LD terms, D_λl and are defined in the GWAS sample while and are defined in their parents (S1 Text Section S1.2). V_λ is the genotypic variance at λ, equal to , where F_λ is Wright’s coefficient of inbreeding at λ.

The environmental confound is ; all non-local cis- and trans-LD terms in the study sample (D_λl and ) are direct genetic confounds (Fig 1C and 1D), and all cis- and trans-LD terms among parents of sampled individuals ( and ), together with all trans-LD terms in the study sample (), are indirect genetic confounds.

The direct genetic confounds arise because an allele carried by an offspring at λ is correlated with the alleles that they carry at other loci l∈L (via D_λl and ) that directly affect the trait value. The indirect genetic confounds arise because an allele carried by the offspring at λ—say, the maternal allele—is correlated with alleles carried by the offspring’s mother at other loci ( and ) and alleles carried by their father (as reflected by the trans-LD in the offspring, ). These alleles in the parents can indirectly affect the offspring’s trait value.

Thus, as is now well appreciated, population-based GWASs potentially suffer from many types of confounds [1,2]. In practice, they can be reduced by including principal components—which capture genome-wide relatedness among GWAS participants—as regressors in a GWAS, or by using relatedness matrices in mixed models [15,16]. However, it is often unclear exactly what these methods control for in a given application [1,2], they can be sensitive to sample size [47], and they have been shown to be inadequate in important cases (e.g., [20,21]). When principal components (or other controls) fail to account fully for stratification, then Eq (3) can be interpreted as a decomposition of the remaining, uncontrolled-for confounding in the GWAS. (By the Frisch–Waugh–Lovell theorem ([48], pg. 36), Eq (3) is the estimate one obtains by first regressing the focal-locus genotype on the PCs, collecting the residuals from this regressions—which can be thought of as focal-locus genotype values stripped of whatever signal the PCs captured—and regressing the trait value on these residuals.)

2.2 Within-family association studies

The 2 within-family association study designs that we consider are parent–offspring GWASs and sibling GWASs. Other designs have been proposed to control for genetic and environmental confounding in the estimation of aggregate quantities such as heritability (e.g., [49]), but our primary focus is on the estimation of single-marker effect sizes. We do later turn to the interpretation of polygenic score (PGS) regressions within families.

2.3 Polygenic scores and their phenotypic associations

A current drawback to family-based GWASs is that sample sizes are often small, limiting power to estimate direct genetic effects. Because of this limitation, instead of estimating per-locus effect sizes in family designs, investigators often measure the within-family phenotypic association of a combined linear predictor, a PGS, constructed using effect-size estimates across many loci from a population GWAS. In the sibling-based version of this study design, the difference in siblings’ population-based PGSs is regressed on their difference in phenotypes (e.g., [30,31]). In parent–offspring designs, offsprings’ trait values are regressed on their population-based PGSs, controlling for the midparent PGSs (e.g., [35]), or, equivalently, offsprings’ trait values are regressed jointly on their population-based PGSs constructed separately for transmitted and untransmitted alleles, and the difference in the slopes in this joint regression is taken (e.g., [27]).

When such PGS regressions are used within families for the same phenotype as the population GWAS, a non-zero slope of the PGS is usually interpreted as reflecting the fact that the PGS—despite having been calculated from a population GWAS and therefore subject to many potential confounds—nevertheless does capture the direct genetic effects of alleles. When the PGS for one phenotype is regressed within families on the value of another phenotype, non-zero slopes are often interpreted as evidence that direct genetic effects on the 2 phenotypes are causally related, for example, through pleiotropic effects of the alleles involved.

Suppose that we have performed a population GWAS for trait 1, generating effect-size estimates at a set of genotyped loci λ∈Λ. To construct a PGS for trait 1, these effect-size estimates are used as weights in a linear sum across an individual’s genotype: (10)

In a sibling-based study (the results and intuition below will be the same for a parent–offspring study), the difference between siblings’ trait-1 PGSs, ΔPGS₁, is regressed on the difference in their values for trait 2, ΔY₂ (note that trait 2 could be the same as trait 1). If L is the set of loci that causally underlie variation in trait 2, and β_l are the true direct genetic effects of variants at these loci on trait 2, then the numerator of the slope in this regression can be written as follows: (11)

(see S1 Text Section S2). Note that, while the population-based effect-size estimates depend on cis- and trans-LD, as detailed by Eq (3), the patterns of LD may differ from those in the family study (the term in Eq (11)) if the population- and family-based studies differ in relevant aspects of sample composition.

The intuition for Eq (11) is similar to that for the single-locus effect-size estimate in a sibling GWAS (Eq (7)). The numerator of the difference in slopes of transmitted and untransmitted PGSs in a parent–offspring design takes a similar form to Eq (11).

In the absence of confounding and under some simplifying assumptions, the sibling PGS covariance measures the contribution of each locus included in the PGS to the additive genic covariance between traits 1 and 2 that is tagged by the genotyped variants included in the PGS (see Eq S.38 in S1 Text Section S2). Under these assumptions, the sibling PGS slope therefore does provide a measure of the underlying pleiotropy between the traits.

Interpretation of the sibling PGS slopes is more complicated in the presence of genetic confounding (see Eq S.37 in S1 Text Section S2), which is absorbed into the effect-size estimates (Eq (3)) so that the PGS applies a potentially strange set of weights to the genotyped loci it includes. (A related problem occurs when indirect genetic effects absorbed by the population-based PGS change the interpretation of within-family PGS slopes [51,52].) A non-zero sibling PGS slope still establishes that the trait-1 PGS loci are in systematic signed intra-chromosomal LD with loci that causally affect trait 2. However, it no longer necessarily implies that traits 1 and 2 are causally related via pleiotropy, for 2 reasons. To understand these reasons, suppose that the causal loci for traits 1 and 2 are distinct, i.e., that there is in fact no pleiotropy. First, a SNP included in the trait-1 PGS could tag local variants that causally affect trait 1 but which are also, via sources of confounding such as cross-trait assortative mating, in systematic long-range LD with variants on the same chromosome that causally affect trait 2. Such SNPs will be predictive of sibling differences in trait 2, even though they locally tag only trait-1 causal variants. Second, LD between variants on the same or distinct chromosomes that are causal for trait 1 and trait 2 will cause some SNPs that locally tag trait-2 causal variants to be significantly associated with trait 1 in a population GWAS, and therefore to be included in the trait-1 PGS. These SNPs, since they tag trait-2 causal variants, will be predictive of sibling differences in trait 2.

In summary, in the presence of confounding, non-zero sibling PGS slopes cannot be viewed as de facto evidence for causal relationships between traits.

3.1 Assortative mating

Assortative mating is the tendency for mating pairs to be correlated for particular traits—either the same trait (same-trait assortative mating) or distinct traits (cross-trait assortative mating). For example, humans are known to exhibit same-trait assortative mating for height and cross-trait assortative mating for educational attainment and height (among many other examples, reviewed in refs. [37,55]). Assortative mating generates both cis- and trans-LD: It generates positive trans-LD among trait-increasing alleles because genetic correlations between mates translate to genetic correlations between maternally and paternally inherited genomes, and it generates positive cis-LD among trait-increasing alleles because, over generations, recombination converts trans-LD into cis-LD [9]. (In some cases, assortative mating can generate cis-LD by mechanisms additional to recombination—see ref. [56].)

Constant-strength assortative mating.

If the strength of assortative mating, measured by the phenotypic correlation among mates ρ, is constant over time and there are no other sources of genetic confounding such as population structure, then, for a given pair of loci l,l′∈L, the positive cis-LD D_ll′ will initially be smaller than the positive trans-LD , but will gradually grow towards an equilibrium value equal to the trans-LD (); in this equilibrium, assortative mating generates new cis-LD at the same rate as old cis-LD is destroyed by recombination (S1 Text Section S3.1; [9]).

Therefore, in a population GWAS, effect-size estimates will initially be biased upwards because of positive trans-LD, and the magnitude of the bias will grow over time as positive cis-LD too is generated from this trans-LD (Eq (3); Fig 2). In contrast, in a family-based GWAS, effect-size estimates will initially be biased downwards because the positive trans-LD exceeds the positive cis-LD (Eqs (6) and (7); Fig 2). However, as the cis-LD grows over time towards the value of the trans-LD, the magnitude of the downward bias will shrink, and, in equilibrium, the family-based GWAS will not be confounded by assortative mating (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Assortative mating systematically biases effect-size estimation in population and within-family GWASs, although the bias in within-family GWASs is expected usually to be small.

Here, cross-trait assortative mating between traits 1 and 2 occurs among parents for the first 19 generations, after which mating is random. Assortative mating is sex-asymmetric, with strength ρ = 0.2. Distinct sets of loci underlie variation in trait 1 and 2, with effect sizes at causal loci normalized to 1. Plotted are average estimated effects of the focal alleles at loci causal for trait 1 in population and within-family GWASs on trait 2, for a hypothetical genome with 1 chromosome of length 1 Morgan (A) and for humans (B). Since the traits have distinct genetic bases, the true effects on trait 2 of the alleles at trait-1 loci are zero. The horizontal lines at 0.1 are a theoretical “first-order” approximation of the asymptotic bias in a population GWAS (S1 Text Section S3.1). Profiles are averages across 10,000 replicate simulation trials. Simulation details can be found in the Methods and the code can be found at https://doi.org/10.5281/zenodo.10520811.

https://doi.org/10.1371/journal.pbio.3002511.g002

Under certain simplifying assumptions, we can calculate the average bias that assortative mating induces in a population GWAS in equilibrium, in the absence of other sources of genetic confounding such as population structure (S1 Text Section S3.1). In the case of same-trait assortative mating, effect-size estimates are inflated by an average factor of approximately 1/(1−h²ρ), where ρ is the phenotypic correlation among mates and h² is the trait heritability (for similar calculations, see refs. [22,57]). As an example, the strength of assortative mating for human height has been estimated as ρ~0.25 [58], which, together with a heritability of h²~0.8, implies that effect-size estimates in a population-based GWAS would be inflated by a factor of about 1/(1−h²ρ)~1.25, a 25% amplification.

In the case of cross-trait assortative mating, if assortative mating is directional/asymmetric with respect to sex—i.e., the correlation ρ is between female trait 1 and male trait 2—then assortative mating generates spurious associations between trait 1 and alleles that affect trait 2 (and vice versa). If the loci underlying the 2 traits are distinct, then, in equilibrium, the spurious effect-size estimate at non-causal loci is approximately h²ρ/2 times the effect at causal loci, assuming the traits to have the same heritabilities and genetic architectures (horizontal dashed line in Fig 2; see S1 Text Section S3.1 for relaxations of these assumptions). If cross-trait assortative mating is bidirectional/symmetric with respect to sex, then, in equilibrium, the average spurious effect-size estimate at non-causal loci is approximately h²ρ times the effect at causal loci. Upward biases in effect-size estimates at causal loci are also expected under cross-trait assortative mating, but these are second-order relative to the biases at non-causal loci (S1 Fig).

The systematic over- and underestimation of effect sizes that assortative mating induces in population and family-based GWASs, respectively, will also affect our second measure of interest, the heterozygosity-weighted average squared effect-size estimate (and therefore, also downstream quantities such as SNP heritabilities). In a population GWAS, the presence of trans-LD and the gradual creation of cis-LD under assortative mating will increase the biases in effect-size estimates over time (Fig 2), which will concomitantly increase the average value of (Fig 3; also see ref. [22]). Moreover, cross-trait assortative mating will generate signals of genetic correlations among traits even in the absence of any pleiotropic effects of underlying variants [37]. In a family-based GWAS, the temporary attenuation of effect-size estimates owing to a transient excess of trans-LD over cis-LD under assortative mating will lead to a similar attenuation in the average squared effect-size estimate (Fig 3), although, like the bias in effect-size estimates themselves, this attenuation is expected to be small in humans (Fig 3B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The impact of assortative mating on the average squared effect-size estimate in population and within-family GWASs.

Same-trait assortative mating of strength ρ = 0.2 occurs among parents in generations 0–20; mating is random before and after this period. Under random mating, the average squared effect-size estimates exceed the true average squared effect size (yellow line) because random drift generates chance local LD with causal alleles that inflates the variance of effect-size estimation (e.g., [18]). The magnitude of this variance inflation depends on the GWAS design and sample size, and the effect of assortative mating and its cessation should be judged in reference to it. To guide the eye in this judgment, the faint horizontal lines in (A) and (B) show the average squared effect-size estimate in the last 20 generations of the initial burn-in period of random mating. The inset in (B) zooms in on the sibling GWAS profile, omitting the population GWAS profile for clarity. Profiles are averages across 5,000 replicate simulation trials. Simulation details can be found in the Methods and the code can be found at https://doi.org/10.5281/zenodo.10520811.

https://doi.org/10.1371/journal.pbio.3002511.g003

As shown by Border and colleagues [22,37], the effects of assortative mating on estimates of heritability and genetic correlations described above are not well controlled for by LD Score regression [17,18]. The LD score of a variant proxies the amount of local causal variation the SNP tags, but because assortative mating generates long-range signed LD among causal variants, it causes local causal variants to be in long-range signed LD with other causal variants throughout the genome. Therefore, the slope of the LD score regression absorbs the effects of assortative mating, causing its estimates of heritability and of the degree of pleiotropy to be inflated (S1 Text Section S4). We note that the definition of heritability itself, and its measurement using various study designs, can be complicated by processes such as assortative mating (e.g., [59]).

Assortative mating between traits with different genetic architectures.

An important practical question is how genetic confounding affects the GWAS loci we prioritize for functional follow-up and for use in the construction of PGSs. SNPs are usually prioritized on the basis of their GWAS p-value, which is proportional to the estimated variance explained by a SNP, (where p_λ is the minor allele frequency). The results above assume, in the case of cross-trait assortative mating, that the traits involved have similar genetic architectures (distribution of p_l and α_l at causal loci, and the total number of causal loci). In that case, if there is no pleiotropy between the traits, then while SNPs that tag trait-1 causal loci are predictive of the value of trait 2 owing to LD between trait-1 and trait-2 causal loci, we nonetheless expect the SNPs that tag trait-2 causal loci to be better predictors of trait 2, such that GWAS investigators would primarily pick out SNPs tagging trait-2 causal loci for prioritization and use in PGSs.

However, analysis of human GWASs suggests that quantitative traits can have widely different genetic architectures, with, in particular, substantial differences in the effective numbers of causal loci involved and in the distribution of minor allele frequencies ([60] and references therein). If 2 traits with distinct genetic bases show cross-trait assortative mating, but trait 1 has a denser genetic architecture (fewer causal loci) than trait 2, then the genetic signal of assortative mating—systematic LD between trait-1 and trait-2 causal loci—will be more heavily loaded per-locus onto trait-1 loci than onto trait-2 loci. In a GWAS on trait 2, this will inflate the magnitude of spurious effect-size estimates at SNPs that tag trait-1 loci relative to effect-size estimates at SNPs that tag causal trait-2 loci. In S1 Text Section S3.1, we quantify this effect, showing that, in a population GWAS for trait 2, the average magnitude of spurious effect-size estimates at trait-1 loci is proportional to , where |L₁| and |L₂| are the numbers of loci underlying variation in traits 1 and 2, respectively. Thus, when trait 1 has a denser genetic architecture than trait 2 (|L₂|/|L₁| is large), the magnitudes of effect-size estimates at non-causal trait-1 loci could substantially overlap with those at causal trait-2 loci (as illustrated in Fig 4), potentially causing part of the apparent, mappable genetic architecture of the trait-2 GWAS to actually tag trait-1 loci.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Cross-trait assortative mating for traits with different genetic architectures can generate large spurious effect-size estimates in population GWASs.

Shown, for a population GWAS on trait 2, are estimated distributions of the magnitude of effect-size estimates at loci causal for trait 2 (gray) and at loci causal for trait 1 (greens), under random mating (light green) and after 20 generations of cross-trait assortative mating (sex-asymmetric, of strength ρ = 0.2) for traits 1 and 2 (dark green). Although the true effects of trait-1 loci on trait 2 are zero in these simulations (no pleiotropy), there is sampling noise in effect-size estimation at trait-1 loci under random mating (light green line), so that the mean magnitude of effect-size estimates is shifted away from zero (light green dot; dashed line displays 95th percentile under random mating). In (A), the traits have equal heritability, but the number of loci contributing variation to trait 1 is 10-fold smaller than that for trait 2. Under assortative mating, the magnitudes of the spurious effect-size estimates at trait-1 loci shift significantly rightward (dark green line), coming to overlap substantially with the distribution of effect-size estimate magnitudes at causal trait-2 loci (gray line; the distribution for trait-2 loci does not appreciably differ under random and assortative mating). In (B), equal numbers of loci contribute variation to the 2 traits. In this case, the rightward shift of the distribution of effect-size estimate magnitudes at trait-1 loci is not as pronounced as in (A). Densities are estimated from pooled effect-size estimates from 1,000 replicate simulations. Simulation details in Methods and the code can be found at https://doi.org/10.5281/zenodo.10520811.

https://doi.org/10.1371/journal.pbio.3002511.g004

3.2 Population structure

When a population GWAS draws samples from individuals of dissimilar ancestries, differences in the distribution of causal genotypes, and potentially of environmental exposures, can confound the association study [1,4]. Correcting for confounds due to population structure has therefore been an important pursuit in the GWAS literature [14,23,61].

For concreteness, consider a simple model where 2 populations diverged recently, with no subsequent gene flow between them. Genetic drift—and possibly selection—in the 2 populations will have led to allele frequency differences between them at individual loci. If allele frequencies have diverged at both a genotyped study locus and at loci that causally influence the study trait, these frequency differences will manifest as linkage disequilibria between the study locus and the causal loci in a sample taken across both populations, even if the loci are not in LD within either population. Specifically, if the frequencies of the focal allele at a given locus k are and in populations 1 and 2, then the cis-LD between the focal alleles at the association study locus λ and a causal locus l is (12) in a sample that weights the 2 populations equally, with the superscript (S) denoting that this LD is due to stratification. The trans-LD takes exactly the same form: . From Eq (3), locus l therefore confounds estimation of the direct effect at λ in a population GWAS, by an amount proportional to (13)

These genetic confounds are in addition to environmental confounding that would arise if the environments of the 2 populations alter their average trait values by different amounts.

In contrast, estimates of direct effects obtained from within-family association studies are not genetically confounded, because cis- and trans-LD are equal (Eqs (6) and (7)). Another way of seeing this is to consider that, by controlling for family, within-family GWASs control for the population, and in the scenario considered, by construction, there are no within-population LDs to confound effect-size estimation.

Allele frequency divergence due to drift.

How do the confounds introduced by population structure affect the first of our measures of interest, the average deviation of effect-size estimates from their true values? The answer depends on the source of allele frequency differences between the 2 populations. If the differences are due to neutral genetic drift, they will be independent of each other (assuming causal loci are sufficiently widely spaced) and independent of the direction and size of effects at individual loci. Therefore, the LD induced by these allele frequency differences will, on average, not bias effect-size estimates in a population GWAS: (14) since at any locus k.

However, the LD induced by population structure will inflate the average squared effect-size estimate, and by extension the variance of effect-size estimates (Fig 5). In S1 Text Section S3.2, we quantify this effect for the same simple case of 2 separate populations. We find that the average squared effect-size estimate in a population GWAS is an increasing function of the divergence between the 2 populations (as measured by F_ST), the number of loci contributing variation to the study trait, and the true average squared effect size per locus (see also refs. [38,62]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The impact of population structure and admixture on the average squared effect-size estimate in population and within-family GWASs.

Here, 2 populations are isolated until generation 0, at which point they mix in equal proportions. Initial allele frequencies are chosen independently for the 2 populations, such that allele frequency differences between the populations resemble those that would accumulate over time via random drift. As in Fig 3, the equilibrium value of the mean squared effect-size estimate under random mating is greater than the true mean squared effect size, in both the population and within-family GWAS, owing to linkage disequilibria among causal alleles that arise due to drift. This explains why, in the insets, the blue (population) and red (within-family) profiles do not shrink all the way down to the yellow (true) line after admixture, when mating is random. Note too the difference in scale of the y-axes in the insets: the return to equilibrium is much more rapid under the human genetic map (B) than for a hypothetical genome of one chromosome of length 1 Morgan (A), since, with more recombination, the ancestry-based linkage disequilibria are broken down more rapidly. Profiles are averages across 10,000 replicate simulation trials. Simulation details can be found in the Methods and the code can be found at https://doi.org/10.5281/zenodo.10520811.

https://doi.org/10.1371/journal.pbio.3002511.g005

In contrast, because effect-size estimates from within-family GWASs are not confounded in this model of isolated populations, the average squared effect-size estimate will not differ substantially from its expectation in an unstructured population (Fig 5).

While we have focused on a simple model of 2 isolated populations, the result that within-family association studies are not confounded holds for other kinds of population structure as well. Specifically, we may be concerned that a population GWAS suffers from genetic confounding along some given axis of population stratification. However, the family-based estimates will be unbiased by confounding along such an axis if the maternal and paternal genotypes at each locus are exchangeable with respect to each other along this axis (S1 Text Section S3.2). This requirement will be met in expectation under many models of local genetic drift in discrete populations or along geographic gradients. However, as we will shortly argue, migration and admixture introduce further complications.

3.3 Admixture

When populations that have previously been separated come into contact, alleles from the same ancestral population remain associated with each other in the admixed population until they are dissociated by recombination. If allele frequencies had diverged between the ancestral populations, this “ancestry disequilibrium” can translate to cis-LD between loci affecting a trait [67], potentially confounding GWASs performed in the admixed population. More generally, long-range LD will be an issue when there is genetic stratification and ongoing migration between somewhat genetically distinct groups.

For concreteness, we again consider a simple model where 2 populations have been separated for some time, allowing allele frequencies to diverge between them. The populations then come into contact and admix in the proportions A and 1−A. We assume that mating is random with respect to ancestry in the admixed population.

Suppose that, just before admixture, the frequencies of the focal allele at a given locus k were and in the 2 populations. Then, the initial degree of cis-LD between loci λ and l in the admixed population is given by Eq (12), weighted by the proportions in which the populations admix: (16) see, e.g., ref. [68]. This cis-LD subsequently decays at a rate c_λl per generation, so that, t generations after admixture, (17)

Because we assume that mating is random in the admixed population, the trans-LD is zero in every generation after admixture: . Note that the decay of cis-LD in an admixed population will be slowed if individuals mate assortatively by ancestry, because the trans-LD generated by assortative mating is continually converted by recombination to new cis-LD (as in our assortative mating model above; see ref. [69] for more discussion of this point in the context of population admixture).

3.4 Stabilizing selection

Stabilizing selection—selection against deviations from an optimal phenotypic value—is thought to be common [73], and has recently been argued to be consistent with the genetic architectures of many human traits [60]. By disfavoring individuals with too many or too few trait-increasing alleles, stabilizing selection generates negative LD among alleles with the same directional effect on the trait [10]. Thus, stabilizing selection will attenuate GWAS effect-size estimates at genotyped loci that tag these causal loci.

To quantify these biases, we consider the model of Bulmer [10,11], in which a large number of loci contribute to variation in a trait under stabilizing selection, with the population having adapted such that the mean trait value is equal to the optimum. Under this model, stabilizing selection rapidly reduces variance in the trait by generating, within each generation, negative cis- and trans-LD among trait-increasing alleles; this negative LD is then partially transmitted (because of recombination) to the next generation in the form of cis-LD.

If we make the simplifying assumption that all loci have equal effect sizes, then the equilibrium reduction in trait variance in a given generation, −d* (where d*<0), measured before the action of selection in that generation, can be calculated as a function of the genic variance V_g, the environmental noise V_E, the strength of stabilizing selection V_S/V_P (scaled according to the phenotypic variance V_P), and the harmonic mean recombination rate, , among loci underlying variation in the trait ([11]; S1 Text Section S3.4). Selection within the same generation will reduce the trait variance further, to a degree that can also be calculated as a function of V_g, V_E, V_S/V_P, and (S1 Text Section S3.4; [11]).

Under the assumption of equal effect sizes across loci, we calculate in S1 Text Section S3.4 the average per-locus attenuation bias in effect-size estimates induced by stabilizing selection, . In a population GWAS performed before selection has acted in the sampled individuals’ generation, the attenuation bias is approximately while in a population GWAS performed after selection has acted in the sampled individuals’ generation, the attenuation bias is approximately that is, a factor greater than in the pre-selection population GWAS.

In a within-family GWAS, the average proportionate bias in effect-size estimation is approximately irrespective of whether the GWAS is performed before or after selection has acted (or is ongoing) in the generation from which the sample is drawn. The attenuation bias in a family-based GWAS is therefore smaller than that in a pre-selection population GWAS by a factor of , and smaller than that in a post-selection population GWAS by a factor of .

Thus, the bias in effect-size estimation can be calculated given estimates of the phenotypic variance and heritability of the trait, the harmonic mean recombination rate, and the strength of stabilizing selection (S1 Text Section S3.4). In the Methods, making some simplifying assumptions about the genetic architecture of the trait in question, we calculate an approximate value for humans. Using this value, Fig 6 shows the average proportionate reduction in GWAS effect-size estimates for various strengths of stabilizing selection and heritabilities of the trait. The range of selection strengths was chosen to match that inferred for human traits by Sanjak and colleagues [74].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Stabilizing selection attenutates GWAS effect-size estimates.

The calculations displayed here assume that genetic variation in the trait is contributed by 1,000 loci of equal effect spaced evenly along the human genome. Stabilizing selection is stronger if the width of the selection function scaled by the phenotypic variance, V_S/V_P, is smaller. The placement of the point for human height assumes a heritability of 0.8 and a strength of stabilizing selection of V_S/V_P = 30, as estimated in ref. [74]. Details of these calculations can be found in S1 Text Section S3.4. Note the different scales of the y-axes in (A), (B), and (C).

https://doi.org/10.1371/journal.pbio.3002511.g006

Attenuation of effect-size estimates is larger if stabilizing selection is stronger or if the trait is more heritable. Taking height as an example, heritability is ~0.8, V_P≈7cm², and Sanjak and colleagues [74] estimate a sex-averaged strength of stabilizing selection of V_S/V_P≈30. From these values, we calculate that a pre-selection population GWAS would systematically underestimate effect sizes at loci that causally influence height by about 2.7% on average, in the absence of other sources of LD (Fig 6A), while a post-selection population GWAS would systematically underestimate effect sizes by about 5.2%, on average (Fig 6B). More generally, within the range of reasonable strengths of stabilizing selection inferred by Sanjak and colleagues [74], we calculate average attenutations of pre-selection population-based effect-size estimates of up to 5% for highly heritable traits (h²≈1) under strong stabilizing selection (V_S/V_P≈20), down to 0.25% for less heritable traits (h²≈0.4) under weak stabilizing selection (V_S/V_P≈170) (Fig 6A); the analogous range is 0.5% to 10% for post-selection population-based effect-size estimates (Fig 6B).

Given the estimate , the proportionate bias that stabilizing selection induces in within-family GWASs is expected to be a fraction that in pre-selection population GWASs and that in post-selection population GWASs. Thus, for height, a within-family GWAS would underestimate effect sizes by only about 0.2% on average (Fig 6C).

The quantitative importance of these biases will vary by application. In situations where the goal is gene discovery, for example, 5% to 10% reductions in effect-size estimates are unlikely to flip the statistical significance of variants with large effects on a trait. However, the attenuations in effect-size estimates caused by stabilizing selection are systematic across loci, and therefore could substantially affect aggregate quantities based on these estimates. For example, the range of average reductions in population effect-size estimates calculated above for human traits would translate to reductions in naive estimates of SNP-based heritabilities of between 0.5% and 20% (approximately 5.4% in the case of height using effect-size estimates from a pre-selection population GWAS; approximately 10% using estimates from a post-selection population GWAS). If effect sizes are estimated by within-family GWAS, on the other hand, the reductions in these SNP-based heritability estimates would be much smaller.

As a further example, by generating negative LD between alleles with the same directional effect on the trait, the impact of stabilizing selection opposes, and therefore masks, the genetic impact of assortative mating [75]. A practical consequence is that stabilizing selection will tend to attenuate estimates of the strength of assortative mating based on GWAS effect sizes, which often use cross-chromosome correlations of PGSs (e.g., [57,76]). In humans, the phenotypic correlation among mates for height has been measured at about approximately 0.25 [58]. In S1 Text Section S3.4, we calculate that estimates of this correlation based on cross-chromosome correlations in PGSs will be biased downwards by about 20% (to approximately 0.20) because of stabilizing selection on height, if selection has not yet acted in the measured generation, and by about 40% (to 0.15) if selection has acted in the measured generation. Were assortative mating weaker, or stabilizing selection stronger, the genetic impact of assortative mating would be masked to an even greater extent (S1 Text Section S3.4).

As in our analysis of assortative mating above, if stabilizing selection ceases in some generation, the negative LD that built up during the period of stabilizing selection will decay over subsequent generations, rapidly for pairs of loci on different chromosomes and more slowly for linked pairs of loci. Patterns of selection on human traits have changed over time—for example, the strength of stabilizing selection on birth weight has relaxed [77]. In general, therefore, patterns of confounding reflect a composite of contemporary and historic processes.

3.5 Sibling indirect effects

Indirect effects of siblings’ genotypes on each other’s phenotypes are known to be a potential source of bias in sibling-based GWASs [34,52], and can be measured and corrected for only if, in addition to sibling genotypes, parental genotypes are also available, either directly or via imputation [27,34]. To generate intuition for their impact on GWASs, we consider a simple model of indirect sibling effects in the absence of G×E interactions and other confounding effects, focusing on a single-locus model for simplicity. We suppose that the indirect effect of an individual’s phenotype on their sibling’s (same) phenotype is β, so that the phenotypes of 2 siblings i and j can be written as follows: (18)

Taking their difference and rearranging, we find that (19)

Therefore, in the absence of genetic confounding and G×E interactions, a sibling-based association study would return an effect-size estimate of (20) in expectation. Thus, if sibling indirect effects are synergistic (β>0), they lead to underestimation of the direct genetic effect at the locus. In contrast, if sibling indirect effects are antagonistic (β<0), they lead to overestimation of the direct genetic effect.

How would a population GWAS be affected by the same sibling indirect effects? Sibling i’s phenotype can be written as follows: (21) where Y** = (1+β)Y*. Therefore, if we were to randomly choose 1 sibling from each sibship and estimate the effect size at the locus using a population association study across families, we would obtain (22) where is the genotypic correlation between sibs at the locus. Sibling indirect effects alter the effect-size estimate in a population GWAS via 2 channels. The first is through the factor 1/(1−β²) in Eq (22), which reflects second-order feedbacks of an individual’s phenotype on itself, via the sibling (see also ref. [78]). Since 1/(1−β²)>1, these feedbacks act to exacerbate the effects of causal alleles. For example, if sibling indirect effects are antagonistic (β<0), then a sibling with a large trait value will tend to indirectly reduce the trait value of their sibling, which in turn will indirectly further increase the trait value of the focal individual. This channel therefore pushes population GWASs towards overestimating the magnitude of direct genetic effects.

The other channel by which sibling indirect effects can influence a population GWAS is driven by the genotypic correlation among siblings, and is easiest to understand if we assume that sibling indirect effects are weak (β²≪1). In this case, . Since the genotypic correlation , this channel of sibling indirect effects has the opposite effect to the one it has on a sibling GWAS: if sibling indirect effects are synergistic (β>0), the population GWAS overestimates the direct genetic effect at the locus, while if sibling indirect effects are antagonistic (β<0), the population GWAS underestimates the direct genetic effect. The reason for this difference is that a sibling GWAS is based on siblings whose genotypes differ at the focal locus, and whose genotypic values are therefore anticorrelated. If sibling indirect effects are synergistic (β>0), they will tend to attenuate the phenotypic differences between such siblings, and therefore attenuate effect-size estimates. In contrast, because siblings’ genotypes are positively correlated across the entire population, synergistic sibling indirect effects (β>0) will tend to exacerbate phenotypic differences across families, leading a population GWAS to overestimate effect sizes.

Sources of genetic confounding

Genetic confounding is caused by long-range LD between loci that affect the trait or traits under study. To illustrate the potential for genetic confounds to bias GWAS effect-size estimates, we have considered several sources of long-range LD. Some of these—assortative mating, selection on GWAS traits, and phenotype-biased migration—can cause systematic directional biases in GWAS effect-size estimates. Others, such as neutral population structure, cause random biases that influence the variance of effect-size estimates and related quantities. Assortative mating and neutral population structure have received considerable theoretical attention in the GWAS literature (e.g., [22,37,38,57]). Here, we have further outlined how both selection and phenotyped-biased migration can drive systematic genetic confounding that may not be well accounted for by current methods of controlling for stratification (see also ref. [81]).

We wish to emphasize stabilizing selection in particular as a potential source of systematic confounding in GWASs. Stabilizing selection has been well studied in the quantitative genetics literature but less so in the context of GWASs, despite its expected ubiquity. By selecting for compensating combinations of trait-increasing and trait-decreasing alleles, stabilizing selection generates negative LD between alleles with the same directional effect on the trait [10,11], and can therefore bias GWAS effect-size estimates downwards. While the potential for stabilizing selection to confound effect-size estimation has been noted (e.g., [66,75,82]), the resulting biases have not, to our knowledge, been quantified. Our calculations suggest that these downward biases could, for some human traits, be as large as 5% to 10% systematically across all causal loci in population GWASs. While biases of this magnitude are unlikely to compromise some goals of GWASs, such as gene discovery, they could be quantitatively problematic for other GWAS aims, such as estimation of SNP heritabilities and the strength of assortative mating. Moreover, while our results pertain to (a particular model of) stabilizing selection, many kinds of selection generate LD between genetically distant loci—in fact, only multiplicative selection among loci does not ([83], pgs. 50 and 177). Therefore, the general result that selection can generate genetic confounding will hold more broadly. Even in the absence of natural and sexual selection, the ascertainment of samples for a population- or family-based GWAS is a form of selection. When participation in a GWAS sample is based partly on a genetically influenced phenotype [84], this sample-selection bias will generate LD between loci underlying participation. In family studies, this participation bias can violate the assumption of random assignment of genotypes, and thus potentially undermine the interpretation of these studies’ results.

For a given genotyped locus in a GWAS, there is no bright line between local “tagged” LD and long-range confounding LD, and one reasonable objection to the approach taken here is that that we have used an arbitrary definition of the causal loci that are locally tagged by a genotyped locus (L_local in Eq (2)). All of the sources of genetic confounding that we have considered generate LD among causal loci both within and across chromosomes. Under these models, the within-chromosome LD that is generated is, in a sense, a continuation of the LD generated across chromsomes (moving from a recombination rate = 0.5 to ≤ 0.5). Thus, while investigators may prefer some looser definition of “local” when thinking about genotyped GWAS loci as tag SNPs, to extend that definition to include all loci on the same chromosome as the SNP would, by reasonable interpretation, be to include confounding into the desired estimator.

The extent to which the absorption of genetic confounding in estimated effect sizes is a problem depends on the application. In the case of polygenic prediction, the absorption of environmental effects, indirect effects, and the effects of untyped loci throughout the genome can help to improve prediction accuracy, although this does come at a cost to interpretability (and potentially also to portability across contexts). For GWAS applications focused on understanding genetic causes and mechanisms, the biases in effect-size estimates and spurious signals of pleiotropy among traits generated by genetic confounding will be more problematic.

Indirect genetic effects

Family GWASs are often interpreted as providing the opportunity to ask to what extent parental genotypes (or other family genotypes) causally affect a child’s phenotype (“genetic nurture” [27]). Viewed in this way, the association between untransmitted parental alleles and the child’s phenotype would seem, at first, a natural estimate of indirect genetic effects.

In practice, however, if the population GWAS suffers from genetic and environmental confounds, then the estimated effects of untransmitted alleles will absorb that confounding in much the same way that estimates of direct genetic effects from a population GWAS do (Eqs (8) and (9)) [50]. For example, in the case of assortative mating, a given untransmitted allele is correlated with alleles that were transmitted both by this parent and by their mate, and these transmitted alleles can directly affect the offspring’s phenotype. Thus, while family-based estimates of direct genetic effects benefit from the randomization of meiosis and from controlling for the environment, family-based estimates of indirect genetic effects lack both of these features and should be interpreted with caution. Indeed, recent work using parental siblings to control for grandparental genotypes has shown that little of the estimated “indirect genetic effect” may be causally situated in parents [36]. With empirical estimates of indirect genetic effects potentially absorbing a broad set of confounds [34,85], and few current studies of indirect effects having designs that allow such confounding to be disentangled, it is premature—and potentially invalid—to interpret associations of untransmitted alleles causally in terms of indirect genetic effects [3]. Rather, they should be treated agnostically in terms of “non-direct” effects.

Direct genetic effects

Mendelian segregation provides a natural randomization experiment within families [86], and so crosses in experimental organisms and family designs have long been an indispensable tool to geneticists in exploring genetic effects and causation. Growing concerns about GWAS confounding and the increasing availability of genotyped family members have led to a return of family-based studies to the association study toolkit [2]. Family-based estimates of direct genetic effects are often interpreted as being unbiased and discussed in terms of the counterfactual effect of experimentally substituting one allele for another [34,87,88].

As we have shown, family-based GWASs are indeed less subject to confounding than population-based GWASs: in the presence of genetic and environmental confounding, the family-based estimate of the effect size at a given locus provides a much closer approximation to the true effects of tightly linked causal loci than a population-based estimate does. The family-based estimate is not biased by environmental variation across families and avoids the correlated effects of the many causal loci that lie on other chromosomes. Still, the family-based estimate does absorb the effects of non-local causal loci on the same chromosome, and so cannot truly be said to be free of genetic confounding. Rather than considering a single allele being substituted between individuals, a better experimental analogy for the effect-size estimate would be to say that we are measuring the mean effect of transmission of a large chunk of chromosome surrounding the focal locus, potentially carrying many causal loci.

In addition, while within-family GWASs offer these advantages, in other ways, they move us further away from the questions about the sources and causes of variation among unrelated individuals that motivate population GWASs in the first place. Indeed, the presence of confounding and of G×E/G×G interactions introduces a number of conceptual issues in moving from within-family GWAS to the interpretation of differences among individuals from different families [44,89,90]. For example, in the presence of genetic confounding, the estimated effect of a causal allele of interest will depend on a set of weights: its LD to many other causal alleles. Family-based approaches weight these LD terms differently to population-based approaches, which, we argue, can complicate the interpretation of these estimates. For example, when previously isolated populations admix, same-ancestry alleles will be held together in long genomic blocks until these are broken up by recombination, which will happen very quickly for alleles on different chromosomes but more slowly for alleles on the same chromosome. A few generations after admixture, therefore, cross-chromosome ancestry LD will largely have dissipated, but contiguous ancestry tracts will still span substantial portions of chromosome lengths. Since both population and within-family GWASs are similarly confounded by the same-chromosome LD, their mean squared effect sizes will be similar in this case (Fig 5). Bearing in mind that the LD resulting from admixture is not present in the source populations, it becomes unclear which weighting of ancestry LD is appropriate if we want to interpret the resulting effect-size estimates as direct effects. As this example illustrates, while family-based GWASs are a useful device for dealing with confounding, it is not always obvious how to interpret the quantities that they measure.

A number of additional complications arise when, to compensate for the small effect sizes of individual loci, researchers combine many SNPs into a PGS and study the effects of PGSs within families (or use them as instruments in mendelian randomization analyses). For one, SNPs are usually chosen for inclusion in the PGS on the basis of their statistical significance in a population GWAS. This approach prioritizes SNPs whose effect-size estimates are amplified (or even wholly generated) by confounding (for an example of how this leads to residual environmental confounding in applications of sibling-based effect-size estimates, see ref. [79]). Second, the weights given to SNPs that are included in the PGS absorb the effects of confounding, and this confounding is heterogeneous across SNPs. Thus, when we study the correlates of trait-A PGS differences between siblings in the presence of GWAS confounding, we are not observing the average phenotypic outcomes of varying the genetic component of trait A between siblings. Rather, we are varying a potentially strangely weighted set of genetic correlates of trait A.

An observation that a population-GWAS PGS is predictive of phenotypic differences among siblings demonstrates that the PGS SNPs tag nearby causal loci, but beyond that, interpretation is difficult. Notably, if there is cross-trait assortative mating for traits A and B, but no pleiotropic link between the traits, then some of the SNPs identified as significant in a GWAS on trait A may be tightly linked to loci that causally affect trait B but not trait A. If these loci are included in the trait-A PGS, then when we study the effect of variation in the trait-A PGS on sibling differences, we are accidentally absorbing some components of the variation in trait B across siblings. Thus, we might observe a correlation between the trait-A PGS and differences in trait B between siblings, and this correlation may be lower than is observed at the population level, without there existing any pleiotropic (or causal) link between A and B. These effects can be exacerbated if the 2 traits have different genetic architectures (Fig 4). Instead of using a set of SNPs and weights from a population GWAS, genetic correlations between traits due to pleiotropy could be estimated from the correlation of effect sizes estimated within families [33]. Given current sample size constraints in family-based studies, the confidence intervals on these estimates are large. Moreover, significant family-based correlations need not reflect pure pleiotropy, since, as we have shown, they are not completely free of genetic confounding due to intra-chromosomal LD.

Also complicating the interpretation of family-based effect-size estimates are various types of interactions. Indirect effects between siblings can bias family estimates of direct genetic effects (Eq (20); [2,34,52]) in ways that are conceptually different from the biases they introduce to population-based estimates (Eq (22)). These sibling effects can potentially be addressed with fuller family information (e.g., parental genotypes in addition to sibling genotypes [27,34]).

In summary, family-based studies are a clear step forward towards quantifying genetic effects, with large-scale family studies carrying the potential to resolve long-standing issues in human genetics. However, these designs come with their own sets of caveats, which will be important to understand and acknowledge as family-based genetic studies become a key tool in the exploration of causal effects across disparate fields of study.

Assortative mating

For our general cross-trait assortative mating setup, traits 1 and 2 are influenced by variation at sets of bi-allelic loci L₁ and L₂, respectively. The effect sizes of the reference allele at locus l on traits 1 and 2 are α_l and β_l, respectively. An individual’s PGS is then for trait 1 and for trait 2. In all the scenarios we simulated, traits had heritability 1, so that individuals’ trait values are the same as their PGSs.

Our aim is to simulate a scenario where assortative mating is based on females’ values for trait 1 and males’ values for trait 2, such that, across mating pairs, the correlation of the mother’s PGS for trait 1, , and the father’s PGS for trait 2, , is a constant value ρ (in all of our simulations, ρ = 0.2). To achieve this, we use an algorithm suggested by Zaitlen and colleagues [69]: At the outset, we choose an accuracy tolerance ε such that, if by some assignment of mates the correlation of their PGSs falls within ε of the target value ρ, we accept that assigment. Each generation in which assortative mating occurs, we rank females in order of their PGSs for trait 1, and males in order of their PGSs for trait 2. We then calculate the PGS correlation across mating pairs, ρ₀, if females and males were matched according to this ranking. If this (maximal) correlation is smaller than the upper bound of our target window (ρ₀<ρ+ε, which very seldom occurred in our simulations), then females and males mate precisely according to their PGS rankings and we move on to the next generation. If, instead, ρ₀ exceeds ρ+ε, then we follow the following iterative procedure until we have found a mating structure under which the correlation of PGSs falls within ε of the target value ρ.

First, we choose initial “perturbation sizes” ξ₀ and ξ₁ = 2ξ₀. Suppose that, in iteration k of the procedure, the perturbation size is ξ_k and the chosen mating structure leads to a correlation among mates of ρ_k. If |ρ_k−ρ|<ε, we accept the mating structure and move on to the next generation. Otherwise, we choose a new perturbation size ξ_k+1: (i) if ρ_k−1, ρ_k>ρ, then ξ_k+1 = 2ξ_k; (ii) if ρ_k−1>ρ>ρ_k or ρ_k−1<ρ<ρ_k, then ξ_k+1 = (ξ_k−1+ξ_k)/2; (iii) if ρ_k−1, ρ_k<ρ, then ξ_k+1 = ξ_k/2. Once we have chosen ξ_k+1, for each individual we perturb their PGS (trait 1 for females; trait 2 for males) by a value chosen from a normal distribution with mean 0 and standard deviation ξ_k+1, independently across individuals. We then rank females and males according to their perturbed PGSs, and calculate the correlation ρ_k+1 of their true PGSs if they mate according to this ranking. (Since, in our experience, there can be substantial variance in the ρ_k+1 values that result from this procedure, we repeat it 5 times and choose the mating structure that produces the value of ρ_k+1 closest to the target value ρ.) We then decide if another iteration—i.e., another perturbation size ξ_k+2—is required.

Fig 2. Cross-trait assortative mating for traits with the same genetic architecture

In the simulations displayed in Fig 2, ρ = 0.2, and traits 1 and 2 have identical but non-overlapping genetic architectures: L₁ and L₂ are distinct sets of 500 loci each, with α_l = 1 and β_l = 0 for l∈L₁, and α_l = 0 and β_l = 1 for l∈L₂. Loci in L₁ and L₂ alternate in an even spacing along the physical (bp) genome. Fig 2A shows results for the “single chromosome” case where the recombination fraction between adjacent loci is c = 1/999 in both sexes (such that the single-chromosome genome receives, on average, one crossover per transmission). Fig 2B shows results for the case where recombination fractions between loci are calculated from the human female and male linkage maps generated by Kong and colleagues [54]. In both cases, we assumed no crossover interference.

At each locus, the initial frequency of the reference allele was 1/2, with reference alleles assigned randomly across diploid individuals and independently across loci such that, in expectation, Hardy–Weinberg and linkage equilibrium initially prevail. The assortative mating algorithm above was run for 19 generations, with a target correlation ρ = 0.2, a tolerance parameter ε = ρ/100, and an initial perturbation size . Thereafter, assortative mating was switched off, with mating pairs (still monogamous) being chosen randomly.

Fig 3. Same-trait assortative mating

The algorithm we followed to ensure assortative mating of a given strength was the same as that for Fig 2 above, but here traits 1 and 2 are identical, and 1,000 loci underlie variation in the trait, and are evenly spread along the physical genome. The effect size of the reference allele at each locus was drawn from a normal distribution with mean 0 and standard deviation 1, independently across loci. The initial frequency of the reference allele at each locus was drawn, independently across loci, from a uniform distribution on [MAF, 1−MAF]; in our simulations, we chose a minimum minor allele frequency of MAF = 0.1. Since here we are interested in quantifying the mean squared effect size estimate, which is directionally affected by drift-based local LD that may not be present in our initial configuration, we allowed 150 generations of random mating before switching on assortative mating (only the final 20 generations of this random mating burn-in are displayed in Fig 3). Assortative mating occurred for 20 generations, after which random mating occurred for a further 20 generations.

Fig 4. Cross-trait assortative mating for traits with different architectures

For Fig 4A, we again followed a similar procedure to that for Fig 2 above, but now, while traits 1 and 2 have distinct genetic bases, the numbers of loci contributing variation to traits 1 and 2 are |L₁| = 100 and |L₂| = 1,000. Trait-1 loci are placed evenly along the physical genome, with trait-2 loci then evenly spaced among the trait-1 loci; we used the human linkage map for these simulations. At both trait-1 and trait-2 loci, the initial frequency of the focal allele was drawn from a uniform distribution on [MAF, 1−MAF], with MAF = 0.1. At trait-2 loci, true effect sizes were randomly drawn from a normal distribution with mean zero and standard deviation 1; at trait-1 loci, true effect sizes were randomly drawn from a normal distribution with mean zero and standard deviation , so that traits 1 and 2 have equal genic variances. After a burn-in of 150 generation of random mating, assortative mating was switched on. We performed a population GWAS at the end of the period of random mating and after 20 generations of assortative mating. These GWASs were performed across 1,000 replicate trials, with the effect size estimates then pooled across trials. From these, we estimated the densities of the absolute values of effect size estimates using Matlab’s kernel density estimator ksdensity, specifying that the support of the distributions be positive. For Fig 4B, as a control, we also carried out the same simulations with |L₁| = |L₂| = 1,000, drawing effect sizes from a standard normal distribution.

Fig 5. Population structure and admixture

We wished first to simulate a situation where 2 populations of size N/2 have been separated for a length of time such that the value of F_ST between them is some predefined level (in our case, a mean F_ST per locus of 0.1). To do so without having to run the full population dynamics of 2 allopatric populations for a prohibitively large number of generations, we simply assigned allele frequencies to achieve the desired level of F_ST. We assumed 1,000 loci spread evenly over the physical genome. At each locus l, we chose an “ancestral” frequency for the reference allele independently from a uniform distribution on [MAF, 1−MAF], with MAF = 0.2. We then perturbed this allele frequency in populations 1 and 2 by independent draws from a normal distribution with mean 0 and variance ; if a perturbed allele frequency fell below 0 or above 1, we set it to 0 or 1, respectively. The population dynamics described above, with monogamous mating pairs chosen randomly, were then run for 50 generations.

In generation 50, the 2 populations merge, forming an admixed population of size N. The same population dynamics, with monogamous mating pairs chosen randomly, were then run for a further 50 generations.

Fig 6. Stabilizing selection

To calculate the bias in GWAS effect size estimation caused by stabilizing selection, we must first calculate the harmonic mean recombination rate. We focus on humans and consider only the autosomal genome. The set of loci underlying variation in the trait is L, which we apportion among the 22 autosomes according to their physical (bp) lengths (as reported in GRCh38.p11 of the human reference genome; https://www.ncbi.nlm.nih.gov/grc/human/data?asm=GRCh38.p11). For each chromosome, we spread its allotment of loci evenly over its sex-averaged genetic (cM) length, using the male and female linkage maps produced by Kong and colleagues [54]. (We use genetic lengths instead of physical lengths because, were we to spread loci evenly over the physical lengths of the chromosomes, some pairs of adjacent loci on some chromosomes might have a sex-averaged recombination fraction of 0, in which case the harmonic mean recombination rate would be undefined.) For each pair of linked loci, the recombination rate between them was estimated separately from the male and female genetic distance between them using Kosambi’s map function [91]. Pairs of loci on separate chromosomes have a recombination fraction of 1/2. With the sex-averaged recombination fraction c_ll′ thus calculated for every pair of loci (l,l′), the harmonic mean recombination fraction was calculated as , where is the number of pairs of distinct loci in L.

Performing this calculation with |L| = 1,000 loci, we obtain an estimate of for human autosomes. Substituting this estimate into S1 Text Eqs (S.106), (S.107), and (S.108) then defines the curves plotted in Figs 6A, 6B and 6C, respectively.