## Figures

## Abstract

*F*_{ST} and kinship are key parameters often estimated in modern population genetics studies in order to quantitatively characterize structure and relatedness. Kinship matrices have also become a fundamental quantity used in genome-wide association studies and heritability estimation. The most frequently-used estimators of *F*_{ST} and kinship are method-of-moments estimators whose accuracies depend strongly on the existence of simple underlying forms of structure, such as the independent subpopulations model of non-overlapping, independently evolving subpopulations. However, modern data sets have revealed that these simple models of structure likely do not hold in many populations, including humans. In this work, we analyze the behavior of these estimators in the presence of arbitrarily-complex population structures, which results in an improved estimation framework specifically designed for arbitrary population structures. After generalizing the definition of *F*_{ST} to arbitrary population structures and establishing a framework for assessing bias and consistency of genome-wide estimators, we calculate the accuracy of existing *F*_{ST} and kinship estimators under arbitrary population structures, characterizing biases and estimation challenges unobserved under their originally-assumed models of structure. We then present our new approach, which consistently estimates kinship and *F*_{ST} when the minimum kinship value in the dataset is estimated consistently. We illustrate our results using simulated genotypes from an admixture model, constructing a one-dimensional geographic scenario that departs nontrivially from the independent subpopulations model. Our simulations reveal the potential for severe biases in estimates of existing approaches that are overcome by our new framework. This work may significantly improve future analyses that rely on accurate kinship and *F*_{ST} estimates.

## Author summary

Kinship coefficients and *F*_{ST}, which measure relatedness and population structure, respectively, are important quantities needed to accurately perform various analyses on genetic data, including genome-wide association studies and heritability estimation. However, existing estimators require restrictive assumptions of independence that are not met by real human and other datasets. In this work we find that existing estimators can be severely biased under reasonable scenarios, first by theoretically determining their properties, and then using an admixture simulation to illustrate our findings. In particular, we find that existing *F*_{ST} estimators are downwardly biased, and that existing kinship matrix estimators have related biases that are on average downward and of similar magnitude but vary for every pair of individuals. These insights led us to a new estimation framework for kinship and *F*_{ST} that is practically unbiased for any population structure, as demonstrated by theory and simulations. Our new approaches—available as open-source R packages—are easy to use and are more widely applicable than existing approaches, and they are likely to improve downstream analyses that require accurate kinship and *F*_{ST} estimates.

**Citation: **Ochoa A, Storey JD (2021) Estimating *F*_{ST} and kinship for arbitrary population structures. PLoS Genet 17(1):
e1009241.
https://doi.org/10.1371/journal.pgen.1009241

**Editor: **Marcus W. Feldman,
Stanford University, UNITED STATES

**Received: **September 11, 2019; **Accepted: **November 2, 2020; **Published: ** January 19, 2021

**Copyright: ** © 2021 Ochoa, Storey. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability: **The data and computer code for these manuscripts can be found at https://github.com/StoreyLab/human-differentiation-manuscript.

**Funding: **This research was supported in part by National Institutes of Health, National Human Genome Research Institute grant R01 HG006448 (JDS). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

**Competing interests: ** The authors have declared that no competing interests exist.

## Introduction

In population genetics studies, one is often interested in characterizing structure, genetic differentiation, and relatedness among individuals. Two quantities often considered in this context are *F*_{ST} and kinship. *F*_{ST} is a parameter that measures structure in a subdivided population, satisfying *F*_{ST} = 0 for an unstructured population and *F*_{ST} = 1 if every locus has become fixed for some allele in each subpopulation. More generally, *F*_{ST} is the probability that alleles drawn randomly from a subpopulation are “identical by descent” (IBD) relative to an ancestral population [1, 2]. The kinship coefficient is a measure of relatedness between individuals defined in terms of IBD probabilities, and it is closely related to *F*_{ST}, since the mean kinship of the parents in a subpopulation is the *F*_{ST} of the following generation [1].

This work focuses on the estimation of *F*_{ST} and kinship from biallelic single-nucleotide polymorphism (SNP) data. Existing estimators can be classified into parametric estimators (methods that require a likelihood function) and non-parametric estimators (such as the method-of-moments estimators we focus on, which only require low-order moment equations). There are many likelihood approaches that estimate *F*_{ST} and kinship, but these are limited by assuming independent subpopulations or Normal approximations for *F*_{ST} [3–11] or totally outbred individuals for kinship [12, 13]. Additionally, more complete likelihood models such as that of Jacquard [14] are underdetermined for biallelic loci [15]. Non-parametric approaches such as those based on the method of moments are considerably more flexible and computationally tractable [16], so they are the natural choice to study arbitrary population structures.

The most frequently-used *F*_{ST} estimators are derived and justified under the “independent subpopulations model,” in which non-overlapping subpopulations evolved independently by splitting all at the same time from a common ancestral population. The Weir-Cockerham (WC) *F*_{ST} estimator assumes subpopulations of differing sample sizes and equal per-subpopulation *F*_{ST} relative to the common ancestral population [17]. The Weir-Hill *F*_{ST} estimator generalized WC for subpopulations with different *F*_{ST} values, and first considered arbitrary coancestry between subpopulations, resulting in estimates of a linearly-transformed *F*_{ST}, namely (where is the unknown mean coancestry value between subpopulations) [4, 18, 19]. Weir-Hill has further evolved into the Weir-Goudet approach, incorporating relatedness for subpopulations and individuals based on allele matching, also estimating a linearly-transformed *F*_{ST} [20–22]. Note that the Weir-Hill and Weir-Goudet approaches intended to estimate such linearly-transformed quantities, which may be negative, and they did not aim to estimate IBD probabilities [4, 18–22]; in contrast, our goal is to estimate IBD probabilities, which must be non-negative and valid probabilities. The “Hudson” *F*_{ST} estimator [23] assumes two subpopulations with different *F*_{ST} values. All of the previous *F*_{ST} estimators are ratio estimators derived using the method of moments to have unbiased numerators and denominators, which gives approximately unbiased ratio estimates when their assumptions are met [4, 17, 23]. We also evaluate BayeScan [10], which estimates population-specific *F*_{ST} values using a Bayesian model and the Dirichlet-Multinomial likelihood function—thus representing non-method-of-moments approaches—but which like other existing *F*_{ST} estimators also assumes that subpopulations are non-overlapping and evolve independently. These *F*_{ST} estimators are important contributions, used widely in the field.

Kinship coefficients are now commonly calculated in population genetics studies to capture structure and relatedness. Kinship is utilized in principal components analyses and linear-mixed effects models to correct for structure in Genome-Wide Association Studies (GWAS) [16, 24–30] and to estimate genome-wide heritability [31, 32]. Often absent in previous models is a clear identification and role of the ancestral population *T* that sets the scale of the kinship estimates used. Omission of *T* makes sense when kinship is estimated on an unstructured population (where only a few individual pairs are closely related; there *T* is the current population). Our more complete notation brings *T* to the fore and highlights its key role in kinship estimation and its applications. The most commonly-used kinship estimator [16, 27, 30–36] is also a method-of-moments estimator whose operating characteristics are largely unknown in the presence of structure. We show here that this popular estimator is accurate only when the average kinship is zero, which implies that the population must be unstructured.

The goal of our work is to consistently estimate IBD probabilities, namely kinship coefficients and *F*_{ST}, for which there are currently no consistent estimators under general relatedness. Estimation of these as probabilities, as opposed to linearly-transformed quantities that may be negative, is important since the probabilistic definition of these parameters was required to derive their fundamental connections to many applications in genetics, including allele fixation [1, 2, 37], DNA forensics [3], and heritability [38, 39]. Although IBD probabilities are not absolute, but rather depend on the choice of ancestral population [40], their values become fixed upon agreeing to estimate them in terms of the Most Recent Common Ancestor (MRCA) population, which has long been the choice for models of *F*_{ST} [17, 23, 41] and kinship estimation from pedigrees [42, 43] or markers [12, 13].

Recent genome-wide studies have revealed that humans and other natural populations are structured in a complex manner that break the assumptions of the above estimators. Such complex population structures has been observed in several large human studies, such as the Human Genome Diversity Project [44, 45], the 1000 Genomes Project [46], Human Origins [47–49], and other contemporary [50–54] and archaic populations [55, 56]. We have also demonstrated that the global human population has a complex kinship matrix and no independent subpopulations [57–59]. Therefore, there is a need for innovative approaches designed for complex population structures. To this end, we reveal the operating characteristics of these frequently-used *F*_{ST} and kinship estimators in the presence of arbitrary forms of structure, which leads to a new estimation strategy for *F*_{ST} and kinship.

Here, we study existing *F*_{ST} and kinship method-of-moments estimators in models that allow for arbitrary population structures (see Fig 1 for an overview of the results). First, in section **The generalized F_{ST} for arbitrary population structures** we present the generalized definition of

*F*

_{ST}for arbitrary population structures [57]. In section

**The kinship and coancestry models**we review the kinship model for genotype covariance [1, 14] and the coancestry model for individual-specific allele frequencies [57, 60, 61]. In section

**Assessing the accuracy of genome-wide ratio estimators**we obtain new strong convergence results for a family of ratio estimators that includes the most common

*F*

_{ST}and kinship estimators. Next, we calculate the convergence values of these

*F*

_{ST}(section

**) and kinship (section**

*F*_{ST}estimation based on the independent subpopulations model**Characterizing a kinship estimator and its relationship to**) estimators under arbitrary population structures, where we find biases that are not present under their original assumptions about structure (panels “Indep. Subpop.

*F*_{ST}*F*

_{ST}Estimator” and “Existing Kinship Estimator” in Fig 1). We characterize the limit of the standard kinship estimator, identifying complex biases or distortions, in agreement with recent work [21, 62].

Our analysis is based on the generalized *F*_{ST} definition (section **The generalized F_{ST} for arbitrary population structures**) and two parallel models: the “Coancestry Model” for individual-specific allele frequencies (

*π*

_{ij}), and the “Kinship Model” for genotypes (

*x*

_{ij}). The “Coancestry in Terms of Kinship” panel connects kinship (, ) and coancestry () parameters (section

**The kinship and coancestry models**). We use these models to study the accuracy of

*F*

_{ST}and kinship method-of-moment estimators under arbitrary population structures. The “Indep. Subpop.

*F*

_{ST}Estimator” panel shows the bias resulting from the misapplication of

*F*

_{ST}estimators for independent subpopulations () to arbitrary structures (section

**), as calculated under the coancestry model. The “Existing Kinship Estimator” panel shows the bias in the standard kinship model estimator () and its resulting plug-in**

*F*_{ST}estimation based on the independent subpopulations model*F*

_{ST}estimator (; section

**Characterizing a kinship estimator and its relationship to**), as calculated under the kinship model. The “New Kinship Estimator” panel presents a new statistic

*F*_{ST}*A*

_{jk}that estimates kinship with a uniform bias, which together with a consistent estimator of its minimum value () results in our new kinship () and

*F*

_{ST}() estimators, which are consistent under arbitrary population structure (section

**A new approach for kinship and**).

*F*_{ST}estimationIn section **A new approach for kinship and F_{ST} estimation** we introduce a new approach for kinship and

*F*

_{ST}estimation for arbitrary population structures, and demonstrate the improved performance using a simple implementation of these estimators (panel “New Kinship Estimator” in Fig 1). There are two key innovations. First, based on the method of moments, we derive a statistic that estimates kinship coefficients up to a shared unknown scaling factor. Second, we propose a new condition, the identification of unrelated individual pairs in the data, which yields the value of the unknown scaling factor and enables the consistent estimation of kinship matrices and

*F*

_{ST}. We present a simple implementation of this second estimator, based on taking the minimum average statistic value between subpopulations, which in section

**Simulations evaluating**is shown to perform well under some misspecification, namely in an admixture scenario that does not actually have subpopulations [63–65]. Elsewhere, we analyze the Human Origins and 1000 Genomes Project datasets with our novel kinship and

*F*_{ST}and kinship estimators*F*

_{ST}estimation approach, where we demonstrate its coherence with the African Origins model, and illustrate the shortcomings of previous approaches in these complex data [59]. In summary, we identify a new approach for unbiased estimation of

*F*

_{ST}and kinship, and we provide new estimators that are nearly unbiased.

## Results

### The generalized *F*_{ST} for arbitrary population structures

The existing *F*_{ST} definition requires individuals to belong to discrete, non-overlapping subpopulations, so it must be generalized in order to apply to arbitrary population structures (such as the admixture model with individual-specific ancestry proportions considered in our simulations). Our generalized *F*_{ST} can be understood as a two-step strategy: (1) we define *F*_{ST} on a per-individual basis, and (2) we define *F*_{ST} for a group of individuals as a weighted average of the per-individual *F*_{ST} values [57].

The inbreeding coefficient of an individual *j* relative to an ancestral population *T* is defined as the probability that the two alleles at a random locus are *identical by descent* (IBD) [37]. Note that the ancestral population *T* determines what is IBD: only relationships since *T* count toward IBD. This *total* inbreeding coefficient () is the individual analog of Wright’s total inbreeding coefficient *F*_{IT}, the latter of which is the mean over a group of individuals [2]. Wright partitioned *total* inbreeding (*F*_{IT}) into *local* (*F*_{IS}) and *structural* (*F*_{ST}) coefficients defined by a subpopulation *S* that contains all individuals in question and evolved from the ancestral population *T*, so that *F*_{IS} is the inbreeding of individuals relative to *S* (as opposed to *T*) and *F*_{ST} is inbreeding of the subpopulation *S* relative to *T*, and these coefficients satisfy (1 − *F*_{IT}) = (1 − *F*_{IS})(1 − *F*_{ST}) [2]. In our generalized definitions for one individual *j*, we restrict the subpopulation of interest (*S*) to be *L*_{j}, called the local subpopulation of *j*, which is the most recent subpopulation from which *j* drew its alleles. In this case, is the *local* inbreeding coefficient of *j* (always relative to its local subpopulation *L*_{j}), and is the *structural* inbreeding coefficient of *j* (equal to the inbreeding of the subpopulation *L*_{j} relative to *T*), and being a special case of Wright’s equation, they also satisfy [57]
(1)

Now we discuss estimating the three quantities we just introduced. First, the total inbreeding coefficient () should be estimated from the variance of genotypes, using the practically unbiased approach we introduce in this work. Second, note that the local inbreeding coefficient () corresponds to (non-population) family relatedness, so it can be taken to be the inbreeding calculated from a pedigree if it is available [42]. Note that estimation of the various inbreeding coefficients from pedigrees was the only approach available to Wright when he studied cattle and defined inbreeding and *F*_{ST} [2, 37]. Alternatively, in the absence of pedigrees, local inbreeding can be estimated from inferred self-IBD blocks or unusually-large runs of homozygosity [66–68]. Lastly, since the structural inbreeding coefficient () is given by the previous two quantities (solving from Eq (1)) by
(2)
then we propose estimating using this equation, from the above estimates of and .

As a toy example, suppose we estimate a total inbreeding coefficient of for a given individual whose parents are first cousins, then the pedigree expectation for its local inbreeding is , and the structural inbreeding (i.e. the *F*_{ST} of this individual) using Eq (2) is . However, if in the same example () the individual instead had parents who were second cousins, then , then the structural estimate becomes , which is much closer to the total inbreeding value. Thus, when total inbreeding estimates are much larger than local inbreeding estimates, correcting for the latter via Eq (2) may not change the numerical estimate of structural inbreeding by a meaningful amount. Conversely, as the local inbreeding coefficient is reduced exponentially with the degree of relatedness of the parents ( for *n*-th cousins), and as local inbreeding is required to be recent (to exclude population-level inbreeding), then sufficiently-accurate estimates of structural inbreeding can be obtained by estimating non-zero local inbreeding only for individuals with the most related parent pairs (above a certain degree of relatedness).

We define the generalized *F*_{ST} across *n* individuals as the weighted average of the per-individual structural inbreeding coefficients (*i.e.*, individual *F*_{ST} values) [57],
(3)
where *w*_{j} is the weight of individual *j* and the weights are required to sum to one and be non-negative. The above is a straightforward generalization of Wright’s *F*_{ST}: if every individual *j* has *L*_{j} = *S* as its local subpopulation, then Eq (3) becomes , where is the inbreeding coefficient of subpopulation *S* relative to *T*, so it has the same meaning as Wright’s *F*_{ST} (the exact weights here do not matter as long as , as required). Moreover, if each individual *j* belongs to one of *K* subpopulations *S*_{u} (*u* ∈ {1, …, *K*}) and if subpopulations are weighted equally ( for every *S*_{u}), then Eq (3) becomes , so it equals the (unweighted) average subpopulation-specific *F*_{ST} (*i.e.*, ), which is the *F*_{ST} definition for multiple subpopulations prevalent in modern work [4, 21, 23]. The last case illustrates the need for weights, which above downweights individuals that belong to subpopulations with greater numbers of observations. In general, weights allow adjustment for skewed or unbalanced samples. However, in complicated scenarios without subpopulations and no obvious sampling biases, for simplicity we recommend using uniform weights () for the target generalized *F*_{ST}.

In terms of total and local inbreeding coefficients (using Eq (2)), the generalized *F*_{ST} equals
which immediately suggests the estimation strategy when estimates of the total and local inbreeding coefficients are available. For simplicity, in the remainder of this work we shall consider only locally-outbred individuals ( for all *j*), for which the generalized *F*_{ST} simply equals the weighted mean total inbreeding coefficient:
(4)
This greatly simplifies our discussion of bias for all of the *F*_{ST} estimators we analyzed; determining the statistical properties of local inbreeding estimators is beyond the scope of this work. Moreover, the assumption of locally-outbred individuals is satisfied in all of the simulations presented in this work.

### The kinship and coancestry models

The generalized *F*_{ST} above is given solely in terms of inbreeding coefficients. In order to establish our results and framework, it is necessary to consider kinship coefficients as well. The kinship coefficient is the extension of the inbreeding coefficient for a pair of individuals: the kinship coefficient of two individuals *j* and *k* relative to an ancestral population *T* is the probability that two alleles, chosen at random from each individual at a random locus, are IBD [1]. Note that the self-kinship coefficient is related to the inbreeding coefficient by [16].

Kinship coefficients determine the covariance structure of genotypes, which is the key to estimating kinship and *F*_{ST} from genotype data. We shall concentrate on biallelic variants, which include single-nucleotide polymorphisms, and are the dominant data from genotyping microarrays and whole-genome sequencing studies. We shall also restrict our attention to diploid organisms in this present work. Genotypes are encoded into variables *x*_{ij} for each locus *i* and individual *j* that count the number of alleles (dosage) of a given reference type, so for diploid organisms *x*_{ij} = 2 is homozygous for the reference allele, *x*_{ij} = 0 is homozygous for the alternative allele, and *x*_{ij} = 1 is heterozygous. Based on the definition of the IBD probabilities, the kinship model determines the mean and covariance structure of the genotype random variables at neutral loci [1, 2, 14, 16, 37]:
(5)
where is the allele frequency at locus *i* in the ancestral population *T* and .

The coancestry model resembles the kinship model, but it is formulated in terms of allele frequencies, which simplifies our analysis of *F*_{ST} estimators for subpopulations as well as yielding kinship coefficients under the admixture model we simulate from in this work. Let *π*_{ij} be the *individual-specific allele frequency* (IAF) at locus *i* for individual *j*, which is a real number between zero and one [60, 61]. Our coancestry model assumes that [57]
(6)
where is the coancestry coefficient between individuals *j* and *k* relative to the ancestral population *T*. This model is inspired by coancestry models for subpopulations common in the *F*_{ST} literature [4, 5, 21, 23], and exactly equals those models when subpopulation sizes go to infinity, in which case *j* and *k* index subpopulations rather than individuals, and *π*_{ij} is interpreted as the true allele frequency at locus *i* for subpopulation *j*.

The coancestry model connects to the kinship model under the additional assumption that the alleles of an individual *j* are drawn independently from its IAF,
(7)
In this case, marginalizing the intermediate IAF random variables (*π*_{ij}) and matching the resulting genotype moments results in the following equivalence [57]:
(8)
The coancestry coefficient equals the kinship coefficient between two different individuals, but the self-coancestry coefficient equals the inbreeding coefficient (rather than the self-kinship coefficient). However, since in the coancestry model alleles are drawn independently conditional on the IAF in Eq (7), then the only structure present is the population structure, so these coancestry models cannot generate family structures, unlike the more general kinship model that also encompasses pedigrees. Therefore, despite Eq (8), the kinship and coancestry are not equivalent models except under the more restrictive assumptions of the coancestry model. Thus, individuals drawn from this model are always locally-outbred, so also equals the structural inbreeding coefficient, and the generalized *F*_{ST} under the coancestry model is therefore
(9)
which also generalizes previous definitions of *F*_{ST} under coancestry for subpopulations [4, 5, 21, 23]. The kinship and coancestry models, and their connection, is included in the overview Fig 1.

### Assessing the accuracy of genome-wide ratio estimators

In this section we change gears to focus on theoretical convergence properties of two broad estimator families. The resulting theory will be applied repeatedly to various *F*_{ST} and kinship estimators of interest in later sections.

Many *F*_{ST} and kinship coefficient method-of-moments estimators are *ratio estimators*, a general class of estimators that tends to be biased and to have no closed-form expectation [69]. In the *F*_{ST} literature, the expectation of a ratio is frequently approximated with a ratio of expectations [4, 17, 23]. Specifically, ratio estimators are often called “unbiased” if the ratio of expectations is unbiased, even though the ratio estimator itself may be biased [69]. Here we characterize the behavior of two ratio estimator families calculated from genome-wide data, known as “ratio-of-means” and “mean-of-ratios” estimators [23], detailing conditions where the previous approximation is justified and providing additional criteria to assess the accuracy of such estimators.

#### Ratio estimators.

The general problem of forming ratio estimators involves random variables *a*_{i} and *b*_{i} calculated from genotypes at each locus *i*, such that E[*a*_{i}] = *Ac*_{i} and E[*b*_{i}] = *Bc*_{i} and the goal is to estimate . *A* and *B* are constants shared across loci (given by *F*_{ST} or ), while *c*_{i} depends on the ancestral allele frequency and varies per locus. The problem is that the single-locus estimator is biased, since , which applies to ratio estimators in general [69]. Below we study two estimator families that combine large numbers of loci to better estimate .

#### Convergence.

The solution we recommend is the “ratio-of-means” estimator , where , and , which is common for *F*_{ST} estimators [4, 17, 19, 23, 70]. Note that and , where . We will assume bounded terms (|*a*_{i}|, |*b*_{i}| ≤ *C* for some finite *C*), a convergent , and *Bc* ≠ 0, which are satisfied by common estimators. Given independent loci, we prove almost sure convergence to the desired quantity (S1 Text),
(10)
a strong result that implies , justifying previous work [4, 17, 23]. Moreover, the error between these expectations scales with (S1 Text), just as for standard ratio estimators [69]. Although real loci are not independent due to genetic linkage, their dependence is very localized, so this estimator will perform well if the effective number of independent loci is large.

In order to test if a given ratio-of-means estimator converges to its ratio of expectations as in Eq (10), the following three conditions can be tested. (i) The expected values of each term *a*_{i}, *b*_{i} must be calculated and shown to be of the form E[*a*_{i}] = *Ac*_{i} and E[*b*_{i}] = *Bc*_{i} for some *A* and *B* shared by all loci *i* and some *c*_{i} that may vary per locus *i* but must be shared by both E[*a*_{i}], E[*b*_{i}]. In the estimators we study, *A* and *B* are functions of IBD probabilities such as and *F*_{ST}, while *c*_{i} is a function of only. (ii) The mean *c*_{i} must converge to a non-zero value for infinite loci. (iii) Both |*a*_{i}|, |*b*_{i}| ≤ *C* must be bounded for all *i* by some finite *C* (the estimators we study usually have *C* = 1 or *C* = 4). If these conditions are satisfied, then Eq (10) holds for independent loci and the *A* and *B* found in the first step. See the next section for an example application of this procedure to an *F*_{ST} estimator.

Another approach is the “mean-of-ratios” estimator , used often to estimate kinship coefficients [16, 27, 30–35] and *F*_{ST} [46]. If each is biased, their average across loci will also be biased, even as *m* → ∞. However, if for all loci *i* = 1, …, *m* as the number of individuals *n* → ∞, and is bounded, then
(11)
Therefore, mean-of-ratios estimators must satisfy more restrictive conditions than ratio-of-means estimators, as well as large *n* (in addition to the large *m* needed by both estimators), to estimate well. We do not provide a procedure to test whether a given mean-of-ratios estimator converges as shown above.

*F*_{ST} estimation based on the independent subpopulations model

Now that we have detailed how ratio estimators may be evaluated for their accuracy, we turn to existing estimators and assess their accuracy under arbitrary population structures. We study the *F*_{ST} estimators Weir-Cockerham (WC) [17], Weir-Hill [4], “Hudson” [23], and Weir-Goudet (equals HudsonK below for biallelic loci; S1 Text) [21]. The panel “Indep. Subpop. *F*_{ST} Estimator” in Fig 1 provides an overview of our results, which we detail in this section.

#### The *F*_{ST} estimator for independent subpopulations and infinite subpopulation sample sizes.

The WC, Weir-Hill, and Hudson method-of-moments estimators have small sample size corrections that remarkably make them consistent (as the number of independent loci *m* goes to infinity) for finite numbers of individuals. However, these small sample corrections also make the estimators unnecessarily cumbersome for our purposes (see Methods, section **Previous F_{ST} estimators for the independent subpopulations model** for complete formulas). In order to illustrate clearly how these estimators behave, both under the independent subpopulations model and for arbitrary structure, here we construct simplified versions that assume infinite sample sizes per subpopulation (Methods, section

**Previous**). This simplification corresponds to eliminating statistical sampling, leaving only genetic sampling to analyze [71]. Note that our simplified estimator nevertheless illustrates the general behavior of the WC, Weir-Hill, and Hudson estimators under arbitrary structure, and the results are equivalent to those we would obtain under finite sample sizes of individuals. While the Hudson

*F*_{ST}estimators for the independent subpopulations model*F*

_{ST}estimator compares two subpopulations [23], based on that work we derive a generalized “HudsonK” estimator for more than two subpopulations in Methods, section

**Generalized HudsonK**. Note that HudsonK, first derived in [58], also equals the Weir-Goudet

*F*_{ST}estimator*F*

_{ST}estimator for subpopulations [21] when loci are biallelic, which was derived independently using allele matching (S1 Text).

Under infinite subpopulation sample sizes, the allele frequencies at each locus and every subpopulation are known. Let *j* ∈ {1, …, *n*} index subpopulations rather than individuals and *π*_{ij} be the true allele frequency in subpopulation *j* at locus *i*. Note that *π*_{ij} are not estimated allele frequencies, but rather true subpopulation allele frequencies; this abstraction does not result in a practical estimation approach, but it greatly simplifies understanding of bias for subpopulations in a setting where there there is no statistical sampling. Although in this analysis of *F*_{ST} estimators the *π*_{ij} values are applied to subpopulations, for coherence with our previous work we shall call them “individual-specific allele frequencies” (IAF) [60, 61]. Whether for individuals or subpopulations, the key assumption is that IAFs satisfy the coancestry model of Eq (6). In this special case of infinite subpopulation sample sizes, all of WC, Weir-Hill, and HudsonK simplify to the following *F*_{ST} estimator for independent subpopulations (“indep”; derived in Methods, section **Previous F_{ST} estimators for the independent subpopulations model**):
(11)
(12)
(13)
The goal is to estimate , which is the special case of Eq (9) that weighs every subpopulation

*j*equally ().

*F*_{ST} estimation under the independent subpopulations model.

Under the independent subpopulations model for *j* ≠ *k*, where *T* is the most recent common ancestor (MRCA) population of the set of subpopulations. Note that the estimator in Eq (13) can be derived directly from Eq (6) and these assumptions using the method of moments (ignoring the existence of previous *F*_{ST} estimators; S1 Text). The expectations of the two recurrent terms in Eq (13) are
Eliminating and solving for *F*_{ST} in this system of equations recovers the estimator in Eq (13).

Before applying the convergence result in Eq (10), we test that the three conditions listed in section **Assessing the accuracy of genome-wide ratio estimators** are met. Condition (i): The locus *i* terms are and , which satisfy E[*a*_{i}] = *Ac*_{i} and E[*b*_{i}] = *Bc*_{i} with *A* = *F*_{ST}, *B* = 1, and . Condition (ii): over the distribution across loci. Condition (iii): Since , then and , and since *n* ≥ 2, *C* = 1 bounds both |*a*_{i}| and |*b*_{i}|. Therefore, for independent loci,

*F*_{ST} estimation under arbitrary coancestry.

Now we consider applying the independent subpopulations *F*_{ST} estimator to dependent subpopulations. The key difference is that now for every (*j*, *k*) will be assumed in our coancestry model in Eq (6), and now *T* may be either the MRCA population of all subpopulations or a more ancestral population. In this general setting, (*j*, *k*) may index either subpopulations or individuals. The two terms of now satisfy
where is the mean coancestry with uniform weights. There are two equations but three unknowns: *F*_{ST}, , and . The independent subpopulations model satisfies , which allows for the consistent estimation of *F*_{ST}. Therefore, the new unknown precludes consistent *F*_{ST} estimation without additional assumptions. As shown later, our additional assumption is that we can identify unrelated individuals in the data, which determines all unknowns. We defer our complete solution to this problem until kinship and its estimation challenges have been presented.

The *F*_{ST} estimator for independent subpopulations converges more generally to
(14)
(the conclusion of panel “Indep. Subpop. *F*_{ST} Estimator” in Fig 1), where
is the average of all between-subpopulation coancestry coefficients, in agreement with related calculations regarding the WC and Weir-Hill estimators [4, 21]. Therefore, under arbitrary structure the independent subpopulations estimator’s bias is due to the coancestry between subpopulations. While the limit in Eq (14) appears to vary depending on the choice of *T*, it is in fact a constant with respect to *T* (proof in S1 Text).

Since (S1 Text), this estimator has a downward bias in the general setting: it is asymptotically unbiased () only when , while bias is maximal when , where . For example, if so the MRCA population *T* is fixed, but *n* is large and for most pairs of subpopulations, then as well, and . Therefore, the magnitude of the bias of is unknown if is unknown, and small estimates may arise even if *F*_{ST} is very large.

#### Coancestry estimation as a method of moments.

Since the generalized *F*_{ST} is given by coancestry coefficients in Eq (9), a new *F*_{ST} estimator could be derived from estimates of . Here we attempt to define a method-of-moments estimator for , and find an underdetermined estimation problem, just as for *F*_{ST}. This is consistent with IBD parameters in general requiring a reference population to be determined [40], whereas in this subsection this reference population is unspecified.

Given IAFs and the coancestry model of Eq (6), the first and second moments that average across loci are (15) (16) where , , and is as before.

Suppose first that only are of interest. There are *n* estimators given by Eq (16) with *j* = *k*, each corresponding to an unknown . However, all these estimators share two nuisance parameters: and . While can be estimated from Eq (15), there are no more equations left to estimate , so this system is underdetermined. The estimation problem remains underdetermined if all estimators in Eq (16) are considered rather than only the *j* = *k* cases. Therefore, we cannot estimate coancestry coefficients consistently using only the first two moments without additional assumptions.

### Characterizing a kinship estimator and its relationship to *F*_{ST}

Given the biases we see for under arbitrary structures in the previous section, we now turn to the generalized definition of *F*_{ST} and pursue an estimate of it. Recall that our generalized *F*_{ST} in Eq (3) is defined in terms of inbreeding coefficients, which are a special case of the kinship coefficient. Kinship coefficients also determine the bias of in Eq (14) (since coancestry and kinship coefficients are closely related: see panel “Coancestry in Terms of Kinship” in Fig 1). Therefore, we will consider estimates of kinship and inbreeding in this section. Estimating kinship is also important for GWAS approaches that control for population structure [16, 24–35, 72, 73].

In this section, we focus on a standard kinship method-of-moments estimator and calculate its limit for the first time (panel “Existing Kinship Estimator” in Fig 1). We study estimators that use genotypes or IAFs, and construct *F*_{ST} estimators from their kinship estimates. We find biases comparable to those of in the previous section, and define unbiased *F*_{ST} estimators that require knowing the mean kinship or coancestry, or its proportion relative to *F*_{ST}. The results of this section directly motivate and help construct our new kinship and *F*_{ST} estimation approach in the following section.

#### Characterization of the standard kinship estimator.

Here we analyze a standard kinship estimator that is frequently used [16, 27, 30–36]. We generalize this estimator to use weights in estimating the ancestral allele frequencies, and we write it as a ratio-of-means estimator due to the favorable theoretical properties of this format as detailed in the earlier section **Assessing the accuracy of genome-wide ratio estimators**:
(17)
(18)
The estimator in Eq (18) resembles the sample covariance estimator applied to genotypes, but centers by locus *i* rather than by individuals *j* and *k*, and normalizes using estimates of . We derive Eq (18) directly using the method of moments in S1 Text. The weights in Eq (17) must satisfy *w*_{j} > 0 and , so that and .

Utilizing the kinship model for genotypes from Eq (5), we find that Eq (18) converges to
(19)
where and , which agrees with related derivations [21, 62]. (This is the conclusion of panel “Existing Kinship Estimator” in Fig 1; see S1 Text for intermediate calculations that lead to Eq (19).) Therefore, the bias of varies per pair of individuals *j* and *k*. Analogous distortions have been observed for sample covariances of genotypes [74]. The limit of in Eq (19) is constant with respect to *T* (proof in S1 Text). Similarly, inbreeding coefficient estimates derived from Eq (18) converge to
(20)
The difference between the bias of for *j* ≠ *k* in Eq (19) and in Eq (20) is visible in the kinship estimates shown toward the end of the results section. The limits of the ratio-of-means versions of two more estimators [32] are, if uses Eq (17),
(21)

The estimators in Eqs (18) and (21) are unbiased when is replaced by [16, 32, 36], and are consistent when is consistent [60]. Surprisingly, in Eq (17) is not consistent (it does not converge almost surely to ) for arbitrary population structures, which is at the root of the bias in Eqs (19) to (21). In particular, although is unbiased, its variance (S1 Text, and some special cases shown elsewhere, *e.g.*, [19]),
(22)
may be asymptotically non-zero as *n* → ∞, since is fixed and may take on any value between zero and one for arbitrary population structures. Further, as *n* → ∞ if and only if for almost all pairs of individuals (*j*, *k*). These observations hold for any weights such that . An important consequence is that the plug-in estimate of is biased (S1 Text),
which is present in all estimators we have studied.

#### Estimation of coancestry coefficients from IAFs.

Here we form a coancestry coefficient estimator analogous to Eq (18) but using IAFs. Assuming the moments in Eq (6), this estimator and its limit are (23) (24) where and are analogous to and . Eq (23) generalizes Eq (11) for arbitrary weights. Thus, use of IAFs does not ameliorate the estimation problems we have identified for genotypes. Like Eq (22), in Eq (23) is not consistent because may not converge to zero for arbitrary population structures, which causes the bias observed in Eq (24).

*F*_{ST} estimator based on the standard kinship estimator.

Since the generalized *F*_{ST} is defined as a mean inbreeding coefficient in Eq (3), here we study the *F*_{ST} estimator constructed as where is the inbreeding estimator derived from the standard kinship estimator. Although is biased, we nevertheless plug it into our definition of *F*_{ST} so that we may study how bias manifests. Note that we do not recommend utilizing this *F*_{ST} estimator in practice, but we find these results informative for identifying how to proceed in deriving new estimators in the following section.

Remarkably, the three estimators in Eqs (20) and (21) give exactly the same plug-in if the weights in *F*_{ST} and in Eq (17) match, namely
(25)
where the limit assumes locally-outbred individuals so Eq (4) holds. The analogous *F*_{ST} estimator for IAFs and its limit are
(26)
The estimators in Eqs (25) and (26) for individuals and their limits resemble those of classical *F*_{ST} estimators for populations of the form [4, 5]. in Eq (26) for subpopulations *j* with uniform weight and one locus is also *G*_{ST} for two alleles [75]. Compared to in Eq (13), in Eq (26) admits arbitrary weights and, by forgoing bias correction under the independent subpopulations model, is a simpler target of study.

Like in Eq (13), in Eqs (25) and (26) are downwardly biased since . in Eq (26) may converge arbitrarily close to zero since can be arbitrarily close to *F*_{ST} (S1 Text). Moreover, although for large *n* (see Eq (8) and panel “Coancestry in Terms of Kinship” in Fig 1), in extreme cases can exceed *F*_{ST} under the coancestry model (where ) and also under extreme local kinship, where in Eq (25) converges to a negative value.

#### Adjusted consistent oracle *F*_{ST} estimators and the “bias coefficient”.

Here we explore two adjustments to from IAFs in Eq (26) that rely on having minimal additional information needed to correct its bias. These “oracle” approaches require information that is not known in practice, but this exercise helps us understand the problem more deeply and finds further connections between the various *F*_{ST} estimators.

If is known, the bias in Eq (26) can be reversed, yielding the consistent estimator
(27)
Consistent estimates are also possible if a scaled version of is known, namely
(28)
which we call the “bias coefficient” and which has interesting properties. The bias coefficient quantifies the departure from the independent subpopulations model by comparing the mean coancestry () to the mean inbreeding coefficient (), and given *F*_{ST} > 0 satisfies 0 < *s*^{T} ≤ 1 (S1 Text). The limit in Eq (26) in terms of *s*^{T} is
(29)
Treating the limit as equality and solving for *F*_{ST} yields the following consistent estimator:
(30)
(31)
Note that and from Eqs (12) and (13) are the special case of Eqs (30) and (31) for uniform weights and ; hence, generalizes .

Lastly, using either Eqs (26) or (29), the relative error of converges to
(32)
which is approximated by *s*^{T} if *F*_{ST} ≪ 1, hence the name “bias coefficient”. Note *s*^{T} varies depending on the choice of *T*, which is necessary since *F*_{ST} (and hence the relative bias of from *F*_{ST}) depends on the choice of *T*.

### A new approach for kinship and *F*_{ST} estimation

Here, we propose a new estimation approach for kinship coefficients that has properties favorable for obtaining nearly unbiased estimates (panel “New Kinship Estimator” in Fig 1). These new kinship estimates yield an improved *F*_{ST} estimator. We present the general approach and implement a simple version of one key estimator that results in the complete proof-of-principle estimator that is evaluated in the next section and applied to human data in [59]. We also compare our approach to a related estimator of non-IBD linearly-transformed kinship values [20–22] that was proposed concurrently to ours [58].

#### General approach.

In this subsection we develop our new estimator in two steps. First, we compute a new statistic *A*_{jk} that is proportional in the limit of infinite loci to times a nuisance factor *v*^{T}. Second, we estimate and remove *v*^{T} to yield the proposed estimator . —an estimator of the limit of the minimum *A*_{jk}—yields *v*^{T} if the least related pair of individuals in the data has , which sets *T* to the MRCA population of all the individuals in the data. The new kinship estimator immediately results in new inbreeding () and *F*_{ST} () estimators. This general approach leaves the implementation of open; the simple implementation applied in this work is described in subsection **Proof-of-principle kinship estimator using subpopulation labels**, but our method can be readily improved by substituting in a better in the future.

Applying the method of moments to Eq (5), we derive the following statistic (S1 Text), whose expectation is proportional to :
(33)
Compared to the standard kinship estimator in Eq (19), which has a complex asymptotic bias determined by *n* parameters ( for each *j* ∈ {1, …, *n*}), the *A*_{jk} statistics estimate kinship with a bias controlled by the sole unknown parameter shared by all pairs of individuals. The key to estimating is to notice that if then . Thus, assuming , which sets *T* to the MRCA population, then the minimum *A*_{jk} yields the nuisance parameter. However, we recommend using a more stable estimate than the minimum *A*_{jk} to unbias all *A*_{jk}, such as the estimator presented in the next subsection.

In general, suppose is a consistent estimator of the limit of the minimum *E*[*A*_{jk}|*T*], or equivalently,
along with the assumption that for some *v*^{T} ≠ 0. Our new kinship estimator follows directly from replacing with and solving for in Eq (33), which results in a consistent kinship estimator (given the convergence proof of section **Assessing the accuracy of genome-wide ratio estimators**):
(34)
The resulting new inbreeding coefficient estimator is
(35)
and the new *F*_{ST} estimator is consistent for locally-outbred individuals (estimates Eq (4)):
(36)
Thus, only the implementation of is left unspecified from this general estimation approach of kinship and *F*_{ST}. The implementation of used in the analyses in this work is given in the next subsection.

#### Proof-of-principle kinship estimator using subpopulation labels.

To showcase the potential of the new estimators, we implement a simple proof-of-principle version of needed for our new kinship estimator ( in Eq (34)). This relies on an appropriate partition of the *n* individuals into *K* subpopulations (denoted *S*_{u} for *u* ∈ {1, …, *K*}), where the only requirement is that the kinship coefficients between pairs of individuals across the two most unrelated subpopulations is zero, as detailed below. Note that, unlike the the independent subpopulations model of section ** F_{ST} estimation based on the independent subpopulations model**, these

*K*subpopulations need not be independent nor unstructured. The desired estimator is the minimum average

*A*

_{jk}over all subpopulation pairs: (37) This consistently estimates the limit of the minimum

*A*

_{jk}if for the least related pair of subpopulations

*S*

_{u},

*S*

_{v}.

This estimator should work well for individuals truly divided into subpopulations, but may be biased for a poor choice of subpopulations, in particular if the minimum mean between subpopulations is far greater than zero. For this reason, inspection of the kinship estimates is required and careful construction of appropriate subpopulations may be needed. See our analysis of human data for detailed examples [59]. Future work could focus on a more general that circumvents the need for subpopulations of our proof-of-principle estimator.

#### Comparison to the Weir-Goudet kinship estimator for individuals.

Here we analyze the Weir-Goudet (WG) kinship estimator for individuals [20–22]. This has connections to our new estimator but differs in having the goal of estimating linearly-transformed kinship values. In our framework, the WG estimator is given by
Therefore, this estimator differs from our proposal [58] by replacing our with . Under the kinship model, the expectation of is
Therefore, the limit of this estimator is
(38)
which agrees with calculations in the original WG work [20–22]. Note that, assuming that kinship coefficients must be non-negative, the above estimator recovers the kinship IBD probabilities if and only if which holds if and only if for every pair of individuals *j* ≠ *k*. The resulting WG inbreeding coefficient estimator is
which estimates linearly-transformed inbreeding values [21]. Therefore, the resulting WG *F*_{ST} estimator (for individuals) also targets a linearly-transformed *F*_{ST} value (under locally-outbred individuals, where *F*_{ST} is given by Eq (4)), namely
The WG authors also briefly consider a variant of their kinship estimator that is normalized using the minimum kinship value as we did, developed concurrently with our approach [58], but was largely dismissed as an unnecessary correction [21, 76]. See S1 Text for a detailed proof that the general estimator framework we propose here (Eqs (33) and (34)) is algebraically equivalent to our original formulation in [58].

Note that the original WG does not estimate *F*_{ST} from individuals as considered above; instead, *F*_{ST} is estimated from coancestry estimates for subpopulations (which equals our HudsonK for biallelic loci, S1 Text) [20–22]. For completeness, we consider both kinds of *F*_{ST} estimates in the evaluations that follow.

### Simulations evaluating *F*_{ST} and kinship estimators

#### Overview of simulations.

We simulate genotypes from two models to illustrate our results when the true population structure parameters are known. Both simulations have clearly-defined IBD probability parameters in terms of the MRCA population. The first simulation satisfies the independent subpopulations model that existing *F*_{ST} estimators assume. The second simulation is from an admixture model with no independent subpopulations and pervasive kinship designed to induce large downward biases in existing kinship and *F*_{ST} estimators (Fig 2). This admixture scenario resembles the population structure we estimated for Hispanics in the 1000 Genomes Project [59]: compare the simulated kinship matrix (Fig 2B) and admixture proportions (Fig 3C) to our estimates on the real data [59]. Both simulations have *n* = 1000 individuals, *m* = 300, 000 loci, and *K* = 10 subpopulations or intermediate subpopulations. These simulations have *F*_{ST} = 0.1, comparable to previous estimates between human populations (in 1000 Genomes, the estimated *F*_{ST} between CEU (European-Americans) and CHB (Chinese) is 0.106, between CEU and YRI (Yoruba from Nigeria) it is 0.139, and between CHB and YRI it is 0.161 [23]).

Both panels have *n* = 1000 individuals along both axes, *K* = 10 subpopulations (final or intermediate), and *F*_{ST} = 0.1. Color corresponds to between individuals *j* and *k* (equal to off-diagonal, along the diagonal). (A) The independent subpopulations model has between subpopulations, and varying per subpopulation, resulting in a block-diagonal coancestry matrix. (B) Our admixture scenario models a 1D geography with extensive admixture and intermediate subpopulation differentiation that increases with distance, resulting in a smooth coancestry matrix with no independent subpopulations (no between blocks). Individuals are ordered along each axis by geographical position.

We model a 1D geography population that departs strongly from the independent subpopulations model. (A) *K* = 10 intermediate subpopulations, evenly spaced on a line, evolved independently in the past with *F*_{ST} increasing with distance, which models a sequence of increasing founder effects (from left to right) to mimic the global human population. (B) Once differentiated, individuals in these intermediate subpopulations spread by random walk modeled by Normal densities. (C) *n* = 1000 individuals, sampled evenly in the same geographical range, are admixed proportionally to the previous Normal densities. Thus, each individual draws most of its alleles from the closest intermediate subpopulation, and draws the fewest alleles from the most distant populations. Long-distance random walks of intermediate subpopulation individuals results in kinship for admixed individuals that decays smoothly with distance in Fig 2B. (D) For *F*_{ST} estimators that require a partition of individuals into subpopulations, individuals are clustered by geographical position (*K* = 10).

The independent subpopulations simulation satisfies the HudsonK and BayeScan estimator assumptions: each independent subpopulation *S*_{u} has a different *F*_{ST} value of relative to the MRCA population *T* (Fig 2A). Ancestral allele frequencies are drawn uniformly between 0.01 and 0.5. Allele frequencies for *S*_{u} and locus *i* are drawn independently from the Balding-Nichols (BN) distribution [3] with parameters and . Every individual *j* in subpopulation *S*_{u} draws alleles randomly with probability . Subpopulation sample sizes are drawn randomly (Methods, section **Simulations**).

The admixture simulation corresponds to a “BN-PSD” model [6, 27, 34, 60, 77]: the intermediate subpopulations are independent subpopulations that draw from the BN model, then each individual *j* constructs its allele frequencies as , which is a weighted average of the subpopulation allele frequencies with the admixture proportions *q*_{ju} of individual *j* and subpopulation *u* as weights (which satisfy ), as in the Pritchard-Stephens-Donnelly (PSD) admixture model [63–65]. We constructed *q*_{ju} that model admixture resulting from spread by random walk of the intermediate subpopulations along a one-dimensional geography, as follows. Intermediate subpopulations *S*_{u} are placed on a line with differentiation that grows with distance, which corresponds to a serial founder effect (Fig 3A). Upon differentiation, individuals in each *S*_{u} spread by random walk, a process modeled by Normal densities (Fig 3B). Admixed individuals derive their ancestry proportional from these Normal densities, resulting in a genetic structure governed by geography (Figs 3C and 2B) and departing strongly from the independent subpopulations model (Fig 3D). The amount of spread—which sets the mean kinship across all individuals—was chosen to give a bias coefficient of , which by Eq (32) results in a large downward bias for (in contrast, the independent subpopulations simulation has *s*^{T} = 0.1). The true coancestry and *F*_{ST} parameters of this simulation are given by the values of the intermediate subpopulations and the admixture coefficients *q*_{ju} of the individuals via the following equations [57]:
(39)
The first equation above connecting coancestry to admixture proportions was derived independently in other work [62], but the *F*_{ST} for the admixed individuals was absent and instead follows from our generalized *F*_{ST} definition given in Eq (9). See Methods, section **Simulations** for additional details regarding these simulations.

#### Evaluation of *F*_{ST} estimators.

Our admixture simulation illustrates the large biases that can arise if existing *F*_{ST} estimators that require independent subpopulations or *F*_{ST} estimates derived from existing kinship estimators are misapplied to arbitrary population structures to estimate the generalized *F*_{ST}, and demonstrate the higher accuracy of our new *F*_{ST} estimator ( given by the combination of Eqs (36) and (37)). The WC *F*_{IT} (total inbreeding) estimator was also evaluated.

First, we test these estimators in our independent subpopulations simulation. The HudsonK (Methods, section **Generalized HudsonK F_{ST} estimator**) and BayeScan

*F*

_{ST}estimators are consistent in this simulation, since their assumptions are satisfied (Fig 4A). The WC

*F*

_{ST}estimator assumes that for all subpopulations

*S*

_{u}, which does not hold; nevertheless, WC has only a small bias (Fig 4A). The WC

*F*

_{IT}estimator arrives at similar estimates, as it should since there is no local inbreeding, so the true

*F*

_{IT}also equals

*F*

_{ST}. The Weir-Hill estimator permits different values per subpopulation, but assigns equal weight to individuals rather than subpopulations (Methods, section

**The Weir-Hill**), resulting in a slightly different target

*F*_{ST}estimator*F*

_{ST}(we verified that these estimates are unbiased for this

*F*

_{ST}). For comparison, we show the standard kinship-based in Eq (25) (weights from Methods, section

**Simulations**) and based on the Weir-Goudet kinship estimates for individuals, both of which do not have corrections that would make them consistent under the independent subpopulations model. Since the number of subpopulations

*K*is large, has a small relative bias of about (Fig 4A); greater bias is expected for smaller

*K*. Our new

*F*

_{ST}estimator has a very small bias in this simulation resulting from estimating the minimum kinship from the smallest kinship between subpopulations (see Eq (37)) rather than their average as HudsonK does implicitly (Fig 4A).

The Weir-Cockerham, Weir-Hill, Weir-Goudet (for individuals), HudsonK (equal to Weir-Goudet for subpopulations, S1 Text), BayeScan, in Eq (25) derived from the standard kinship estimator, and our new *F*_{ST} estimator in Eqs (34) and (37), are evaluated on simulated genotypes from our two models (Fig 2). The Weir-Cockerham *F*_{IT} estimator was also included to show that estimation of total inbreeding behaves similarly to *F*_{ST} estimators. (A) The independent subpopulations model required by the Weir-Hill, HudsonK, and BayeScan *F*_{ST} estimators. All but standard kinship () and Weir-Goudet (for individuals) recover the target *F*_{ST} IBD probability in Eq (9) (red line) with small errors. (B) Our admixture scenario, which has no independent subpopulations, was constructed so . Only our new estimates are accurate. The rest of these estimators give values smaller than the target *F*_{ST} IBD probability, which result from treating kinship as zero between every subpopulations imposed by geographic clustering (or between individuals for Standard Kinship and Weir-Goudet). The estimator limit in Eq (14) (green dotted line) overlaps the true *F*_{ST} (red line) in (A) but not (B). Estimates (blue) include 95% prediction intervals (often too narrow to see) from 39 independently-simulated genotype matrices for each model (Methods, section **Prediction intervals**).

Next we test these estimators in our admixture simulation. To apply the *F*_{ST} estimators that require subpopulations to the admixture model, individuals are clustered into subpopulations by their geographical position (Fig 3D). We find that estimates of all existing methods are smaller than the true *F*_{ST} by nearly half, as predicted by the limit of in Eq (14) (Fig 4B). The WC *F*_{IT} estimator obtains slightly larger estimates than the WC *F*_{ST} estimator, but overall remains as biased as the other *F*_{ST} estimators, showing that the use of a total inbreeding estimator for independent subpopulations displays the same bias as the corresponding *F*_{ST} estimator. By construction, the kinship-based also has a large relative bias of about *s*^{T} = 50%; remarkably, all existing *F*_{ST} estimators for subpopulations suffer from comparable biases. Thus, the corrections for independent subpopulations present in the WC, Weir-Hill and HudsonK estimators, or the Bayesian likelihood modeling of BayeScan, are insufficient for accurate estimation of the target generalized *F*_{ST} (Eq (9)) in this admixture scenario. Only our new *F*_{ST} estimator achieves accurate estimates of the generalized *F*_{ST} in the admixture simulation (Fig 4B).

#### Evaluation of kinship estimators.

Our admixture simulation illustrates the distortions of the standard kinship estimator in Eq (18), the linearly-transformed kinship values given by the Weir-Goudet estimator, and demonstrates the improved accuracy of our new kinship estimator given by the combination of Eqs (34) and (37). Kinship matrix estimates and their limits are visualized as heatmaps in Fig 5, whereas estimator accuracy is shown directly in Fig 6. The limit of the standard estimator in Eq (18) would have had a uniform bias if held for all individuals *j*. For that reason, our admixture simulation has varying differentiation per intermediate subpopulation *S*_{u} (Fig 3A), which causes large differences in per individual *j* and therefore large distortions in . The Weir-Goudet approach estimates the linearly-transformed kinship values calculated in Eq (38).

Observed accuracy for two existing kinship coefficient estimators is illustrated in our admixture simulation and contrasted to the nearly unbiased estimates of our new estimator. Plots show *n* = 1000 individuals along both axes, and color corresponds to between individuals *j* ≠ *k* and to along the diagonal ( is in the same scale as for *j* ≠ *k*; plotting , which have a minimum value of , would result in a discontinuity in this figure). (A) True kinship matrix. (B) Estimated kinship using our new estimator in Eqs (34) and (37) from simulated genotypes recovers the true kinship matrix with high accuracy. (C) Theoretical limit of in Eq (19) as the number of independent loci goes to infinity demonstrates the accuracy of our bias predictions under the kinship model. (D) Standard kinship estimates given by Eq (18) from simulated genotypes are downwardly biased on average and distorted by pair-specific amounts. (E) Theoretical limit of the Weir-Goudet kinship estimator given by Eq (38). (F) Weir-Goudet kinship estimates from the same simulated genotypes agree with our calculated limit.

Here the estimated kinship values are directly compared to their true values, in the same admixture simulation data (*n* = 1000 individuals) shown in the previous figure. (A) Kinship between different individuals (excluding inbreeding). The new estimator has practically no bias in this evaluation (falls on the 1-1 dashed gray line). The standard estimator has a complex, non-linear bias that covers a large area of errors. (B) Inbreeding comparison, shows the bias of the standard estimate follows a different pattern for inbreeding compared to kinship between individuals. To better visualize and compare data across panels, a random subset of *n* points (out of the original *n*(*n* − 1)/2 unique individual pairs) were plotted in (A), matching the number of individuals (number of points in (B)).

Our new kinship estimator (Fig 5B) recovers the true kinship matrix of this complex population structure (Fig 5A), with an RMSE of 2.83% relative to the mean (Fig 6). In contrast, estimates using the standard estimator have a large overall downward bias (Fig 5C), resulting in an RMSE of 115.72% from the true relative to the mean (Fig 6). Additionally, estimates from are very distorted, with an abundance of cases—some of which are negative estimates (blue in Fig 5C)—but remarkably also cases with (top left corner of Figs 5C and 6).

Now we compare the convergence of the ratio-of-means and mean-of-ratios versions of the standard kinship estimator to their biased limit we calculated in Eq (19) (Fig 5D). The ratio-of-means estimate (Fig 5C) has an RMSE of 2.14% from its limit relative to the mean . In contrast, the mean-of-ratios estimates that are prevalent in the literature have a greater RMSE of 10.77% from the same limit in Eq (19). Thus, as expected from our theoretical results in section **Assessing the accuracy of genome-wide ratio estimators**, the ratio-of-means estimate is much closer to the desired limit than the mean-of-ratio estimate. The distortions are similar for the estimator that uses IAFs in Eq (24), with reduced RMSEs from its limit of 0.32% and 8.82% for the ratio-of-means and mean-of-ratios estimates, respectively.

#### Evaluation of oracle-adjusted *F*_{ST} estimators.

Here we verify additional calculations for the bias of the standard kinship-based estimator and the unbiased adjusted “oracle” *F*_{ST} estimators that require the true mean kinship or the bias coefficient *s*^{T} to be known. Note that in Eq (36) is related but not identical to these oracle estimators. We tested both IAF (Fig 7A) and genotype (Fig 7B) versions of these estimators. The unadjusted in Eq (26) is severely biased (blue in Fig 7) by construction, and matches the calculated limit for IAFs and genotypes (green lines in Fig 7, which are close because ). In contrast, the two consistent adjusted estimators and in Eqs (27) and (31) estimate *F*_{ST} quite well (blue predictions overlap the true *F*_{ST} red line in Fig 7). However, and are oracle methods, since they require parameters (, , *s*^{T}) that are not known in practice.

The convergence values we calculated for the standard kinship plug-in and adjusted *F*_{ST} estimators are validated using our admixture simulation. All adjusted estimators are unbiased but are “oracle” methods, since the mean kinship (), mean coancestry (), or bias coefficient ( for IAFs, replaced by for genotypes) are usually unknown. (A) Estimation from individual-specific allele frequencies (IAFs): is the standard coancestry plug-in estimator in Eq (26); “Adj. ” is in Eq (27); “Adj. *s*” is in Eq (31). (B) For genotypes, is given in Eq (25), and the adjusted estimators use rather than . Lines: true *F*_{ST} (red line), limits of biased estimators (green lines, which differ slightly per panel). Estimates (blue) include 95% prediction intervals (too narrow to see) from 39 independently-simulated genotype matrices for our admixture model (Methods, section **Prediction intervals**).

Prediction intervals were computed from estimates over 39 independently-simulated IAF and genotype matrices (Methods, section **Prediction intervals**). Estimator limits are always contained in these intervals because the number of independent loci (*m* = 300, 000) is sufficiently large. Estimates that use genotypes have wider intervals than estimates from IAFs; however, IAFs are not known in practice, and use of estimated IAFs might increase noise. Genetic linkage, not present in our simulation, will also increase noise in real data.

## Discussion

We studied analytically the most commonly-used estimators of *F*_{ST} and kinship, which can be derived using the method of moments. We determined the estimation limits of convergence of these approaches under two models of arbitrary population structure (Fig 1). We found that no existing approaches estimate the generalized *F*_{ST} (an IBD probability) accurately (but note that some of these approaches intended to estimate a linearly-transformed *F*_{ST} quantity and not the IBD probability). We also showed that the standard kinship estimator is biased on structured populations (particularly when the average kinship is comparable to the kinship coefficients of interest), and this bias varies for each pair of individuals. These results led us to a new kinship estimator, which is consistent if the minimum kinship is estimated consistently (Fig 1). We presented an implementation of this approach, which is practically unbiased in our simulations. Our kinship and *F*_{ST} estimates in human data are consistent with the African Origins model while suggesting that human differentiation is considerably greater than previously estimated [59].

Estimation of *F*_{ST} in the correct scale is crucial for its interpretation as an IBD probability, for obtaining comparable estimates in different datasets and across species, as well as for DNA forensics [3, 7, 19, 20, 78–80]. Our framework results in a new unbiased genome-wide *F*_{ST} estimator. However, our findings may not have direct implications for single-locus *F*_{ST} estimate approaches where only the relative ranking matters, such as for the identification of loci under selection [8, 10, 81–86], assuming that the bias of the genome-wide estimator carries over uniformly to all single-locus estimates. Our convergence calculations in section **Assessing the accuracy of genome-wide ratio estimators** require large numbers of loci, so they do not apply to single-locus estimates. Moreover, various methods for single-locus *F*_{ST} estimation for multiple alleles suffer from a strong dependence to the maximum allele frequency and heterozygosity [83–85, 87–90] that suggests that a more complicated bias is present in these single-locus *F*_{ST} estimators.

We have shown that the misapplication of existing *F*_{ST} estimators for independent subpopulations may lead to downwardly-biased estimates that can approach zero even when the true generalized *F*_{ST} is large. Weir-Cockerham [17], Weir-Hill [4], HudsonK (which generalizes the Hudson pairwise *F*_{ST} estimator [23] to *K* independent populations; also equals the Weir-Goudet approach for subpopulations [21]; S1 Text), and BayeScan [10]*F*_{ST} estimates in our admixture simulation are all smaller than the *F*_{ST} target by nearly a factor of two (Fig 4B), and differ from our new *F*_{ST} estimates in humans by nearly a factor of three [59]. To be accurate, existing *F*_{ST} estimators require independent subpopulations, so the observed biases arise from their misapplication to subpopulations that are neither independent not homogeneous. Nevertheless, natural populations—particularly humans—often do not adhere to the independent subpopulations model [59, 91–95].

The standard kinship coefficient estimator we investigated is often used to control for population structure in GWAS and to estimate genome-wide heritability [16, 27, 30–35]. While this estimator was known to be biased [16, 35], no closed-form limit had been calculated until very recently [21, 62]. These kinship estimates are biased downwards on average, but bias also varies for each pair of individuals (Figs 1 and 5). Thus, the use of these distorted kinship estimates may be problematic in GWAS or for estimating heritability, but the extent of the problem remains to be determined.

We developed a theoretical framework for assessing genome-wide ratio estimators of *F*_{ST} and kinship. We proved that common ratio-of-means estimators converge almost surely to the ratio of expectations for infinite independent loci (S1 Text). Our result justifies approximating the expectation of a ratio-of-means estimator with the ratio of expectations [4, 17, 23]. However, mean-of-ratios estimators may not converge to the ratio of expectations for infinite loci. Mean-of-ratios estimators are potentially asymptotically unbiased for infinite individuals, but it is unclear which estimators have this behavior. We found that the ratio-of-means kinship estimator had much smaller errors from the ratio of expectations than the more common mean-of-ratios estimator, whose convergence value is unknown. Therefore, we recommend ratio-of-means estimators, whose asymptotic behavior is well understood.

Our new framework enables accurate *F*_{ST} estimation in more complex datasets than before, but challenges remain. One challenge is the estimation of local inbreeding coefficients, which are required for estimating the generalized *F*_{ST} when not all individuals are locally outbred. To this end, we suggest employing existing approaches that infer inbreeding from large runs of homozygosity or related strategies [66–68], particularly when such self-IBD blocks are much larger than observed between individuals in the same subpopulation. A streamlined approach for jointly estimating total and local inbreeding is desirable, but will require an appropriate evaluation featuring realistic simulation of local inbreeding in a complex population structure. Another challenge is the estimation of the minimum kinship value without the use of subpopulation labels, so that accurate *F*_{ST} estimates can be obtained with even less user supervision. A more general unsupervised method could better ensure accuracy under extreme cases, such as when there are few unrelated individual pairs. These challenges can be overcome with the estimators we have presented, although supervision is needed to ensure that local inbreeding and the minimum kinship are estimated correctly.

We have demonstrated the need for new models and methods to study complex population structures, and have proposed a new approach for kinship and *F*_{ST} estimation that provides nearly unbiased estimates in this setting. Extending our implementation to deliver consistent accuracy in arbitrary population structures will require further innovation, and the results provided here may be useful in leading to more robust estimators in the future.

## Methods

### Previous *F*_{ST} estimators for the independent subpopulations model

Here we summarize the previous Weir-Cockerham, Weir-Hill, and Hudson *F*_{ST} estimators for independent subpopulations and derive the generalized HudsonK estimator for more than two subpopulations (which also equals the recent Weir-Goudet *F*_{ST} estimator for subpopulations under biallelic loci; S1 Text). We show that each of these estimators reduces, under infinite subpopulation sizes, to in Eqs (11) to (13) that was studied in the results. In this section, let *i* index the *m* loci, *j* index the *n* subpopulations, *n*_{j} be the number of individuals sampled from subpopulation *j*, and be the sample reference allele frequency at locus *i* in subpopulation *j*.

#### The Weir-Cockerham *F*_{ST} estimator.

The Weir-Cockerham (WC) *F*_{ST} estimator [17] estimates the coancestry parameter *θ*^{T} shared by each of the *n* independent subpopulation in consideration. Let denote the fraction of heterozygotes in subpopulation *j* for locus *i*. The ratio-of-means WC *F*_{ST} estimator and its limit for independent subpopulations ( for *j* ≠ *k*) with equal differentiation () is
Note that above weighs every individual equally by weighing subpopulation *j* proportional to its sample size *n*_{j}, so it equals the estimator in Eq (17) with uniform weights.

Now we simplify this estimator as the sample size of every subpopulation becomes infinite. First set the sample size of every subpopulation *n*_{j} equal to their mean , which implies *C*^{2} = 0 and
Now we take the limit as the sample size , which results in sample allele frequencies converging to the true subpopulation allele frequencies for every subpopulation *j* and locus *i*, and
which matches the in Eqs (11) to (13) as desired. Note the number of subpopulations *n* remains finite, and the sample heterozygosity is not needed in the limit.

#### The Weir-Hill *F*_{ST} estimator.

Weir and Hill developed new estimators for subpopulation-specific *F*_{ST} values and considered the effects of non-independent subpopulations [4]. However, these estimators target linearly-transformed *F*_{ST} values, and recover the *F*_{ST} defined in Eq (9) only when subpopulations are independent [4], so we group them here with other estimators that strictly assume independent subpopulations. For simplicity, here we only consider the global *F*_{ST} estimator; the estimators of the coancestry matrix of the subpopulations was found to have the same overall linear transformation [4]. In the limit of infinite subpopulation sizes, this estimator also converges to the asymptotic *F*_{ST} estimator for independent subpopulations () discussed in the main text.

The Weir-Hill (WH) *F*_{ST} estimator, simplified here for biallelic loci but extended to average over loci, and its limit, are given by
where the target *F*_{ST} and both weigh individuals (rather than subpopulations) equally [4]:
For equal sample sizes *n*_{j} = *n*_{S}∀*j*, we have , , and the estimator becomes
Therefore, as sample sizes per subpopulation go to infinity (*n*_{S} → ∞, which results in for every (*i*, *j*)), we again recover the desired limiting *F*_{ST} estimator for independent subpopulations ( in Eqs (11) to (13)).

#### The Hudson *F*_{ST} estimator.

The Hudson pairwise *F*_{ST} estimator [23] measures the differentiation of two subpopulations (*j*, *k*). The estimator and its limit for two independent subpopulations () is
(40)

#### Generalized HudsonK *F*_{ST} estimator.

Here we derive the “HudsonK” estimator (first made available in [58]), which generalizes the Hudson pairwise *F*_{ST} estimator in Eq (40) to *n* independent subpopulations. This estimator also equals the recent Weir-Goudet *F*_{ST} estimator for subpopulations [21] (for biallelic loci; S1 Text). Note that for independent subpopulations, the *F*_{ST} of all the subpopulations equals the mean pairwise *F*_{ST} of every pair of subpopulations:
For that reason, averaging numerators and denominators of the pairwise estimator in Eq (40) before computing the ratio, we obtain the generalized estimator and a limit under independent subpopulations of
Note that unlike the WC and Weir-Hill estimators, above weighs every subpopulation equally, so every individual is weighed inversely proportional to the sample sizes *n*_{j} of their subpopulation *j*.

Like WC and Weir-Hill, simplifies to in Eqs (11) to (13) in the limit of infinite sample sizes *n*_{j} → ∞, where for every (*i*, *j*).

### Simulations

#### Construction of subpopulation allele frequencies.

We simulate *K* = 10 subpopulations *S*_{u} and *m* = 300, 000 independent loci. Every locus *i* draws We set where *τ* ≤ 1 tunes *F*_{ST}. For the independent subpopulations model, so gives the desired *F*_{ST} (*τ* ≈ 0.18 for *F*_{ST} = 0.1). For the admixture model, *τ* is found numerically (*τ* ≈ 0.90 for *F*_{ST} = 0.1; see last subsection). Lastly, values are drawn from the Balding-Nichols distribution,
which results in subpopulation allele frequencies that obey the coancestry model of Eq (6), with and [3], as desired.

#### Random subpopulation sizes.

We randomly generate sample sizes **r** = (*r*_{u}) for *K* subpopulations and individuals, as follows. First, draw **x** ∼ Dirichlet (1, …, 1) of length *K* and **r** = round(*n* **x**). While , draw a new **r**, to prevent small subpopulations (they do not occur in real data). Due to rounding, may not equal *n* as desired. Thus, while , a random *u* is updated to *r*_{u} ← *r*_{u} + sgn(*δ*), which brings *δ* closer to zero at every iteration. Weights for individuals *j* in *S*_{u} are so the generalized *F*_{ST} matches from the independent subpopulations model (see section **The generalized F_{ST} for arbitrary population structures**), which HudsonK estimates.

#### Admixture proportions from 1D geography.

We construct *q*_{ju} from random-walk migrations along a one-dimensional geography. Let *x*_{u} be the coordinate of intermediate subpopulation *u* and *y*_{j} the coordinate of a modern individual *j*. We assume *q*_{ju} is proportional to *f*(|*x*_{u} − *y*_{j}|), or
where *f* is the Normal density function with *μ* = 0 and tunable *σ*. The Normal density models random walks, where *σ* sets the spread of the populations (Fig 5). Our simulation uses *x*_{u} = *u* and , so the intermediate subpopulations span between 1 and *K* and individuals span between and . For the *F*_{ST} estimators that require subpopulations, individual *j* is assigned to the nearest subpopulation *S*_{u} (the *u* that minimizes |*x*_{u} − *y*_{j}|; Fig 3D); these subpopulations have equal sample size, so is appropriate.

#### Choosing *σ* and *τ*.

Here we find values for *σ* (controls *q*_{jk}) and *τ* (scales ) that give and *F*_{ST} = 0.1 in the admixture model. In our simulation, and , so applying those parameters to Eq (39) gives and . Therefore,
depends only on *σ*. A numerical root finder finds that *σ* ≈ 1.78 gives . For fixed *q*_{ju},
*F*_{ST} = 0.1 is achieved with *τ* ≈ 0.901.

### Prediction intervals

Prediction intervals with *α* = 95% correspond to the range of *n* = 39 independent *F*_{ST} estimates. In the general case, *n* independent statistics are given in order *X*_{(1)} < … < *X*_{(n)}. Then *I* = [*X*_{(j)}, *X*_{(n+1−j)}] is a prediction interval with confidence [96]. In our case, *j* = 1 and *n* = 39 gives *α* = 0.95, as desired. Each estimate was constructed from simulated data with the same dimensions and structure as before (fixed and *q*_{ju}; fixed sample sizes for the independent subpopulations model), but with drawn separately for each estimate.

### BayeScan and Weir-Goudet implementations

Weir-Goudet (WG) kinship estimates [20–22] were calculated using the function snpgdsIndivBeta in the R package SNPRelate 1.20.1 available on Bioconductor and GitHub. We found identical estimates using the function beta.dosage in the R package hierfstat 0.4.30 available on GitHub. WG (individuals) *F*_{ST} estimates were computed from the kinship estimates as described in section **Comparison to the Weir-Goudet kinship estimator for individuals**.

BayeScan 2.1 was downloaded from http://cmpg.unibe.ch/software/BayeScan/. To estimate *F*_{ST}, first the per-subpopulation *F*_{ST} values were estimated across loci assuming no selection, then the global *F*_{ST} was given by the mean *F*_{ST} across subpopulations.

### Software

An R package called popkin, which implements the kinship and *F*_{ST} estimation methods proposed here, is available on the Comprehensive R Archive Network (CRAN) at https://cran.r-project.org/package=popkin and on GitHub at https://github.com/StoreyLab/popkin.

An R package called bnpsd, which implements the BN-PSD admixture simulation, is available on CRAN at https://cran.r-project.org/package=bnpsd and on GitHub at https://github.com/StoreyLab/bnpsd.

An R package called popkinsuppl, which implements memory-efficient algorithms for the Weir-Cockerham, Weir-Hill, and HudsonK *F*_{ST} estimators, and the standard kinship estimator, is available on GitHub at https://github.com/OchoaLab/popkinsuppl.

Public code reproducing these analyses are available at https://github.com/StoreyLab/human-differentiation-manuscript.

## Supporting information

### S1 Text. Supplementary information.

Includes mathematical proofs and other calculations, including proof of convergence of ratio-of-means estimators, proof that the Weir-Goudet *F*_{ST} estimator for subpopulations equals HudsonK, derivation of existing method-of-moment estimators, proof that *F*_{ST} and kinship estimator limits are constants with respect to the ancestral population *T*, mean coancestry bounds, moments of estimator building blocks, the derivation of our new kinship estimator, and proof that our estimator from our original 2016 manuscript is algebraically equivalent to the one presented here.

https://doi.org/10.1371/journal.pgen.1009241.s001

(PDF)

## References

- 1.
Malécot G. Mathématiques de l’hérédité. Masson et Cie; 1948.
- 2. Wright S. The genetical structure of populations. Ann Eugen. 1951;15(4):323–354.
- 3. Balding DJ, Nichols RA. A method for quantifying differentiation between populations at multi-allelic loci and its implications for investigating identity and paternity. Genetica. 1995;96(1-2):3–12.
- 4. Weir BS, Hill WG. Estimating F-Statistics. Annual Review of Genetics. 2002;36(1):721–750.
- 5. Nicholson G, Smith AV, Jónsson F, Gústafsson O, Stefánsson K, Donnelly P. Assessing population differentiation and isolation from single-nucleotide polymorphism data. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2002;64(4):695–715.
- 6. Falush D, Stephens M, Pritchard JK. Inference of population structure using multilocus genotype data: linked loci and correlated allele frequencies. Genetics. 2003;164(4):1567–1587.
- 7. Balding DJ. Likelihood-based inference for genetic correlation coefficients. Theoretical Population Biology. 2003;63(3):221–230.
- 8. Beaumont MA, Balding DJ. Identifying adaptive genetic divergence among populations from genome scans. Molecular Ecology. 2004;13(4):969–980.
- 9. Foll M, Gaggiotti O. Identifying the Environmental Factors That Determine the Genetic Structure of Populations. Genetics. 2006;174(2):875–891.
- 10. Foll M, Gaggiotti O. A genome-scan method to identify selected loci appropriate for both dominant and codominant markers: a Bayesian perspective. Genetics. 2008;180(2):977–993.
- 11. Coop G, Witonsky D, Rienzo AD, Pritchard JK. Using Environmental Correlations to Identify Loci Underlying Local Adaptation. Genetics. 2010;185(4):1411–1423.
- 12. Thompson EA. The estimation of pairwise relationships. Ann Hum Genet. 1975;39(2):173–188.
- 13. Milligan BG. Maximum-likelihood estimation of relatedness. Genetics. 2003;163(3):1153–1167.
- 14.
Jacquard A. Structures génétiques des populations. Paris: Masson et Cie; 1970.
- 15. Csűrös M. Non-identifiability of identity coefficients at biallelic loci. Theor Popul Biol. 2014;92:22–29.
- 16. Astle W, Balding DJ. Population Structure and Cryptic Relatedness in Genetic Association Studies. Statist Sci. 2009;24(4):451–471.
- 17. Weir BS, Cockerham CC. Estimating F-Statistics for the Analysis of Population Structure. Evolution. 1984;38(6):1358–1370.
- 18. Weir BS, Cardon LR, Anderson AD, Nielsen DM, Hill WG. Measures of human population structure show heterogeneity among genomic regions. Genome Res. 2005;15(11):1468–1476.
- 19. Buckleton J, Curran J, Goudet J, Taylor D, Thiery A, Weir BS. Population-specific FST values for forensic STR markers: A worldwide survey. Forensic Science International: Genetics. 2016;23:91–100.
- 20. Weir B, Zheng X. SNPs and SNVs in forensic science. Forensic Science International: Genetics Supplement Series. 2015;5(Dec):e267–e268.
- 21. Weir BS, Goudet J. A Unified Characterization of Population Structure and Relatedness. Genetics. 2017;206(4):2085–2103.
- 22. Goudet J, Kay T, Weir BS. How to estimate kinship. Mol Ecol. 2018;27(20):4121–4135.
- 23. Bhatia G, Patterson N, Sankararaman S, Price AL. Estimating and interpreting FST: the impact of rare variants. Genome Res. 2013;23(9):1514–1521.
- 24. Xie C, Gessler DD, Xu S. Combining different line crosses for mapping quantitative trait loci using the identical by descent-based variance component method. Genetics. 1998;149(2):1139–1146.
- 25. Yu J, Pressoir G, Briggs WH, Vroh Bi I, Yamasaki M, Doebley JF, et al. A unified mixed-model method for association mapping that accounts for multiple levels of relatedness. Nat Genet. 2006;38(2):203–208. pmid:16380716
- 26. Aulchenko YS, de Koning DJ, Haley C. Genomewide rapid association using mixed model and regression: a fast and simple method for genomewide pedigree-based quantitative trait loci association analysis. Genetics. 2007;177(1):577–585.
- 27. Price AL, Patterson NJ, Plenge RM, Weinblatt ME, Shadick NA, Reich D. Principal components analysis corrects for stratification in genome-wide association studies. Nat Genet. 2006;38(8):904–909.
- 28. Kang HM, Zaitlen NA, Wade CM, Kirby A, Heckerman D, Daly MJ, et al. Efficient control of population structure in model organism association mapping. Genetics. 2008;178(3):1709–1723. pmid:18385116
- 29. Kang HM, Sul JH, Service SK, Zaitlen NA, Kong SY, Freimer NB, et al. Variance component model to account for sample structure in genome-wide association studies. Nat Genet. 2010;42(4):348–354. pmid:20208533
- 30. Zhou X, Stephens M. Genome-wide efficient mixed-model analysis for association studies. Nat Genet. 2012;44(7):821–824.
- 31. Yang J, Benyamin B, McEvoy BP, Gordon S, Henders AK, Nyholt DR, et al. Common SNPs explain a large proportion of the heritability for human height. Nat Genet. 2010;42(7):565–569. pmid:20562875
- 32. Yang J, Lee SH, Goddard ME, Visscher PM. GCTA: a tool for genome-wide complex trait analysis. Am J Hum Genet. 2011;88(1):76–82.
- 33. Rakovski CS, Stram DO. A kinship-based modification of the armitage trend test to address hidden population structure and small differential genotyping errors. PLoS ONE. 2009;4(6):e5825.
- 34. Thornton T, McPeek MS. ROADTRIPS: case-control association testing with partially or completely unknown population and pedigree structure. Am J Hum Genet. 2010;86(2):172–184.
- 35. Speed D, Balding DJ. Relatedness in the post-genomic era: is it still useful? Nat Rev Genet. 2015;16(1):33–44.
- 36. Wang B, Sverdlov S, Thompson E. Efficient Estimation of Realized Kinship from SNP Genotypes. Genetics. 2017; p. genetics.116.197004.
- 37. Wright S. Systems of Mating. V. General Considerations. Genetics. 1921;6(2):167–178.
- 38. Lush JL. Heritability of Quantitative Characters in Farm Animals. Hereditas. 1949;35(S1):356–375.
- 39.
Falconer DS, Mackay TFC. Introduction to Quantitative Genetics. 4th ed. Harlow: Pearson; 1996.
- 40. Thompson EA. Identity by descent: variation in meiosis, across genomes, and in populations. Genetics. 2013;194(2):301–326.
- 41. Slatkin M. Inbreeding coefficients and coalescence times. Genetics Research. 1991;58(2):167–175.
- 42. Emik LO, Terrill CE. Systematic procedures for calculating inbreeding coefficients. J Hered. 1949;40(2):51–55.
- 43. García-Cortés LA. A novel recursive algorithm for the calculation of the detailed identity coefficients. Genetics Selection Evolution. 2015;47(1):33.
- 44. Rosenberg NA, Pritchard JK, Weber JL, Cann HM, Kidd KK, Zhivotovsky LA, et al. Genetic Structure of Human Populations. Science. 2002;298(5602):2381–2385. pmid:12493913
- 45. Ramachandran S, Deshpande O, Roseman CC, Rosenberg NA, Feldman MW, Cavalli-Sforza LL. Support from the relationship of genetic and geographic distance in human populations for a serial founder effect originating in Africa. Proc Natl Acad Sci U S A. 2005;102(44):15942–15947.
- 46. Consortium TGP. A map of human genome variation from population-scale sequencing. Nature. 2010;467(7319):1061–1073.
- 47. Lazaridis I, Patterson N, Mittnik A, Renaud G, Mallick S, Kirsanow K, et al. Ancient human genomes suggest three ancestral populations for present-day Europeans. Nature. 2014;513(7518):409–413. pmid:25230663
- 48. Lazaridis I, Nadel D, Rollefson G, Merrett DC, Rohland N, Mallick S, et al. Genomic insights into the origin of farming in the ancient Near East. Nature. 2016;536(7617):419–424. pmid:27459054
- 49. Skoglund P, Posth C, Sirak K, Spriggs M, Valentin F, Bedford S, et al. Genomic insights into the peopling of the Southwest Pacific. Nature. 2016;538(7626):510–513. pmid:27698418
- 50. Tishkoff SA, Reed FA, Friedlaender FR, Ehret C, Ranciaro A, Froment A, et al. The Genetic Structure and History of Africans and African Americans. Science. 2009;324(5930):1035–1044. pmid:19407144
- 51. Moreno-Estrada A, Gravel S, Zakharia F, McCauley JL, Byrnes JK, Gignoux CR, et al. Reconstructing the Population Genetic History of the Caribbean. PLOS Genetics. 2013;9(11):e1003925. pmid:24244192
- 52. Moreno-Estrada A, Gignoux CR, Fernández-López JC, Zakharia F, Sikora M, Contreras AV, et al. The genetics of Mexico recapitulates Native American substructure and affects biomedical traits. Science. 2014;344(6189):1280–1285. pmid:24926019
- 53. Leslie S, Winney B, Hellenthal G, Davison D, Boumertit A, Day T, et al. The fine-scale genetic structure of the British population. Nature. 2015;519(7543):309–314. pmid:25788095
- 54. Baharian S, Barakatt M, Gignoux CR, Shringarpure S, Errington J, Blot WJ, et al. The Great Migration and African-American Genomic Diversity. PLoS Genet. 2016;12(5):e1006059. pmid:27232753
- 55. Haak W, Lazaridis I, Patterson N, Rohland N, Mallick S, Llamas B, et al. Massive migration from the steppe was a source for Indo-European languages in Europe. Nature. 2015;522(7555):207–211. pmid:25731166
- 56. Allentoft ME, Sikora M, Sjögren KG, Rasmussen S, Rasmussen M, Stenderup J, et al. Population genomics of Bronze Age Eurasia. Nature. 2015;522(7555):167–172. pmid:26062507
- 57.
Ochoa A, Storey JD.
*F*_{ST}and kinship for arbitrary population structures I: Generalized definitions. bioRxiv. 2016; https://doi.org/10.1101/083915 - 58.
Ochoa A, Storey JD.
*F*_{ST}and kinship for arbitrary population structures II: Method of moments estimators. bioRxiv. 2016; https://doi.org/10.1101/083923 - 59.
Ochoa A, Storey JD. New kinship and
*F*_{ST}estimates reveal higher levels of differentiation in the global human population. bioRxiv. 2019; https://doi.org/10.1101/653279 - 60. Thornton T, Tang H, Hoffmann TJ, Ochs-Balcom HM, Caan BJ, Risch N. Estimating kinship in admixed populations. Am J Hum Genet. 2012;91(1):122–138.
- 61. Hao W, Song M, Storey JD. Probabilistic models of genetic variation in structured populations applied to global human studies. Bioinformatics. 2016;32(5):713–721.
- 62. Zheng X, Weir BS. Eigenanalysis of SNP data with an identity by descent interpretation. Theoretical Population Biology. 2016;107:65–76.
- 63. Pritchard JK, Stephens M, Donnelly P. Inference of population structure using multilocus genotype data. Genetics. 2000;155(2):945–959.
- 64. Tang H, Peng J, Wang P, Risch NJ. Estimation of individual admixture: analytical and study design considerations. Genet Epidemiol. 2005;28(4):289–301.
- 65. Alexander DH, Novembre J, Lange K. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 2009;19(9):1655–1664.
- 66. Browning BL, Browning SR. A Fast, Powerful Method for Detecting Identity by Descent. The American Journal of Human Genetics. 2011;88(2):173–182.
- 67. Gazal S, Sahbatou M, Perdry H, Letort S, Génin E, Leutenegger AL. Inbreeding Coefficient Estimation with Dense SNP Data: Comparison of Strategies and Application to HapMap III. HHE. 2014;77(1-4):49–62.
- 68. Joshi PK, Esko T, Mattsson H, Eklund N, Gandin I, Nutile T, et al. Directional dominance on stature and cognition in diverse human populations. Nature. 2015;523(7561):459–462. pmid:26131930
- 69.
Cochran WG. Sampling techniques. 3rd ed. Wiley; 1977.
- 70. Reynolds J, Weir BS, Cockerham CC. Estimation of the Coancestry Coefficient: Basis for a Short-Term Genetic Distance. Genetics. 1983;105(3):767–779.
- 71.
Weir BS. Genetic data analysis II. Methods for discrete population genetic data. Sunderland, USA: Sinauer Associates; 1996.
- 72. Bourgain C, Hoffjan S, Nicolae R, Newman D, Steiner L, Walker K, et al. Novel case-control test in a founder population identifies P-selectin as an atopy-susceptibility locus. Am J Hum Genet. 2003;73(3):612–626. pmid:12929084
- 73. Choi Y, Wijsman EM, Weir BS. Case-Control Association Testing in the Presence of Unknown Relationships. Genet Epidemiol. 2009;33(8):668–678.
- 74. Pickrell JK, Pritchard JK. Inference of population splits and mixtures from genome-wide allele frequency data. PLoS Genet. 2012;8(11):e1002967.
- 75. Nei M. Analysis of Gene Diversity in Subdivided Populations. PNAS. 1973;70(12):3321–3323.
- 76.
Weir BS, Goudet J. A unified characterization of population structure and relatedness. bioRxiv. 2016; p. 088260.
- 77. Raj A, Stephens M, Pritchard JK. fastSTRUCTURE: variational inference of population structure in large SNP data sets. Genetics. 2014;197(2):573–589.
- 78. Nelis M, Esko T, Mägi R, Zimprich F, Zimprich A, Toncheva D, et al. Genetic Structure of Europeans: A View from the North–East. PLOS ONE. 2009;4(5):e5472. pmid:19424496
- 78. Silva NM, Pereira L, Poloni ES, Currat M. Human Neutral Genetic Variation and Forensic STR Data. PLOS ONE. 2012;7(11):e49666.
- 80. Steele CD, Court DS, Balding DJ. Worldwide FST Estimates Relative to Five Continental-Scale Populations. Annals of Human Genetics. 2014;78(6):468–477.
- 81. Cavalli-Sforza LL. Population Structure and Human Evolution. Proceedings of the Royal Society of London Series B, Biological Sciences. 1966;164(995):362–379.
- 82. Lewontin RC, Krakauer J. Distribution of Gene Frequency as a Test of the Theory of the Selective Neutrality of Polymorphisms. Genetics. 1973;74(1):175–195.
- 83. Beaumont MA, Nichols RA. Evaluating Loci for Use in the Genetic Analysis of Population Structure. Proceedings of the Royal Society of London B: Biological Sciences. 1996;263(1377):1619–1626.
- 84. Vitalis R, Dawson K, Boursot P. Interpretation of Variation Across Marker Loci as Evidence of Selection. Genetics. 2001;158(4):1811–1823.
- 85. Akey JM, Zhang G, Zhang K, Jin L, Shriver MD. Interrogating a high-density SNP map for signatures of natural selection. Genome Res. 2002;12(12):1805–1814.
- 86. Porter AH. A test for deviation from island-model population structure. Molecular Ecology. 2003;12(4):903–915.
- 87. Bowcock AM, Kidd JR, Mountain JL, Hebert JM, Carotenuto L, Kidd KK, et al. Drift, admixture, and selection in human evolution: a study with DNA polymorphisms. PNAS. 1991;88(3):839–843. pmid:1992475
- 88. Hedrick PW. A Standardized Genetic Differentiation Measure. Evolution. 2005;59(8):1633–1638.
- 89. Jakobsson M, Edge MD, Rosenberg NA. The Relationship Between FST and the Frequency of the Most Frequent Allele. Genetics. 2013;193(2):515–528.
- 90. Edge MD, Rosenberg NA. Upper bounds on FST in terms of the frequency of the most frequent allele and total homozygosity: the case of a specified number of alleles. Theor Popul Biol. 2014;97:20–34.
- 91. Lewontin RC. The Apportionment of Human Diversity. Evolutionary Biology. 1972;6:381–398.
- 92. Barbujani G, Magagni A, Minch E, Cavalli-Sforza LL. An apportionment of human DNA diversity. PNAS. 1997;94(9):4516–4519.
- 93. Novembre J, Johnson T, Bryc K, Kutalik Z, Boyko AR, Auton A, et al. Genes mirror geography within Europe. Nature. 2008;456(7218):98–101. pmid:18758442
- 94. Coop G, Pickrell JK, Novembre J, Kudaravalli S, Li J, Absher D, et al. The Role of Geography in Human Adaptation. PLoS Genet. 2009;5(6):e1000500. pmid:19503611
- 95. Patterson N, Moorjani P, Luo Y, Mallick S, Rohland N, Zhan Y, et al. Ancient admixture in human history. Genetics. 2012;192(3):1065–1093. pmid:22960212
- 96. Beran R, Hall P. Interpolated Nonparametric Prediction Intervals and Confidence Intervals. Journal of the Royal Statistical Society Series B (Methodological). 1993;55(3):643–652.