Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Inbreeding, Allee effects and stochasticity might be sufficient to account for Neanderthal extinction

  • Krist Vaesen ,

    Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

    Affiliations School of Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands, Human Origins Group, Faculty of Archaeology, University of Leiden, Leiden, The Netherlands

  • Fulco Scherjon,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Writing – review & editing

    Affiliation Human Origins Group, Faculty of Archaeology, University of Leiden, Leiden, The Netherlands

  • Lia Hemerik,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing

    Affiliation Biometris, Mathematical and Statistical Methods, Wageningen University, Wageningen, The Netherlands

  • Alexander Verpoorte

    Roles Conceptualization, Investigation, Writing – original draft, Writing – review & editing

    Affiliation Human Origins Group, Faculty of Archaeology, University of Leiden, Leiden, The Netherlands

Inbreeding, Allee effects and stochasticity might be sufficient to account for Neanderthal extinction

  • Krist Vaesen, 
  • Fulco Scherjon, 
  • Lia Hemerik, 
  • Alexander Verpoorte


The replacement of Neanderthals by Anatomically Modern Humans has typically been attributed to environmental pressure or a superiority of modern humans with respect to competition for resources. Here we present two independent models that suggest that no such heatedly debated factors might be needed to account for the demise of Neanderthals. Starting from the observation that Neanderthal populations already were small before the arrival of modern humans, the models implement three factors that conservation biology identifies as critical for a small population’s persistence, namely inbreeding, Allee effects and stochasticity. Our results indicate that the disappearance of Neanderthals might have resided in the smallness of their population(s) alone: even if they had been identical to modern humans in their cognitive, social and cultural traits, and even in the absence of inter-specific competition, Neanderthals faced a considerable risk of extinction. Furthermore, we suggest that if modern humans contributed to the demise of Neanderthals, that contribution might have had nothing to do with resource competition, but rather with how the incoming populations geographically restructured the resident populations, in a way that reinforced Allee effects, and the effects of inbreeding and stochasticity.


A long-standing enigma in palaeoanthropology is the demise of Neanderthals approximately 40 kya [1]. There is general agreement that―after a long period of largely separated coexistence [26]―their disappearance roughly coincides with migration events starting around ~60 kya by Anatomically Modern Humans (AMHs) from Africa into the Near East and Europe and that, accordingly, AMHs took over the territories previously occupied by our sister species [712]. What is uncertain however, are the causes of Neanderthal extinction. It has been attributed to a wide variety of intensely debated factors, including climatic change ([1317], but see [1819]), epidemics [2021], a superiority of AMHs over Neanderthals in competing for the same resources ([2234] but see [18]). Models have, not surprisingly, confirmed that if such superiority is assumed [3541], Neanderthals would indeed have been likely to go extinct.

Here we argue that no such contested factors might be needed to account for the demise of Neanderthals. We present two independent models that capture the internal dynamics of Neanderthal populations―the models thus ignore, among other things, competitive interactions with AMHs―and that suggest that the disappearance of Neanderthals might have resided in the small size of their population(s) alone. Accordingly, our study substantiates the suggestion, made in passing by French [42], that “it may simply be the case that Neanderthal populations declined below their minimum viable population threshold”.

Genetic studies, indeed, indicate that Neanderthal effective population size―the size of the ideal population that would undergo the same amount of random genetic drift as the actual population [43]―was already small by ~400 kya, amounting to no more than 3,000–3,900 individuals, a level that was sustained almost up till the species’ extinction ~38 kya [4446]. Rogers et al. [47] report a higher number for the effective size of the global Neanderthal population, but also show that this global population was subdivided in very small and highly isolated local populations (each, so to speak, with its own, small effective size). Likewise, Castellano et al. [48] conclude, based on a comparison between the genome of three Neanderthal individuals and the genome of present-day modern humans, that Neanderthal genetic diversity was limited and that Neanderthal populations were small and isolated from one another. The archaeological data point in the same direction ([49], and references therein); and Bocquet-Appel and Degioanni [50] estimate that the Neanderthal census metapopulation was in the range of a mere 5,000–70,000 individuals.

The models presented here implement three basic factors that, according to conservation biology (the field from which our models were drawn), would put such small populations at risk of extinction: inbreeding, Allee effects and stochasticity ([51]; in archaeology, see Finlayson [52]). Inbreeding depression refers to the reduction in fitness of individuals that arise from matings between genetic relatives, matings thus that are more likely to occur in small populations. Inbreeding, which seems to have been common in Neanderthals [44, 5355], might lead to a lower fitness because it increases the chances of the expression of recessive, deleterious traits and because homozygotes often have a general disadvantage relative to heterozygotes. Harris and Nielsen [56] estimate that, due to inbreeding, Neanderthals had at least 40% lower fitness than modern humans on average. Allee effects refer to the effects that population density has on reproduction and, thus, on population growth [51]. At lower densities, the case we are concerned with here, growth rates might drop due to problems in mate-finding, and to several problems that highly cooperative species, such as Neanderthals, are particularly susceptible to, including low availability of helpers in cooperative hunting, defending kills from kleptoparasites, and allo-parenting [57]. Finally, stochastic, annual fluctuations in births, deaths and sex ratio are more likely to place smaller populations on a trajectory towards extinction than bigger ones [51].

Our models indicate that these factors alone could have resulted in Neanderthal extinction, even if Neanderthals and AMHs were identical in terms of individual-level traits that are deemed relevant to persistence or extinction (e.g., cognitive and technological ability, sociality).

Materials and methods

The specific question our study aims to address is whether inbreeding, Allee effects and stochasticity are sufficient to explain the disappearance of Neanderthals. To that end, we develop two separate models, both of which track Neanderthal population growth over time. Crucially, since we want to avoid making assumptions about the superiority of AMHs over Neanderthals, both models are parameterized based on estimates for AMHs, estimates that pertain to, among other things, reproduction, mortality, and inbreeding (see below, S1 Appendix, and S6 and S7 Tables).

The first model is a deterministic matrix model, the second an individual-based stochastic simulation model (IBM). The matrix model served two purposes: to calibrate the IBM and to validate some of the results of the IBM. Since, in contrast to the matrix model, the IBM allowed us straightforwardly to introduce inbreeding and stochasticity, it was our primary resource for obtaining the results reported here. More specifically, we used the IBM to determine, separately, the levels of inbreeding and the levels of Allee effects that put Neanderthal populations at risk of extinction; and to determine Neanderthal’s vulnerability to extinction in a scenario involving both inbreeding and Allee effects. Subsequently, we set these results against what―as it comes to Neanderthal population size, inbreeding and Allee effects―can be inferred from the literature.

Below we provide a general, non-technical overview of the research set-up. For a full description, please consult S1 Appendix.

General description of the basic models

The deterministic model consists of a Leslie matrix, a type of matrix that is commonly used in conservation biology to model population growth [58]. The matrix summarizes, for each of a population’s age classes, yearly survival and reproduction. Our model assumes a sex ratio of 1:1 and, because they are the limiting factor for reproduction, only considers females. Accordingly, the first row of the matrix contains, for each of the age classes, the average number of female offspring per female per year. The subdiagonals represent the yearly survivals of females; these values are on the subdiagonals because, whenever a female survives in a given year, she will move to the next age class. The growth factor of the population―the factor by which the population will annually be multiplied―is given by the dominant eigenvalue of the matrix. If this factor is less than 1, the population will have a negative growth rate and goes extinct.

For the IBM, we relied on VORTEX [5961], a software package used by conservation biologists to perform Population Viability Analyses of endangered wildlife species. VORTEX simulates the annual life events (e.g., sex determination, breeding, mortality) that might occur to each of the individuals within a given population, and records over time, in discrete time steps, the characteristics of these individuals as well as those of the population as a whole. Occurrences of events are probabilistic. Demographic stochasticity (relating to annual fluctuations in, e.g., sex ratios, births and deaths) is thus inherent to VORTEX models.


We calibrated the two models by checking whether, under similar parameter settings, they were able to produce similar trends in Neanderthal population growth. At this stage, we didn’t include inbreeding or Allee effects.

The primary input parameters for both models were derived from estimates of female reproduction in extant hunter-gatherers [62] and from the West model life table, Level 5 [63]. In general, such model life tables summarize the mortality events of an ideal population. West tables are the most commonly used in human palaeo-demography; Level 5 is closely matched by the mortality profiles of extant hunter-gatherers [64].

The parameters used in the matrix model are shown in S1 Appendix, S1 Table. Calculating the dominant eigenvalue of the corresponding matrix yields a population growth factor of 1.008, which corresponds to a relative growth rate of 0.80%.

The parameters used in the VORTEX model are summarized in S1 Appendix, S5 and S6 Tables. For various initial population sizes (N0 = 50; 100; 500; 1,000; 5,000), we ran ten simulation runs, each simulating a time span of 100 years. The relative growth rate of these simulations was, on average, 0.76%. This value is slightly lower than the relative growth rate obtained from the matrix model. However, it has been theoretically established that, in general, growth rates of stochastic models are less than the growth rates of deterministic models [65].

Inbreeding and stochasticity

In conservation biology there is no standard procedure for implementing inbreeding in deterministic matrix models. Therefore, we relied on our IBM to model inbreeding. In VORTEX, inbreeding depression is modeled in terms of its effects on infant survival. It is governed by two parameters: the number of lethal equivalents, I, which is the number of recessive alleles carried in a heterozygous genome that would be lethal if carried in the homozygous state; and the percentage, fi, of the inbreeding depression caused by such lethal alleles rather than by other genetic mechanisms (e.g., a general disadvantage of homozygotes). Based on observations of several species, VORTEX sets fi at 50% by default.

For various initial population sizes (N0 = 50; 100; 500; 1,000; 5,000) and various value of fi (viz., fi = 30;50;70), and again starting from West table Level 5, we determined, by simulating widely over I, the parameter Irisk. Irisk is the lowest value of I that yields at least one extinction event in ten simulation runs, each run simulating over a time span of 10,000 years. Furthermore, we determined, Isure, which is the lowest value of I that yields extinction in all of the ten runs. Note that VORTEX models that incorporate inbreeding generally run very slow, and do so especially when carrying capacity is high. Therefore we set K at 5,000, unlike we did in the basic model (where carrying capacity was set at K = 10,000). This doesn’t affect our conclusions (see below).

Allee effects, with and without stochasticity

The dependence of female breeding rates on population size is commonly (including in VORTEX) captured by an equation that, when simplified to cover the most conservative scenario, tells us that the fraction of females breeding at population size N, E, is given by (1) where p0 is the percentage of females breeding in the absence of population size constraints, and A is a parameter describing the strength of the Allee effect (see equations S4 and S5 in the S1 Appendix).

Accordingly, in the IBM we simulated widely over A, and determined, for various initial population sizes (N0 = 50; 100; 500; 1,000; 5,000), Arisk, which is the lowest value of A that leads, in ten simulation runs (each run comprises 10,000 years), to at least one extinction event. Also, we determined Asure, which is the lowest value of A for which all ten runs result in extinction.

In order to validate our results, we introduced the Allee equation used in the IBM into the matrix model, and set the results against the results obtained from the IBM.

Inbreeding, Allee effects and stochasticity

Estimates of I for modern humans [6670] range from 0.58 [69] to 2.2 [66]. Gao et al. [69] point out that, since these estimates are based on reported deaths after birth and thus do not take into account prenatal deaths, the actual number of lethal equivalents might be higher; the authors surmise that prenatal deaths might increase I by one additional lethal equivalent (resulting in a maximum value of I = 3.2).

To assess the combined effects of inbreeding, Allee effects and stochasticity, we ran simulations for the highest I-value just reported (i.e., I = 3.2). More specifically, by widely varying over A, we determined for this I-value, and for various initial population sizes (N0 = 50; 100; 500), Arisk.

Although I = 3.2 is the least conservative value among the values found in the literature, our choice was motivated by the results obtained in scenarios that involved inbreeding alone (i.e., the results suggested that even at I = 3.2 the impact of inbreeding would be relatively small; see Results), and by the large computational demands of VORTEX in scenarios that combine Allee effects, inbreeding and very lengthy timespans (10,000 years). Also due to resource constraints, we did not determine Asure, restricted N to the range 0–500, and set carrying capacity at K = 5,000.


Inbreeding and stochasticity

Fig 1 plots Irisk and Isure for initial populations of size N0 = 50; 100; 500; 1,000; 5,000, assuming fi = 50. Fig 1 comprises three regimes: below the lower, green squares populations can be expected to survive (SURVIVAL); above the upper, red triangles populations die out (EXTINCTION); populations in between run a risk of extinction (RISK). The latter regime is where stochastic effects occur, with higher risks of extinction closer to the upper, red triangles.

Fig 1. Inbreeding and stochasticity.

Irisk (lowest value of inbreeding depression parameter I that results in at least one extinction event in ten simulation runs) and Isure (lowest value of inbreeding depression that results in extinction in all simulation runs) for various initial population sizes N0 and fi = 50% (for fi = 30;70, see S1 Appendix). The horizontal dotted lines mark the range of values of I observed in AMHs.

In virtually all of the scenarios, the available estimates for modern humans (ranging from 0.58 to 3.2; see above) fall in the regime of survival. Only when N0 = 50, Neanderthals face a risk of extinction, i.e., Irisk falls in the range 0.58–3.2 (the same holds when fi = 30; 70, see S4 Fig). Given that the lowest Neanderthal census size reported in the literature is 5,000 [50], and on the assumption that Neanderthals carried the same average number of lethal equivalents as AMHs, it is unlikely that Neanderthals would have disappeared due to inbreeding alone. Even if the meta-population comprised hundred isolated bands of 50 individuals each, e.g., because these bands were interspersed by AMH bands (see Discussion), it is unlikely that inbreeding alone would have resulted in extinction. Recall that Irisk is the lowest value of I that yields at least one extinction event in ten simulation runs, each run simulating over a time span of 10,000 years. Accordingly, if inbreeding were the primary cause of Neanderthals extinction, one would need to make the extraordinary assumption that each of the hundred bands experienced an extinction event that had only a one out of ten chance of occurring.

In the simulations, K was set at K = 5,000. But our conclusions would also hold if, as we did in the basic model, K had been set at K = 10,000. Carrying capacity exerts a downward push on populations that are approaching it. Increasing K thus would make it even more unlikely that inbreeding alone would have yielded extinction in populations that did not go extinct at lower K.

Allee effects, with and without stochasticity

Fig 2A represents Arisk and Asure for populations of size N0 = 50; 100; 500; 1,000; 5,000. Again, Fig 2A comprises three regimes (viz., SURVIVAL, RISK and EXTINCTION).

Fig 2. Allee effects, with and without stochasticity.

(a) Arisk (lowest value of Allee parameter A that results in at least one extinction event in ten simulation runs) and Asure (lowest value of Allee parameter A that results in extinction in all simulation runs) for various initial population sizes N0. (b) Birth intervals corresponding to Arisk and Asure, as well as birth intervals corresponding to the Asure values obtained in the non-stochastic matrix model (labeled Asure(MM)). The horizontal dotted lines mark the range of values of birth intervals observed in contemporary hunter-gatherers.

For most animal species, including AMHs and Neanderthals, estimates of A are unavailable [71]. Yet, Eq (1) allows us indirectly to assess Allee effects. In all of our simulations, the percentage of females breeding without Allee effects, p0, was set at 0.33, which corresponds to a birth interval of 3 years, which in turn corresponds to the average birth interval reported by Kelly [62] among extant hunter-gatherers and with the birth interval estimated for Neanderthals by Lalueza-Fox and colleagues [72]. Consider now a population of N = 1,000, for which Asure equals 301 (see Fig 2A). Eq (1) tells us that Allee effects need to reduce the percentage of females breeding to 0.25 for the population to become extinct. This percentage corresponds to a birth interval of 4 years, a value that falls well within the range of birth intervals reported by Kelly [62] (viz. 2.3–5.4). In other words, for extinction to occur, it is enough that Allee effects produce birth intervals that are common among extant hunter-gatherers. Performing the same calculation for all N’s results in Fig 2B. It appears that all of the birth intervals neatly fall in the range observed among extant hunter-gatherers.

The regime in between the red and green dots illustrates the effects of stochasticity. In this regime, fluctuations in births, deaths and sex ratio determine whether or not, in a time span of 10,000 years, an actual extinction event takes place. So even at Arisk, when Allee effects are relatively small (e.g., for N = 1,000, Arisk is 200, which corresponds to a birth interval of 3.6 years), random events might lead to extinction.

Fig 2B also plots the birth intervals inferred from the results of the matrix model (grey dots). The values, again, fall within Kelly’s range, as does the value for a population of N = 70,000 (not depicted in Fig 2B; birth interval of 3.84 years). Importantly, the results of the matrix model and of the IBM fall within the same range; and at higher population sizes, where stochastic effects are expected to be small, the values of Asure obtained in the matrix model and in the IBM start to converge.

Note that in our models, Allee effects only comprise the effects that population size has on the percentage of females breeding. Our models thus exclude the effects that low population numbers might have on survival rates (e.g., lower infant survival due to shortages in allo-parents or cooperative hunts), and therefore likely underestimate the challenges faced by small populations. In sum, Allee effects probably were a key, and perhaps even a sufficient, factor in the demise of Neanderthals.

Inbreeding, Allee effects and stochasticity

Fig 3 plots, for various initial population sizes N0, Arisk(0.0) and Arisk(3.2), which denote, respectively, Arisk obtained in a scenario with only Allee effects, and Arisk obtained in a scenario with Allee effects and inbreeding—the latter set at the highest I-value reported in the literature (viz. I = 3.2). It appears that even at this highest I-value inbreeding leads to a reduction in Arisk only at lower N0; already at N0 = 500, the values of Arisk(0.0) and Arisk(3.2) converge. In order to check whether at lower I-values inbreeding would still have an effect on Arisk we performed some extra simulations, now setting I at I = 0.58; 2.2, and N0 at N0 = 50; 100. It turned out that such lower levels of inbreeding reduce Arisk when N0 = 50, but not when N0 = 100.

Fig 3. Inbreeding, Allee effects, and stochasticity.

Arisk (lowest value of Allee parameter A that results in at least one extinction event in ten simulation runs) for a scenario without inbreeding (blue triangles; Arisk(0)) and a scenario with inbreeding (orange squares; Arisk(3.2)) (inbreeding depression parameter at I = 3.2).

These findings are in line with the observation above that, if Neanderthals carried the same number of lethal equivalents as AMHs, inbreeding would have had an effect only at the sub-population level. In such a scenario, inbreeding would occasionally—i.e., when Neanderthals had the stochastic odds against them—lead to a local extinction event, thereby accelerating a process of decline that was primarily driven by Allee effects.

Conclusion and discussion

Our results support the hypothesis that the disappearance of Neanderthals might have been the result of a demographic factors alone, that is, the result merely of the internal dynamics that operate in small populations. Our conclusions are consistent with but go beyond the conclusions of a recent study by Kolodny and Feldman [73]. Based on a series of mathematical models, these authors too argue that no external factors (climate, epidemics) nor a superiority of AMHs in resource competition are needed to account for Neanderthal extinction. Their models suggest that migratory dynamics―with more migration happening from Africa into Europe by AMHs than migration from Europe into Africa by Neanderthals―might have been sufficient to result in the replacement of Neanderthals by AMHs. While Kolodny and Feldman’s models indeed do not assume a competitive advantage for either species (but see S1 Appendix), they do take for granted that Neanderthals and AMHs competed for the same habitats. Our study shows that even without this contested assumption ([74]; see also the literature on competition avoidance among extant hunter-gatherers, e.g., [75] and references therein), Neanderthal extinction might have taken place.

If Neanderthals lived in small populations since ~400 kya [4446], why did it take so long for them to become extinct? A first relevant consideration concerns demographic stochasticity. We have seen that annual fluctuations in births, deaths and sex ratio might determine whether and when a small population disappears. So our results are consistent with a scenario in which a small population of Neanderthals persists for several thousands of years, and then, due to a stroke of bad luck, disappears. Furthermore, for the sake of simplicity, our models do not take into account environmental stochasticity. That is, the models work with fixed probabilities for mortality, fertility and sex ratios―fluctuations are thus simply caused by probabilistic sampling (demographic stochasticity). In natural conditions, though, the probabilities themselves will vary, according to random fluctuations in the environment (environmental stochasticity). Note that these fluctuations do not correspond to the millennial trends observed in the palaeoclimatic record, but occur at much lower temporal scales (e.g., a couple of years of drought, an epidemic among prey). They should be understood as natural variations around a mean, rather than as external forcings, such as the ones that some scholars have claimed to be responsible for the demise of Neanderthals (e.g., climatic change [1316] or volcanic eruptions [17]). Importantly, in a given year, demographic and environmental stochasticity might very well work in opposite directions; environmental conditions, for instance, might be favorable and alleviate the stress induced by demographic stochasticity. In fact, it will very rarely happen that a metapopulation comprising several sub-populations has all the stochastic odds against it, that is, it will only very rarely happen that, for a significant amount of time, environmental variability produces low fertility rates and high death rates, and additionally, demographic stochasticity produces low fertility rates and high death rates and unfavorable sex ratios, and this in all of the metapopulation’s sub-populations. But in the very long run, such an unfavorable scenario eventually will take place. Accordingly, it is not implausible that, despite regular local extinction events [76], a small metapopulation manages to survive over prolonged stretches of time but eventually dies out due to its overall size and stochasticity. Noteworthy, there is nothing unusual about the persistently small size of Neanderthal populations. Hominin populations likely were small throughout the Pleistocene [77].

We suggest that AMHs might still have contributed to the extinction, but not necessarily by engaging in competition with or outcompeting Neanderthals. The mere interspersal of AMH sub-populations between Neanderthal sub-populations reduced the opportunities for intra-breeding and migratory activity among the latter. The resulting small and isolated sub-populations (as documented by [4749,76,78]) would be increasingly vulnerable precisely to the factors examined in the current paper (viz. inbreeding, Allee effects and stochasticity), and thus to extinction. As such, the presence of modern humans in Eurasia would have accelerated a process that, at some point, was likely to have occurred anyway. Stated otherwise, the arrival of AMHs would have been a contributory factor rather than the cause of the extinction. Importantly, population-level characteristics―e.g., many of the characteristics that conservation biology has shown to be critical for a species’ persistence, including population size, species distribution, intraspecific variability, and patterns of dispersal―might also account for the successful range expansion of AMHs. In other words, our species’ success need not be the result of a superiority in its individual-level traits.

An explanation solely in terms of the internal dynamics of the Neanderthal population, as the one presented here, serves as a null hypothesis against which competing, and less parsimonious, hypotheses are to be assessed. Regardless of whether external factors (climate or epidemics) or factors related to resource competition played a role in the actual demise of Neanderthals, our study suggests that any plausible explanation of the demise also needs to incorporate demographic factors as key variables.

Supporting information

S1 Table. Yearly survival probabilities based on the life tables published by [63].


S2 Table. Sensitivities and elasticities in the density-independent model (see third and fourth column).


S3 Table. Sensitivities and elasticities in the density-dependent model (see third and fourth column). Note that the Allee effect is included with specific parameter values.


S4 Table. Asure(MM) and corresponding birth interval, for various initial population sizes N0 (see Fig 2B in main text).


S5 Table. Comparison of 5y-to-1y conversion of mortality rates. Column 2 pertains to the conversion (1-M5)^(1/5); Column 3 pertains to the conversion used in our study, viz., 1-(M5/5).


S6 Table. Annual mortality rates (as percentages) implemented in Vortex, derived from West model life Table 5 [63].


S7 Table. Parameter settings for the basic Vortex model. In the third column, conservative estimates refer to estimates that can be expected to counteract the negative effects of inbreeding, Allee effects, and stochasticity.


S1 Fig. Population growth factor (contour lines), for different values of the first year survival (s0) and the yearly adult reproduction rate (m).

The red dashed line indicates combinations of s0 and m that yield a stable populations (i.e., populations with a growth factor of 1). Lower values for either of the two parameters result in extinction.


S2 Fig. Screenshot of Vortex’s graphical user interface.


S3 Fig. Screenshot of the first part of a model input text-file.


S4 Fig. Inbreeding and stochasticity: Irisk (lowest value of inbreeding depression parameter I that results in at least one extinction event in ten simulation runs) and Isure (lowest value of inbreeding depression that results in extinction in all simulation runs) for various initial population sizes N0.

(a) fi = 30%; (b) fi = 50%; and fi = 70%. The horizontal dotted lines mark the range of values of I observed in AMHs.



We thank the members of the Human Origins Group, Faculty of Archaeology, Leiden University, and two anonymous reviewers for their feedback on earlier drafts of this paper.


  1. 1. Higham T, Douka K, Wood R, Ramsey CB, Brock F, Basell L et al. The timing and spatiotemporal patterning of Neanderthal disappearance. Nature 512(7514): 306–309 (2014). pmid:25143113
  2. 2. Liu W, Martinón-Torres M, Cai Y, Xing S, Tong H, Pei S et al. The earliest unequivocally modern humans in southern China. Nature 526: 696–700 (2015). pmid:26466566
  3. 3. Hublin JJ, Ben-Ncer A, Bailey SE, Freidline SE, Neubauer S, Skinner MM et al. New fossils from Jebel Irhoud, Morocco and the pan-African origin of Homo sapiens. Nature 546: 289–292 (2017). pmid:28593953
  4. 4. Posth C, Wißing C, Kitagawa K, Pagani L, van Holstein L, Racimo F et al. Deeply divergent archaic mitochondrial genome provides lower time boundary for African gene flow into Neanderthals. Nature communications 8: 16046 (2017). pmid:28675384
  5. 5. Hershkovitz I, Weber GW, Quam R, Duval M, Grün R, Kinsley L et al. The earliest modern humans outside Africa. Science 359: 456–459 (2018). pmid:29371468
  6. 6. Hajdinjak M, Fu Q, Hübner A, Petr M, Mafessoni F, Grote S et al. Reconstructing the genetic history of late Neanderthals. Nature 555(7698): 652–656 (2018). pmid:29562232
  7. 7. Stringer C. Human evolution: Out of Ethiopia. Nature 423(6941): 692–695 (2003). pmid:12802315
  8. 8. Klein RG. Out of Africa and the evolution of human behavior. Evol Anthropol 17(6): 267–281 (2008).
  9. 9. Smith FH, Janković I, Karavanić I. The assimilation model, modern human origins in Europe, and the extinction of Neandertals. Quat Int 137(1): 7–19 (2005).
  10. 10. Benazzi S, Douka K, Fornai C, Bauer CC, Kullmer O. Early dispersal of modern humans in Europe and implications for Neanderthal behaviour. Nature 479: 525–528 (2011). pmid:22048311
  11. 11. Higham T, Compton T, Stringer C, Jacobi R, Shapiro B, Trinkaus E et al. The earliest evidence for anatomically modern humans in northwestern Europe. Nature 479: 521–524 (2011). pmid:22048314
  12. 12. Hublin JJ. The modern human colonization of western Eurasia: when and where? Quat. Sci. Rev. 118: 194–210 (2015).
  13. 13. Finlayson C. On the importance of coastal areas in the survival of Neanderthal populations during the Late Pleistocene. Quat Sci Rev 27(23): 2246–2252 (2008).
  14. 14. Müller UC, Pross J, Tzedakis PC, Gamble C, Kotthoff U, Schmiedl G et al. The role of climate in the spread of modern humans into Europe. Quat Sci Rev 30(3): 273–279 (2011).
  15. 15. Tzedakis PC, Hughen KA, Cacho I, Harvati K. Placing late Neanderthals in a climatic context. Nature 449(7159): 206–208 (2007). pmid:17851522
  16. 16. Jimenez-Espejo FJ, Jiménez-Espejo FJ, Martínez-Ruiz F, Finlayson C, Paytan A, Sakamoto T, Ortega-Huertas M et al. Climate forcing and Neanderthal extinction in southern Iberia: insights from a multiproxy marine record. Quat Sc Rev 26: 836–852 (2007).
  17. 17. Golovanova LV, Doronichev VB, Cleghorn NE, Koulkova MA, Sapelko TV, Shackley MS. Significance of ecological factors in the middle to upper-paleolithic transition. Curr Anthropol 51: 655–691 (2010).
  18. 18. Sørensen B. Demography and the extinction of European Neanderthals. J Anthropol Arch 30: 17–29 (2011).
  19. 19. Villa P, Roebroeks W. Neandertal Demise: An Archaeological Analysis of the Modern Human Superiority Complex. PLoS ONE 9(4): e96424 (2014). pmid:24789039
  20. 20. Underdown S. A potential role for transmissible spongiform encephalopathies in Neanderthal extinction. Med Hypotheses 71(1): 4–7 (2008). pmid:18280671
  21. 21. Wolff H, Greenwood AD. Did viral disease of humans wipe out the Neandertals? Med Hypotheses 75(1): 99–105 (2010). pmid:20172660
  22. 22. Mellars PA. The Impossible Coincidence. A Single-Species Model for the Origins of Modern Human Behavior in Europe. Evol Anthropol 14: 12–27 (2005).
  23. 23. Marean CW. From the tropics to the colder climates: contrasting faunal exploitation adaptations of modern humans and Neanderthals. In From Tools to Symbols. From Early Hominids to Modern Humans (eds d’Errico F, Backwell L) 333–371 (Witwatersrand University Press 2005).
  24. 24. Marean CW, Bar-Matthews M, Bernatchez J, Fisher E, Goldberg P, Herries AIR, Jacobs Z et al. Early human use of marine resources and pigment in South Africa during the Middle Pleistocene. Nature 449: 905–908 (2007). pmid:17943129
  25. 25. Brown KS, Marean CW, Jacobs Z, Schoville BJ, Oestmo S, Fisher EC et al. An early and enduring advanced technology originating 71,000 years ago in South Africa. Nature 491: 590–593 (2012). pmid:23135405
  26. 26. Shea JJ, Sisk ML. Complex projectile technology and Homo sapiens dispersal into western Eurasia. PaleoAnthropology 2010: 100–122 (2010).
  27. 27. Shea JJ. The ecological impact of projectile weaponry in Late Pleistocene human evolution. In The evolution of hominid diets: integrating approaches to the study of Paleolithic subsistence (eds Hublin JJ, Richards MP) 189–199 (Springer 2009).
  28. 28. Brown KS, Marean CW, Herries AIR, Jacobs Z, Tribolo C, Braun D et al. Fire As an Engineering Tool of Early Modern Humans. Science 325: 859–862 (2009). pmid:19679810
  29. 29. Wadley L, Hodgskiss T, Grant M. Implications for complex cognition from the hafting of tools with compound adhesives in the Middle Stone Age, South Africa. Proc Natl Acad Sci 106: 9590–9594 (2009). pmid:19433786
  30. 30. Wynn T. Hafted spears and the archaeology of mind. Proc Natl Acad Sci 106: 9544–9545 (2009). pmid:19506246
  31. 31. Nash D, Coulson S, Staurset S, Stewart Ullyott JS, Babutsi M, Hopkinson L et al. Provenancing of silcrete raw materials indicates long-distance transport to Tsodilo Hills, Botswana, during the Middle Stone Age. J Hum Evol 64: 280–288 (2013). pmid:23453438
  32. 32. Hockett B, Haws JA. Nutritional ecology and the human demography of Neandertal extinction. Quat Int 137: 21–34 (2005).
  33. 33. Wynn T, Coolidge FL. The implications of the working memory model for the evolution of modern cognition. Int J Evol Biol 2011: 741357 (2011). pmid:21716664
  34. 34. Banks WE, d'Errico F, Peterson AT, Kageyama M, Sima A, Sánchez-Goñi M. Neanderthal extinction by competitive exclusion. PLoS ONE 3(12): e3972 (2008). pmid:19107186
  35. 35. Zubrow EBW. The demographic modelling of Neanderthal extinction. In The human revolution (eds Mellars P, Stringer C) 212–231 (Edinburgh University Press 1989).
  36. 36. Flores JC. A Mathematical Model for Neanderthal Extinction. J Theor Biol 191: 295–298 (1998). pmid:9631569
  37. 37. Flores JC. Diffusion coefficient of Modern Humans outcompeting Neanderthals. J Theor Biol 280: 189–190 (2011). pmid:21540038
  38. 38. Murray JD. Mathematical Biology (Springer 2004).
  39. 39. Gilpin W, Feldman MW, Aoki K. An ecocultural model predicts Neanderthal extinction through competition with modern humans. Proc Natl Acad Sci 113(8): 2134–2139 (2016). pmid:26831111
  40. 40. Horan RD, Bulte E, Shogren JF. How trade saved humanity from biological exclusion: an economic theory of Neanderthal extinction. J Econ Behav Organ 58: 1–29 (2005).
  41. 41. Goldfield AN, Booton R, Marston JM. Modeling the role of fire and cooking in the competitive exclusion of Neanderthals. J Hum Evol 124: 91–104 (2018). pmid:30177445
  42. 42. French JC, Demography and the Palaeolithic archaeological record, J Archaeol Meth Theor 1;23(1):150–99 (2016).
  43. 43. Soulé ME. Viable populations for conservation (Cambridge University Press 1987).
  44. 44. Prüfer K, Racimo F, Patterson N, Jay F, Sankararaman S, Sawyer S et al. The complete genome sequence of a Neanderthal from the Altai Mountains. Nature 505: 43–49 (2014). pmid:24352235
  45. 45. Kuhlwilm K, Gronau I, Hubisz MJ, de Filippo C, Prado-Martinez J, Kircher M et al. Ancient gene flow from early modern humans into Eastern Neanderthals. Nature 530: 429–433 (2017).
  46. 46. Mafessoni F, Prüfer K. Better support for a small effective population size of Neandertals and a long shared history of Neandertals and Denisovans. Proc Nat Ac Sc 114: 16918 (2017).
  47. 47. Rogers AR, Bohlender RJ and Huff CD. Early history of Neanderthals and Denisovans. Proc Nat Ac Sc 114(37): 9859–9863 (2017).
  48. 48. Castellano S, Parra G, Sánchez-Quinto FA, Racimo F, Kuhlwilm M, Kircher M et al., Patterns of coding variation in the complete exomes of three Neandertals, Proc Nat Acad Sc 111(18):6666–71 (2014)
  49. 49. Churchill SE, Thin on the ground: Neandertal biology, archeology and ecology (John Wiley & Sons 2014)
  50. 50. Bocquet-Appel J, Degioanni A. Neanderthal Demographic Estimates. Curr Anthropol 54(8): 202–2013 (2013).
  51. 51. Courchamp F, Berec L, Gascoigne B. Allee effects in Ecology and Conservation (Oxford University Press 2008).
  52. 52. Finlayson C. Neanderthals, Modern Humans (Cambridge University Press 2004).
  53. 53. Ríos L, Rosas A, Estalrrich A, García-Tabernero A, Bastir M, Huguet R et al. Possible Further Evidence of Low Genetic Diversity in the El Sidrón (Asturias, Spain) Neandertal Group: Congenital Clefts of the Atlas. PLoS ONE 10(9): e0136550 (2015). pmid:26418427
  54. 54. Prüfer K, de Filippo C, Grote S, Mafessoni F, Korlevic P, Hajdinjak M at al. A high-coverage Neandertal genome from Vindija Cave in Croatia. Science, (2017). pmid:28982794
  55. 55. Sikora M, Seguin-Orlando A, Sousa VC, Albrechtsen A, Korneliussen T, Ko A et al. Ancient genomes show social and reproductive behavior of early Upper Paleolithic foragers, Science 358(6363):659–62 (2017). pmid:28982795
  56. 56. Harris K, Nielsen R, The genetic cost of Neanderthal introgression, Genetics 203(2):881–91 (2016). pmid:27038113
  57. 57. Courchamp F, Clutton-Brock T, Grenfell B. Multipack dynamics and the Allee effect in the African wild dog, Lycaon pictus. Animal Conservation 3: 277–285 (2000).
  58. 58. Caswell H. Matrix Population Models: Construction, Analysis, and Interpretation. (Sinauer 2001).
  59. 59. Lacy RC. VORTEX: A computer simulation model for Population Viability Analysis. Wildlife Research 20: 45–65 (1993).
  60. 60. Lacy RC. Structure of the VORTEX simulation model for population viability analysis. Ecological Bulletins 48: 191–203 (2000).
  61. 61. Lacy RC, Pollak JP. Vortex: A stochastic simulation of the extinction process. Version 10.0. Chicago Zoological Society, Brookfield, Illinois, USA (2014).
  62. 62. Kelly RL. The lifeways of hunter-gatherers: the foraging spectrum. (Cambridge University Press 2013).
  63. 63. Coale AJ, Demeny P. Regional Model Life Tables and Stable Populations (2nd edition). (Princeton University Press 1983).
  64. 64. Chamberlain A. Demography in archaeology. (Cambridge University Press 2006).
  65. 65. Tuljapurkar SD, Orzack SH. Population dynamics in variable environments, I: Long-run growth rates and extinction. Theor Pop Biol 18: 314–342 (1980).
  66. 66. Cavalli-Sforza LL, Bodmer WF. The genetics of human populations. (Free-man 1971).
  67. 67. Chakraborty R, Chakravarti A. On consanguineous marriages and the genetic load. Hum Genet 36(1): 47–54 (1977). pmid:870410
  68. 68. Bittles AH, Neel JV. The costs of human inbreeding and their implications for variations at the DNA level. Nat Genet 8(2): 117–121 (1994). pmid:7842008
  69. 69. Gao Z, Waggoner D, Stephens M, Ober C, Przeworski M. An estimate of the average number of recessive lethal mutations carried by humans. Genetics 199: 1243–1254 (2015). pmid:25697177
  70. 70. Narasimhan VM, Hunt K, Mason D, Baker CL, Karczewski KJ, Barnes MR et al. Health and population effects of rare gene knockouts in adult humans with related parents. Science 352(6284): 474–477 (2016). pmid:26940866
  71. 71. Molnár PK, Lewis MA, Derocher AE. Estimating Allee Dynamics before They Can Be Observed: Polar Bears as a Case Study. PLoS ONE 9(1): e85410 (2014). pmid:24427306
  72. 72. Lalueza-Fox C, Rosas A, Estalrrich A, Gigli E, Campos PF, García-Tabernero A et al., Genetic evidence for patrilocal mating behavior among Neandertal groups, Proc Nat Acad Sc 108(1):250–3 (2011).
  73. 73. Kolodny O, Feldman MW. A parsimonious neutral model suggests Neanderthal replacement was determined by migration and random species drift. Nature communications 8(1): 1040 (2017). pmid:29089499
  74. 74. Pettit PB. Disappearing from the world: An archaeological perspective on Neanderthal extinction. Oxf J Archaeol 18(3): 217–240 (1999).
  75. 75. Hrdy SB. Mother and Others: the Evolutionary Origins of Mutual Understanding. (Harvard University Press 2009).
  76. 76. Hublin JJ, Roebroeks W. Ebb and flow or regional extinctions? On the character of Neandertal occupation of northern environments. Pal Evol 8: 503–509 (2009).
  77. 77. Pennington R, Hunter-gatherer demography. In: Panter-Brick C (ed), Hunter-gathers: an interdisciplinary perspective (Cambridge University Press 2001, pp. 170–204)
  78. 78. Finlayson C, Pacheco FG, Rodríguez-Vidal J, Fa DA, Gutierrez López JM, Pérez ASet al. Late survival of Neanderthals at the southernmost extreme of Europe. Nature 443(19): 850–854 (2006).