Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Abundance estimation for line transect sampling: A comparison of distance sampling and spatial capture-recapture models

  • Nathan J. Crum ,

    Roles Conceptualization, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Fish and Wildlife Research Institute, Florida Fish and Wildlife Conservation Commission, St Petersburg, Florida, United States of America

  • Lisa C. Neyman,

    Roles Formal analysis, Writing – original draft, Writing – review & editing

    Affiliation Fish and Wildlife Research Institute, Florida Fish and Wildlife Conservation Commission, St Petersburg, Florida, United States of America

  • Timothy A. Gowan

    Roles Formal analysis, Methodology, Writing – review & editing

    Affiliation Fish and Wildlife Research Institute, Florida Fish and Wildlife Conservation Commission, St Petersburg, Florida, United States of America


Accurate and precise abundance estimation is vital for informed wildlife conservation and management decision-making. Line transect surveys are a common sampling approach for abundance estimation. Distance sampling is often used to estimate abundance from line transect survey data; however, search encounter spatial capture-recapture can also be used when individuals in the population of interest are identifiable. The search encounter spatial capture-recapture model has rarely been applied, and its performance has not been compared to that of distance sampling. We analyzed simulated datasets to compare the performance of distance sampling and spatial capture-recapture abundance estimators. Additionally, we estimated the abundance of North Atlantic right whales in the southeastern United States with two formulations of each model and compared the estimates. Spatial capture-recapture abundance estimates had lower root mean squared error than distance sampling estimates. Spatial capture-recapture 95% credible intervals for abundance had nominal coverage, i.e., contained the simulating value for abundance in 95% of simulations, whereas distance sampling credible intervals had below nominal coverage. Moreover, North Atlantic right whale abundance estimates from distance sampling models were more sensitive to model specification compared to spatial capture-recapture estimates. When estimating abundance from line transect data, researchers should consider using search encounter spatial capture-recapture when individuals in the population of interest are identifiable, when line transects are surveyed over multiple occasions, when there is imperfect detection of individuals located on the line transect, and when it is safe to assume the population of interest is closed demographically. When line transects are surveyed over multiple occasions, researchers should be aware that individual space use may induce spatial autocorrelation in counts across transects. This is not accounted for in common distance sampling estimators and leads to overly precise abundance estimates.


The cornerstone of effective wildlife population management is the ability to estimate population abundance accurately and precisely. Two widely used methods for estimating abundance are distance sampling (DS) and spatial capture-recapture (SCR), a recent development of mark-recapture [1], and there are situations in which both methods could be suitable. Although studies have compared abundance estimates of DS and mark-recapture models [2, 3] and SCR and mark-recapture models [e.g. 4], and demonstrated how these models are related [5], to the best of our knowledge, DS and SCR model performances have not been compared. We compare DS and SCR abundance estimation herein after providing a brief introduction for both approaches. For in-depth descriptions we refer interested readers to Buckland et al. [6] for DS and Royle et al. [7] for SCR.

During a DS survey, one or more observers conduct at least one search for individuals of a species of interest. When observers detect an individual, they record the distance between themselves and the detected individual. Distances to detected individuals are used to estimate the scale of a detection function, which describes the probability that an individual is detected, given its distance from the observer. The detection function and abundance can be estimated simultaneously as in one-stage, hierarchical distance sampling [8]. Alternatively, a two-stage approach can be taken where the detection function is estimated first, and abundance second using a Horvitz-Thompson estimator based on the average detection probability over the surveyed region and the number of individuals detected [6].

There are two primary DS survey types, line transect and point transect [1, 6]. While both methods are useful, depending on the species or object of interest, we focus solely on line transect surveys, which involve observers traveling along a transect line and recording the perpendicular distance between detected individuals and the line. Individuals in the population of interest need not be individually identifiable for DS, although it is assumed that an individual is seen at most once over a transect line.

A key assumption underlying DS is certain detection on the transect line [6]. However, detection is often imperfect on the transect line due to availability and perception biases. For instance, some individuals are unavailable for detection or observers do not detect all individuals that are located on the transect line and are available for detection, respectively. When either bias is present, survey or analytical methods can be modified to relax the assumption of certain detection on the transect line and reduce the negative bias of resulting abundance estimates [9]. One such approach to estimate perception bias is to use a double-observer design, or mark-recapture distance sampling (MRDS), which involves two independent observers scanning the survey area [6, 9, 10]. The proportion of detections made by one observer to detections made by both observers is used to estimate the probability of detection given an individual is located on the transect line [6, 11]. Alternatively, a separate study may be conducted to estimate detection probability on the transect line for the species of interest [1215].

SCR, unlike DS, requires that individuals in the population of interest be individually identifiable. Identifying marks may be natural, such as humpback whale fluke or tiger stripe patterns [16, 17], a DNA sample collected from hair or scat [18, 19], or human imposed, such as leg bands on birds [20]. A surveyor attempts to capture or detect individuals, often using an array of traps or detectors, from the population of interest. When individuals can be detected at most once during a sampling occasion, multiple sampling occasions are needed for a SCR analysis. The surveyor records encounter histories for each individual, denoting the occasions during which each individual was and was not detected. Additionally, the location of each detector and each detection is recorded for SCR analyses. Encounter histories can be used to estimate detection probability, which, along with the number of individuals detected at least once, informs abundance estimation.

SCR was developed to address two shortcomings of traditional mark-recapture [7, 21, 22]. First, abundance estimates from mark-recapture are not associated with a well-defined study area, so population density cannot be formally estimated. Second, mark-recapture does not account for individual heterogeneity in detection probability due to individuals’ locations relative to detectors. SCR addresses these issues by incorporating the spatial data inherent to an individual’s encounter history, i.e. where they were detected and where they were not. Specifically, each individual is associated with an unobservable activity center, which represents the center of an individual’s movement or average location across sampling occasions. SCR estimates the density of activity centers across a spatial state-space defined by the researcher. The probability that an individual is detected at a detector on an occasion is modeled as a function of the distance between the individual’s activity center and the detector. This detection function is analogous to the detection function used in DS.

SCR models can also accommodate data collected through search encounter field methods where individuals are identifiable [22]. These field methods include surveying points, plots, or along lines, such as in DS surveys. Search encounter SCR models differ from traditional SCR models in that individuals can be detected anywhere in the state space, not just at the location of detectors. However, to the best of our knowledge, all applications of the search encounter SCR model treat surveys as a trapping grid, where survey effort and the locations of detections are aggregated into a grid of cells [e.g., 2325]. If survey effort occurs in a cell, then the “trap” at the center of the cell is considered active, i.e., individuals can be detected there. The amount of survey effort expended within each cell can be used as a covariate on detection probability. However, when detections can occur away from a transect line, as in DS surveys, an individual may be detected in cells other than the cell containing the line transect from which the individual was detected. Therefore, this approach presents a challenge to modeling survey effort when detections occur away from the line. The search encounter SCR model proposed by Royle et al. [22], which has yet to be broadly applied, avoids this challenge. It models detection probability as a function of the distance between an individual’s location and the location of survey effort, rather than aggregating survey effort into a trapping grid. Like DS models for multiple occasions, SCR assumes population closure (no births, deaths, immigration, or emigration) across occasions and, like all mark-recapture models, that heterogeneity in detection probability across individuals is modeled. Unlike DS, search encounter SCR models do not require certain detection along the transect line and can model the space use of individuals across occasions.

One example of a species whose abundance is often estimated with DS but could be estimated with SCR is the endangered North Atlantic right whale (NARW) Eubalaena glacialis [26, 27]. Aerial line transect surveys are conducted off the southeastern US coast (SEUS) on most days with suitable weather each winter (December through March) to monitor the NARW population while on their winter calving grounds [28, 29]. Each NARW is uniquely identifiable by callosity patterns on its head [30], and survey protocol dictates that planes break track to photograph and identify every detected individual and obtain their exact location [29]. Although this survey protocol of multiple sampling occasions and identifying individuals is not needed for DS, this combination produces data that can be modeled with both DS and SCR.

Conceptually, search encounter SCR may be more appropriate for use in estimating NARW abundance than DS. NARWs are not always available for detection during a survey (i.e., are at a depth at which they cannot be seen) and may be missed by observers even when they are located along the transect line and are available to be detected. SCR estimates the probability of detecting an individual located on the transect line based on information from individual identifications, while DS requires additional data to be collected to correct cases with imperfect detection on the transect line. Moreover, NARWs often move between sampling occasions, and inference about NARW movement may be of interest. SCR explicitly models individuals’ space use across sampling occasions, while DS does not.

However, the performance of the search encounter SCR model has not been compared to DS. Given the lack of understanding around how the increased flexibility of SCR models may influence abundance estimation, our objective was to compare the accuracy and precision of abundance and other parameter estimates from DS and SCR models. We analyzed simulated datasets and NARW data from January 2010 with DS and SCR models for this comparison.

Material and methods

Simulation scenarios

We analyzed simulated line transect datasets with DS and SCR models to compare their performance. For each dataset, we simulated a population of individuals where each individual was associated with an activity center, s = (sx, sy), located within the state space (study area). Individuals moved about these activity centers, resulting in locations or movement outcomes, u = (ux, uy), during each occasion of the simulation. We overlaid transect lines on the state space, and the probability that an individual was detected on an occasion was a function of the individual’s distance to the closest point on a transect line. Each transect line was surveyed during each occasion in these simulations, although this is not a requirement for either DS or SCR. From each simulation we generated capture histories that included each individual’s location on the occasions it was detected.

We simulated datasets based on four values of the probability of detection on the transect line, commonly referred to as g(0), g(0) ϵ (1, 0.8, 0.5, 0.3), because our SCR model allows estimation of g(0) without additional data, while DS models require additional data to be collected to estimate g(0). These additional data often come in the form of having an additional observer during surveys (e.g. MRDS) or by estimating g(0) independently of the line transect surveys. Therefore, we simulated datasets generated from single- and double-observer sampling designs for each value of g(0) less than one, resulting in seven scenarios (single- and double- observer designs for g(0) ϵ (0.8, 0.5, 0.3) and a single-observer design for g(0) = 1).

We simulated 1,000 datasets for each scenario with each dataset having a population size of 100 individuals. The number of occasions varied across different scenarios, so that the average number of detections was similar across scenarios (between 50 and 70). Activity centers were uniformly distributed across a state space (0 ≤ x ≤ 40, 0 ≤ y ≤ 40; Fig 1), where we denote the activity center of individual i: (si,x, si,y).

Fig 1. Simulated datasets.

Examples of simulated datasets following a spatial capture-recapture model where density was uniform across the state space (top row) or a function of a spatial covariate in the state space (bottom row). Each individual was associated with an activity center, depicted by the points in the left column. Individuals were located around their respective activity centers on each sampling occasion, represented for one individual by the open circles in the middle column. The probability of an individual moving to a particular location during an occasion was a function of the distance from its activity center, with the individual being more likely to be located close to rather than far from the activity center. These locations were also distributed as a function of the spatial covariate in the spatial covariate scenario (bottom row). Line transect surveys, represented by the horizontal lines, were conducted during each sampling occasion. The probability that an individual was detected on an occasion was a function of the distance between its location on that occasion and the closest point on a transect line, with the probability being higher the closer an individual was to a line (right column). The locations of detections are depicted by the crossed-out circles in the right column.

Individuals’ locations on each occasion were distributed according to a bivariate normal distribution around their respective activity centers, with movement scale parameter, σm = 2.5, such that the probability of the location of individual i on occasion j, ui,j, occurring within some area of space, A, such as a grid cell in a gridded state space, is a function of the distance, dmovement,i,j, between the individual’s location and its activity center, following: (1)

Ten horizontal line transects were surveyed each occasion, although each line need not be surveyed every occasion in practice (Fig 1). Line transects were defined as y ϵ (11, 13, 15, 17, 19, 21, 23, 25, 27, 29), with endpoints at x = 10 and 30. The probability that individual i was detected on occasion j, denoted as yi,j = 1, was a function of g(0) and the distance between the individual and the closest transect on occasion j, ddetection,i,j, with a scale parameter, σd, of 1/3.


We chose the above sampling configuration and values for σm and σd such that individuals with activity centers located at the edge of the state space would have a negligible probability of being detected over the course of sampling. In single-observer scenarios, we simulated whether an individual was detected or not on an occasion from a Bernoulli distribution. In double-observer scenarios, we simulated the number of observers that detected an individual on an occasion from a binomial distribution with two draws.

We were also interested in comparing the performance of DS and SCR models when estimating how density is associated with a spatial covariate. We simulated 250 datasets with 150 individuals, 10 sampling occasions, and g(0) = 0.8 for this spatial covariate scenario. We used the same state space as in previous scenarios but discretized it into G = 100 4-by-4-unit grid cells, where each cell was associated with a covariate value (Fig 1). Activity centers were distributed across cells of the state space according to a multinomial distribution, with the probability of the activity center of individual i, si, occurring in cell k being a log-linear function of the spatial covariates. (3) Here, we set the values for parameters β0 and β1 such that the expected population size, , was 150.

Individuals’ locations were modeled on two spatial scales. The coarser scale was the cell in which an individual was located on an occasion, which followed a multinomial distribution, with the probability of being located in cell k being a function of that cell’s distance from the individual’s activity center, with a scale parameter σm = 4, and the cell’s covariate value [31].


Here we used the same β1 parameter describing the effect of the covariate from the activity center distribution, although SCR models can model these effects separately. Then, for the finer spatial scale, the individual’s location within a cell followed a uniform distribution.

For this spatial covariate scenario, five single-observer line transects were surveyed each occasion (Fig 1). Line transects were defined as y ϵ (12, 16, 20, 24, 28) with endpoints x = 10 and 30. The probability that an individual was detected follows from Eq 2, with a detection scale parameter, σd, of 2/3.

Distance sampling

We formulated our line transect DS models following Kery and Royle [32], who describe hierarchical, one-stage DS models. We used a Bayesian approach with data augmentation to estimate total abundance within the state space (N), abundance within the sampled region (Nsampled), the detection scale (σd), and the probability of detection at zero distance from the survey line (g(0)) in double observer, MRDS scenarios. Ntotal was derived from the estimate of Nsampled following: (5) where Areastate space is the area of the state space. The area sampled, Areasampled, is based on the cumulative length of transect lines surveyed across sampling occasions and the truncation distance, which is described below.

Data augmentation is performed by adding M—n latent detection distances to the model, where M is the largest population size permitted by the model and n is the number of detections. Abundance within the sampled region followed a binomial distribution: where ψ is the data augmentation parameter used to estimate the proportion of the M potential individuals that were truly part of the population [32]. In uniform density scenarios ψ followed a uniform distribution: In scenarios where density was modeled as a function of a spatial covariate, ψ was a function of expected abundances across grid cells: (6) (7) where λk and Covariatek are the expected abundance and covariate value of cell k, respectively, and β0 and β1 are estimated parameters. Additionally, in the spatial covariate scenario the cell in which an individual was located followed a categorical distribution: where ui is the cell in which individual i is located.

For line transect data the distances between individuals and transect lines follow a uniform distribution: where di is the distance from individual i to the transect line and dtruncation is the truncation distance, which defines the boundary of the sampled region around a transect. In our analyses the truncation distance was half the distance between adjacent transect lines. However, our simulations and the NARW application allowed for detections beyond the endpoints of transect lines, which is atypical for line transect distance sampling. Distances between individuals and the endpoints of transect lines follow a triangular distribution, as in point transect distance sampling [33]:

Additional details describing the simultaneous analysis of detection distances within and outside transect line endpoints can be found in the S3 Appendix. Finally, the number of times individual i was detected or not, yi, followed a binomial distribution: where x, the number of draws, was one for single observer models and two for double observer models. In double observer models we estimated g(0) from the data with a non-informative uniform prior distribution, whereas in single observer models g(0) was either fixed at the simulating value of g(0) or drawn from an informative beta prior distribution. Informative beta prior distributions had a mean equal to the value of g(0) used to simulate the dataset being analyzed and a standard deviation of 0.1 in uniform density models and 0.05 in models with spatial covariates. Use of informative priors was intended to replicate the approach of using independent data to estimate g(0). In all models we estimated σd from the data with a non-informative uniform prior distribution.

Spatial capture-recapture

We formulated our SCR models following Royle et al. [22]. We again used a Bayesian approach with data augmentation to estimate total abundance within the state space (N), the movement scale (σm), the detection scale (σd), and the probability of detection at zero distance from the survey line (g(0)). Abundance within the state space followed a binomial distribution: As in the distance sampling models, ψ followed a uniform distribution in uniform density models. In models that included a spatial covariate, ψ was a function of expected abundances across grid cells: (8) (9) In uniform density models the activity center of individual i, si, followed a uniform distribution: where S is the state space. In models with a spatial covariate, the grid cell in which si was located followed a categorical distribution:

The location of individual i on occasion j, ui,j, followed a bivariate normal distribution in uniform density models: (10)

In models with a spatial covariate, the grid cell in which ui,j was located followed a categorical distribution: (11) where dk is the distance from si to the center of cell k. The location of ui,j within a grid cell followed a uniform distribution. We estimated σm from the data using a non-informative uniform prior distribution. Whether individual i was detected or not on occasion j, yi,j, followed a binomial distribution: where di,j is the distance between ui,j and the closest transect line and the number of draws, x, is two in double observer designs and one in single observer designs. We estimated g(0) and σd from the data using non-informative uniform prior distributions. This detection model is identical to that of our distance sampling model and can accommodate continuous line transect survey effort. In contrast, Royle et al. [22] present hazard models for detection and approximated line transect survey effort using a set of evenly spaced points along the transects. Details for calculating the distance between ui,j and continuous transect lines can be found in the S3 Appendix.

Analysis of simulated datasets

We used the Nimble package [34] in program R 3.6.2 [35] to write and implement all DS and SCR models [S3 Appendix]. For each simulated dataset we ran three Markov chain Monte Carlo (MCMC) chains of 25,000 iterations for each model. We checked for convergence of MCMC chains using the r-hat statistic [36]. We discarded the fewest of 5,000, 10,000, or 15,000 iterations as burn-in for each simulation while ensuring that r-hat values were less than 1.1 for all parameters. For instance, if all chains converged within the first 5,000 iterations, the first 5,000 iterations were discarded, but if all chains did not converge within the first 5,000 iterations but did within 10,000 iterations, the first 10,000 iterations were discarded, and so on. If the chains of any parameter did not converge by 15,000 iterations, all analyses conducted on that dataset were omitted and a new dataset was simulated and analyzed to replace it. However, we did not simulate new datasets in such cases for the spatial covariate scenario because of the greater runtime required for those models. We report the average posterior mean and 95% credible interval, as well as the 95% credible interval coverage, average root mean squared error (RMSE), and average bias for each parameter in each scenario. The 95% credible interval coverage was calculated as , where n is the number of simulations, θ is the true value of a parameter, and are the 0.025 and 0.975 quantiles of the posterior distribution of parameter θ of the i-th simulation, and I is an indicator function that equals one if its argument is true and zero if it is false. Average root mean squared error and bias were calculated as and , respectively, where T is the number of MCMC iterations after discarding the burn-in, and is the estimate of θ at the t-th MCMC iteration for the i-th simulated dataset .

Estimating abundance of North Atlantic right whales

Using single-observer DS and SCR models, we estimated abundance of NARWs in the SEUS from aerial survey data collected between 16 January 2010 and 31 January 2010. Specifically, we estimated abundance over a 76,800 km2 state space that ranges from 28.0° N to 33.8° N and from the coast of Florida, Georgia, and South Carolina to waters 70 m deep (Fig 2). Aerial line transect surveys were flown for 10 of the 16 days. Subsets of 82 transect lines were flown each of these 10 days. The transect lines flown ranged from 29.6° N to 33.0° N and from shore to as far as 65 km offshore. When a NARW was detected, the aerial survey team broke from the transect line to photograph the NARW for identification purposes and to document the exact location of the NARW. See Gowan and Ortega-Ortiz [29] for a further description of field methods used. Over these 10 survey days 91 individual NARWs were detected (38 on one occasion, 27 twice, 12 three times, 11 four times, and 3 five times).

Fig 2. North Atlantic right whale density.

Expected North Atlantic right whale (NARW) density per 400 km2. NARW density was estimated as a quadratic function of depth within the state space using distance sampling (DS; bottom left panel and red in top panel) and spatial capture-recapture (SCR; bottom right panel and blue in top panel). Posterior mean expected density and 95% credible intervals across the range of depth in the state space is illustrated in the top panel for both DS (red) and SCR (blue) models. Bottom panels illustrate posterior mean expected density across the state space for DS (left) and SCR (right) models that included depth and depth2 as covariates on density.

We present two DS and SCR models of right whale density to demonstrate the application of both models to the same dataset. The first assumes density is constant across space, and the second models spatial variation in density as a quadratic function of depth analogous to our spatial covariate scenario. Additional models that include different detection functions and density models would be needed to make robust multi-model inference regarding NARW abundance, however such a model set is beyond the scope of this study. In our DS analyses of right whale data we fixed the value of g(0) to 0.6 following the work of Hain et al. [37], who found right whale availability for detection in the Southeast was approximately 0.6. Uncertainty in this estimate of availability was high and including that uncertainty into a prior distribution for g(0) led to slow model convergence and unreasonably large abundance credible intervals in preliminary tests; therefore we did not account for this uncertainty in g(0). We used a Bayesian approach for all models and augmented the encounter histories of the 91 observed individuals with 259 unobserved encounter histories of individuals that may have been part of the population, resulting in a maximum possible abundance of 350. Preliminary analyses indicated that enough unobserved encounter histories were added through data augmentation, as the posterior distribution for abundance was not truncated. We ran three MCMC chains of 20,000 iterations for the uniform density models and three chains of 40,000 iterations for the models that included depth as a covariate on density. We discarded 5,000 iterations as burn-in for each model and report the posterior means and credible intervals of estimated parameters.



SCR abundance estimates were positively biased, from 1–9%, across all scenarios (however, see S1 Appendix, which shows SCR abundance estimates of larger datasets are unbiased), while the bias of DS abundance estimates was more varied (Table 1). DS models with a fixed g(0) were negligibly biased (-1%–2%) across all scenarios, while double-observer and informed g(0) DS models had increasingly positive bias as the true value of g(0) decreased (bias <5% with g(0) = 0.8 and 25–30% with g(0) = 0.3) (Table 1).

RMSE of SCR abundance estimates was lower than that of all DS models in all scenarios (Table 1). DS models with fixed g(0) had RMSE 6–22% higher than SCR models across scenarios. RMSE of abundance estimates from informed g(0) and double-observer DS models increased as the true value of g(0) decreased. Additionally, RMSE of abundance estimates from these models was 12% and 26% higher than that of SCR in scenarios where g(0) was 0.8. When g(0) was 0.3 RMSE was 133 and 215% higher than that of SCR (Table 1).

The estimated abundance credible interval range was 17–25% narrower for fixed g(0) DS models than for SCR models across all scenarios. The credible interval range increased as the true value of g(0) decreased for double-observer and informed g(0) models, increasing 206% and 137%, respectively, from when g(0) was 0.8 to 0.3 The credible interval ranges of these two models were wider than that of SCR in all scenarios except the double-observer scenario with g(0) equal to 0.8 (Table 1).

Abundance 95% credible interval coverage was nominal, i.e. not significantly different from 95%, for SCR models in all scenarios. Coverage was below nominal for fixed g(0) and double-observer DS models in all scenarios. Coverage was below nominal for the informed g(0) DS model when g(0) was 0.8 and was above nominal when g(0) was 0.5 and 0.3 (Table 1).

Both the bias and RMSE of the detection scale estimates were lower for SCR than for all DS models across all scenarios. Likewise, bias and RMSE of the estimated effect of the spatial covariate were lower for SCR than for DS in the spatial covariate scenario. The estimated movement scale from SCR models was negatively biased across all scenarios. In uniform density scenarios with g(0) equal to 1, 0.8, and 0.5 all top level parameters (N, g(0), σd, and σm) converged for >99% of both SCR and DS models. In the scenario with g(0) equal to 0.3, top level parameters converged in all DS models that fixed the value g(0), in 73.0% of DS models that drew g(0) from an informative prior distribution, and in 79.3% of SCR models. In the spatial covariate scenario convergence was reached for all top level parameters (N, g(0), β0, β1, σd, and σm) in 64% of fixed g(0) DS analyses, in 60.8% of informed g(0) analyses, and in 79.2% of SCR analyses [S1 Appendix].

North Atlantic right whale abundance

The uniform density DS model estimated a population size of 247.5 (210–285 95% CI), while the uniform density SCR model estimated a population size of 160.2 (134–193 95% CI). The value of g(0) for the DS model was fixed at 0.6, and the uniform density SCR model estimated a value of g(0) of 0.832 (0.662–0.984 95% CI). The uniform density SCR and DS models estimated a similar σd of 3.51 km (2.94–4.24 km 95% CI) and 3.47 km (2.91–4.18 km 95% CI), respectively. The SCR model estimated a σm of 142.6 km (119.3–182.4 km 95% CI).

The SCR and DS models that modeled density as a quadratic function of depth estimated similar population sizes of 159.0 (131–193 95% CI) and 156.4 (133–182 95% CI), respectively. Additionally, both models estimated density to be similarly associated with depth (Fig 2). The SCR model estimated g(0) to be 0.558 (0.441–0.696 95% CI). Again, the SCR and DS models estimated similar values of σd, 3.79 km (3.11–4.67 km 95% CI) and 3.82 km (3.12–4.69 km 95% CI). The SCR model estimated a σm of 153.9 km (124.0–201.0 km 95% CI).


We compared the accuracy and precision of search encounter spatial capture-recapture (SCR) and hierarchical distance sampling (DS) abundance estimators using simulated data sets and found that the SCR estimator outperformed all DS estimators. The SCR abundance estimator had lower RMSE than all DS estimators and had nominal credible interval coverage, whereas DS estimators had below nominal coverage. Additionally, while the SCR abundance estimator was slightly positively biased, the DS estimators were as or more positively biased when uncertainty in g(0) was modeled. The search encounter SCR model has not been applied beyond its conception [22, 38], and we are unaware of any such comparison between DS and SCR models. However, the search encounter SCR model outperformed DS models in our scenarios, suggesting that future research and use of the search encounter SCR model is warranted.

Assumptions of conventional DS include the following: all animals on the transect line are detected (g(0) = 1), animals do not move in response to the observer, distance to animals is measured without error, and animals are detected independently of one another [6]. Our simulated datasets did not violate these assumptions, other than g(0) ≠ 1, which we accounted for in our modeling approaches. Despite this, the DS abundance estimates were overly precise (i.e. variance was underestimated and credible interval coverage was below nominal) and became more so in scenarios with more sampling occasions [S1 Appendix]. We suspect that this over-precision is due to the DS model underestimating variance by not explicitly accounting for the space use of individuals relative to the spatial configuration of transect lines across sampling occasions. This space use of individuals can cause spatial autocorrelation in the number of detections across transect lines. Models have been developed for repeated DS surveys to account for individuals moving in to and out of the sampled region, i.e., temporary emigration [39]. Still, variance is not appropriately estimated by these models when individuals have different probabilities of temporarily emigrating, as in our simulations where the overlap of an individual’s space use with the sampled region depended on the location of its activity center [39]. Developments in hierarchical DS have been proposed to account for non-independence of detections at discrete sites over repeat sampling occasions [40]. Advancing this idea to cases where the study area is continuous, rather than composed of discrete sites, could address the over-precision we found in DS abundance estimates. Additionally, variance estimation has received more attention for two-stage DS models than the one-stage hierarchical model we applied [41]. However, variance is still underestimated by a two-stage DS model that accounts for repeated sampling occasions, whether variance is estimated analytically or through non-parametric bootstrapping [S2 Appendix]. Moving block bootstrapping [42, 43] and parametric bootstrapping [44] are additional specialized approaches that may provide unbiased variance estimation but were not tested here. We evaluated simple space use models here that may not approximate the space use patterns of many species. Moreover, the space use models of our simulations met the assumptions of the SCR models. Therefore, we did not evaluate whether SCR abundance estimates are robust to any misspecification of the space use model. Additional work would be needed to determine if DS underestimates the variance of abundance and if SCR abundance estimates are robust under other individual space use models.

The DS models required modification to account for the location along the transect line where individuals were detected, i.e. within or outside the transect line’s endpoints. Detections within endpoints were generated from line transect sampling, while detections outside endpoints were generated from point transect sampling. This modification is not available in common analytical tools for DS, such as the Distance package [45] in program R, and required a custom sampler built through the Nimble package [33; S3 Appendix]. Researchers using DS to model line transect surveys should be aware that detections outside the endpoints of line transects should be discarded to meet the assumptions of line transect DS. Survey configurations with fewer and longer line transects and narrower search widths will require a smaller proportion of detections to be discarded. Retaining these detections in a line transect DS analysis could lead to an overestimation of the detection scale (more detections occur farther from the sampler in point transect than line transect sampling with the same detection scale) and an underestimation of abundance. SCR inherently accounts for this obstacle by explicitly modeling the locations of transect lines and individuals and deriving the distance between them. Therefore, survey effort and detections that are typically discarded in a DS analysis can be incorporated into a SCR analysis without modification of the SCR model.

The SCR model performed well except for the small positive bias of abundance point estimates. However, this bias decreased in scenarios with more recaptures, suggesting that the estimator has negligible bias for large sample sizes [S1 Appendix]. Additionally, the DS abundance estimates from models that accounted for uncertainty in g(0) were as or more positively biased than SCR estimates. The DS abundance estimates from models that fixed g(0) at the true value were unbiased. However, this approach is not ideal in applications of DS, as g(0) is unknown and fixing its value does not propagate any of its associated uncertainty into the abundance estimate [12]. Researchers using these models should be aware of these biases when analyzing small datasets and should consider using the median of the posterior distribution as a point estimate for abundance because it is less biased than the mean [S1 Appendix].

DS has been used to produce spatially explicit estimates of North Atlantic right whale (NARW) abundance [26, 27]; however, SCR should be the preferred approach to estimate NARW abundance based on common NARW sampling procedures and SCR’s outperforming DS in our comparisons. NARWs are individually identifiable, and their population is surveyed by aerial line transect surveys conducted over multiple occasions in the same region [27, 29]. Additionally, g(0) ≠ 1, because NARWs, like other marine mammals, spend time below the water’s surface where they are unavailable for detection and may be difficult to detect at the surface, depending on environmental conditions and behavior [37, 46]. SCR overcomes these sources of bias, while design or modeling modifications would be needed for a DS approach, which would still result in a poorer performing abundance estimator. Furthermore, SCR may produce more robust abundance estimates across competing model formulations than DS and therefore be less sensitive to model misspecification. This is evidenced by both SCR models producing similar abundance estimates (159.0 and 160.2), whereas abundance estimates from the two DS models were so different that their credible intervals did not overlap. The uniform density DS model assumes random placement of transect lines and estimated abundance to be considerably higher than the other three models, which may be due to the non-random placement of transect lines with respect to depth or other environmental characteristics with which NARW abundance is associated. However, we did not apply any model selection techniques or goodness of fit statistics that would be needed to make inference on NARW abundance.

Still, a closed population SCR model is not ideal for estimating NARW abundance. NARWs are a migratory species, and their regional abundance in the SEUS changes throughout the winter as individuals migrate into and out of the region [47]. Therefore, although we estimated NARW abundance in the SEUS over a short 2-week interval to try to meet our SCR model’s assumption that the population is closed [7], NARWs may have arrived or departed the SEUS during the study period. Future efforts to estimate NARW abundance in the SEUS using SCR should focus on longer time periods using an open population model, which will allow for estimation of abundance over time and rates of immigration, emigration, and movement within the SEUS region [4850].

Additional development of the search encounter SCR model may be needed to accommodate NARW social behavior and movement too. SCR assumes individuals in the population of interest are distributed and detected independently, however NARWs form temporary social groups [51]. Future work developing methods to account for group fission/fusion dynamics and the non-independence of detections of individuals in a group could address this violation of SCR assumptions. Although, Bischof et al. [52] found that SCR abundance estimates are robust to non-independence of individuals’ activity centers and detections. Furthermore, the SCR models used herein model the space use of a NARW as distributed about a central activity center. This type of space use model is often used for species that, unlike NARWs, exhibit home range behavior. Instead, a movement model could be developed such that the location of a NARW on one sampling occasion is modeled as a function of its location on the previous sampling occasion. Similar models have been developed to describe the movement of individuals’ activity centers in open population SCR models [53].

Estimating abundance accurately and precisely is critical to wildlife population management and conservation. Both DS and SCR models estimated abundance with bias of less than 10% across most scenarios. Therefore, it is unlikely that using one model over the other would inform management decisions differently. Still, researchers in the planning and analysis stages of abundance estimation studies should think carefully about their study system when choosing a modeling approach. We advise them to consider additional simulation studies, because the costs of sampling and individual identification and value of accurate and precise abundance estimates vary across study systems. Generally, DS will be more cost efficient than SCR as it does not require repeat sampling occasions or expending effort to identify individuals. The search encounter SCR model requires multiple sampling occasions and the assumption of population closure, which a single occasion DS study does not. However, for rare species such as the NARW, more than one sampling occasion may be required to detect enough individuals for a DS analysis. Still, DS is a good choice when it is safe to assume that g(0) = 1 or perception bias and availability biases can be estimated, when individuals cannot be uniquely identified, and when surveys are conducted over a single sampling occasion. However, researchers must be aware that a combination of transect line placement and animal movement over multiple occasions can lead to overly precise abundance estimates if variance is inappropriately estimated, which may give management decision-makers false confidence and inaccurate information. Survey design may ameliorate this issue, with surveys conducted in the same region over multiple occasions being distributed widely enough in space or time. This could prevent movement of individuals from causing over-precision in abundance estimates, although surveys should not be so widely distributed in time that abundance changes over sampling occasions. Conversely, search encounter SCR is more flexible and robust than DS for modeling line transect data but requires that individuals are identifiable and surveys are conducted over multiple occasions, which is not possible for every study system. SCR relaxes the DS assumption that detection probability on the transect line is certain without survey design or modeling modifications by using information from individual recaptures across occasions. Additionally, SCR can provide inference on space use and resource selection, which are unaccounted for in DS models and can lead to over-precision and higher error in abundance estimates. Therefore, SCR should be the model of choice to estimate abundance from line transect data collected over multiple occasions on a population of identifiable individuals.

Supporting information

S2 Appendix. Effects of movement on distance sampling abundance estimation.



We thank the aerial survey teams that collected the data used to estimate abundance of North Atlantic right whales. We also thank Richard Glennie, Colin Shea, and three anonymous reviewers for providing constructive reviews that improved the manuscript.


  1. 1. Borchers DL, Marques TA. From distance sampling to spatial capture-recapture. AStA Adv. Stat. Anal. 2017;101: 475–494.
  2. 2. Calambokidis J, Barlow J. Abundance of blue and humpback whales in the eastern North Pacific estimated by capture-recapture and line-transect methods. Publications, Agencies and Staff of the U.S. Department of Commerce; 2004. pmid:14768880
  3. 3. Minton G, Peter C, Poh ANZ, Ngeian J, Braulik G, Hammond PS, et al. Population estimates and distribution patterns of Irrawaddy dolphins (Orcaella brevirostris) and Indo-Pacific finless porpoises (Neophocaena phocaenoides) in the Kuching Bay, Sarawak. Raffles B. Zool. 2013;6: 877–888.
  4. 4. Noss AJ, Gardner B, Maffei L, Cuéllar E, Montaño , Romero-Muñoz A, et al. Comparison of density estimation methods for mammal populations with camera traps in the Kaa-lya del Gran Chaco landscape. Anim. Conserv. 2012;15: 527–535.
  5. 5. Borchers DL, Stevenson BC, Kidney D, Thomas L, Marques TA. A Unifying Model for Capture-Recapture and Distance Sampling Surveys of Wildlife Populations. J. Am. Stat. Assoc. 2015;110: 195–204. pmid:26063947
  6. 6. Buckland ST, Anderson DR, Burnham KP, Laake JL. Introduction to distance sampling: estimating abundance of biological populations. Oxford University Press; 2001.
  7. 7. Royle JA, Chandler RB, Sollmann R, Gardner B. Spatial capture-recapture. Academic Press; 2013.
  8. 8. Royle JA, Dawson DK, Bates S. Modeling abundance effects in distance sampling. Ecology 2004;85: 1591–1597.
  9. 9. Buckland ST, Rexstad EA, Marques TA, Oedekoven CS. Distance sampling: methods and applications. Springer International Publishing; 2015.
  10. 10. Hammond PS, Macleod K, Berggren P, Borchers DL, Burt L, Cañadas A, et al. Cetacean abundance and distribution in European Atlantic shelf waters to inform conservation and management. Biol. Conserv. 2013;164: 107–122.
  11. 11. Borchers DL, Buckland ST, Goedhart PW, Clarke ED, Hedley SL. Horvitz-Thompson estimators for double platform line transect surveys. Biometrics 1998;54: 1221–1237.
  12. 12. Laake JL, Calambokidis J, Osmek SD, Rugh DJ. Probability of detecting harbor porpoise from aerial surveys: estimating g(0). J. Wildl. Manag. 1997;61: 63–75.
  13. 13. Cañadas A, Desportes G, Borchers B. The estimation of the detection function and g(0) for short-beaked common dolphins (Delphinus delphis), using double-platform data collected during the NASS-95 Faroese survey. J. Cetacean Res. Manag. 2004;6: 191–198.
  14. 14. Bächler E, Liechti F. On the importance of g(0) for estimating bird population densities with standard distance-sampling: implications from a telemetry study and a literature review. Ibis 2007;149: 693–700.
  15. 15. Oyster JH, Keren IN, Hansen SJK, Harris RB. Hierarchical Mark-Recapture Distance Sampling to Estimate Moose Abundance. J. Wildl. Manag. 2018;82: 1668–1679.
  16. 16. Straley JM, Quinn TJ II, Gabriele CM. Assessment of mark–recapture models to estimate the abundance of a humpback whale feeding aggregation in Southeast Alaska. J. Biogeogr. 2009;36: 427–438.
  17. 17. Karanth KU. Estimating tiger Panthera tigris populations from camera-trap data using capture-recapture models. Biol. Conserv. 1995;71: 333–338.
  18. 18. Wilton CM, Beringer J, Puckett EE, Eggert LS, Belant JL. Spatiotemporal factors affecting detection of black bears during noninvasive capture–recapture surveys. J. Mammal. 2016;97: 266–273.
  19. 19. López-Bao JV, Godinho R, Pacheco C, Lema FJ, García E, Llaneza V, et al. Toward reliable population estimates of wolves by combining spatial capture-recapture models and non-invasive DNA monitoring. Sci. Rep. 2018;8: 2177. pmid:29391588
  20. 20. Sedinger JS, Riecke TV, Leach AG, Ward DH. The black brant population is declining based on mark recapture. J. Wildl. Manag. 2019;83: 627–637.
  21. 21. Borchers DL. A non-technical overview of spatially explicit capture-recapture models. J. Ornithol. 2012;152: 435–444.
  22. 22. Royle JA, Kéry M, Guélat J. Spatial capture-recapture models for search encounter data. Methods Ecol. Evol. 2011;2: 602–611.
  23. 23. Pirotta E, Thompson PM, Cheney B, Donovan CR, Lusseau D. Estimating spatial, temporal and individual variability in dolphin cumulative exposure to boat traffic using spatially explicit capture-recapture methods. Anim. Conserv. 2015;18: 20–31.
  24. 24. Russell RE, Royle JA, Desimone R, Schwartz MK, Edwards VL, Pilgrim KP, et al. Estimating abundance of mountain lions from unstructured spatial sampling. J. Wildl. Manag. 2012;76: 1551–1561.
  25. 25. Thompson CM, Royle JA, Garner JD. A framework for inference about carnivore density from unstructured spatial sampling of scat using detector dogs. J. Wildl. Manag. 2012;76:863–871.
  26. 26. Roberts JJ, Best BD, Mannocci L, Fujioka E, Halpin PN, Palka DL, et al. Habitat-based cetacean density models for the U.S. Atlantic and Gulf of Mexico. Sci. Rep. 2016;6: 22615. pmid:26936335
  27. 27. Ganley LC, Brault S, Mayo CA. What we see is not what there is: estimating North Atlantic right whale Eubalaena glacialis local abundance. Endanger. Species Res. 2019;38: 101–113.
  28. 28. Keller CA, Ward-Geiger LI, Brooks WB, Slay CK, Taylor CR, Zoodsma BJ. North Atlantic right whale distribution in relation to sea-surface temperature in the southeastern United States calving grounds. Mar. Mamm. Sci. 2006;22: 426–445.
  29. 29. Gowan TA, Ortega-Ortiz JG. Wintering habitat model for the North Atlantic right whale (Eubalaena glacialis) in the southeastern United States. PLoS ONE 2014;9: e95126. pmid:24740091
  30. 30. Kenney RD. Right Whales–Eugbalaena glacialis, E. japonica, and E. australis. In: Würsig B., Thewissen J. G. M., and Kovacs K. M., Encyclopedia of Marine Mammals, 3rd ed. Academic Press; 2018.
  31. 31. Royle JA, Chandler RB, Sun CC, Fuller AK. Integrating resource selection information with spatial capture-recapture. Methods Ecol. Evol. 2013;4: 520–530.
  32. 32. Kéry M, Royle JA. Applied Hierarchical Modeling in Ecology: Analysis of distribution, abundance and species richness in R and BUGS, 1st edition. Academic Press, London, UK; 2015.
  33. 33. Buckland ST. Advanced distance sampling. Oxford University Press; 2004.
  34. 34. de Valpine P, Turek D, Paciorek CJ, Anderson-Bergman C, Lang DT, Bodik R. Programming with models: writing statistical algorithms for general model structures with NIMBLE. J. Comput. Graph. Stat. 2017;26: 403–413.
  35. 35. R Core Team. 2019. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
  36. 36. Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB. Bayesian data analysis, 2nd ed. Chapman and Hall; 2003.
  37. 37. Hain JHW, Ellis SL, Kenney RD, Slay CK. Sightability of right whales in coastal waters of the southeastern United States with implications for the aerial survey monitoring program. In: Garner GW, Amstrup SC, Laake JL, Manley BFJ, McDonald LL, and Robertson DG (eds.), Marine Mammal Survey and Assessment Methods. A. A. Balkema; 1999. pp. 191–207.
  38. 38. Royle JA, Young KV. A hierarchical model for spatial capture-recapture data. Ecology 2008;89: 2281–2289. pmid:18724738
  39. 39. Chandler RB, Royle JA, King DI. Inference about density and temporary emigration in unmarked populations. Ecology 2011;92:1249–1435. pmid:21870617
  40. 40. Oedekoven CS, Buckland ST, Mackenzie ML, Evans KO, Burger LW Jr. Improving distance sampling: accounting for covariates and non-independency between sampled sites. J. Appl. Ecol. 2013;50:786–793.
  41. 41. Fewster RM, Buckland ST, Burnham KP, Borchers DL, Jupp PE, Laake JL, et al. Estimating the encounter rate variance in distance sampling. Biometrics 2009;65: 225–236. pmid:18363772
  42. 42. Efron B, Tibshirani RJ. An introduction to the bootstrap. Chapman Hall/CRC; 1993.
  43. 43. Miller DL, Burt ML, Rexstad EA, Thomas L. Spatial models for distance sampling data: recent developments and future directions. Methods Ecol. Evol. 2013;4: 1001–1010.
  44. 44. Hedley SL, Buckland ST. Spatial models for line transect sampling. J. Agr. Biol. Envir. St. 2004;9: 181–199.
  45. 45. Miller DL, Rexstad E, Thomas L, Marshall L, Laake JL. Distance sampling in R. J. Stat. Softw. 2019;89: 1–28.
  46. 46. Marsh H, Sinclair DF. Correcting for visibility bias in strip transect aerial surveys of aquatic fauna. J. Wildl. Manag. 1989;53: 1017–1024.
  47. 47. Krzystan AM, Gowan TA, Kendall WL, Martin J, Ortega-Ortiz JG, Jackson K, et al. Characterizing residence patterns of North Atlantic right whales in the southeastern USA with a multistate open robust design model. Endanger. Species Res. 2018;36: 279–295.
  48. 48. Gardner B, Reppucci J, Lucherini M, Royle JA. Spatially explicit inference for open populations: estimating demographic parameters from camera-trap studies. Ecology 2010;91: 3376–3383. pmid:21141198
  49. 49. Glennie R, Borchers DL, Murchie M, Harmsen BJ, Foster RJ. Open population maximum likelihood spatial capture-recapture. Biometrics 2019;75: 1345–1355. pmid:31045249
  50. 50. Gowan TA, Crum N, Roberts JJ. An open spatial capture-recapture model to estimate density, movement, and population dynamics from line-transect surveys. Ecology and Evolution Forthcoming.
  51. 51. Parks SE, Brown MW, Conger LA, Hamilton PK, Knowlton AR, Kraus SD, et al. Occurrence, composition, and potential functions of North Atlantic right whale (Eubalaena glacialis) surface active groups. Mar. Mamm. Sci. 2007;23: 868–887.
  52. 52. Bischof R, Dupont P, Milleret C, Chipperfield J, Royle JA. Consequences of ignoring group association in spatial capture-recapture analysis. Wildlife Biol. 2020;1: 1–10.
  53. 53. Schaub M, Royle JA. Estimating true instead of apparent survival using spatial Cormack-Jolly-Seber models. Methods Ecol. Evol. 2014;5:1316–1326.