Introduction

It has been observed that in general, the effect size of a variant or single nucleotide polymorphism (SNP) tends to be lower in a replication study than in the genome-wide association study (GWAS) that discovered the SNP-trait association. This observation is due to the phenomenon known as Winner’s Curse. In the context of a single discovery GWAS, the term Winner’s Curse describes how the estimates of association strength for SNPs that have been deemed most significant are very likely to be exaggerated compared with their true underlying values. These estimated effect sizes can take the form of log odds ratios (log-OR) resulting from a logistic regression for a binary outcome, e.g. disease status, or regression coefficients (beta) derived from a linear regression for a quantitative trait.

Dudbridge & Newcombe [1] distinguish between the concepts of ranking bias and selection bias, both of which are lenses through which we can view the effect of Winner’s Curse. Ranking bias stems from ranking many SNPs, often close to a million or more, by some measure of effect size or statistical significance. In practice, p-values are generally used. It is then expected that the bias will be greatest for those variants which have been ranked highly. Selection bias describes how the use of a stringent threshold, such as 5 × 10⁻⁸, can result in overestimated effect sizes for SNPs that exceed this threshold.

Winner’s Curse bias can have many practical consequences, especially with respect to techniques which are reliant on SNP-trait association estimates obtained from GWASs. One such example is Mendelian randomization (MR), a statistical framework which uses genetic variants as instrumental variables to estimate the magnitude of the casual effect of an exposure on an outcome. In the case of two-sample MR, if the same GWAS is used to identify instrument SNPs and estimate their effects relative to the exposure, Winner’s Curse will result in the overestimation of these SNP-exposure associations. This bias will then propagate into the causal estimate, resulting in a deflation of this estimate. On the other hand, if instrument SNPs are discovered in the same GWAS as that used to estimate the SNP-outcome associations, the causal estimate will be inflated [2]. In addition, Winner’s Curse has been shown to greatly increase the magnitude of weak instrument bias in these MR analyses [3]. Another implication of Winner’s Curse bias is in the use of polygenic risk scores which employs GWAS results for prediction purposes. Enlarged association estimates of significant variants used in creating the polygenic score can lead to reduced accuracy in out-of-sample prediction [4].

In this paper, we review existing Winner’s Curse correction methods and explore possible modifications that could be made in order to improve some of these methods. However, eliminating this bias induced by Winner’s Curse is known to be a difficult task. The challenge in doing so, in terms of genome-wide scans, was first pointed out by Göring et al. [5]. In their paper, they concluded that “attempts at bias correction give unsatisfactory results” and claimed that accurate locus-specific effect size estimates could only be obtained using a sample independent to that used for locus mapping. However, in the absence of readily available replication samples, this recommendation encourages the employment of sample-splitting, in which a portion of the original dataset is withheld for estimation. This approach can have major disadvantages as it reduces the power to detect linkage, whilst also increasing variance in the estimated effect sizes. Fortunately, since this issue was initially highlighted, several methods have been proposed for both linkage and association studies to reduce the bias in large observed effects, without requiring an independent sample.

Sun and Bull [6] are responsible for first proposing the use of the bootstrap to facilitate a solution to the problem of Winner’s Curse bias. In their paper, they consider estimating locus-specific effects by using statistical resampling procedures, such as cross-validation and the bootstrap. These techniques result in the original data set being randomly split into a detection and estimation sample, repeatedly. A bias reduction factor can then be generated by comparing the estimates obtained in the detection and estimation samples at each resampling step. Concentrating on the single-locus model, Sun and Bull [6] presented three potential estimators, namely the shrinkage, out-of-sample and weighted estimators, and applied these to the location corresponding to the largest test statistic that also exceeded a genome-wide significance criterion in the original sample. Both simulations and a genome-wide linkage analysis example with an Affected Sib Pair design showed great reduction in bias due to method application, with the bootstrap methods behaving best as judged by smaller MSE. However, for small effect sizes, the bootstrap estimates were still biased upward, which the authors attribute to a form of residual bias.

In the work of Wu et al. [7], additional support was provided for this bootstrap approach. Wu et al. [7] investigated the ability of the three estimators, with respect to reducing the selection bias for effect estimation in locus-specific quantitative trait linkage analysis. Similar to Sun and Bull [6], the methods were used for effect estimation at the location of the maximum test statistic. The results obtained demonstrated the capacity of the estimators to reduce the selection bias of the naïve approach, with their relative effectiveness depending on the size of the true effect and the power to discover it.

In contrast to the above uses of the bootstrap, Jeffries [8] discusses its suitability for the alleviation of bias in multiple comparison settings in general. Jeffries [8] ignores the role that a significance threshold may have in this bias, postulating that for these large observed effect sizes, overestimation is due to the existence of a large random component. In this application of bootstrap techniques to individual-level data, an empirical distribution of overestimation is generated, and utilized to construct alternative confidence intervals. Although the description of the suggested approach is given with respect to differences in gene expression, its potential application to genome-wide scans is recognised. In fact, it is noted that its use is not limited to the highest ranked location, in terms of test statistic, but can also be extended to other highly ranked locations, attenuating the bias described for those effect sizes.

As the popularity of GWASs began to grow, Garner [9] recognised the need to investigate Winner’s Curse bias in the context of these association studies. Similar to earlier research, Garner [9] stressed that the lack of success in replicating findings was due to overestimation of the most significant associations, as these estimates were used to elect suitable sample sizes for follow-up studies. Inflation of association estimates is attributed to the selection over many statistical tests and the fact that most studies lack adequate power. Garner [9] attests that increasing sample sizes will result in considerable bias reduction. In addition, in this paper, the asymptotic sampling distribution for the estimated effect size of a SNP after selection using a significance threshold, which is a left-truncated normal distribution, is derived.

A related development was seen in the emergence of Winner’s Curse correction methods based on conditional likelihood, such as that of Zöllner and Pritchard [10]. Focussing on case-control studies, Zöllner and Pritchard [10] detail a two-stage computational algorithm which determines the approximate maximum likelihood estimates (MLEs) of the population frequencies of all genotypes for a SNP of interest and corresponding penetrance parameters, conditional on the detection of a significant association signal and known population prevalence of the disease. They demonstrate large reductions in the bias of the parameter estimates, even with a low powered study, although a small degree of downward bias was still observed.

Even though we have chosen the approaches described in Ghosh et al. [11] to represent methods based on conditional likelihood with respect to method evaluation in this manuscript, as discussed below, other notable contributions to the field which alleviate Winner’s Curse using conditional likelihood must be mentioned. These include the work of Zhong and Prentice [12], and Xiao and Boehnke [13]. Firstly, Zhong and Prentice [12] focus on a 2-stage GWAS design, in which many SNPs are genotyped in a portion of samples in stage 1, while in stage 2, selected SNPs are genotyped in the remaining samples. For each significant SNP, the 2-stage design can yield a combined estimated effect size, namely the inverse variance weighted log OR. In this context, they propose several bias-reduced estimators, using the asymptotic approximation to the conditional probability density of the estimated log OR given the standardized estimated log OR is greater than a specified cut-point, to derive effect size estimates for selected SNPs. Their best performing estimator, the weighted MSE adjusted estimator, demonstrated nearly unbiased performance, except in very low power settings. However, it was also in these low to moderately powered situations in which all of proposed estimators tended to have greater variance compared with the naïve estimator. In addition, Zhong and Prentice [12] demonstrated that any bias present in the standard error of the estimated effect size of a significant SNP is essentially negligible in comparison to that in the corresponding estimated effect size. The method proposed by Xiao & Boehnke [13] can be considered a simpler version of that of Zöllner and Pritchard [10], which only considers the allele frequency difference and odds ratio parameters. They propose an “ascertainment-corrected” maximum likelihood approach that conditions on observing evidence for association. The conditional likelihood is maximised, as a function of the risk allele frequency in controls and risk allele difference, using the Nelder-Mead simplex technique, to obtain corrected MLEs for both allele frequency difference and odds ratio. With respect to log OR, the corrected MLE typically shows a reduction in both MSE and absolute bias, unless the effect size is large with high power.

On a genome-wide scale, bias correction methods that require only summary data, i.e. regression coefficients and standard errors, have notable advantages compared to methods that require individual-level genotyped data. First, methods based on summary data tend to be more computationally efficient, in terms of run time and memory efficiency. Furthermore, GWAS summary statistics are much more accessible and are more widely used in epidemiological techniques such as MR. Therefore, in this manuscript, we evaluate three more recently proposed methods which require only summary data, as described by Ghosh et al. [11], Ferguson et al. [14] and Bigdeli et al. [15]. These methods are introduced briefly below, and described in technical detail in the ‘Materials and methods’ section.

In a similar manner to the previously discussed condition likelihood methods, Ghosh et al. [11] use an approximate conditional likelihood to provide improved association estimates for SNPs with p-values less than the specified genome-wide significance threshold. The problem at hand is reframed as simply the estimation of a single parameter, i.e. the standardized effect size, for a truncated normal distribution. Interestingly, the authors admit that “the conditional MLE has no special optimality properties” in this context. However, their three proposed estimators are shown to greatly improve bias and MSE in the range of small to moderate effect size. Although the method is described in relation to a logistic regression and disease study framework, it can also be used for analyses concerning quantitative traits.

In contrast to the practice above in which correction is performed to each SNP separately, independently of estimated associations of other SNPs, alternative methods have been suggested which involve the use of all SNPs, including those which do not pass the threshold, in order to produce bias-reduced estimated effect sizes. We note that it is only through this joint consideration of all effect sizes that ranking bias can be properly addressed. Thus, the conditional likelihood-based techniques are limited in the sense that they just have the ability to account for the effect of selection bias alone. The empirical Bayes method described by Ferguson et al. [14] uses the collective distribution of all effect sizes to determine a suitable correction for each SNP. The method is shown to be squared error optimal if the true density of observed standardized effect sizes over differing SNPs is known. However, in practice, empirical Bayes needs to estimate the tails of this distribution and as a result, it is possible that rival correction methods can outperform empirical Bayes for larger standardized effects. That said, the empirical Bayes demonstrated superiority in terms of obtaining more accurate association estimates for SNPs with lower effect sizes. Ferguson et al. [14] also note that an additional attractive characteristic of the method is that the correction applied to each SNP is independent of the chosen significance threshold.

Bigdeli et al. [15] suggested the use of False Discovery Rate (FDR) Inverse Quantile Transformation (FIQT) to deal with Winner’s Curse bias. Identifying the similarity between correcting for Winner’s Curse bias on the scale of standardized effect sizes and adjusting p-values for multiple testing, they propose a two-step method. The approach first uses FDR for multiple testing adjustment of SNP p-values and then, obtains corrected standardized effect size estimates by means of the Gaussian quantiles corresponding to the adjusted p-values. When a sufficient number of signals are present and sample sizes are relatively large, they demonstrate that FIQT has superior MSE compared with competing approaches over a range of simulated scenarios. This performance is credited to the lower variance of FIQT in comparison with other methods.

Returning to the discussion of the evolution of the bootstrap method, Faye et al. [16] considered the extension of the approach beyond application to the most significant locus in a genome-wide linkage setting, as detailed by Sun and Bull [6], and propose a multi-locus bootstrap estimator for use in large-scale GWAS. By implementing identical selection criteria as those imposed on the original data set in each bootstrap sample, the developed method explicitly deals with both ranking and selection bias. Faye et al. [16] also incorporate two modifications in their procedure that aim to reduce overcorrection, especially under low power circumstances. The first takes into account variance differences associated with minor allele frequency while the second removes the negative correlation between the bootstrap and out-of-bootstrap estimates used in determining the bias reduction factor. Even though it was observed that the bootstrap had greater downward bias in comparison to the estimator of Ghosh et al. [11], it consistently produced smaller RMSE values. As this described approach of Faye et al. [16] requires individual-level data, we have chosen to continue the evolvement of the bootstrap in this manuscript by proposing and evaluating an alternative form of the method, which makes corrections using summary statistics.

The concentration of our work detailed here is on techniques which have been designed for use when a replication sample is unavailable. However, we note that there exist methods which use both a discovery and a replication GWAS in order to make suitable corrections to estimated effect sizes of significant SNPs. Examples include the uniformly minimum variance conditionally unbiased estimator (UMVCUE) of Bowden and Dudbridge [17], as well as the conditional likelihood approach of Zhong and Prentice [12], discussed above.

As mentioned above, we have made amendments to existing Winner’s Curse correction methods to address certain weaknesses. In particular, we investigated modifications that could be made to the empirical Bayes method in order to ensure that it makes better adjustments to association estimates. Following this review of correction methods, a rigorous evaluation and comparison of these methods was performed. This assessment took place by means of a simulation study as well as engagement with three real data sets. Simulations allowed us to compare methods easily over a wide range of different possible genetic architectures. We then used UK Biobank (UKBB) body mass index (BMI), type 2 diabetes (T2D) and height data sets to see how these techniques would perform in more realistic settings in which a large degree of linkage disequilibrium (LD) exists. In both instances, assessment of methods was predominantly based on the computation of estimated mean squared error (MSE) over significant SNPs. A notable challenge that was encountered at the start of the work discussed in this paper was the lack of available software to implement these various correction methods. Therefore, to complement this review, we have developed an R package, namely ‘winnerscurse’ (https://github.com/amandaforde/winnerscurse), which can be used to apply a number of Winner’s Curse adjustment methods to GWAS summary statistics. Techniques which require a replication GWAS are also included in this package.

Materials and methods

Throughout this paper, we let and with the assumption that (1) asymptotically, in which β_i denotes the true effect size of SNP i for i = 1, …, N, its estimated effect size, with respect to a trait of interest, and its estimated standard error. N represents the total number of SNPs in the discovery GWAS. Depending on the type of phenotype, be it a disease or a quantitative trait, this estimated effect size can represent a log odds ratio or a regression coefficient attained from a linear regression, respectively.

Results

Simulation study

When is winner’s curse bias most prominent?

A simulation study in which a simple correlation structure was imposed on the set of N = 1,000,000 SNPs was first executed, as described in ‘Materials and methods’. Before application of the Winner’s Curse correction methods to the sets of summary statistics, an attempt to gain an insight into the simulation scenarios in which Winner’s Curse bias is most prominent was made. This was done by computing the average number of significant SNPs, the average naïve MSE of significant SNPs and the average proportion of significant SNPs that had significantly overestimated effect sizes in each setting with respect to two significance thresholds, 5 × 10⁻⁸ and 5 × 10⁻⁴. A SNP is defined as being significantly overestimated or as having a significantly more extreme effect size estimate, at the 2.5% significance level, if it satisfies the condition: (15)

Thus, the proportion of significant SNPs that are significantly overestimated is considered to be representative of the proportion of significant SNPs with effect size estimates that greatly suffer from Winner’s Curse bias. As detailed in S1 Table, it was clear that as sample size was increased from 30,000 to 300,000, this proportion of significant SNPs decreased. The other two parameters which played key roles in defining the various simulated genetic architectures were heritability and polygenicity. It was observed that the proportion of significantly overestimated significant SNPs decreased when heritability was increased from 0.3 to 0.8, but when the value representing trait polygenicity was increased from 0.001 to 0.01, this proportion increased.

In fact, it was also noted that as the number of significant SNPs increased, both the MSE of significant SNPs and the proportion of these that were significantly overestimated decreased. This can be clearly seen in S1 Fig. This indicates that as the number of samples in a study and as the number of SNPs passing the significance threshold increases, bias induced by Winner’s Curse will be less of an issue among significant SNPs. In terms of genetic architecture characteristics, these results suggest that the presence of Winner’s Curse bias in the estimated effect sizes of significant SNPs should be of a greater concern when investigating traits with lower heritability or traits which have a larger proportion of effect SNPs. Of course, these factors mentioned here all directly pertain to the power of genome-wide significance of true effects, which is likely the most important one variable summary of the degree of Winner’s Curse, as shown in much of the existing literature. The extent of Winner’s Curse bias will partially depend on the true effect size distribution. This true effect size distribution is dependent on a combination of factors such as polygenicity, heritability and allele frequency distribution, with all of these features relating to power and in turn, determining the number of SNPs passing the significance threshold. In addition, increasing the sample size will result in increased power and less Winner’s Curse bias. However, differing combinations of polygenicity, heritability, allele frequency distribution and sample size can generate the same distribution of power, and thus it is worth taking into account how these other factors jointly affect the degree of Winner’s Curse bias.

Evaluation of performance at p < 5 × 10⁻⁸.

With respect to the evaluation of methods, we focus on the results of computing the quantity ‘change in RMSE over all significant SNPs due to method implementation’ for each method, with obtaining a negative value being desirable. These results are provided in S2 and S6 Tables. At a threshold of 5 × 10⁻⁸, several observations were notable. Firstly, for scenarios in which sample size has been designated the greater value of 300,000, the effect of applying the methods is on a much smaller scale to those scenarios with sample sizes of 30,000. This is evident from the large difference in the values on the y-axis between plots (A) and (B) of Fig 1. This observation ties in with the fact that the magnitude of Winner’s Curse bias is greater at n = 30,000. At this 5 × 10⁻⁸ significance threshold, the conditional likelihood methods are seen to perform poorly, especially when sample sizes are increased to 300,000. In most instances, these methods provide worse association estimates than the naïve approach, often increasing the RMSE. In addition to an increase in variance, the reason for this observation is the overcorrection of estimated effect sizes, especially those that lie close to the significance threshold. See the supplementary material for further discussion.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Estimated change in RMSE of significant SNPs at threshold 5 × 10⁻⁸ for each method and simulation setting, with a simple correlation structure imposed on the set of SNPs.

RMSE is evaluated for 100 simulated sets of summary statistics for each setting. (A) contains results for sample sizes of 30,000 while (B) concerns sample sizes of 300,000. Heritability is denoted by ‘h2’ while ‘prop_effect’ represents the proportion of effect SNPs. Methods are abbreviated as: ‘CL1’,’CL2’ and ‘CL3’ = conditional likelihood estimators of Ghosh et al. [11], ‘EB’ = original empirical Bayes method with tail restriction, ‘EB_df’ = empirical Bayes with fixed degrees of freedom of 7, ‘EB_gam_po’ = empirical Bayes with smoothing splines assuming poisson count distribution, ‘EB_gam_nb’ = empirical Bayes with smoothing splines assuming negative binomial count distribution, ‘EB_scam’ = empirical Bayes with shape constrained additive models, ‘boot’ = proposed bootstrap approach and ‘FIQT’ = FDR Inverse Quantile Transformation. The solid black lines represent no change in RMSE.

https://doi.org/10.1371/journal.pgen.1010546.g001

The novel bootstrap method is one of the most consistent methods at reducing the RMSE of association estimates for significant SNPs at this threshold. In all situations depicted, it has one of the largest negative values for the change in RMSE of significant SNPs and on average, improves the RMSE by 15% across the 8 settings, as shown in S3 Table. FIQT tends to perform in a somewhat similar manner to this bootstrap method. With respect to the empirical Bayes method and its variations, for sample sizes of 300,000, the best performing versions were the original empirical Bayes method (‘EB’), ‘EB-gam-nb’ and ‘EB-scam’. With the lower sample size, all variations perform very similarly. However, the empirical Bayes variants ‘EB-df’ and ‘EB-gam-po’ performed less well overall. In the case of ‘EB-gam-po’, this might be partly due to convergence problems that sometimes occurred in obtaining the poisson regression fit. In general, we advise caution in utilizing the empirical Bayes results in the context of convergence warnings from R.

In terms of the average bias over all significant SNPs at this 5 × 10⁻⁸ threshold, the results obtained may be viewed in S3 and S4 Figs, as well as S4 and S5 Tables. As expected, due to the symmetric nature of the simulated effect size distributions, the values computed for the average bias of positive significant SNPs are essentially mirrored by the negative significant SNPs. The plots produced for sample sizes of 30,000 and proportion of effect SNPs of 0.01 provide support for the use of the conditional likelihood methods. However, their tendency to overcorrect the estimated effect sizes is evident as negative values for the average bias of positive SNPs are witnessed regularly. Similar to above, it is deduced that the most consistent performances at producing average bias values close to zero are demonstrated by the empirical Bayes versions ‘EB’, ‘EB-gam-nb’, ‘EB-scam’, the bootstrap method and FIQT.

Evaluation of performance at p < 5 × 10⁻⁴.

A lower threshold of 5 × 10⁻⁴ was also investigated. Less emphasis is placed on the results obtained at this threshold as it is possible that many false positives are detected here, i.e. many SNPs that in fact have a true effect size of zero pass the significance threshold, although lower thresholds may be useful for the construction of polygenic risk scores. Therefore, as these Winner’s Curse correction methods are all considered to be shrinkage methods, improvements in the RMSE over all significant SNPs would be expected. However, as can be seen in S5 Fig, positive values are witnessed at this threshold when sample sizes are large. However, these positive values most often occur for the conditional likelihood methods which seem to be the worst performers overall. It seems here that the most consistent and best performing methods are the bootstrap method, the original empirical Bayes method and the empirical Bayes method which uses shape constrained additive models (SCAMs). These methods all reduce the RMSE of significant SNPs at this threshold by an average of at least 18.7%, as shown in S7 Table.

Additional simulations in absence of linkage disequilibrium.

In order to demonstrate potential performance of the Winner’s Curse methods in the context of SNP-trait associations from genome-wide arrays with lower SNP density or LD-pruned datasets, we also examined the less complex situation in which SNPs are independent. The results of these extra simulations are shown in S8–S15 Figs, and described in depth in the supplementary material. In this setting, with a normal effect size distribution, the most consistent methods in terms of reducing the RMSE of significant SNPs were the original empirical Bayes method and ‘EB-gam-nb’. Just as was observed in the simulations with linkage disequilibrium, the conditional likelihood methods perform poorly and often result in an increase in the evaluation metric in comparison to the naïve approach, while the proposed bootstrap method continued to exhibit competitive performance.

Empirical analysis

The results of an initial exploration of the six UK Biobank sets of summary statistics are detailed in Table 1. From trait to trait, there is a large difference in the number of SNPs with p-values lower than the genome-wide significance threshold of 5 × 10⁻⁸. Values for the proportion of these SNPs with significantly overestimated effect sizes in each discovery GWAS are included. A comparison of BMI and height GWASs at the 5 × 10⁻⁸ threshold tends to indicate that as the number of significant SNPs increases, the proportion that are significantly overestimated decreases. This trend is even more apparent at the larger threshold of 5 × 10⁻⁴, and as stated above, was also clearly observed in the simulated data.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The number of significant SNPs at two significance thresholds, 5 × 10⁻⁸ and 5 × 10⁻⁴, with proportions that indicate the extent of Winner’s Curse bias for each data set.

https://doi.org/10.1371/journal.pgen.1010546.t001

The problem of Linkage Disequilibrium in real data.

Naturally, the results of engagement with real data sets are more complex than those of the simulation study. For example, it was noted that for BMI, in one instance, all significant SNPs which had a z-value greater than 15 in the discovery GWAS had association estimates in the replication GWAS which were in fact greater. This observation can be clearly seen in S16 Fig in which z-statistics are plotted against estimated bias for each data set, with estimated bias of SNP i defined as: (16)

This finding is of course contrary to what is expected. However, these SNPs with z-values greater than 15 were all in strong linkage disequilibrium and thus, represented a single independent signal. It can be seen in Table 1 that a similar result was noted when the first T2D data set was used as the discovery GWAS. When using a significance threshold of 5 × 10⁻⁸, most of the 31 significant SNPs had larger estimated effect sizes in the replication GWAS than in the discovery GWAS. In these cases, we need to be careful not to over-generalize or interpret the results of applying a Winner’s Curse correction, given that there may be very few independent association signals at p < 5 × 10⁻⁸.

Evaluation of performance at p < 5 × 10⁻⁸.

As stated in ‘Materials and methods’, the methods were evaluated using the estimated MSE of SNPs which passed the chosen significance threshold. Using the threshold of 5 × 10⁻⁸, the estimated MSE for each method and GWAS combination are displayed in Table 2 while Fig 2 provides a corresponding illustration of these values. In this figure, the light blue bar as well as the black dotted horizontal line mark the estimated MSE obtained using the naïve approach, i.e. when no Winner’s Curse correction method has been applied and the raw effect estimates are used. This provides a standard to which the performance of each method can be directly compared with, in which it is desired that method application will result in an estimated MSE less than this approach. Similar to the section above describing the results of the simulation study, the poor performance of the conditional likelihood methods is evident. In 5 out of the 6 independent instances, it was observed that at least one of these methods had a greater estimated MSE than that of the naïve approach.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Estimated MSE of significant SNPs at threshold 5 × 10⁻⁸ for each method and data set.

Methods are abbreviated as: ‘naïve’ = no correction method applied, ‘CL1’,’CL2’ and ‘CL3’ = conditional likelihood estimators of Ghosh et al. (11), ‘EB’ = original empirical Bayes method with tail restriction, ‘EB_df’ = empirical Bayes with fixed degrees of freedom of 7, ‘EB_gam_po’ = empirical Bayes with smoothing splines assuming poisson count distribution, ‘EB_gam_nb’ = empirical Bayes with smoothing splines assuming negative binomial count distribution, ‘EB_scam’ = empirical Bayes with shape constrained additive models, ‘boot’ = proposed bootstrap approach, and ‘FIQT’ = FDR Inverse Quantile Transformation. The darker green bar and dashed black line represent the estimated MSE of the naïve approach.

https://doi.org/10.1371/journal.pgen.1010546.g002

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Estimated MSE of significant SNPs at threshold 5 × 10⁻⁸ for each method and data set.

https://doi.org/10.1371/journal.pgen.1010546.t002

On real data, the original empirical Bayes method proposed by Ferguson et al. [14] performed poorly, sometimes failing to adjust estimated associations downwards. This observation motivated the proposal of possible modifications as mentioned in ‘Materials and methods’. These suggested improvements, in particular the inclusion of the shape-constrained additive models and the use of the generalized additive model, resulted in slightly more consistent reductions in MSE over significant SNPs. In fact, taking all six data sets into account, it is ‘EB-scam’ and ‘EB-gam-po’, which tend to be the best performing methods, having an average improvement on estimated MSE of greater than 29.4% over the naïve approach.

However, as stated previously, we must be cautious when using the first T2D data set to evaluate methods, and also when using the second T2D data set. The problem with linkage disequilibrium and very few independent signals is common to both data sets. In Fig 2 for the first T2D data set, it is witnessed that all methods result in greater estimated MSE values than the naïve approach, with the original empirical Bayes method, bootstrap and FIQT clearly greatly shrinking the estimated effect sizes of significant SNPs away from those larger replication effect sizes. Therefore, if we exclude these two T2D data sets and re-compute the average improvement in estimated MSE for each method, it is our proposed bootstrap method which is seen to be the dominant method with an average improvement of approximately 40.2%.

Given the issues regarding method performance that occur in the presence of linkage disequilibrium, we also acquired a pruned set of SNPs and investigated the behaviour of the correction methods in that setting. The results obtained are provided in S15 and S16 Tables, and illustrated in S21 and S22 Figs, with further details included in the supplementary material. Comparing these results with those attained for the original set of SNPs, a large degree of similarity is evident. The single most notable difference is seen at the 5 × 10⁻⁸ threshold for the T2D datasets, in which most methods are now producing a lower estimated MSE than that of the naïve approach. However, it must be noted that these calculations are based on only two and four significant SNPs for the first and second T2D data sets, respectively. At this 5 × 10⁻⁸ threshold, using the pruned set of SNPs, ‘EB’, ‘EB-gam-nb’ and FIQT were the most reliable correction methods, producing an average improvement in estimated MSE of greater than 50% across the six data sets.

Applying the methods to the T2D data sets highlighted the importance of having a sufficient number of independent signals in the set of significant SNPs, in order to ensure appropriate method performance. This observation motivated a brief empirical investigation, which aimed to establish how many independent signals are required so that these methods perform better than the naïve approach of using no correction. Our work, detailed in the supplementary material, suggests that one would need at least 30 independent significant signals to guarantee confidence in the application of the empirical Bayes methods, the bootstrap method or FIQT.

In addition to the above, we also considered the average bias over all positive and negative significant SNPs at the 5 × 10⁻⁸ threshold, as shown in S17 and S18 Figs. We note that both T2D data sets had zero significant SNPs with negative estimated effect sizes at this threshold. The results obtained here for the height and BMI data sets further emphasize the overcorrection tendency of the conditional likelihood approaches, as demonstrated by large positive values for the average bias of negative SNPs. In most instances, the proposed bootstrap method, ‘EB’ and ‘EB-scam’ yield small values for the average bias closer to zero than that of the naïve approach.

Evaluation of performance at p < 5 × 10⁻⁴.

This evaluation procedure was repeated using a larger significance threshold of 5 × 10⁻⁴. The results of which can be found summarised in S12 Table and Fig 3. At this threshold, for all 6 data sets, all of the methods produce estimated MSE values less than the naïve approach. Each version of the empirical Bayes method along with the bootstrap and FIQT lead to an average improvement in estimated MSE of between 65 and 70% with the implementation of the empirical Bayes algorithm which incorporates shape constrained additive models (SCAMs) having the greatest average improvement of just over 70%.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Estimated MSE of significant SNPs at threshold 5 × 10⁻⁴ for each method and data set.

Methods are abbreviated as: ‘naïve’ = no correction method applied, ‘CL1’,’CL2’ and ‘CL3’ = conditional likelihood estimators of Ghosh et al. [11], ‘EB’ = original empirical Bayes method with tail restriction, ‘EB_df’ = empirical Bayes with fixed degrees of freedom of 7, ‘EB_gam_po’ = empirical Bayes with smoothing splines assuming poisson count distribution, ‘EB_gam_nb’ = empirical Bayes with smoothing splines assuming negative binomial count distribution, ‘EB_scam’ = empirical Bayes with shape constrained additive models, ‘boot’ = proposed bootstrap approach, and ‘FIQT’ = FDR Inverse Quantile Transformation. The darker green bar and dashed black line represent the estimated MSE of the naïve approach.

https://doi.org/10.1371/journal.pgen.1010546.g003

Discussion

In this article, we investigated the problem of Winner’s Curse bias which results in the estimated effect sizes of significant SNPs often being greater than their true values. Our work concentrated on methods that could be used to reduce this bias in settings in which only summary statistics of the GWAS that discovered these SNP-trait associations were available. We chose to focus on this particular situation as Winner’s Curse correction methods which only require GWAS summary data tend to be very computational efficient and furthermore, this summary data is often much easier to access than individual-level data.

We performed a thorough evaluation and comparison of these methods using both simulated and real data sets. Our simulation study considered a wide range of genetic architectures including data sets in which a simple correlation structure had been imposed on the set of SNPs as well as data sets of independent SNPs. In addition, three UKBB real data sets were used for method evaluation purposes. As well as assessing currently published correction methods, we also explored several possible modifications that could be made in order to improve these methods. In particular, we looked at a number of variations of the empirical Bayes method and proposed an additional approach which uses the parametric bootstrap in order to establish suitable corrections for the estimated effect size of each SNP. The estimated mean squared error (MSE) was chosen as an appropriate metric in order to compare the methods. Due to the notable lack of software for implementation of Winner’s Curse correction methods, we developed an R package, ‘winnerscurse’, as an accompaniment to the work described in this paper. This allows users to apply all methods discussed here, as well as the proposed modifications, to their sets of GWAS summary data.

As a first step in both our simulation study and engagement with real data, we computed the proportion of significant SNPs that were significantly overestimated and observed the common trend that as the number of SNPs passing the significance threshold increased, the proportion of those that were significantly overestimated decreased. This aligns with the postulation that as sample sizes increase, Winner’s Curse bias becomes less of a concern although it still exists. However, caution must be taken when working with real data sets, especially those of binary traits, in which a very small number of SNPs have been deemed significant at a certain threshold. In this instance, it may be that the significant SNPs are representative of only one or two independent signals. For example, in our first T2D data set, while using a threshold of 5 × 10⁻⁸, we witnessed 93.5% of significant SNPs having greater replication effect size estimates than those obtained in the discovery GWAS. We note that our brief empirical investigation, as detailed in the supplementary material, suggested that at least 30 independent significant signals are required in order to ensure appropriate performance of the Winner’s Curse correction methods. Fortunately, as sample sizes increase in the future and different diseases have greater numbers of true signals captured by their respective sets of significant SNPs, this issue will only be seen to present itself in rare circumstances.

With respect to method performance, it was clear that the conditional likelihood methods performed poorly as in most instances, especially for p < 5 × 10⁻⁸, these methods resulted in greater values for the estimated MSE among significant SNPs than the naïve approach. The other considered methods behaved much more similarly to the extent that we cannot state that there is a clear advantage of one method over another. Thus, the choice of which method a user should apply to their set of GWAS summary statistics in order to correct for Winner’s Curse is dependent on personal preference. However, it is advised that when doing so, the possible limitations of the chosen method are understood well. Notably, the empirical Bayes methods have a clear theoretical advantage, that is, when the true effect size distribution is known, empirical Bayes is fully Bayesian and minimizes the average MSE of the standardized estimated effects as estimates of the true standardized effects over all competing estimators. Unfortunately, performance of empirical Bayes estimators can be restricted in practice due to inaccurate estimation of the extreme tails of the z-statistic distribution. This estimation difficulty is particularly problematic when the existence of strong linkage disequilibrium results in clusters of associations in the tails. These clusters can be falsely detected as local modes in the distribution by automatic fitting algorithms. Some progress on improving estimation in the tails has been made here with the proposal of modifications that employ generalized additive models or shape constrained additive models. However, these adaptations have not resulted in large enough improvements in order to claim objective superiority of the empirical Bayes methods over other approaches such as the bootstrap method or FIQT. In a setting in which the distribution of effect sizes is asymmetric, methods like the empirical Bayes and bootstrap, where the correction rule is not a function of absolute value z-statistics, possess the potential to perform better than FIQT and conditional likelihood methods. In spite of this fact, no tangible evidence of improved performance over FIQT was observed on the real data sets that we examined.

With both our set of simulations and real data analysis, we have aimed to be as comprehensive as possible as it is possible that differing method performance results may occur under differing genetic architectures, but this is an obviously difficult task. Informing these simulations appropriately is particularly challenging, especially when attempting to define the true effect size distribution. However, under the assumption of independent SNPs, we also investigated scenarios which had a bimodal or skewed distribution of effect sizes, as described in the supplementary material. Furthermore, for simulations involving correlated SNPs, we have assumed a very simplistic linkage disequilibrium (LD) structure in which the minor allele frequencies have been simulated independently of this LD structure. In contrast, the use of real data permitted the analysis of method performance in a realistic setting where a large degree of LD exists. However, this was limited to only three UKBB data sets. In the case of the binary trait T2D, it must be noted that due to the very small number of significant SNPs at p < 5 × 10⁻⁸, the results are deemed rather questionable here.

Due to space considerations, Winner’s Curse correction methods which require both a discovery and replication GWAS in order to make suitable adjustments to estimated effect sizes have not been examined in this manuscript, even though several of these methods have been implemented in our developed R package, ‘winnerscurse’. Furthermore, computation of standard errors of the adjusted estimated effect sizes have not been considered here. However, for methods such as the empirical Bayes, bootstrap and FIQT, the R package, ‘winnerscurse’, utilizes the parametric bootstrap in order to obtain these standard errors. This package can also be used to provide confidence intervals for estimated effect sizes which have been corrected for Winner’s Curse using the conditional likelihood methods. In two-sample Mendelian randomization, it is known that as this Winner’s Curse bias can be present in the estimated SNP-exposure associations, the causal estimate will then suffer from bias. Thus, the Winner’s Curse correction methods explored in this paper can also be potentially used as plug-in corrections for two-sample MR. In addition, these methods could prove beneficial in the computation of polygenic risk scores, in order to reduce the effect of Winner’s Curse bias there.

Supporting information