Reader Comments

Post a new comment on this article

Does butter matter?

Posted by Jesper_M_Kivelä on 13 Jul 2016 at 13:09 GMT

Pimpin et al showed in their meta-analysis that butter consumption 14 grams per day was associated with an average relative risk (RR) 1.0134 (95% CI 1.0003 to 1.0266; P=0.045) for total mortality (1). The authors concluded in the abstract that relatively small association was observed.

This raises an interesting aspect of statistically significant small effect sizes observed within large studies or meta-analyses that might be trivial, impossible to differentiate from biases or even differ from a Bayesian approach.

I calculated Bayes factors (BF) based on this statistically significant association using study by Ioannidis as a groundwork (2). These BFs can also be calculated with the use of Excel-based calculator where link has provided elsewhere (3).
Briefly, BF was calculated with the use of lump-and-smear prior. Data supports null hypothesis (i.e. no association) relative to alternative hypothesis (i.e. association exists) when BF > 1 and more support is given by data for alternative hypothesis relative to null when BF < 1.

I extracted relevant data from Pimpin et al meta-analysis (1) based on the aforementioned association with the use of reported RR and 95% CIs. This approach yielded Z-value of 2.011 which corresponds 2-tailed P-value of 0.044 with the method detailed by Altman and Bland (4).
I used median RRs of 1.96 and 1.59 as priors based on a total of 272 observational studies and a subsample of 74 studies related to dietary factors, respectively (2). In addition, I used RRs of 1.20 and 1.04 based on meta-analyses of observational studies between single dietary factors and all-cause mortality (5,6).

According to my calculations, BFs were 16.9, 11.6, 4.6 and 1.03 when the average expected value of RR (natural logarithm of RR) 1.96 (0.673), 1.59 (0.464), 1.20 (0.182) and 1.04 (0.039), respectively, were assumed under the alternative hypothesis for harmful association between butter and all-cause mortality. In other words, the association became less credible (i.e BFs > 1) after the meta-analysis by Pimpin et al (1).

This is an intriguing example of Lindley's paradox where statistical inference based on Bayesian and frequentist approach differ. Bayes factor favours null relative to alternative despite the fact that null hypothesis is rejected based on a statistically significant result. This disparity depends on choice of priors. Lindley's paradox was also evident in both observational studies and meta-analyses of genetic-disease associations when more extreme priors were used (2).

In sensitivity analysis, I used RR 1.01 (1/0.99) as a prior under the alternative based on a recent meta-analysis between saturated fat intake and all-cause mortality (7). Bayes factor was 0.44. As evident, the BF favours the alternative relative to the null hypothesis. However, association between butter and all-cause mortality is still only around 2.3-fold more credible after the meta-analysis by Pimpin et al (1).

One can calculate posterior probability for association to be true positive (3). For example, with a prior probability of 20% (odds 0.25) one gets a posterior probability of 36% [0.25 / (0.25 + 0.44)] for the association between butter and all-cause mortality. One might argue that chances for a single dietary factor or even combination of few dietary factors to be associated with the all-cause mortality are low, and therefore the choice of skeptic prior probability might be persuasive.

Researchers did not express any limitations considering statistically significant tiny effects (RR 0.95 to 1.05) in 55% (28/51) of the cases (8). Lindley´s paradox was not mentioned when limitations of tiny effects were discussed (8).
This was also evident in Pimpin et al meta-analysis (1). However, the authors stated in the discussion that “...such residual confounding may overestimate potential harms of butter for mortality...” (1).

This re-analysis should be not taken as a face value. Subjective choices were made although the magnitude of priors were based on published literature. This re-analysis, however, emphasizes that statistics might matter more than butter by itself.


Jesper M Kivelä, MD PhD student (pediatrics)
Institute of Clinical Medicine, University of Helsinki
Helsinki, Finland



References


1. Pimpin L, Wu JH, Haskelberg H, Del Gobbo L, Mozaffarian D. Is Butter Back? A Systematic Review and Meta-Analysis of Butter Consumption and Risk of Cardiovascular Disease, Diabetes, and Total Mortality. PLoS One. 2016;11:e0158118. doi: 10.1371/journal.pone.0158118

2. Ioannidis JP. Effect of formal statistical significance on the credibility of observational associations. Am J Epidemiol. 2008;168:374-83.

3. Pereira TV, Ioannidis JP. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J Clin Epidemiol. 2011;64:1060-9.

4. Altman DG, Bland JM. How to obtain the P value from a confidence interval. BMJ. 2011;343:d2304.

5. Aune D, Keum N, Giovannucci E, Fadnes LT, Boffetta P, Greenwood DC, et al. Whole grain consumption and risk of cardiovascular disease, cancer, and all cause and cause specific mortality: systematic review and dose-response meta-analysis of prospective studies. BMJ. 2016;353:i2716.

6. O'Sullivan TA, Hafekost K, Mitrou F, Lawrence D. Food sources of saturated fat and the association with mortality: a meta-analysis. Am J Public Health. 2013;103:e31-42.

7. de Souza RJ, Mente A, Maroleanu A, Cozma AI, Ha V, Kishibe T, et al. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies. BMJ. 2015;351:h3978.

8. Siontis GC, Ioannidis JP. Risk factors and interventions with statistically significant tiny effects. Int J Epidemiol. 2011;40:1292-307.

Competing interests declared: I like statistics.