Skip to main content
Advertisement
  • Loading metrics

A Stronger Post-Publication Culture Is Needed for Better Science

  • Hilda Bastian

    hilda.bastian@nih.gov

    Affiliation Scientist and Editor, National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, Maryland, United States of America

A research report or idea needs to clamber over more than the hurdle of publication to move science, practice, or policy forward. It's not only a matter of authors waiting for kudos and citations to roll in. If their work is not to sink into oblivion, or be acted on when it shouldn't be, publication is just the beginning. Both improving research quality [1],[2] and reducing waste in science [3] require a stronger post-publication culture.

Early Enlightenment science was rooted in ongoing discussion among scientists. Scientific discourse in a small, widely scattered community was in person and via books and the “erudite letters” that were the precursor of journal articles [4]. The journal system, capturing fragments of research, enabled massive expansion and acceleration of scientific activity [4].

These days the system does not keep up well with the speed of activity and the volume of research from a vast community. Articles are, by and large, too uncorrectable and unconnected [5], and much significant intellectual effort is not captured at all. Substantive discussions in journal clubs, in email lists, in social media, and at conferences are not distilled into a concise, permanent, accessible record. Most of the unaddressed content of pre-publication peer review is also lost.

Post-publication evaluation is highly fragmented. It often appears within future articles, either embedded in the introduction and discussion sections, or in formal research syntheses. Dedicated review journals (and journal sections) select, summarize, and critique publications, usually in an “expert picks” way. There are also rigorously structured systems of post-publication evaluation inside and outside journals [6],[7].

There are more immediate channels to respond to published research, such as letters and comments to the editor, commentaries, and editorials in journals, and discussion in blogs. Dedicated websites have been developed for discussing and sharing research among authors [8], and PubMed Commons (for which I am editor) enables post-publication commenting and linkages by the PubMed authorship community and journal clubs [9].

Somewhere within this activity is the amorphous phenomenon that people call post-publication peer review. For some, post-publication peer review is simply shifting pre-publication peer review to after an article's release [10]. For others, it's any evaluation of an article that is similar to pre-publication peer review. Post-publication peer review overlaps with post-publication commenting, but does not encompass all of that activity.

Post-Publication Commenting

Many associate post-publication commentary with only the negative “yin” of criticism, correction, retraction, and failed replication. It is essential to prevent research-led error, harm, and futile studies. But there is a vital positive “yang” aspect, too, incorporating research aftercare [11]. Answers to questions may be critical for other studies, for adequate research assessment and synthesis, and for considering practice and policy implications [12]. Discussion can build, apply, connect, and update ideas and ongoing work.

For some, though, the success of post-publication commentary is concerned only with the “yin” of correction and retraction. For others, post-publication evaluation only “works” if it occurs for all articles, making pre-publication peer review redundant. From these perspectives, post-publication evaluation would always be shortchanged, and be seen to fall short. However, success includes rescuing important work from obscurity, and building work and capacity, not just tearing it down. Updating is at least as critical as correction to improving published research.

Furthermore, the scientific evidence base for effects of routine pre-publication peer review on article quality remains weak [13]. Pre-publication peer review can also worsen the quality of research, as when peer reviewers demand unplanned analyses of clinical trials [14]. With an oversaturation of publication in many areas, assessing it all only exacerbates the waste. Post-publication review faces the same problems.

Cultural Challenges to Post-Publication Activity

Many are wary or worse about post-publication culture. For some, any un-peer-reviewed response to peer-reviewed work is impertinent, and the Internet's removal of constraints to adding both substantive and trivial post-publication commentary to the public space is hard to accept. The Internet has also increased the quantity of incivility out in public view (Figure 1).

thumbnail
Figure 1. The melding of Internet culture and traditional communication.

(Wackaloon: Internet slang for a kook; believed to derive from wacky and loon [33].)

https://doi.org/10.1371/journal.pmed.1001772.g001

Disputes between scientists have always been common, and it has always been the case that “in the bitter social conflict that ensues, the standards governing behavior deteriorate” [15]. According to sociologist Robert Merton, the example of Edmond Halley calling another astronomer a “lazy and malicious thief” in the 17th century was, and remains, more commonplace than aberrant [15]. He saw these conflicts as arising from the same “deep devotion to the advancement of knowledge” that fuels the passion for engaging in intellectual labor. We need to study and improve the way we communicate and cope with our errors and criticisms of our work.

The fear of repercussions for junior scientists in particular is high. This fear lies at the heart of the contentious issue of anonymous post-publication commenting. Some argue, though, that the risks for young scientists of openly commenting on others' work do not necessarily outweigh the advantages of visibility and recognition [16].

Even if the cost is reticence about participating, I believe the balance tips towards the requirement for transparency. Readers need to be able to judge whether writers are commenting outside their areas of expertise. Concerned readers need to have a chance of recognizing writers who have conflicts of interest, or be able to investigate whether or not potential conflicts exist.

However, addressing the obstacle to scientific progress posed by social dominance and aggression is a critical cultural issue, and not only—or even necessarily predominantly—for young researchers. Stereotype threat (anticipating discrimination) and other social issues may deter women scientists and other groups from commenting, too. Social influences can make women less talkative and less assertive than men in mixed gender groups [17], especially where “participants' concerns for self-presentation are heightened” [18].

Women scientists seem to be underrepresented in science activities that make their reflections public. In some fields and countries at least, women may still publish less [19][21], present less at conferences [22],[23], and blog less [24],[25]. A small body of research since the 1990s has identified some disturbingly low rates of participation by women as peer reviewers [26][28], though double-blind peer review might increase women's participation [29].

During the first year of PubMed Commons, less than 20% of those commenting were women. Research on gender bias in research and editorial peer review has been somewhat reassuring [26],[27],[30]. But the subject of this research has been the effect on publication fairness. The effect of under-participation on the development of confidence with the core science career skill of formulating valuable and effective critique was not considered.

I don't think that anonymity is a good solution. We need to consider skill development in critiquing research [13],[31]. That may also be valuable for those who are not scientific peers, but have contributions to make [32]. Developing a much more encouraging communication climate about errors and weaknesses of scientific communication is critical. This situation reminds me of the imperative identified decades ago to create a safety and quality culture in hospitals. A mature culture of responsiveness to complaints and problem identification is as much a prerequisite for research quality improvement as it was in health care.

Rewards for substantive intellectual effort post-publication and for the aftercare of research publications and sharing of data would help. Formal recognition is also necessary to undo the perverse incentive for authors to keep important insights and additional data until a subsequent publication. Such delays can last for months, if not years.

Passive consumption of scientific papers, and the withholding of adequate information by authors, cannot advance science. Thinking and talking about our responses to research reports is still science's vibrant and compelling intellectual core. Capturing that post-publication intellectual effort more rigorously is essential for better science.

Acknowledgments

The views expressed are personal, and do not necessarily reflect those of the National Institutes of Health. My views and knowledge on post-publication evaluation have benefited greatly from ongoing discussions with David Lipman and Melissa Vaught.

Author Contributions

Wrote the first draft of the manuscript: HB. Wrote the paper: HB. ICMJE criteria for authorship read and met: HB. Agree with manuscript results and conclusions: HB.

References

  1. 1. Collins FS, Tabak LA (2014) Policy: NIH plans to enhance reproducibility. Nature 505: 612–613 Available: http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586 Accessed 30 October 2014.
  2. 2. Ioannidis JP (2014) How to make more published research true. PLoS Med 11: e1001747 Available: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001747 Accessed 30 October 2014.
  3. 3. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, et al. (2014) Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383: 267–276 Available: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62228-X/fulltext Accessed 30 October 2014.
  4. 4. Kronick DA (1984) Literature of the life sciences: the historical background. Bull NY Acad Med 60: 857–875 Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1911798/ Accessed 30 October 2014.
  5. 5. The PLOS Medicine, Editors Getting closer to a fully correctable and connected research literature. PLoS Med 10: e1001408 Available: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001408 Accessed 30 October 2014.
  6. 6. Haynes RB, Cotoi C, Holland J, Walters L, Wilczynski N, et al. (2006) Second-order peer review of the medical literature for practitioners. JAMA 295: 1801–1808 Available: http://jama.jamanetwork.com/article.aspx?articleid=202708 Accessed 30 October 2014.
  7. 7. NHS Centre for Reviews and Dissemination (2002) The Database of Abstracts of Reviews of Effects (DARE). Eff Matters 6: 1–4 Available: http://www.york.ac.uk/inst/crd/EM/em62.pdf Accessed 30 October 2014.
  8. 8. Van Noorden R (2014 Mar 14) The new dilemma of online peer review: too many places to post? Nature News Blog. Available: http://blogs.nature.com/news/2014/03/the-new-dilemma-of-online-peer-review-too-many-places-to-post.html. Accessed 30 October 2014.
  9. 9. National Center for Biotechnology Information (2014) PubMed Commons. Available: http://www.ncbi.nlm.nih.gov/pubmedcommons/. Accessed 30 October 2014.
  10. 10. Hunter J (2012) Post-publication peer review: opening up scientific conversation. Front Comput Neurosci 6: 63 Available: http://journal.frontiersin.org/Journal/10.3389/fncom.2012.00063/full Accessed 30 October 2014.
  11. 11. Rennie D (1998) Freedom and responsibility in medical publication: setting the balance right. JAMA 280: 300–302 Available: http://jama.jamanetwork.com/article.aspx?articleid=187765 Accessed 30 October 2014.
  12. 12. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, et al. (2014) Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 348: g1687 Available: http://www.bmj.com/content/348/bmj.g1687.long Accessed 30 October 2014.
  13. 13. Jefferson T, Rudin M, Brodney Folse S, Davidoff F (2007) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007: MR000016.
  14. 14. Hopewell S, Collins GS, Boutron I, Yu LM, Cook J, et al. (2014) Impact of peer review on reports of randomized trials published in open peer review journals: retrospective before and after study. BMJ 349: g4145 Available: http://www.bmj.com/content/349/bmj.g4145.long Accessed 30 October 2014.
  15. 15. Merton RK (1963) The ambivalence of scientists. Bull Johns Hopkins Hosp 112: 77–97.
  16. 16. Goetz A (2014) Reexamining reviewer anonymity—more costs than benefits. Open Science Collaboration. Available: http://osc.centerforopenscience.org/2014/10/22/reexamining-reviewer-anonymity/. Accessed 30 October 2014.
  17. 17. Leaper C, Robnett RD (2011) Women are more likely than men to use tentative language, aren't they? A meta-analysis testing for gender differences and moderators. Psychol Women Q 35: 129–142.
  18. 18. Leaper C, Ayres MM (2007) A meta-analytic review of gender variations in adults' language use: talkativeness, affiliative speech, and assertive speech. Pers Soc Psychol Rev 11: 328.
  19. 19. Jagsi R, Guancial EA, Worobey CC, Henault LE, Chang Y, et al. (2006) The “gender gap” in authorship of academic medical literature—a 35-year perspective. N Engl J Med 355: 281–287 Available: http://www.nejm.org/doi/full/10.1056/NEJMsa053910 Accessed 30 October 2014.
  20. 20. Kelly CD, Jennions MD (2006) The h index and career assessment by numbers. Trends Ecol Evol 21: 167–170.
  21. 21. Larivière V, Ni C, Gingras Y, Cronin B, Sugimoto CR (2013) Bibliometrics: global gender disparities in science. Nature 504: 211–213 Available: http://www.nature.com/news/bibliometrics-global-gender-disparities-in-science-1.14321 Accessed 30 October 2014.
  22. 22. Isbell LA, Young TP, Harcourt AH (2012) Stag parties linger: continued gender bias in a female-rich scientific discipline. PLoS ONE 7: e49682 Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0049682 Accessed 30 October 2014.
  23. 23. Jones TM, Fanson KV, Lanfear R, Symonds MR, Higgie M (2014) Gender differences in conference presentations: a consequence of self-selection? PeerJ 2: e627 Available: https://peerj.com/articles/627/ Accessed 30 October 2014.
  24. 24. Shema H, Bar-Ilan J, Thelwall M (2012) Research blogs and the discussion of scholarly information. PLoS ONE 7: e35869 Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0035869 Accessed 30 October 2014.
  25. 25. Maynard A (2014 Mar 24) Gender balance in science blog networks: how has the male/female science blogger ration changed over the past three and a half years? Medium. Available: https://medium.com/2020-science-comms/gender-balance-in-science-blog-networks-71a1efb79958. Accessed 30 October 2014.
  26. 26. Gilbert JR, Williams ES, Lundberg GD (1994) Is there gender bias in JAMA's peer review process? JAMA 272: 139–142.
  27. 27. Zuber MA (2001) [Underrepresentation of women among peer reviewers and textbook authors in medicine in Germany.] Med Klin (Munich). 96: 173–180.
  28. 28. Mutz R, Bornmann L, Daniel HD (2012) Does gender matter in grant peer review? Z Psychol 220: 121–129 Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3414231/ Accessed 30 October 2014.
  29. 29. Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R (2008) Double-blind review favours increased representation of female authors. Trends Ecol Evol 23: 4–6.
  30. 30. Grod ON, Budden AE, Tregenza, Koricheva J, Leimu R, et al. (2008) Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PLoS ONE 3: e3202 Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0003202 Accessed 30 October 2014.
  31. 31. Galipeau J, Moher D, Skidmore B, Campbell C, Hendry P, et al. (2013) Systematic review of the effectiveness of training programs in writing for scholarly publication, journal editing, and manuscript peer review (protocol). Syst Rev 2: 41 Available: http://www.systematicreviewsjournal.com/content/2/1/41 Accessed 30 October 2014.
  32. 32. Bastian H (2003) Non-peer review: consumer involvement in research review. In Godlee F, Jefferson T, editors. Peer review in health sciences, 2nd edition. London: BMJ Books. pp. 248–262.
  33. 33. Wiktionary (2014) Wackaloon. Available: https://en.wiktionary.org/w/index.php?title=wackaloon&oldid=21379171. Accessed 4 November 2014.