PLoSPLoS MedplosmedPLoS Medicine1549-1676Public Library of ScienceSan Francisco, USAPMEDICINE-D-14-0344910.1371/journal.pmed.1001772EditorialA Stronger Post-Publication Culture Is Needed for Better ScienceBastianHilda*Scientist and Editor, National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, Maryland, United States of America* E-mail: hilda.bastian@nih.gov
HB is a member of the Editorial Board of PLOS Medicine, and lead editor for PubMed Commons, a forum for scientific discourse.
Wrote the first draft of the manuscript: HB. Wrote the paper: HB. ICMJE criteria for authorship read and met: HB. Agree with manuscript results and conclusions: HB.
122014301220141112e10017722014This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Hilda Bastian considers post-publication commenting and the cultural changes that are needed to better capture this intellectual effort.
Please see later in the article for the Editors' Summary
HB has received funding from the Intramural Research Program of the National Center for Biotechnology Information at the National Library of Medicine, National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
A research report or idea needs to clamber over more than the hurdle of publication to move science, practice, or policy forward. It's not only a matter of authors waiting for kudos and citations to roll in. If their work is not to sink into oblivion, or be acted on when it shouldn't be, publication is just the beginning. Both improving research quality [1],[2] and reducing waste in science [3] require a stronger post-publication culture.
Early Enlightenment science was rooted in ongoing discussion among scientists. Scientific discourse in a small, widely scattered community was in person and via books and the “erudite letters” that were the precursor of journal articles [4]. The journal system, capturing fragments of research, enabled massive expansion and acceleration of scientific activity [4].
These days the system does not keep up well with the speed of activity and the volume of research from a vast community. Articles are, by and large, too uncorrectable and unconnected [5], and much significant intellectual effort is not captured at all. Substantive discussions in journal clubs, in email lists, in social media, and at conferences are not distilled into a concise, permanent, accessible record. Most of the unaddressed content of pre-publication peer review is also lost.
Post-publication evaluation is highly fragmented. It often appears within future articles, either embedded in the introduction and discussion sections, or in formal research syntheses. Dedicated review journals (and journal sections) select, summarize, and critique publications, usually in an “expert picks” way. There are also rigorously structured systems of post-publication evaluation inside and outside journals [6],[7].
There are more immediate channels to respond to published research, such as letters and comments to the editor, commentaries, and editorials in journals, and discussion in blogs. Dedicated websites have been developed for discussing and sharing research among authors [8], and PubMed Commons (for which I am editor) enables post-publication commenting and linkages by the PubMed authorship community and journal clubs [9].
Somewhere within this activity is the amorphous phenomenon that people call post-publication peer review. For some, post-publication peer review is simply shifting pre-publication peer review to after an article's release [10]. For others, it's any evaluation of an article that is similar to pre-publication peer review. Post-publication peer review overlaps with post-publication commenting, but does not encompass all of that activity.
Post-Publication Commenting
Many associate post-publication commentary with only the negative “yin” of criticism, correction, retraction, and failed replication. It is essential to prevent research-led error, harm, and futile studies. But there is a vital positive “yang” aspect, too, incorporating research aftercare [11]. Answers to questions may be critical for other studies, for adequate research assessment and synthesis, and for considering practice and policy implications [12]. Discussion can build, apply, connect, and update ideas and ongoing work.
For some, though, the success of post-publication commentary is concerned only with the “yin” of correction and retraction. For others, post-publication evaluation only “works” if it occurs for all articles, making pre-publication peer review redundant. From these perspectives, post-publication evaluation would always be shortchanged, and be seen to fall short. However, success includes rescuing important work from obscurity, and building work and capacity, not just tearing it down. Updating is at least as critical as correction to improving published research.
Furthermore, the scientific evidence base for effects of routine pre-publication peer review on article quality remains weak [13]. Pre-publication peer review can also worsen the quality of research, as when peer reviewers demand unplanned analyses of clinical trials [14]. With an oversaturation of publication in many areas, assessing it all only exacerbates the waste. Post-publication review faces the same problems.
Cultural Challenges to Post-Publication Activity
Many are wary or worse about post-publication culture. For some, any un-peer-reviewed response to peer-reviewed work is impertinent, and the Internet's removal of constraints to adding both substantive and trivial post-publication commentary to the public space is hard to accept. The Internet has also increased the quantity of incivility out in public view (Figure 1).
10.1371/journal.pmed.1001772.g001
The melding of Internet culture and traditional communication.
(Wackaloon: Internet slang for a kook; believed to derive from wacky and loon [33].)
Disputes between scientists have always been common, and it has always been the case that “in the bitter social conflict that ensues, the standards governing behavior deteriorate” [15]. According to sociologist Robert Merton, the example of Edmond Halley calling another astronomer a “lazy and malicious thief” in the 17th century was, and remains, more commonplace than aberrant [15]. He saw these conflicts as arising from the same “deep devotion to the advancement of knowledge” that fuels the passion for engaging in intellectual labor. We need to study and improve the way we communicate and cope with our errors and criticisms of our work.
The fear of repercussions for junior scientists in particular is high. This fear lies at the heart of the contentious issue of anonymous post-publication commenting. Some argue, though, that the risks for young scientists of openly commenting on others' work do not necessarily outweigh the advantages of visibility and recognition [16].
Even if the cost is reticence about participating, I believe the balance tips towards the requirement for transparency. Readers need to be able to judge whether writers are commenting outside their areas of expertise. Concerned readers need to have a chance of recognizing writers who have conflicts of interest, or be able to investigate whether or not potential conflicts exist.
However, addressing the obstacle to scientific progress posed by social dominance and aggression is a critical cultural issue, and not only—or even necessarily predominantly—for young researchers. Stereotype threat (anticipating discrimination) and other social issues may deter women scientists and other groups from commenting, too. Social influences can make women less talkative and less assertive than men in mixed gender groups [17], especially where “participants' concerns for self-presentation are heightened” [18].
Women scientists seem to be underrepresented in science activities that make their reflections public. In some fields and countries at least, women may still publish less [19]–[21], present less at conferences [22],[23], and blog less [24],[25]. A small body of research since the 1990s has identified some disturbingly low rates of participation by women as peer reviewers [26]–[28], though double-blind peer review might increase women's participation [29].
During the first year of PubMed Commons, less than 20% of those commenting were women. Research on gender bias in research and editorial peer review has been somewhat reassuring [26],[27],[30]. But the subject of this research has been the effect on publication fairness. The effect of under-participation on the development of confidence with the core science career skill of formulating valuable and effective critique was not considered.
I don't think that anonymity is a good solution. We need to consider skill development in critiquing research [13],[31]. That may also be valuable for those who are not scientific peers, but have contributions to make [32]. Developing a much more encouraging communication climate about errors and weaknesses of scientific communication is critical. This situation reminds me of the imperative identified decades ago to create a safety and quality culture in hospitals. A mature culture of responsiveness to complaints and problem identification is as much a prerequisite for research quality improvement as it was in health care.
Rewards for substantive intellectual effort post-publication and for the aftercare of research publications and sharing of data would help. Formal recognition is also necessary to undo the perverse incentive for authors to keep important insights and additional data until a subsequent publication. Such delays can last for months, if not years.
Passive consumption of scientific papers, and the withholding of adequate information by authors, cannot advance science. Thinking and talking about our responses to research reports is still science's vibrant and compelling intellectual core. Capturing that post-publication intellectual effort more rigorously is essential for better science.
The views expressed are personal, and do not necessarily reflect those of the National Institutes of Health. My views and knowledge on post-publication evaluation have benefited greatly from ongoing discussions with David Lipman and Melissa Vaught.
ReferencesCollinsFS, TabakLA (2014) Policy: NIH plans to enhance reproducibility. 505: 612–613Available: http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586 Accessed 30 October 2014IoannidisJP (2014) How to make more published research true. 11: e1001747Available: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001747 Accessed 30 October 2014GlasziouP, AltmanDG, BossuytP, BoutronI, ClarkeM, et al. (2014) Reducing waste from incomplete or unusable reports of biomedical research. 383: 267–276Available: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62228-X/fulltext Accessed 30 October 2014KronickDA (1984) Literature of the life sciences: the historical background. 60: 857–875Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1911798/ Accessed 30 October 2014The PLOSMedicine, Editors Getting closer to a fully correctable and connected research literature. 10: e1001408Available: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001408 Accessed 30 October 2014HaynesRB, CotoiC, HollandJ, WaltersL, WilczynskiN, et al. (2006) Second-order peer review of the medical literature for practitioners. 295: 1801–1808Available: http://jama.jamanetwork.com/article.aspx?articleid=202708 Accessed 30 October 2014NHS Centre for Reviews and Dissemination (2002) The Database of Abstracts of Reviews of Effects (DARE). 6: 1–4Available: http://www.york.ac.uk/inst/crd/EM/em62.pdf Accessed 30 October 2014Van Noorden R (2014 Mar 14) The new dilemma of online peer review: too many places to post? Nature News Blog. Available: http://blogs.nature.com/news/2014/03/the-new-dilemma-of-online-peer-review-too-many-places-to-post.html. Accessed 30 October 2014.National Center for Biotechnology Information (2014) PubMed Commons. Available: http://www.ncbi.nlm.nih.gov/pubmedcommons/. Accessed 30 October 2014.HunterJ (2012) Post-publication peer review: opening up scientific conversation. 6: 63Available: http://journal.frontiersin.org/Journal/10.3389/fncom.2012.00063/full Accessed 30 October 2014RennieD (1998) Freedom and responsibility in medical publication: setting the balance right. 280: 300–302Available: http://jama.jamanetwork.com/article.aspx?articleid=187765 Accessed 30 October 2014HoffmannTC, GlasziouPP, BoutronI, MilneR, PereraR, et al. (2014) Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. 348: g1687Available: http://www.bmj.com/content/348/bmj.g1687.long Accessed 30 October 2014JeffersonT, RudinM, Brodney FolseS, DavidoffF (2007) Editorial peer review for improving the quality of reports of biomedical studies. 2007: MR000016.HopewellS, CollinsGS, BoutronI, YuLM, CookJ, et al. (2014) Impact of peer review on reports of randomized trials published in open peer review journals: retrospective before and after study. 349: g4145Available: http://www.bmj.com/content/349/bmj.g4145.long Accessed 30 October 2014MertonRK (1963) The ambivalence of scientists. 112: 77–97.Goetz A (2014) Reexamining reviewer anonymity—more costs than benefits. Open Science Collaboration. Available: http://osc.centerforopenscience.org/2014/10/22/reexamining-reviewer-anonymity/. Accessed 30 October 2014.LeaperC, RobnettRD (2011) Women are more likely than men to use tentative language, aren't they? A meta-analysis testing for gender differences and moderators. 35: 129–142.LeaperC, AyresMM (2007) A meta-analytic review of gender variations in adults' language use: talkativeness, affiliative speech, and assertive speech. 11: 328.JagsiR, GuancialEA, WorobeyCC, HenaultLE, ChangY, et al. (2006) The “gender gap” in authorship of academic medical literature—a 35-year perspective. 355: 281–287Available: http://www.nejm.org/doi/full/10.1056/NEJMsa053910 Accessed 30 October 2014KellyCD, JennionsMD (2006) The h index and career assessment by numbers. 21: 167–170.LarivièreV, NiC, GingrasY, CroninB, SugimotoCR (2013) Bibliometrics: global gender disparities in science. 504: 211–213Available: http://www.nature.com/news/bibliometrics-global-gender-disparities-in-science-1.14321 Accessed 30 October 2014IsbellLA, YoungTP, HarcourtAH (2012) Stag parties linger: continued gender bias in a female-rich scientific discipline. 7: e49682Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0049682 Accessed 30 October 2014JonesTM, FansonKV, LanfearR, SymondsMR, HiggieM (2014) Gender differences in conference presentations: a consequence of self-selection?2: e627Available: https://peerj.com/articles/627/ Accessed 30 October 2014ShemaH, Bar-IlanJ, ThelwallM (2012) Research blogs and the discussion of scholarly information. 7: e35869Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0035869 Accessed 30 October 2014Maynard A (2014 Mar 24) Gender balance in science blog networks: how has the male/female science blogger ration changed over the past three and a half years? Medium. Available: https://medium.com/2020-science-comms/gender-balance-in-science-blog-networks-71a1efb79958. Accessed 30 October 2014.GilbertJR, WilliamsES, LundbergGD (1994) Is there gender bias in JAMA's peer review process?272: 139–142.ZuberMA (2001) [Underrepresentation of women among peer reviewers and textbook authors in medicine in Germany.] Med Klin (Munich). 96: 173–180.MutzR, BornmannL, DanielHD (2012) Does gender matter in grant peer review?220: 121–129Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3414231/ Accessed 30 October 2014BuddenAE, TregenzaT, AarssenLW, KorichevaJ, LeimuR (2008) Double-blind review favours increased representation of female authors. 23: 4–6.GrodON, BuddenAE, Tregenza, KorichevaJ, LeimuR, et al. (2008) Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. 3: e3202Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0003202 Accessed 30 October 2014GalipeauJ, MoherD, SkidmoreB, CampbellC, HendryP, et al. (2013) Systematic review of the effectiveness of training programs in writing for scholarly publication, journal editing, and manuscript peer review (protocol). 2: 41Available: http://www.systematicreviewsjournal.com/content/2/1/41 Accessed 30 October 2014Bastian H (2003) Non-peer review: consumer involvement in research review. In Godlee F, Jefferson T, editors. Peer review in health sciences, 2nd edition. London: BMJ Books. pp. 248–262.Wiktionary (2014) Wackaloon. Available: https://en.wiktionary.org/w/index.php?title=wackaloon&oldid=21379171. Accessed 4 November 2014.
Provenance: Commissioned; not externally peer reviewed