https://orcid.org/0000-0003-3978-7552
Figures
Abstract
Objectives
To assess the extent to which peer reviewers and journals editors address study funding and authors’ conflicts of interests (COI). Also, we aimed to assess the extent to which peer reviewers and journals editors reported and commented on their own or each other’s COI.
Study design and methods
We conducted a systematic survey of original studies published in open access peer reviewed journals that publish their peer review reports. Using REDCap, we collected data in duplicate and independently from journals’ websites and articles’ peer review reports.
Results
We included a sample of original studies (N = 144) and a second one of randomized clinical trials (N = 115) RCTs. In both samples, and for the majority of studies, reviewers reported absence of COI (70% and 66%), while substantive percentages of reviewers did not report on COI (28% and 30%) and only small percentages reported any COI (2% and 4%). For both samples, none of the editors whose names were publicly posted reported on COI. The percentages of peer reviewers commenting on the study funding, authors’ COI, editors’ COI, or their own COI ranged between 0 and 2% in either one of the two samples. 25% and 7% of editors respectively in the two samples commented on study funding, while none commented on authors’ COI, peer reviewers’ COI, or their own COI. The percentages of authors commenting in their response letters on the study funding, peer reviewers’ COI, editors’ COI, or their own COI ranged between 0 and 3% in either one of the two samples.
Citation: Makarem A, Mroué R, Makarem H, Diab L, Hassan B, Khabsa J, et al. (2023) Conflict of interest in the peer review process: A survey of peer review reports. PLoS ONE 18(6): e0286908. https://doi.org/10.1371/journal.pone.0286908
Editor: Fares Alahdab, Houston Methodist Academic Institute, UNITED STATES
Received: October 12, 2022; Accepted: May 18, 2023; Published: June 8, 2023
Copyright: © 2023 Makarem et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
We recently proposed the following definition of conflict of interest (COI): “A conflict of interest exists when a past, current, or expected interest creates a significant risk of inappropriately influencing an individual’s judgment, decision, or action when carrying out a specific duty.” [1]. COI may involve a broad spectrum of interests. The financial interests are the most obvious. For example, a researcher may receive significant financial rewards from a pharmaceutical company with interest in the findings of their research. Such financial COIs are common among members of clinical guidelines panels [2, 3]. There is evidence that the quality of the research as well as guidelines may be negatively affected [4].
Non-financial interests can also affect the integrity of research. Such interests include career advancement, fame, social interests, and intellectual beliefs [5]. For instance, an editor may be conflicted when peer reviewing a colleague’s work. Intellectual COI is another type of non-financial COI discussed as far as two decades ago [6]. Lately, intellectual COI has been increasingly acknowledged [7] particularly in clinical practice guidelines (CPG) development [8, 9]. It has been defined as ‘‘academic activities that create the potential for an attachment to a specific point of view that could unduly affect an individual’s judgment about a specific recommendation” [10].
Researchers are expected to avoid and minimize COIs, and disclose them when they exist. The International Committee of Medical Journal Editors (ICMJE) has developed a specific and unified form for disclosure of COIs to facilitate and standardize authors’ disclosures. Moreover, there are recommendations on how to manage and declare COIs among authors of clinical practice [11].
Peer reviewers and editors of journals play a key role in assessing and publishing research manuscripts [12]. On one hand, they need to assess the COI of authors and the funding source of the study. On the other hand, journal peer reviewers and editors may have their own conflict of interests that need to be disclosed [13].
In the current era, the standard for scientific publishing is to have research findings evaluated and published through a peer review process [14]. Peer-reviewed biomedical journals are publishing enormous number of articles each year. As of 2012, about 28000 scholarly journals published more than 2 million peer-reviewed articles [15]. An optimal scientific peer review process is essential to maintain the integrity of the scientific research and to support evidence-based practice.
There is increased media attention to the reported conflicts and concerns about the impact of industry-sponsorship [16, 17]. A transparent handling of conflicts of interest is essential for the public trust in the scientific process and the credibility of peer-reviewed published articles [5]. The reporting of authors’ disclosure of conflicts of interest in publications has become the standard [13]. In order to facilitate and standardize the process of authors’ disclosures, the International Council of Medical Journal Editors (ICMJE) has developed a form for the disclosure of COI [5]. The authors are required to declare via this form all financial and non-financial benefits or personal relationships that might bias their work.
COI issues are relevant to all participants in the peer-reviewed publication process–including peer reviewers, editors, and the editorial board members of the journals. The peer reviewers’ role is to critically assess the manuscript, by constructively commenting on the scientific work, and suggesting how to improve it [18]. Moreover, peer-reviewers and editors are expected to reflect and comment on the authors’ disclosures of conflicts of interest. There are questions about the effectiveness of the current system of COI disclosures. A randomized controlled study found that providing journal reviewers with authors’ conflict of interest information had no significant effect on their rating of the quality of the manuscript [19].
Similarly, journal editors have a core role in managing the review process, through assessing the peer reviewers’ reports and making the final decision on acceptance for publication. In addition, editors can significantly impact the integrity, quality and fairness of the peer review process by how they select the peer reviewers and managing any misconducts by authors or reviewers [20]. In fact, an analysis of the Open Payments database found that journal editors commonly accept industry payments which are often large [21]. Some journals have established policies to deal with editors’ COI [22]. Editors are also expected to review and consider the authors’ disclosures of conflict of interest as part of the peer review process. However, this aspect of their role has not been studied yet.
According to the ICMJE, peer reviewers and editors have to disclose their own conflicts of interest [5]. In some cases, those invited to peer review and editors may need to rescue themselves from being involved. Little is known about the practices and policies of journals regarding disclosures of conflict of interest among peer-reviewers and editors to public [13]. A study assessing the COI policies of health policy and services journals, found only one that described how the COIs of the editorial team are managed during the editorial process [23].
The objective of this study was to assess the extent to which peer reviewers and journal editors address authors’ conflicts of interests and study funding. Also, we aimed to assess the extent to which peer reviewers and journals editors reported and commented on their own or each other’s COI.
Methods
We included two samples in this study. First, we included a sample with any type of original research. However, we found that a very low percentage (9%) had at least one author reporting presence of COI. Given this would not allow us achieve the study objective, we decided to collect a second sample restricted to randomized controlled trials (RCTs) as a survey had found that more than half of clinical trials had authors reporting presence of COI [24].
Eligibility criteria
We included journals that publish in the field of Medicine and in English, are indexed in Medline, and publish all their peer review reports. We included only original articles that are peer reviewed.
Selecting reports
We used the Directory of Open Access Journals (DOAJ) as an initial list of journals. Then, we filtered the journals by language (English) and subject (Medicine). Then, we selected our samples according to the eligibility criteria. For our first sample, we included the latest two original publications from each journal that had peer review reports. For our second sample, we included the latest two randomized controlled trials (RCTs) that had peer review reports. Teams of two reviewers implemented the selection process in duplicate and independently. They resolved any disagreements through discussion, or with the help of a third reviewer if needed.
Data abstraction
We collected and managed study data using the Research Electronic Data Capture (REDCap) tool hosted at the American University of Beirut [25]. REDCap is a secure, web-based software platform designed to support data capture for research studies [25]. We also developed detailed instructions. After conducting calibration exercises, teams of two reviewers abstracted data in duplicate and independently. They resolved any disagreements through discussion, or with the help of a third reviewer if needed.
We abstracted the following data:
- Characteristics of the journal (including impact factor, field of study, publisher)
- Characteristics of the publication (primary research vs. systematic review; source of funding)
- Whether the authors reported their COI and the study funding source
- Whether the peer reviewers commented on authors’ COI and the study funding source
- Whether the journal editors commented on authors’ COI and the study funding source
- Whether the peer reviewers reported their own COI
- Whether the journal editors reported their own COI
- Whether authors and editors commented on the peer reviewers’ COI
- Whether authors and peer reviewers commented on the editors’ COI
Whenever a COI was reported, we further categorized it according to a framework that we recently published [1]. The framework includes seven types of interests relating to either the individuals (direct financial benefit, benefit through professional status, intellectual, and personal) or their institution (direct financial benefit to the institution, benefit through increasing services provided by the institution, and nonfinancial).
Data analysis
We exported all data from REDCap to an Excel sheet for data cleaning and consistency checks. We are sharing that Excel sheet to make all raw data available (see S1 Appendix). We conducted quantitative descriptive analyses of all variables. We used percentage for categorical variables and median and interquartile range for continuous variables. We also conducted a thematic analysis of comments extracted.
Results
Search results
Fig 1 shows the selection process for the 2 samples included in this study. From the initial list of 150 open peer reviewed journals identified from the DOAJ, we included 72 journals meeting our eligibility criteria.
For our first sample, we included 144 articles, two from each of the 72 eligible journals. For our second sample, 58 journals were eligible and published randomized controlled trials. One of these journals published only one RCT. Consequently, we included a total of 115 articles.
General characteristics
Table 1 presents the general characteristics of the journals included in the two samples. For both samples, the majority of journal were from the clinical field (80% and 83%), were published in BioMed Central (BMC) (85% and 89%), and were indexed in Medline (97% and 100%). For the majority of journals peer review was by invitation (99% and 98%), was conducted pre-publication (90% and 91%), and did not reveal the identity of the peer reviewers (56% and 64%).
Table 2 presents the general characteristics of publications included in the two samples. The percentages of RCTs were 3% for sample 1 and 100% for sample 2. For both samples small percentages had COVID-19 as topic (8% and 3%), the median number of authors was 6, the majority had their first author primarily affiliated with academia (87% and 93%), and the majority had the first author affiliated with a high-income country (63% and 61%).
Table 3 presents characteristics of the peer review process. For both samples, the median for number of rounds of peer review, and the number of rounds of revision, and the number of peer reviewers was 2. In the two samples, the median of the number of editors involved was 1. Only 11% and 13% of editors’ letters to the authors were posted publicly, and of those 63% and 53% respectively provided the name of the editor.
Funding of the study
Table 4 presents the characteristics of funding of the included studies. For both samples, the majority of studies were reported as funded (67%and 90%) and included a statement on the funder’s role (57% and 59%). The top two sources of funding were governmental (61% and 46%) and internal funding (41% and 46%).
Declaration of conflict of interest by authors, peer reviewers, and editors
Table 5 summarizes the declaration of conflict of interest by authors, peer reviewers, and editors. For both samples, the majority of studies had all authors report the absence of COI (89% and 68%). For both samples, the majority of studies had reviewers identified by name (55% and 56%), reported absence of COI (70% and 66%). For both samples, none of the editors whose names were publicly posted reported on COI. Substantive percentages did not report on COI of reviewers (28% and 30%) while only small percentages reported any COI (2% and 4%). Most of these declarations reported individual financial interests with direct financial benefit (see footnotes in Table 5 for more details).
Commenting on declarations of conflict of interest
Table 6 summarizes the extent of authors, peer reviewers and editors commenting on study funding and conflicts of Interest. The percentages of peer reviewers commenting on the study funding, authors’ COI, editors’ COI, or their own COI ranged between 0 and 2% in either one of the two samples. 25% and 7% of editors respectively in the two samples commented on study funding, while none commented on authors’ COI, peer reviewers’ COI, or their own COI. The percentages of authors commenting in their response letters on the study funding, peer reviewers’ COI, editors’ COI, or their own COI ranged between 0 and 3% in either one of the two samples. Fig 2 below illustrates these results among our 2 samples.
Discussion
Summary of findings
In summary, the percentages of peer reviewers and editors commenting on study funding and authors’ COI were extremely low. In addition, peer reviewers and journal editors rarely reported their own COI, or commented on their own COI, or on each other’s COI.
Strengths and limitations
To our knowledge, this is one of the first studies that examines the extent to which peer reviewers and editors consider study funding and authors’ COI, and the extent to which they report on their own COI. To ensure our findings are informative, we went beyond our original plan and included a second sample that involved a higher percentage of articles whose authors reported presence of COI, compared with our first sample. We found consistent results for the two samples. One limitation of this study is that our samples represent the peer review processes of articles that ended up being accepted for publication, but not of those that were not survive the peer review process or those that were published in non-open peer review journal.
Comparison to similar studies
The percentage of articles with at least one author reporting presence of COI for our first sample was only 9%. This percentage is much lower compared to that of systematic reviews (41%) and clinical trials (57%) [24, 26]. For our second sample, the percentage was higher at 30% because we included only randomized clinical trials. On the second hand, an article by Ralph et al. highlighted the fact that while there are concerns about authors’ financial COI in health research, there is little discussion on potential editorial financial COI [27]. In their sample of 20 public health journals, editor and peer reviewer financial COI policies were lacking strict disclosure requirements compared to author financial COI policies [27]. Moreover, another cross-sectional study of high-impact medical specialty journals found that although 99% of those journals required author’s COI disclosure, only 12% required editorial’s COI disclosure [28]. A review of the COI policies of 72 health policy and services journals found that only one policy described how the COIs of the editorial team are managed during the editorial process [23].
Implications for practice and research
The results of our study provide insight about the COI disclosure of peer reviewers, authors and editors. One important question is whether the findings are explained by the lack of relevant journal policy or by the non-compliance of editors with such a policy. Reviews of the COI policies of journals in the clinical, surgical and public health areas did not address how the COIs of the editorial team are managed during the editorial process, raising the need for further exploration [29–31].
Our findings may help journals to develop policies to improve how COI is declared and managed during their editorial processes. It would be relevant to conduct qualitative research to explore why some peer reviewers and editors are commenting on authors’ COI and on the study funding and others are not. Such study may help in developing strategies to improve COI declaration and management during the peer review process. Also, it would be important to conduct a similar study of articles that underwent the peer review process but did not survive the peer review process or of articles that were published in non-open peer review journal.
References
- 1. Akl E.A., et al., A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. Journal of Clinical Epidemiology, 2022. 149: p. 236–243. pmid:35697333
- 2. Mendelson T.B., et al., Conflicts of interest in cardiovascular clinical practice guidelines. Arch Intern Med, 2011. 171(6): p. 577–84. pmid:21444849
- 3. Neuman J., et al., Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study. BMJ, 2011. 343: p. d5621. pmid:21990257
- 4. Cosgrove L., et al., Conflicts of interest and the quality of recommendations in clinical guidelines. J Eval Clin Pract, 2013. 19(4): p. 674–81. pmid:23731207
- 5. Editors, I.C.o.M.J., Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. 2018.
- 6. Marshall E., When does intellectual passion become conflict of interest? Science, 1992. 257(5070): p. 620–3. pmid:1496373
- 7. Levinsky N.G., Nonfinancial conflicts of interest in research. N Engl J Med, 2002. 347(10): p. 759–61. pmid:12213950
- 8. Lederer E.D., Development of clinical practice guidelines: are we defining the issues too narrowly? Clin J Am Soc Nephrol, 2007. 2(2): p. 207. pmid:17699410
- 9. Detsky A.S., Sources of bias for authors of clinical practice guidelines. CMAJ, 2006. 175(9): p. 1033, 1035. pmid:17060643
- 10. Guyatt G., et al., The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med, 2010. 152(11): p. 738–41. pmid:20479011
- 11.
Lo B. and Field M.J., Institute of Medicine (US) Committee on Conflict of Interest in Medical Research, Education, and Practice, eds.(2009): Conflict of Interest in Medical Research, Education, and Practice. Washington (DC): National Academies Press (US). www.ncbi.nlm.nih.gov ….
- 12. Glonti K., et al., A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med, 2019. 17(1): p. 118. pmid:31217033
- 13. Cooper R.J., et al., Conflict of Interest Disclosure Policies and Practices in Peer-reviewed Biomedical Journals. J Gen Intern Med, 2006. 21(12): p. 1248–52. pmid:17105524
- 14. Davis C.H., et al., Reviewing the review: a qualitative assessment of the peer review process in surgical journals. Res Integr Peer Rev, 2018. 3: p. 4.
- 15. Manchikanti L., et al., Medical journal peer review: Process and bias. Pain Physician, 2015. 18(1): p. E1.
- 16. Davidoff F., et al., [Sponsorship, authorship and accountability]. Tidsskr Nor Laegeforen, 2001. 121(21): p. 2531–2.
- 17. Fontanarosa P.B., Flanagin A., and DeAngelis C.D., Reporting conflicts of interest, financial aspects of research, and role of sponsors in funded studies. JAMA, 2005. 294(1): p. 110–1. pmid:15998899
- 18. Evans A.T., et al., The characteristics of peer reviewers who produce good-quality reviews. Journal of general internal medicine, 1993. 8(8): p. 422–428. pmid:8410407
- 19. John L.K., et al., Effect of revealing authors’ conflicts of interests in peer review: randomized controlled trial. bmj, 2019. 367: p. l5896. pmid:31694810
- 20. Resnik D.B. and Elmore S.A., Ensuring the quality, fairness, and integrity of journal peer review: A possible role of editors. Science and Engineering Ethics, 2016. 22(1): p. 169–188. pmid:25633924
- 21. Liu J.J., et al., Payments by US pharmaceutical and medical device manufacturers to US medical journal editors: retrospective observational study. bmj, 2017. 359. pmid:29074628
- 22. Gottlieb J.D. and Bressler N.M., How should journals handle the conflict of interest of their editors?: who watches the “watchers”? Jama, 2017. 317(17): p. 1757–1758.
- 23. Khamis A.M., et al., Requirements of health policy and services journals for authors to disclose financial and non-financial conflicts of interest: a cross-sectional study. Health Res Policy Syst, 2017. 15(1): p. 80. pmid:28927424
- 24. Hakoum M.B., et al., Authors of clinical trials reported individual and financial conflicts of interest more frequently than institutional and nonfinancial ones: a methodological survey. Journal of clinical epidemiology, 2017. 87: p. 78–86. pmid:28412465
- 25. Harris P.A., et al., The REDCap consortium: building an international community of software platform partners. Journal of biomedical informatics, 2019. 95: p. 103208. pmid:31078660
- 26. Hakoum M.B., et al., Reporting of financial and non-financial conflicts of interest by authors of systematic reviews: a methodological survey. BMJ open, 2016. 6(8): p. e011997. pmid:27515760
- 27. Ralph A., Petticrew M., and Hutchings A., Editor and peer reviewer financial conflict of interest policies in public health journals. European Journal of Public Health, 2020. 30(6): p. 1230–1232. pmid:33313818
- 28. Dal-Ré R., Caplan A.L., and Marusic A., Editors’ and authors’ individual conflicts of interest disclosure and journal transparency. A cross-sectional study of high-impact medical specialty journals. BMJ Open, 2019. 9(7): p. e029796. pmid:31340971
- 29. Shawwa K., et al., Requirements of clinical journals for authors’ disclosure of financial and non-financial conflicts of interest: a cross sectional study. PLoS One, 2016. 11(3): p. e0152301. pmid:27030966
- 30. El Moheb M., et al., The policies for the disclosure of funding and conflict of interest in surgery journals: a cross-sectional survey. World Journal of Surgery, 2021. 45: p. 97–108. pmid:32914281
- 31. Daou K.N., et al., Public health journals’ requirements for authors to disclose funding and conflicts of interest: a cross-sectional study. BMC public health, 2018. 18: p. 1–9. pmid:29688846