All authors had financial support from the BJS/Wiley and Sons for the submitted work; there were no other financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Peer review is important to the scientific process. However, the present system has been criticised and accused of bias, lack of transparency, failure to detect significant breakthrough and error. At the British Journal of Surgery (BJS), after surveying authors’ and reviewers’ opinions on peer review, we piloted an open online forum with the aim of improving the peer review process.
In December 2014, a web-based survey assessing attitudes towards open online review was sent to reviewers with a BJS account in Scholar One. From April to June 2015, authors were invited to allow their manuscripts to undergo online peer review in addition to the standard peer review process. The quality of each review was evaluated by editors and editorial assistants using a validated instrument based on a Likert scale.
The survey was sent to 6635 reviewers. In all, 1454 (21.9%) responded. Support for online peer review was strong, with only 10% stating that they would not subject their manuscripts to online peer review. The most prevalent concern was about intellectual property, being highlighted in 118 of 284 comments (41.5%). Out of 265 eligible manuscripts, 110 were included in the online peer review trial. Around 7000 potential reviewers were invited to review each manuscript. In all, 44 of 110 manuscripts (40%) received 100 reviews from 59 reviewers, alongside 115 conventional reviews. The quality of the open forum reviews was lower than for conventional reviews (2.13 (± 0.75) versus 2.84 (± 0.71), P<0.001).
Open online peer review is feasible in this setting, but it attracts few reviews, of lower quality than conventional peer reviews.
Formal peer review has been used by many scientific journals since the 18th century. It is regarded as a trusted form of scientific evaluation, which ideally guarantees and improves scientific reporting, thus serving as a quality-control system. To corroborate this there is empirical evidence that conventional peer review improves the quality of papers [
A change in the traditional peer review process seems inevitable and has already been advocated by others [
Thus, the aim of this study was to investigate attitudes towards modernisation, the advantages and disadvantages of an innovative online peer review system in comparison with traditional peer review and to obtain the views of those involved in this new process.
The British Journal of Surgery (BJS) is a monthly, peer-reviewed surgical journal, with a current five-year impact factor of 5.596 (2015). BJS publishes original work in a range of surgical specialties. The journal receives approximately 1800–2000 manuscripts annually. The acceptance rate in 2015 was 12%.
In the BJS standard peer review process, the co-Editor-in-Chief assesses the manuscript for suitability and a small percentage of manuscripts are rejected based on article type. All manuscripts are then allocated to a handling editor, who either selects two to three relevant referees for peer review or consults with at least one other editor before rejecting the manuscript without peer review [
The intention was to develop an innovative peer review system that took full advantage of modern technology to improve the peer review process and draw reviewers and readers into meaningful online scientific debate. The key elements would be: instant electronic real-time communication; an electronic forum; and social media integration, promoting awareness, escalating engagement and propagating participation. The aims of such a system are to: improve the quality of peer review and subsequent post-review manuscripts; draw on a wider range of potential peer reviewers; promote transparency; provide an inside view on critical appraisal, research and peer review to junior surgeons and researchers; and increase readership and impact of the journal and its research.
The present study was divided into three parts: a pre-trial questionnaire, a prospective study on an innovative online forum for peer reviewing and a post-trial questionnaire.
The pre-trial questionnaire aimed to investigate the opinion of authors and reviewers towards an online forum for peer review, and to introduce authors and reviewers to the idea of using an online forum for peer review. The questionnaire was conducted in December 2014, using SurveyMonkey [
The questionnaire (
All manuscripts involving original work (i.e. observational studies, randomised controlled studies and systematic reviews) submitted to BJS between 1st April and 31st May 2015 were considered for possible inclusion. Upon submission the author was informed about the study and their consent for participation was sought. It was made clear to authors that participation was voluntary and would not affect the outcome of their submission. Manuscripts from consenting authors simultaneously went through standard peer review and innovative peer review using an online forum. Manuscripts were posted online, and an invitation to review these online manuscripts was sent every 14 days by email to approximately 7000 referees with an accredited account in BJS ScholarOne. The manuscript was kept online for 3 weeks and was only accessible through the link supplied in the email.
Referees reviewing on the online forum were asked to report using the same standard referee sheets as in the standard peer review process (
According to the pre-specified study protocol, the handling editor decided on the manuscript based on the standard peer review only, without viewing the reports generated in the online forum. After the editorial decision, the online peer reviews were sent to the handling editor, who scored them according to the standard BJS policy rating system, based on a 5-point Likert scale, from cursory (1 point) to outstanding (5 points), as defined above.
In parallel, two editorial assistants (MA, RvA) assessed all reviews, both from the conventional peer review process and from the innovative online forum, using a previously validated instrument by van Rooyen
In order to secure the authors’ copyright during the study, the line “Copyright 2015 The Authors. All rights reserved.” was added at the top of each manuscript. If a manuscript was accepted, standard BJS policy was applied, and the corresponding author received an email prompting him to log in to Author Services, where, via the Wiley Author Licensing Service (WALS), the license agreement had to be completed on behalf of all authors on the paper.
The primary outcome of the present study was the quality of referee reports based on the validated review quality instrument (
The aim of the post-trial questionnaire was to assess the overall satisfaction of authors with the online review process. A web-based questionnaire [
Responses to the pre-trial questionnaire were summarised using descriptive statistics. Factors influencing the likelihood of a respondent’s participation in online peer review were analysed using single and multiple logistic regression, testing associations between characteristics of respondents and responding yes to “would you consider participating in open online peer review”, yielding odds ratios (OR) with 95% confidence intervals (CI). In the multiple models, all items in the questionnaire were included (
A thematic analysis was performed for the only free-text answer and was narratively summarised.
Review quality scores from editors (using a 1–5 point scale) and from assistant editors (using the 8-item validated checklist) were summarised and compared between standard and online review using means and Student t-tests. Agreement between assistant editors’ scoring was measured using kappa statistics. A
Ethical approval was not sought, as patients were not directly involved. It was assumed that consent was given when participants actively chose to participate in the survey. Authors submitting their work received information about the study at submission, and it was made clear to them that participation was voluntary and would not affect the outcome of their manuscript. Authors consented to participating by ticking a box at the submission site.
The pre-trial questionnaire was sent to 6635 listed reviewers. There were 1454 individual responses, corresponding to a response rate of 21.9%. Not every item in the questionnaire was answered by all respondents, but missing responses were few: no more than 2.5% for any item.
Some 87% of respondents were male, 94% of the respondents already acted as a reviewer and 78% had published and/or submitted more than 20 papers (
Numbers (column per cent).
All (n = 1454) | Would participate (n = 973) | OR (95% CI) | ||
---|---|---|---|---|
Male | 1261 (87.5) | 870 (89.9) | 1.99 (1.43–2.78) | |
Female | 180 (12.5) | 98 (10.1) | 1.00 | |
<30 | 22 (1.5) | 14 (1.4) | 0.89 (0.39–2.39) | |
30–40 | 232 (16.0) | 166 (17.1) | 1.00 | |
40–50 | 421 (29.0) | 265 (27.4) | 0.67 (0.47–0.96) | |
50–60 | 462 (31.9) | 311 (32.1) | 0.83 (0.57–1.20) | |
>60 | 312 (21.5) | 21 (21.9) | 0.83 (0.55–1.24) | |
English | 602 (41.8) | 425 (44.0) | 1.33 (1.06–1.68) | |
Other | 839 (58.2) | 542 (56.0) | 1.00 | |
None | 7 (0.5) | 3 (0.3) | 0.57 (0.10–3.12 | |
1–5 | 80 (5.5) | 51 (5.2) | 1.00 | |
5–20 | 225 (15.6) | 160 (16.5) | 1.23 (0.69–2.17) | |
>20 | 1133 (78.4) | 757 (78.0) | 0.99 (0.58–1.69) | |
Yes | 1118 (77.6) | 771 (79.4) | 1.32 (1.01–1.72) | |
No | 322 (22.4) | 200 (20.6) | 1.00 | |
None | 80 (5.6) | 54 (5.6) | 1.02 (0.58–1.80) | |
<10 | 540 (37.5) | 365 (37.6) | 0.96 (0.72–1.28) | |
10–20 | 438 (30.4) | 307 (31.6) | 1.00 | |
>20 | 381 (26.5) | 244 (25.2) | 0.78 (0.58–1.05) | |
Yes | 973 (67.6) | - | - | |
No | 145 (10.1) | - | - | |
Unsure | 321 (22.3) | - | - | |
Yes | 595 (42.1) | 505 (52.8) | - | |
No | 252 (17.8) | 100 (10.5) | - | |
Unsure | 565 (40.0) | 351 (36.7) | - |
Numbers do not always add up since it was possible to leave item blank
Multiple logistic regression with odds ratios and 95% confidence interval for answering yes to “would you submit to open peer review”, adjusted for sex, age, first language, nr of previous published and/or submitted papers, previous submission to BJS, and nr of annual reviews
Free-text answers regarding thoughts and concerns about the online peer review process are summarised in
Categories of concerns and comments | N = 284 |
---|---|
118 (41.5) | |
15 (5) | |
5 (2) | |
61 (21.5) | |
15 (5) | |
31 (11) | |
5 (2) | |
12 (4) | |
6 (2) | |
22 (8) | |
4 (1.5) | |
7 (2.5) |
* Multiple concerns per response were raised, therefore total number of concerns do not add up to total number of responses
In multiple regression analysis, male gender, English as first language and previous submission to BJS were all associated with willingness to participate in online peer review (
Between 1st April and 31st May 2015 a total of 295 manuscripts, including Leading Articles, Snapshots in Surgery, and Original Articles were submitted to the BJS; of these, 265 were eligible for the study (
Some 59 individual referees posted online reviews and the median (range) number of reviews posted per reviewer was 1 (1–15). The number of reviews via the online system declined over time (
The mean (SD) review quality scores are shown in
Means (SD).
Regular peer review (n = 115) | Open peer review (n = 100) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Assessor A | Assessor B | Overall | κ | Assessor A | Assessor B | Overall | κ | P | |
2.99 (0.99) | 2.25 (1.04) | 2.62 (1.08) | 0.12 | 2.13 (1.21) | 2.07 (1.23) | 2.10 (1.22) | 0.26 | 0.001 | |
2.96 (1.18) | 2.43 (1.32) | 2.69 (1.28) | 0.25 | 1.79 (1.14) | 1.98 (1.22) | 1.88 (1.18) | 0.30 | <0.001 | |
3.39 (0.92) | 2.83 (1.12) | 3.11 (1.06) | 0.23 | 2.49 (1.23) | 2.59 (1.19) | 2.54 (1.21) | 0.23 | <0.001 | |
2.57 (0.90) | 1.87 (1.16) | 2.22 (1.09) | 0.16 | 2.15 (1.10) | 1.56 (0.99) | 1.86 (1.09) | 0.16 | 0.017 | |
3.21 (0.92) | 3.10 (1.05) | 3.15 (0.99) | 0.33 | 2.40 (1.24) | 2.62 (1.23) | 2.51 (1.24) | 0.33 | <0.001 | |
3.52 (0.76) | 2.08 (1.19) | 2.79 (1.22) | 0.06 | 1.46 (0.92) | 1.63 (0.95) | 1.55 (0.94) | 0.06 | <0.001 | |
3.27 (1.03) | 1.97 (1.12) | 2.62 (1.26) | 0.11 | 1.74 (0.99) | 1.31 (0.73) | 1.52 (0.89) | 0.11 | <0.001 | |
3.64 (0.69) | 3.32 (0.65) | 3.48 (0.69) | 0.18 | 2.98 (0.89) | 3.15 (0.72) | 3.07 (0.81) | 0.18 | <0.001 | |
3.19 (0.56) | 2.48 (0.66) | 2.84 (0.71) | - | 2.15 (0.76) | 2.12 (0.73) | 2.13 (0.75) | - | <0.001 | |
- | - | 3.52 (0.58) | - | - | - | 2.35 (0.74) | - | <0.001 |
K agreement between assessor A and B. p, Student’s t-test.
The post-trial questionnaire (
In this study, feasibility, advantages and disadvantages of an experimental online forum peer review system in the setting of a high-quality general surgical journal were assessed. The pre-trial survey suggested that a majority of authors and reviewers were willing to participate in a study exploring this new method of peer review. The actual participation rate of authors in the study was 58%, with a median of one online review per manuscript. The quality of the online reviews varied considerably, but was significantly lower than conventional review. Given the large number of potential reviewers who were invited, the participation rate was very low.
In the pre-trial survey, previous submissions/publications, male gender and English as a first language were factors associated with willingness to participate. It is possible that those surveyed, who were all BJS reviewers, differed from authors in these aspects and that worries about the benefits of the online review process and satisfaction with the current system led authors to decline participation. It is also possible that some degree of suspicion around a new system precluded authors from taking part, and if this system became more common and/or accepted, participation rate would increase. Still, over half of the authors submitting during the trial period participated. In this regard, it must be noted that questions 9 and 10 of the survey were somewhat leading (
Despite mailing an invitation to submit a review to more than 7000 referee account holders in the BJS ScholarOne system, we only received reviews from 59 individual reviewers. This has to be considered a disappointingly low rate. This could be due to the invitation itself: reviewers might be more attracted, and feel more responsible, if they receive a personal email from one of the editors than from an impersonal mass mail. Another reason could be a general difficulty in getting reviews. The intention was also to draw reviewers and readers into meaningful online debate about the work under review. The finding that the number of online reviewers declined with time also points towards a loss of interest of the potential reviewers and lack of resources used to win potential participants.
The lack of time and incentives for the reviewer have been pointed out previously[
Placing emphasis on reviewing experience when assigning academic posts within universities and medical schools might also increase the incentive to review. An interesting idea is to create a reviewer index [
The low participation rate might also relate to a lack of competence in a specific field, real or perceived, on the part of the potential reviewer. Surgery and surgical science are, as all branches of medicine, growing more specialised every year. Most manuscripts submitted to BJS deal with a very specific area, in which only a fraction of the invited reviewers might be expected to have competence.
An open online system is attractive because of its transparency and potential to increase communication between reviewers and authors, but, in the present study, some potential reviewers might have refrained from reviewing because their names would have been made public. Thus, non-anonymity might be one reason for the low rate of open peer review. Even if anonymous, online peer review increases transparency, and there is no evidence that unmasking either authors’ or reviewers’ identities increases peer review quality [
Medicine has been slow to take on pre-print publication, which can be considered a special case of open peer review. In areas such as physics, statistics, engineering and mathematics, there are several online pre-print servers, on which authors display their work in progress and receive commentaries from the scientific community. For instance, arXiv.org, which started as early as the 1990s, currently houses over 1,000,000 openly available pre-prints. Many pre-print servers have moderators that function as editors, reviewing submissions, turning down those that are off-topic or simply not scientific papers. Some pre-print servers also make use of endorsers, who are authors in the field, who have the authority to allow unauthorised (non-endorsed) authors to submit papers, thus functioning as peer reviewers.
Recently, there have been some highly publicised, serious cases of fraudulent research. The content of individual papers, journals and publishing houses and review systems have been the subject of fraud and cheating. Getting papers published is, for many academics, directly related to career, status and income, and there is a risk that an online system could be misused. In this light, the authors’ perceived fear of having ideas stolen and plagiarised should be taken seriously. The post-trial survey was only answered by ten authors, which precludes any firm conclusions.
The quality of online review reports as judged using both a validated review quality tool and the conventional BJS scoring system was significantly lower than the quality of the conventional reviews; hence, one may conclude that online reviews cannot replace conventional, invited reviews.
Some limitations of the present study need to be mentioned. One is the lack of randomisation and a control group. The present results might not be applicable to all types of journals; for instance, if a similar study was performed on a strictly open access journal, it might yield different results. Further, the inter-rater variability between the raters using the validated review quality tool was higher than previously reported. Apart from the pre-trial survey, a more extensive engagement strategy could have been employed to increase author and reviewer participation. Such a strategy could include presentation at meetings and advertising in printed, online and social media. Once such a system is up and running and engagement is ensured, papers could come to the editor in a more mature and refined product which has been evaluated from multiple perspectives. Online peer review may not be focussed so much on the scientific credibility of a paper but may provide a better insight in the timeliness, attractive value, desirability and possible impact of a manuscript for the scientific community.
(PDF)
(PDF)
(PDF)
docx.
(DOCX)
dta.
(DTA)
dta.
(DTA)
xlsx.
(XLSX)
dta.
(DTA)