A scoping review on biomedical journal peer review guides for reviewers

Background Peer review is widely used in academic fields to assess a manuscript’s significance and to improve its quality for publication. This scoping review will assess existing peer review guidelines and/or checklists intended for reviewers of biomedical journals and provide an overview on the review guidelines. Methods PubMed, Embase, and Allied and Complementary Medicine (AMED) databases were searched for review guidelines from the date of inception until February 19, 2021. There was no date restriction nor article type restriction. In addition to the database search, websites of journal publishers and non-publishers were additionally hand-searched. Results Of 14,633 database publication records and 24 website records, 65 publications and 14 websites met inclusion criteria for the review (78 records in total). From the included records, a total of 1,811 checklist items were identified. The items related to Methods, Results, and Discussion were found to be the highly discussed in reviewer guidelines. Conclusion This review identified existing literature on peer review guidelines and provided an overview of the current state of peer review guides. Review guidelines were varying by journals and publishers. This calls for more research to determine the need to use uniform review standards for transparent and standardized peer review. Protocol registration The protocol for this study has been registered at Research Registry (www.researchregistry.com): reviewregistry881.


Background
In academic fields, peer review is a fundamental part of assessing a manuscript's potential significance to the field and improving its methodological rigor and its overall quality for publishing in scholarly journals. Editors depend on reviewers to make their editorial decisions since the reviewers have expertise in the subject areas of submissions they are reviewing. In terms of writing a paper, there have been many efforts to address inadequate reporting of researches and to achieve the standard in scientific writing through transparent reporting of researches [1]. Besides these guidelines, almost all journals, if not all, provide guidelines for authors at their websites. In recent years, guidelines and checklists have been disseminated by journals and publishers for each of the major study types to ensure consistent and complete reporting [2]. However, guidelines for reviewers are not easily discovered. In a survey of 116 health research journals on the McMaster list, Hirst and Altman found only 41 journals (35%) to be providing online instructions to peer reviewers [3]. Ensuring consistent and complete reviewing is also needed. Publishers like Elsevier, SAGE, Springer Nature, and Wiley provide openly available guidelines for reviewers to assist them through the review process, in order to maintain and support standards in reviewing. Yet, not all publishers provide such guidance to their reviewers openly without any technical barriers such as signing in as a reviewer.
Scoping reviews are similar to systematic reviews in the way they systematically examine evidence and the way they are conducted, but they differ from each other in their purpose. The main differences relate to critical appraisal or risk of bias, results synthesis, and study inclusion. Critical appraisal or risk of bias assessment is necessary for systematic reviews, while it is optional for scoping reviews. Systematic reviews synthesize findings from each included study using meta-analysis or meta-synthesis, while scoping reviews do not synthesize findings [4,5]. Scoping reviews include more papers broadly to provide an overview of the existing literature regardless of their type or quality [6]. Systematic reviews are used as references for policymaking or clinical practice as they are suitable for clearly defined questions relating to an intervention efficacy, but scoping reviews are descriptive and suitable for examining evidence in a broader sense such as identifying characteristics of the evidence on a topic [7,8] or identifying available evidence in a field [4]. As peer review is a widely used practice for publication of academic research, there is a need to identify available evidence on peer review guidelines, and to clarify definitions and concepts of peer review process with the evidence.
Currently, there are no widely used standard guidelines for peer reviewing manuscripts which may intimidate or frustrate inexperienced peer reviewers [9]. A lack of standardization may lead to difficulty in reviewing as they are uncertain about what are expected from them. Little has been published about peer review and not much resources have been provided [10]. This scoping review will determine existing peer review guidelines and/or checklists intended for reviewers of biomedical journals. The purpose of this study is to determine whether there are enough evidence and research to set standards on how reviewers of biomedical journals should perform peer reviews and write their review reports, and to provide an overview on the review guidelines.

Materials and methods
A scoping review of manuscript peer review guidelines for reviewers of biomedical journals was conducted to identify available evidence on peer review guidelines for reviewers and to determine what is communicated to peer reviewers about how to peer review. In order to improve the methodological and reporting quality of this scoping review, the checklist of PRISMA Extension for Scoping Reviews (PRISMA-ScR) [7], developed in 2018 by the EQUA-TOR Network was used (S1 File).

Registration
The protocol for this study has been registered at Research Registry with its unique identifying number, reviewregistry881 in April 2020.

Databases and search strategy
PudMed, Embase, and Allied and Complementary Medicine Database (AMED) were all searched from the date of inception until February 19, 2021. In the search, there was no date restriction. Reviewer guidelines by some of the major publishers, journals, and non-publishers were additionally hand-searched online and collected.
For comprehensive search on peer review guidelines in biomedical field, the following search terms were used: "reviewing", "reviewer", "peer review", "peer reviewer", "referee", "journal", "manuscript", "guide", "guidelines", "how to", "lesson", "tips", "advice", "recommendation", "report form", and "checklist". In order to search more publications for comprehensive results, Text Word search was performed, and MeSH term for "peer review" was used for search in PubMed. The search strategy was reviewed by all authors and agreed upon. The complete search strategy can be found in S2 File.
For grey literature, websites of major biomedical journal publishers were searched manually by browsing and clicking through the websites to look for guidelines for reviewers. The publishers were selected based on Simba Information (www.simbainformation.com)'s report, Global Medical Publishing 2019-2023, which provides a list of the leading medical publishers. More publishers were selected from those on Journal Citation Report (jcr.clarivate.com) in the Integrative and Complementary Medicine category. Subsidiary websites were not included in the search of major publishers.

Inclusion and exclusion criteria
The inclusion criteria for this study were any publication relating to reviewer guidelines or checklists that a manuscript reviewer may refer to point by point during peer review, websites of publishers and non-publisher organizations to determine the evidence within the biomedicine field. All types of articles in English (or English translation) were eligible for inclusion in this study without restrictions on date or study design. Thus, editorial materials such as editorial and tutorials were included. Among the websites of above-mentioned publishers and nonpublisher organizations in the 2.2 Databases and search strategy section, those with reviewer guidelines or checklists were eligible for inclusion.
The exclusion criteria were statistical review, conference abstracts, non-English articles, checklists primarily for authors, review process related contents, reviewer/editor roles and duties, assessing review quality or reliability, review challenges or issues, conference peer reviews, grant peer reviews, and those not relating to manuscript peer review. Websites that do not provide reviewer guidelines were not eligible for inclusion.
In order to assess and organize the included publications, a modified quality assessment tool was modified from the tool suggested by Hawker et al. [11] and used in this study to assess the publications included.

Outcomes
The primary outcomes were review checklist items such as a list of questions or points to consider during peer review. The secondary outcomes were the review guidance criteria. Before going through the screening, after the search of literature, the authors randomly pre-screened 10 publications to look for possible review guidance criteria, and based on the discussion by all authors with experience as peer reviewers, the criteria were developed. These criteria aimed to group the contents of reviewer guidelines, such as instructions or recommendations on the reviewer's attitude, becoming a reviewer, manuscript contents, review resources, reviewer ethics, manuscript structure, study ethics, time management, and other components relating to reviewing manuscripts, and to provide a clear overview of available guidelines for reviewers. Then for each included publication, the criteria were independently selected by two authors.

Study selection and data extraction
A data extraction form was developed for this scoping review for data charting purposes. The data extracted from the included articles were first author's last name, publication year, article type, guidance criteria, and number of checklist items. Two independent review authors searched the databases and assessed the eligibility of the searched records. Any discrepancies on article inclusion were discussed with a third review author and resolved on consensus.
All checklist items for reviewing manuscripts were extracted into an Excel spreadsheet. Each item was identified and categorized by its relevant section in manuscript review. The section categories used were title, abstract, introduction, methods, results, discussion, conclusion, references, tables and figures, presentations, and other. The categories were based on the general sections and components of manuscripts and also modified from the category structure of the PRISMA 2009 checklist [12]. The items from the included records of publications and websites were categorized first by the original category of the checklist if they match. For instance, for the checklist items that were categorized under the methods category on the checklist, we categorized it accordingly under the methods category. The items without any specific category were reviewed by the two authors and their categories were determined.

Search results
After screening and excluding articles and other records that fell under exclusion criteria, a total of 79 records (65 publications on peer review guidelines and 14 publisher and non-publisher websites with reviewer guidelines) were included for this study (Fig 1). Out of 79 records, 24 publications did not provide a checklist as in what to look for in the manuscript (relating to contents) and only provided guidance for reviewers as to how to review. The 65 included publications and the 14 websites contained 1,077 items and 734 items respectively (S3 File), which amounted to 1,811 items in total. Each item is a question or points to consider, such as "Is the title appropriate?" or "Accurate/Adequate Abstract." The assessment results of each included publication are provided as a supplement (S4 File).

Characteristics of included articles and websites
The descriptive characteristics of the included articles and websites are summarized in Tables  1 and 2 respectively. Publication types of the included studies were mostly editorial materials (n = 48; 75.0%) such as tutorial, editorial, special articles, perspective, opinion, and commentary. Other types included were review articles (n = 14; 21.9%), research articles (n = 2; 3.1%), and guidelines (n = 1; 1.6%). These article types were extracted as they were presented in the publications. Among the included publications that provided information on how to peer review, 41 publications (63.1%) provided a checklist questions or criteria, but 24 publications (36.9%) did not provide a checklist questions or criteria.

Criteria of reviewer guidelines
The suggestions and instructions for reviewers provided in the included publications were grouped into 9 criteria: attitude, becoming a reviewer, contents, resources, reviewer ethics, structure, study ethics, time management, and other as presented in 2.4 Outcomes in the Methods section. Publications were determined to fit into one or more of the criteria as appropriate. The frequency of reviewer guides criteria extracted from the publications is provided at the bottom rows in Table 1.

Categories of reviewer checklist items
The checklist items were extracted from the checklists in the included publications and websites, and the items were grouped into 17 categories ( Table 3). The top 3 categories with the largest number of items were the Methods (548 items, 30.3%) category, the Results (231 items, 12.8%) category, and the Discussion (191 items, 10.5%) category from the publication items and website items combined. Looking into them separately by publications and websites, the  top 3 categories with the most items were also the same 3 categories of Methods, Results, and Discussion. The grouped categories of the identified checklist items and their numbers are shown in a graphical representation (Fig 2). In the figure, the darkness of color and the size of diamond shape are associated with the number of items for each category.

Discussion
From an extensive range of sources, this scoping review produced a comprehensive list of items outlining guidance for conducting peer review. Review guidelines and checklists can be important for high quality review, especially since editors rely on reviewers and peer reviewer recommendations have an influence on editorial decisions [76]. Such influence of reviewer recommendations on editorial decisions becomes an issue when the quality of peer review is doubtful or when reviewer bias is present [77]. In order to prevent reviewer bias and better improve the peer review quality, some journals and publishers published and posted reviewer guidelines. However, Davis et al. [78] found the content of the manuscript review evaluation forms to vary considerably and relatively few journals asked reviewers to rate specific components of a manuscript. In their study, the review evaluation form of higher impact factor journals addressed statistical analysis, ethical considerations, and conflict of interest, while lower impact factor journals addressed scientific importance, validity, and novelty. Similarly, Gardner et al. [79] also stated that a reviewer report form was found to be more important in reviewing Methods and Results sections in manuscripts. In addition, in a study by Strayhorn et al. [80], the use of standardized review checklist was found to increase reviewer agreement. In this review, the review guidance criteria with the two highest frequencies were found to be the Contents (95.3%) criterion and the Structure (75.0%) criterion. These are considered important components to consider during peer review and many of the included publications were highly focused on them. They reflect the quality of the peer review report inside (reviewing on manuscript contents) and outside (organizing review report structure). This is because the main functions of peer review are to assess the manuscript contents and to have the review results clearly delivered to editors and authors. In terms of the manuscript categories among the checklists, the checklist items with the most in number were the Methods, Results, and Discussion categories. This indicates that the authors as well as publishers and non-publisher organizations consider these categories to be the main parts of manuscripts that require careful evaluation over other parts as the manuscript quality depends highly on these parts when reviewed properly. However, bias frequently occurred while reviewing them which eventually affects editors and editorial decisions [81,82]. The quality of the manuscripts could be improved by providing reviewer guidelines, especially touching on such important components. Although the reporting guidelines by the EQUATOR network (www.equator-network. org) are now widely accepted by major publishers and journals, they are most commonly used by authors in their reporting, since the main purpose of those guidelines was to improve author reporting. For this reason, guidelines by publishers or journals placed much less significance on guidelines for reviewers. Yet, reviewers are expected to cover a wide range of tasks. Glonti et al. [77] conducted a scoping review on the roles and tasks of peer reviewers and found reviewer tasks to include: organizing reviewing, making general comments, assessing and addressing content for each section of the manuscript, addressing ethical aspects, assessing manuscript presentation and providing recommendations. Manuscript peer reviews could be improved by increasing the consistency of ratings, narrative comments, and recommendations [83]. In order to improve the reliability of reviewers, one of the ways suggested by Kravitz et al. [84] was providing more effective guidelines for reviewers, and one suggested by Kwon et al. [85] was providing a checklist to reviewers. In more recent study, Stenfors et al. [66] also suggested that guidelines may help to better identify criteria indicating high-quality research in journal publications. DiDomenico et al. [86] discussed about previous studies that investigated the effectiveness of formal training on review quality and found the training to influence the review quality insignificantly or limitedly; thus, the authors suggested that reviewers may benefit from some guidance. Primarily, checklists are meant to be mnemonic devices aimed at supporting authors and reviewers through a process with a general outline. The items for the evaluation of the manuscript structure or format had more overlaps than those for the evaluation of the manuscript content such as methods and results. The general items of the review checklist in the biomedical related field are most likely to overlap with non-biomedical field as well. As the checklist can be designed based on the overall purpose, some checklist can be more useful than others. Therefore, in order to improve current peer review conditions, we suggest that more research should be carried out and the various groups involved should come together to discuss about the need of a more standardized checklist to be developed in a systematic format and useful to many groups. This kind of attempt is still in progress as seen in a study protocol of randomized controlled trial by Speich et al. [87]. This trial aims to evaluate the effectiveness or the need of providing a short version of the CONSORT checklist and its explanation to peer reviewers for them to check during peer review.
Getting the various groups involved and developing a more standardized checklist would have a technical merit. The items employed for a checklist should be from a broad range of academic groups and reliable sources. Various perspectives that could represent the reviewer committee in general should be considered and factored into the design of the checklist. This is particularly important if the checklist is designed to decrease errors of omission during peer review. Although reviewer checklists themselves are generally not included for publication even for open peer review, the development of reviewer guidelines should be evidence based and requires a systematic and comprehensive approach. The design of effective checklist should include utilization of published guidelines, formation of expert panels for consensus and extensive pilot-testing of preliminary checklists to ensure that recommendations are valid, reliable, and applicable.
These days, being transparent and open about peer review process has gained more attention and is a critical issue for publishers and editors because it fosters them to reach fair and consistent decisions [88], which would then facilitate the authors to improve their manuscript and to move forward. Having standards in peer reviewing could make reviewers more at ease and simplify their review. As the guidelines for author reporting has encouraged structure standardization for publications, a uniform review guideline checklist would support the transparent peer review to be more transparent and standardized for the journals in the same field with similar scope. It would also help authors to better understand their insufficient parts of the manuscript. However, more evidence is needed to evaluate the effects of peer review checklist. In addition, on top of the general standardized review checklist, a list of specific questions that are tailored for the journal, depending on the study design or topic, could be added on.
Limitation of this study was the inclusion of mainly editorial publication types. Most of the included publications being editorial materials places concerns for the guidelines and checklists suggested, since editorials present the perspective of only the editorial's author. Scoping reviews cover a wide range of publication types that quality assessment is not mandatory to perform [4,6]. The publications on "how to review" were mostly editorial materials, including editorials, tutorials, and special contributions, and the generally used quality assessment tools could not be applied to them so the modified quality assessment was used for the assessment of the included publications. In addition, there were limitations in extracting more grey literature. Many journals do not provide reviewer checklists openly and they cannot be accessed by anyone. Thus, the checklists openly provided by publishers were only considered. While the manual search of websites for checklists added more to our data, this may have introduced human error. Other limitations were bias related to search strategies and inclusion criteria. Our preselected keyword search strategies may have introduced some bias or unintended exclusion of publications which should have been included. In addition, biomedical databases with access restrictions were also searched to provide more comprehensive data but this may cause barriers to the publications for researchers. The inclusion of only English language publications may have also excluded some content.

Conclusion
This scoping review explored peer reviewer guidelines in biomedical journals. It identified the extent of existing literature on peer review guidelines and provided an overview of peer review guidelines. It demonstrated the current state of peer review guidelines that they were not uniform which may imply the need for more uniform guidelines. This review could provide grounds for future studies related to reviewer guidelines. In order to produce objective criteria for reviewing manuscripts, further research is needed, and various groups involved in publishing academic manuscripts, such as publishers, editors, reviewers, and other experts, should come together to discuss on developing standard guidelines or recommendations for peer reviewers which may be used by itself or along with individual journal's instructions for reviewers.