Peer review

Following Kelly et al. [27, p. 227], we define peer review as “a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field”. The peer review process has been a formal part of scientific communication since the first scientific journals were published over 300 years ago [28]. Although there are differences between journals, peer review typically entails the next steps: (a) an editor makes an initial manuscript selection by checking journal fit and whether a submitted manuscript meets the formal standards of, for example, format and ethics; (b) the editor selects and invites reviewers; (c) the reviewers independently assess the quality of the work and make recommendations concerning publication; and (d) if the editor decides the manuscript is a candidate for publication, she guides the process of resubmission and additional rounds of review and decides upon definitive publication.

The first goal of peer review is to check whether reported research meets the appropriate standards, and therefore whether findings and conclusions can be considered valid. The second goal is to help authors improve the quality of their research and its presentation. The third goal is to assess originality, significance, and broader interest, and finally, to assess the ‘fit’ between a paper and a journal [29].

Despite its ubiquitous use, peer review is intensely debated. Ware [30] found that most academics (85%) believe that peer review greatly helps scientific communication. In contrast, others argued that peer review might increase the use of QRPs due to reviewers demanding ‘perfect’ results that are unrealistic [11,31], and/or researchers expecting reviewers to demand overly clean results [24]. It may well be that critical peer reviewers stimulate, either implicitly or perhaps sometimes even explicitly, authors to leave out unconvincing outcomes or aspects of the study [22,24] or even to misreport certain results. Hence, the quality of reviews remains a concern and little consensus exists on how to even define review quality [32,33].

It is unclear to what extent peer review helps improve research manuscripts. Peer review seems to bring about little change in manuscripts from preprint to post-print [32], which does not need to imply that the original research was flawless. Fraud and QRPs still enter the published literature, and published articles are retracted from publication after peer review failed to detect problems in these articles [27,32,33]. Furthermore, experimental studies have shown that reviewers have a hard time detecting even obvious errors in a paper [34–36]. Although most authors and reviewers believe that the detection of plagiarism is a task of peer review, a minority believes peer review is suited to do so [27].

Reviewers are often inconsistent and agree only slightly above chance in their opinion on whether a paper should be published [37,38]. Inter-reviewer reliability is low [39,40], with quality ratings of the same manuscript ranging from unacceptable to excellent [33,41]. Whereas one reviewer argues ‘I found this paper an extremely muddled paper with a large number of deficits’, another reviewer argues ‘It is written in a clear style and would be understood by any reader’ [38]. Assuming almost all reviewers are honest and are not abusing their power of peer review (e.g., to stall publication of competitors or scoop their ideas), these inconsistencies indicate that the evaluation of manuscript quality is ambiguous. Every peer reviewer has a different background and different knowledge, and editors might be looking for peer reviewers with differing expertise to review an article. While this is a sound strategy using different reviewers’ complementary expertise, unanticipated differences in knowledge and understanding of the same topic, knowledge of previous studies on the same topic, methodological and statistical skills, and personal preferences for some topics but not for others may readily produce inconsistent review assessments. Differences between reviewers in knowledge, understanding, and preferences are difficult to control in research. Therefore, in this study, we focus on the question of which manuscript characteristics are most important for manuscript quality rating that might differ between reviewers and even between manuscripts.

Previous research indicates that reviewers seem to favor their judgements of scientific manuscripts on ‘false cues’, such as statistical significance, often unduly complex procedures, and overly complex writing [42]. Furthermore, reviewers suffer from the same biases as researchers, such as confirmation bias and hind-sight bias [43,44]. They might not even realize these false cues influence their quality assessments. For example, in an experimental study by Atkinson, et al. [45], peer reviewers received a version of a manuscript with either statistically significant results (p < .01), results with p < .10, or results with p < .25. The manuscript that reported a statistically significant result was three times less likely to be rejected than the other two versions. Despite all other characteristics of the manuscript being equal, reviewers often believed they rejected the manuscript based on the design of the study rather than its results.

It is unclear if reviewers differ in their assessment of the quality of a manuscript or whether they assess the suitability of the manuscript for publication. Due to (perceived) expectations of journals and editors, perhaps other characteristics are important for assessing publishability of an article. Statistical significance may be considered more relevant for publishability than for manuscript quality. Reviewers may recognize high quality research but use different characteristics when assessing its ‘publishability’.

Review in education

There are other review issues relevant in assessing a master’s thesis than in reviewing scientific manuscripts as part of the publication process. With their master’s thesis, students show what they have learned during their education. We would expect teachers to positively consider cues that indicate responsible research practices (RRPs), but to negatively assess indications of QRPs. Since the main goal of a master’s thesis is educational, that is, the application of knowledge, insights, and skills previously learned, but usually not publication, statistical significance of the results should be less important than in manuscripts submitted for publication. In addition, students do not believe that there is a causal relation between good science and statistically significant results, and they do not believe their teachers reward statistically significant results [46]. If students’ perspective is correct, we expect academics to uphold different quality standards for master’s theses than for research manuscripts. Furthermore, theses differ from published manuscripts in many aspects that may be relevant for their review and evaluation. For example, previous research showed that theses report effect sizes more frequently than published manuscripts [46–48], on average use larger samples [49], and conduct a priori power analysis more often [46,50]. These differences could also indicate that students and researchers consider different aspects in assessing the quality of these manuscripts.

Students are often confronted with published literature as examples of good research reports during their education. These examples, as well as students’ perceived attitude of their academic teachers towards QRPs shape the students’ own attitudes and behavior towards these QRPs [46] and set the bar for what is considered high-quality research. For example, what students believe may be a sufficient sample size (e.g., to have sufficient statistical power) might be influenced by sample sizes encountered in the literature studied in various courses, and when studying in a field where open science is advocated in the published literature, students might assign greater importance to pre-registration and sharing study materials. Previous research shows that both researchers and students frequently engage in QRPs. Over 90% of psychology researchers admitted having engaged in QRPs during their career [20,21,23,51]. Estimates for students indicate that 64% of them have already engaged in QRPs in their short academic careers (up to graduate level, [52]), whereas 40% of psychology students engaged in QRPs in their thesis [46]. It is currently unknown how academics differ in their quality assessment of theses and articles, and how potential QRPs affect this assessment. Finally, it is unknown how students perceive this quality assessment of their supervisors.

Current study

We know little about the characteristics of submitted manuscripts and master’s theses that affect quality assessments and their publishability or grades. It also is unknown whether students and their supervisors believe quality is affected by the same characteristics. Therefore, the main research question of this study is “Which characteristics of a manuscript affect students and academic researchers when assessing the quality of a scientific manuscript?”. We note that the preregistered research question was originally formulated as: “Which characteristics of a manuscript do students and researcher believe to be of importance when assessing the quality of a scientific manuscript?”. This formulation was changed, because we believed, in hindsight, that this formulation did not cover the full range of our study. We did not only investigate the beliefs of academics and students, but also tested what influenced their quality assessment in the vignette study that we preregistered together with the survey study.

In our vignette study, we focused on three research characteristics that we manipulated in a study’s abstract to examine their effect on the quality evaluation of scientific manuscripts (article or thesis). We studied the impact of sample size, statistical significance, and a statistical reporting error on quality ratings of the research. A statistical reporting error refers to an inconsistency between the reported test statistic, degrees of freedom, and p-value. Furthermore, we studied the two-way interactions of these three characteristics, and the effect of career phase (student versus academic) and manuscript type (thesis versus article) on the ratings of the research.

The following research questions and matching hypotheses were formulated and preregistered (https://osf.io/cd3uw):

RQ1: Which characteristics of research (sample size, significance, reporting error) as described in a manuscript (thesis or article) affect the quality assessment of a scientific manuscript by both students and academics?

H1a: Manuscripts with large sample size are rated as having higher quality compared to manuscripts with smaller sample sizes

H1b: Manuscripts with significant results are rated as having higher quality compared to manuscripts with non-significant results

H1c: Manuscripts with reporting errors are rated the same as manuscripts without reporting errors

RQ2: Do the effects of the characteristics on the quality of the manuscript as assessed by academics differ between a thesis and an article?

H2a: The differences in quality rating between small and large sample size manuscripts is larger for articles compared to theses

H2b: The differences in quality rating between significant and non-significant manuscripts is larger for articles compared to theses

H2c: The differences in quality rating between manuscript with and without reporting error do not differ for articles compared to theses

RQ3: Do the effects of the characteristics on the quality of a thesis differ between students and academics?

H3a: The differences in quality rating between small and large sample size manuscripts differs between students and academics

H3b: The differences in quality rating between significant and non-significant manuscripts is larger for students compared to academics

H3c: The differences in quality rating between manuscripts with and without reporting error does not differ between students and academics

We explored the importance of a broader set of manuscript characteristics that could be relevant for the evaluation of manuscript quality. We formulated no a priori hypotheses for these characteristics but considered a set of research questions to compare different conditions. First, we studied which characteristics were considered relevant when assessing the quality and the grading of a thesis (RQ4) and when assessing the quality and the publishability of an article (RQ5). For academics, we studied the differences in the relevance of the different characteristics when assessing quality and when assessing publishability/grade for both theses (RQ6) and articles (RQ7), differences between the importance of characteristics when assessing quality of theses versus articles (RQ8), differences between the importance of characteristics when assessing the grade or publishability of theses versus articles (RQ9), and differences between the differences of the importance of characteristics when assessing quality versus grade/publishability of theses versus articles (RQ10). For students, we studied differences in the relevance they believe their supervisors assign to characteristics when assessing quality versus grade (RQ11). Finally, we studied differences in relevance of characteristics between students and academics when assessing quality of thesis (RQ12) and grades (RQ13). Fig 1 shows research questions 6–13 and their relations.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of Explorative Research Questions of the Survey Study on the Evaluation of Scientific Manuscripts.

Six different evaluations are made. Academics (top) evaluate both ‘quality’ and ‘grading’ of theses (left) and the ‘quality’ and ‘publishability’ of articles (right). Students (bottom) only evaluate the ‘quality’ and ‘grading’ of’theses’ (left). Arrows 6-9 and 11-13 refer to correlations between two evaluations. For instance, and more specifically, arrow 9 corresponds to the correlation between the evaluated importance of 29 manuscript characteristics for ‘grading theses by academics’ and the evaluated importance of these same manuscript characteristics for the ‘publishability of articles by academics’. Arrow 10 refers to a difference between two correlations. The preregistration lists all research questions verbally.

https://doi.org/10.1371/journal.pone.0355938.g001

Design

The pre-registered project (https://osf.io/cd3uw) consists of a vignette study and a survey. Conditions varied across six design factors. First, participants were either students or academics (factor: career phase). Second, participants were asked about the evaluation of either theses (academics and students) or articles submitted for publication (academics only) (factor: manuscript type). The third, fourth and fifth factors pertained to only the vignette part of the study. The text of the vignette was an abstract describing a study with either a small or a large sample size, main results were either statistically significant or not, and the vignette either contained a statistical reporting error or not. The sixth factor only pertained to the survey part of the study. Participants of the vignette were subsequently asked in the survey to evaluate the importance of a list of manuscript characteristics related to either (i) quality assessment of an article/thesis, or (ii) publishability of an article or a thesis grading (factor: quality assessment type). In total, there were 48 conditions (for academics: 2 manuscript type × 2 sample size × 2 statistical significance × 2 reporting error × 2 quality assessment type = 32. For students: 2 sample size × 2 statistical significance × 2 reporting error × 2 quality assessment type = 16).

Power analysis

To determine the sample size needed to evaluate the hypotheses, we conducted an a priori power analysis. Since we only formulated hypotheses for research questions 1–3, we based our power analysis on these hypotheses. Based on a fixed-effect ANOVA with 24 conditions and the effect of a factor with two levels (df = 1), desiring a power of .8 with α = .05, and assuming a small effect size f = .10, we needed at least 787 participants. Assuming 33 participants per condition, this meant 528 academics and 264 students.

Participants

Data were collected among psychology students and academics who had published within the field of psychology. Exclusion criteria for students were not studying psychology or having finished one’s education program. For academics, exclusion criteria were having no experience with grading theses (in thesis condition) or having no experience with peer reviewing an article (in article condition). PhD candidates were considered as academics rather than students. The sampling strategy between academics and students differed substantially, and we therefore describe their data collection procedures separately.

In November 2019, contact information of academics who published articles or editorial materials in psychology was extracted from Web of Science. E-mail addresses were automatically matched to the author names and academics received a personalized invitation to complete the survey. Three rounds of sampling were needed to generate enough responses. Data collection took place between February 2020 and April 2020. In these three rounds, a total of 11,555 authors and editors was contacted, and 1,239 of them started the survey (response rate of 10.7%). However, many of them did not complete the survey (and were removed from the dataset) or were excluded from the sample due to the exclusion criteria. A total of 687 academics provided responses that were included in the analyses (response rate 5.9%). Fig 2 provides an overview of the sampling procedure of academics. More detailed information can be found on OSF (https://osf.io/j2gpk/). Participants were randomly assigned to one of the 32 academics conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Overview of the Sampling Procedure Amongst Academics.

https://doi.org/10.1371/journal.pone.0355938.g002

The student survey was distributed via social media, as well as through the professional network of the primary researchers and their direct colleagues. An invitation to participate was placed on Twitter, as well as 51 international psychology student Facebook groups. Unfortunately, Facebook flagged the invitation as spam and removed all messages. In a second attempt, moderators from 49 large psychology student Facebook groups were asked to post a message with the survey invitation in their Facebook groups. 24 groups replied positively and either posted the message themselves or allowed us to post the message and pinned it to the top of the Facebook group so it would be noticeable. These Facebook groups had almost 65,000 members in total. The survey was also posted on two survey sharing Facebook groups, with an additional 40,000 international psychology students.

Despite the effort, response rate was very low amongst international psychology students. After 11 months, data collection was closed after 301 students participated in the survey. Participants were randomly allocated to the 16 student-conditions. Unfortunately, 188 of the 301 participants needed to be excluded from analysis, because they did not complete the survey (160), did not study psychology [20], or had recently graduated from their studies or worked already as a PhD candidate [8]. The final number of student responses included was 113. Due to both the low number of responses and the subsequent low statistical power to detect small effect sizes, all hypotheses and analyses that include students (Research questions 3, 11, 12 and 13, see Fig 1) should be considered exploratory instead of confirmatory.

Participant characteristics

Researchers and students included in our sample originated from all continents. Academics were in different stages of their career and worked in various research fields (see Table 1 for participant characteristics). Many researchers (256, 37.2%) indicated they did not work in any of the pre-specified research areas, even though these participants had published on topics within the pre-specified area of psychology. Deviating from our preregistration, we decided not to remove participants from other fields, thereby retaining the planned statistical power for our analyses (note: tests of hypotheses H1a-c to H3a-c excluding those participants yielded the same substantive conclusions, see https://osf.io/84ugb).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of Participating Researchers and Students.

https://doi.org/10.1371/journal.pone.0355938.t001

Ethical review and data management

This research project received an exemption for ethical review by the ethical review board of Tilburg University (ERB, EC-2019.EX135) due to anonymous data collection with minimal risk since the obtained information (a) cannot be traced directly or indirectly to the individual, and (b) does not harm or discomfort the individual in any way. Informed consents were also reviewed by the ERB. Informed consent was obtained by all participants at the start of the survey, by confirming to participate by clicking a button. The data management plan has been approved by the data representatives at Tilburg University. The data management plan describes data storage, open data plans, a pre-DPIA (Data protection Impact Assessment) and a description of GDPR agreements and compliance. Both are available from the preregistration on OSF (https://osf.io/cd3uw).

Procedure

All data were collected using Qualtrics. Participants first answered a set of general questions. Students were asked whether they studied psychology, where they studied (continent), in which phase of their study they were (bachelor/undergraduate, master/graduate, other, namely...), and whether they had ever written a thesis (bachelors’ and/or masters’ thesis). Academics in the thesis condition were asked if they had ever graded a thesis (bachelor/undergraduate level, master/graduate level, PhD level), and in the peer-reviewed manuscript condition they were asked whether they had ever peer reviewed an article submitted for publication. Furthermore, all academics were asked where they worked (continent), what position they held, and in which psychology research field they primarily worked. Categorization was based on the psychology subfields as distinguished in the Clarivate Analytics Master Journal List (https://mjl.clarivate.com). If participants indicated that they did not study psychology (student conditions), had never graded a thesis (academic-thesis condition), or had never peer reviewed a paper (academics-article condition), they were thanked for their participation and redirected to the end of the questionnaire.

Vignette study

Participants were asked to read an abstract of a manuscript that was submitted for publication (academics), or an abstract of a master’s thesis (students and academics). The text of the abstract was written by the principal investigators, and its face validity was assessed by a clinical psychologist in the field of personality psychology (see Box 1). Three characteristics of the abstract were manipulated; sample size (small or large; italics: N = 54 or N = 540), statistical significance of the main finding (yes or no; underlined: p = .005 or p = .6), and reporting error (yes or no; bold; p < .001 (incorrect) or p = .01 (correct)).

Based on the abstract, participants were asked to rate the quality of the master’s thesis/scientific manuscript on a 7-point scale ranging from very low quality to very high quality. Next, they were asked to name three aspects of this thesis/manuscript that were most relevant for their assessment of its quality.

Survey

The questionnaire asked participants how important different manuscript characteristics are when assessing the (i) quality and (ii) grading of a thesis (by students and academics), and the (iii) quality and (iv) publishability of an article (only by academics). The instructions of each condition are provided in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Formulation of Survey Instruction for all six Conditions.

https://doi.org/10.1371/journal.pone.0355938.t002

After these instructions, participants were asked to rate the importance of 29 manuscript characteristics (see Table 3) on a 7-point scale ranging from ‘Not at all important’ to ‘Of the utmost importance’. These 29 characteristics were related to different aspects of a manuscript: theory, design, conduct, (data) analysis, and presentation. All 29 characteristics were presented in a random order different for different participants.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Manuscript Characteristics Evaluated by the Participating Researchers and Students.

https://doi.org/10.1371/journal.pone.0355938.t003

Statistical analysis

Data were analyzed using R (version 4.1.1). In the preregistration, an analysis plan was described, and R-code was included to answer research questions 1–3, as well as R-code for a planned principal component analysis (PCA) that could be used to prepare data for research questions 4–13 (https://osf.io/fn5tu/). Research questions (RQ) 1–3 were answered using linear regression analyses, by testing the main effects of manuscript characteristics (sample size (H1a), statistical significance of the effect (H1b), reporting error (H1c)); by adding to the model and testing the three interactions manuscript characteristic × manuscript type (thesis vs. article) (H2a-H2c); and by adding and testing the three interactions manuscript characteristic × actor (student vs. academic (H3a-H3c) (see preregistration for the equations).

As the outcome variables are Likert scales with seven ordered response categories, we also analyzed the data using quantile (median) regression (Quantreg R-package, version 5.86, [53]) as a sensitivity analysis. For the confirmative analyses (vignette study, hypothesis 1–3), the alpha level was set to.05.

Since hypotheses 1c, 2c and 3c indicated no effect and null hypothesis significance testing cannot provide support for null hypotheses we evaluated these hypotheses using Bayesian statistics. We computed posterior model probabilities for four models (), assuming a zero (), small (), medium (), or large effect (), respectively, all having the same prior probability (uniform prior distribution). The relative posterior model probabilities of the effect size (zero, small, medium, large) were computed as

with L being the likelihood of the observed F-value, given a zero, small, medium or large effect, using the (non-central) F-distribution. The observed F-value is the F-statistic of the test comparing the regression model with the parameter to be estimated (a simple effect of ‘reporting error’, or the main effect in case the interaction was not statistically significant) to the regression model without that parameter. L is distributed as F(1,N-k-1,λ_i), with k the number of predictors in the model (k = 3 for H1c, k = 6 if the interaction is significant, k = 5 if the interaction is not included in the model for H2c and H3c), and non-centrality parameter λ_i = 0, 0.02×(N-k-1), 0.15×(N-k-1), 0.35×(N-k-1) for a zero, small, medium, large effect, respectively (see OSF, https://osf.io/sk5t7/, for R-code). We interpreted a relative posterior probability of at least .75 for a zero true effect-size (corresponding to a Bayes Factor of 3 or more) as evidence in favor of the null-hypothesis (i.e., a zero true effect-size).

The aspects the respondents named in the open-ended question that followed the vignette study (“Please name the three aspects of this manuscript that were most relevant for your assessment of its quality”) were analyzed using content analysis. The aspects named by participants were categorized by the principal investigator (HA). Categories emerged from the data itself, and specific categories with less than five mentions were put in the ‘other category’.

To analyze the importance of the 29 manuscript characteristics in the survey, a PCA was planned for each of the six survey conditions separately to reduce the dimensionality of the dataset if (i) the Kaiser–Meyer–Olkin (KMO) test indicated sufficiently correlated items (KMO > .6 in all six conditions) and at the same time (ii) items loaded similarly on the principal components across the six conditions. If either of these two criteria were not met, analyses would take place at the item-level and the four intended categories (theory, design, conduct and analysis, and presentation). More details can be found in the preregistration.

The statistical analysis of research questions 4–13 was exploratory and was not part of the preregistration. To investigate these research questions, we provide descriptive statistics and correlations between the importance ratings of either the new principal components (RQ 4–5), or their four intended categories (Table 3), of the conditions compared in research questions 6–13 (see also Fig 1). For these exploratory analyses, as a (partial) correction for multiple testing, the alpha level was equal to .01 instead of .05. This provides us with insight in the characteristics most relevant in the quality assessment, according to each group (academics vs students, articles vs theses, quality vs grading/publishability), and the correlations inform us to what extent these groups (dis)agree in their importance ratings.

Both the data and the codebook of the data can be found at osf.io/j2gpk.

Vignette study

The mean quality rating of the research manuscript abstract (Box 1) for all participants was . Students on average provided higher quality ratings than academics (4.42 versus 3.75, , and academics on average provided higher quality ratings to the thesis abstracts than to the scientific manuscript abstracts (3.96 versus 3.55, . Table 4 provides mean quality ratings, standard deviations, number of participants in each of the 18 conditions of the vignette study, and effect sizes (Cohen’s d) of differences between two conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Mean Quality Ratings of Scientific Manuscripts for all Conditions and Effect Sizes Comparing Conditions.

https://doi.org/10.1371/journal.pone.0355938.t004

Below each design factor (sample size, significance, and reporting error) in Table 4, the effect size of the design factor within a condition is provided. None of these factors influenced the quality ratings within a group of respondents and had generally small effect sizes. Effect sizes for small versus large sample size manuscripts differed from to . The effect sizes for manuscripts with statistically significant versus non-significant results differed from through . The presence versus absence of a reporting error resulted in effect sizes between and .

We conducted a linear regression analysis using all respondents (academics and students) to answer the three research questions of hypothesis 1. Sample size, statistical significance, and reporting error were entered as predictor variables to study their impact on the quality rating of the abstract. Only sample size (H1a) predicted quality rating (, suggesting that small sample sizes received lower quality ratings. Statistical significance (H1b; ( and reporting errors (H1c; ( did not predict quality rating. The omnibus test, however, indicated that the model with these three predictors together did not predict quality rating, with only 1% of the variance of quality rating explained (. Exploratively, we also tested the sample size × statistical significance interaction, but found no effect either (B = 0.17, t = 0.85, p = .398). For hypothesis 1c, relative Bayesian posterior model probabilities were.9996,.0004, < .0001 and <.0001 for null, small, medium, and large effects, respectively, indicating strong evidence in favor of H1c (BF₀₁ = 2,826): that is, there is no effect of reporting errors on the quality ratings of the abstract. Given the high response among academics, we also provide results for academics only: Sample size: B = −0.26, SE = 0.11, t = −2.44, p = .015. Statistical significance: B = 0.06, SE = 0.11, t = 0.56, p = .575, reporting error: B = −0.03, SE = 0.11, t = −0.249, p = .803. The model with the three predictors explained 0.9% of the variance (95% CI: [.00,.02], F(3,683) = 2.08, p = .10). Relative Bayesian Posterior Model probabilities (H1c): .9984, .0016, < .0001 and <.0001 for a null, small, medium, and large effect respectively (BF₀₁ = 629)

All 800 participants were asked to name three aspects they considered relevant for their evaluation of manuscript quality. The 800 × 3 = 2,400 named aspects were classified in 17 categories. Sometimes all three aspects named by the same respondent were related to one category (e.g., study design), and sometimes one aspect was related to multiple categories (e.g., ‘Clear description of methodology’ is related to both writing and study design). Therefore, the number of categorized answers was not equal to 2,400. The minimum number of differently categorized answers by a participant was 0 (due to incorrect understanding of the question), and the maximum number of different categories was 4. This resulted in a total of 1,976 categorized answers.

Table 5 shows the numbers of respondents naming an aspect related to each category, ranked by prevalence. Aspects related to sample size or power were mentioned by more than 25% of participants. Aspects related to statistical significance were named by only 14 respondents (1.75%) but 68 different respondents (8.5%) named the ‘results’ as an important aspect. Consequently, the maximum number of participants indicating that they were influenced by the strength of the results was 82 (10.25%). Statistical reporting error was mentioned by 13 participants (1.63%). The actual statistical reporting error was identified in only 3 cases (0.38%). Whereas sample size and statistical significance were likely to be commented upon in both the small/large sample size and the significant/non-significant result condition, mentioning (in)correct reporting was only likely for the 393 participants (49.13% of total sample) in the reporting error condition.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Prevalence of Categories Indicated as Relevant for Assessing the Quality of the Abstract of Scientific Manuscripts and Students’ Theses (Text in Box 1), Provided by Students and Academics.

https://doi.org/10.1371/journal.pone.0355938.t005

We ran a second regression analysis to investigate the interaction between the manipulated characteristics and manuscript type (thesis versus article, H2). Since students only received the thesis condition, only academics were included in this analysis. The regression model with only the four main predictors (sample size, significance, reporting error and manuscript type) explained 2.95% of the variance (. The regression model adding the three two-way interaction terms between sample size, significance, and manuscript type (, shown on the left panel of Table 6, did not significantly improve the model, it explained only 0.04% more variance . Because none of the interactions added to the explanation of quality rating, we cannot support hypothesis 2a or 2b; that is, sample size and statistical significance did not differentially affect quality rating of thesis versus articles for academics. To answer hypothesis 2c, Bayesian relative posterior model probabilities showed that an interaction effect of reporting error and manuscript type is not likely, confirming hypothesis 2c (Bayesian Posterior model probabilities: no effect: .9989, small effect: .0011, medium effect: < .0001, large effect: < .0001; BF₀₁ = 869).

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Results of Regression Analyses Predicting Quality Ratings of Scientific Manuscript with Interactions of Manuscript Type and Career Phase.

https://doi.org/10.1371/journal.pone.0355938.t006

Our third regression analysis investigated the interaction effects of the three manipulated design factors and career phase (student versus academic, H3; Sample×CP, Sig × CP, Error×CP). The regression model with only main effects explained 3.11% of the variance in quality ratings , whereas the model containing both main effects and interactions explained 3.41% of variance in the quality ratings , 0.3% more . Interactions did not improve the prediction, see right the panel of Table 6. That is, neither sample size (H3a) nor statistical significance (H3b) showed an interaction with career phase in the quality ratings of theses. Bayesian posterior model probabilities supported the hypothesis that there was no interaction between reporting errors and career phase (H3c) since the null model received a probability of.9166 (small effect:.0834, medium effect: < .0001, large effect: < .0001; BF₀₁ = 11).

We ran a preregistered exploratory regression analysis on the full model predicting quality ratings of the manuscript, including all interaction terms up to four-way interactions. This full model explained 7.2% of the variance , but of the predictors only sample size had an effect . This full model was not beneficial compared to the model with only main effects . A quantile regression as robustness check also showed no effects for H1 (neither for both academics and students, nor for academics only), but it did show an interaction effect between sample size and type of manuscript (H2a, p = .004). No effects for hypothesis 3 were found. Detailed results of the quantile regression can be found on OSF (https://osf.io/zpk67/). To summarize, we only found a weak effect of sample size on the quality rating of abstracts.

Survey Study

As the items in the survey did not correlate sufficiently (KMO-indices varying from 0.28 to 0.91) in all six conditions (Academics: Thesis quality, Thesis grading, Article quality, Article publishability. Students: Thesis quality, Thesis grading), we analyzed the data at the item-level for each condition separately to answer research questions 4 and 5 (see Table 7 and Fig 3). Table 7 shows the mean rating of each manuscript characteristic in each of the six conditions, as well as the average rating of each category (theory, design, conduct and analysis and presentation). Fig 3 also shows the importance rating for all 29 manuscript characteristics for all six conditions. The order of average importance of the four categories was the same for all six conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Mean Importance Ratings [1–7] of Manuscript Characteristics and Categories for Quality Assessment, in all Six Conditions (Students-Thesis, Quality & Grading; Academics-Thesis, Quality & Grading; Academics-Article, Quality & Publishability) (Standard Deviations Between Brackets).

https://doi.org/10.1371/journal.pone.0355938.t007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Mean Ratings of all Importance Ratings of Manuscript Characteristics for Quality Assessment for all six Conditions (see legend).

https://doi.org/10.1371/journal.pone.0355938.g003

On average, theory ranked highest on importance, closely followed by analysis and presentation (0.2 SD lower), with even lower average scores for design (−0.5 SD) and conduct (−0.9 SD). However, as characteristics within domains varied substantially, it is more meaningful to list those that were considered more important. These were (with average scores above 6): T3 (‘explicitly stating the hypotheses to be tested’), T5 (‘clearly describing the study objective’), D1 (‘using an appropriate study design’), A1 (‘adequately describing all used statistical procedures’), A2 (‘using the appropriate statistical analyses’), A5 (‘clearly presenting the statistical results’), A8 (‘reporting the statistical results without errors’), and A12 (‘drawing the correct conclusion from the statistical analysis’). Considered less important (average scores below 4) were: T4 (‘using a large number of literature references’), D5 (‘using a complex design’), C3 (‘applying complex statistical analyses’), A9 (‘large effect sizes’), A10 (‘effect in hypothesized direction’), and A11 (‘statistical significance’).

We also listed characteristics related to the characteristics we manipulated in the vignette part of this study. For sample size, these were characteristics D4 (large sample size) and C2 (high statistical power). In all six conditions, these characteristics were rated as moderately important for quality assessment. The characteristics related to statistical significance (A9-A11; large effect sizes, effect in hypothesized direction, statistical significance) were rated as having little importance, whereas the characteristic related to reporting error (A8, ‘reporting the statistical results without errors’) was of high importance in all conditions.

Table 8 shows the correlations between the mean ratings for the conditions we compared in research questions 6–13 (see Fig 1), to establish the pair-wise association of all 29 characteristics between pairs of conditions. Correlations were high (r > .9), indicating strong association between mean importance ratings, which is also confirmed by visual inspection of Fig 3. Participants in all conditions seemed to agree with respect to their importance ratings of the different characteristics, regardless of manuscript type (thesis or submitted article), type of evaluation (quality or grading/publishability), and regardless of career phase (students or academics). Correlations between on the one hand thesis quality and thesis grading and on the other hand between article quality and article publishability were all very high (>.93, most > .99) and hence not distinguishable (RQ10). Within-category correlations were generally less strong for the research conduct category.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Correlations Between Mean Importance Ratings Within Categories (Theory, Design, Conduct, Analyses & Presentation) and Across Categories (Total).

https://doi.org/10.1371/journal.pone.0355938.t008

Although correlations were very high, average importance ratings of manuscript characteristics sometimes differed across conditions, as can be seen in Fig 3 as well as Table 7. Focusing on differences of at least 0.5 standard deviation (, using a large number of literature references (T4) was considered more important by students for the quality and grading of theses than by academics for the quality and publishability of articles. Interestingly, preregistration (D6) was rated by students as more important for the quality of a thesis than by academics for anything (all four conditions), and as more important by students for the grading of theses than by academics for the quality of an article.

Limitations and future research

Because we were unable to sample enough students, thus resulting in a lower statistical power, all analyses, hypotheses, and conclusions including a comparison with students should be considered exploratory. We aimed at achieving a power of .80 but only achieved a power of .18 for a small effect (f = .10) for those tests that only include students. Our results strongly suggest that students evaluate the quality of manuscript characteristics like researchers, but larger sample sizes are recommended to confirm these exploratory results. However, we also should be careful drawing general conclusions on the similarity of the evaluation of researchers and students, as researchers and students were recruited with different methods that may have resulted in sampling bias. The academics sample enabled a power of .74 to detect a small effect, whereas the analyses including all 800 respondents had a power of .81.

The samples of academics and students may be biased, possibly due to students and academics participating who are interested above average in meta-science and concerned with good research practices. The abstract we used in the vignette may not be a valid proxy for a scientific article. Readers often use an abstract alone to assess the value of an article [27], and therefore future research might focus on the issue of whether quality assessment of an article based on an abstract alone is a good predictor of the quality of the whole manuscript.

Our survey study might have lower ecological validity than studies with an observational or experimental setup. Unlike participants in [34] and [45], our participants were aware that they were participating in a study. The study characteristics academics find important to the manuscript’s quality are not necessarily the characteristics that influence the assessment the most when peer reviewing a manuscript. For instance, complex design and complex analysis were considered least relevant for manuscript quality in our survey, but Armstrong [42] suggested that reviewers are influenced by these false cues. In our vignette study, we manipulated only three aspects and presented participants with just an abstract. An experimental setup with a manipulation of all 29 aspects we investigated in our survey does not seem feasible. Future research could investigate the impact of some of the characteristics that are considered either irrelevant (e.g., complex design and complex analysis) or highly relevant (e.g., (in)correct conclusions, (in)appropriate study design, describing study objectives, and using (in)appropriate statistical analysis). The aspects that were considered highly relevant are also aspects about which reviewers often disagree. For example, the assessment of the appropriateness of the study design and the appropriateness of the analysis seem to be subject to judgment.

To gain more insight in the characteristics that increase the publishability of a manuscript, one might investigate whether the literature is homogeneous on aspects considered highly relevant (e.g., clearly describing the study objective) and heterogeneous on aspects with little relevance (e.g., the number of references). This could provide an indication that the highly relevant characteristics are a minimum requirement, whereas characteristics with little relevance are indeed arbitrary to the publishability of an article.

Finally, it would be interesting to know how our results relate to results from other fields than psychology, and how the importance of characteristics might change over time. For example, the popularity of open science and sharing materials is increasing and might in part explain the relatively large variance in the importance rating of this characteristic amongst academics.