3.1 Ethics statement

The experiment was reviewed and approved as exempt research by the Microsoft Research IRB, and the data collected were analysed anonymously.

4.1 NeurIPS 2021 conference

• Total number of papers submitted to the conference: 9,034.
• Total number of unique authors who submitted papers to the conference: 23,882.
• Total number of author-paper pairs: 37,100. (Only authors with a profile on the conference management platform (OpenReview.net) could participate in the experiment, yielding 34,713 eligible author-paper pairs).
• Percentage of submitted papers that were eventually accepted to the conference: 25.8%.

We now move on to discuss the responses to the three questions.

4.2 “Acceptance probability” question

• Number of responses: 9,907 (26.7% of author-paper pairs).
• Number of papers with at least one response: 6,278 (69.5%).

4.3 “Paper quality ranking” question

• Number of authors with more than one submission: 6,237.
• Total number of author-paper pairs for these authors: 19,455.
• Number of “rank” responses received (out of 19,455): 6,908 (35.5% response rate).

4.4 “Change of perception” question

• Number of papers remaining after reviews were released (as some were rejected/withdrawn): 8,765
• Number of author-paper pairs remaining: 36,103.
• Number of responses: 4,435 (12.3% response rate).

5.1 Calibration in prediction of acceptance

We begin by looking at the responses to the acceptance probability question, and comparing it with actual acceptance decisions. In Fig 1a, we plot the relation between responses given by authors to the question and the actual acceptance rates for these papers. Here, the blue dots represent responses with at least 50 samples, which together comprise 94% of all responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Author’s predictions on the probability of acceptance of their papers.

(a) Plot of authors’ predictions of chances of acceptance of the paper versus the actual acceptance rates for each response. The diagonal line represents perfect calibration and the (blue) dots represent authors’ responses. (b) Plots of authors’ predictions, for papers that were eventually accepted (thin yellow line) and rejected (thick blue line). The x-axis represents the fraction of responses with predicted percent chance greater or equal to the corresponding value on the y-axis. In other words, the x-axis is the fraction of responses with prediction greater than or equal to the corresponding y value.

https://doi.org/10.1371/journal.pone.0300710.g001

We find that there is a nearly three-fold overestimation overall: The median of the acceptance probabilities estimated by the respondents is 70% and the mean is 67.7%. In comparison, we had primed respondents by mentioning that the acceptance rate in the past four years was about 21%. (The acceptance rate at NeurIPS 2021 ended up being 25.8%.) The fact that participants over-predict aligns with studies in other settings [41, 42] that also find overconfidence effects. Also observe in Fig 1a that interestingly, the authors’ predictions track perfect calibration quite well for responses up to 35%, whereas responses greater than (to the right of) 35% are uncorrelated with the actual acceptance rate.

In Fig 1b, we sort the responses in descending order (on the x axis) and plot the values of these responses (y axis). We make separate plots for papers that were eventually accepted and those that were eventually rejected. We see that these two plots track each other quite closely, with papers that were eventually accepted having slightly higher predictions. We also observe indications of over estimation here—more than 5% of responses predict a 100% chance of their paper getting accepted, about 50% responses predict chances of 75% or higher, whereas fewer than 15% of responses provide a prediction smaller than 40%.

5.2 Role of demographics

Next we look at the role of demographics in calibration. For this we now define the calibration error in prediction of acceptance by any author. First, based on Section 5.1 and Fig 1a, we note that responses were on average overly confident, that is the predicted probability of acceptance was higher than the observed rate of acceptance. Further, we also observe that within each demographic-based subgroup, authors on average predicted a higher acceptance probability of their submission as compared to the acceptance rate within that subgroup. We thus know the direction of miscalibration of each subgroup.

We measure the calibration error of any subgroup in terms of the mean Brier score (i.e., squared loss). The Brier score [43] is a strictly proper scoring rule that measures the accuracy of probabilistic predictions: Given a prediction (value in the interval [0, 1] representing the probability of acceptance) and the outcome (accept = 1, reject = 0), the Brier score equals the square of the difference between the prediction and the outcome. To get a sense of the value of the Brier score, if 25% of the papers are accepted and all respondents provide a prediction of 0.25, then the Brier score equals 0.1875; if all respondents provide a prediction of 0.8 then the Brier score equals 0.49. In our analysis, we had decided in advance to execute statistical tests comparing calibration of male and female authors and of reviewers and meta-reviewers; we had decided to not compare the remaining subgroups due to possibility of high heterogeneity among them. We provide the main details about our analysis in this subsection, and provide additional details in Appendix B.

5.2.1 Gender.

We compute the average calibration error for gender subgroups using the gender data retrieved from participants’ Open Review profiles. Our approach accounts for potential confounding by seniority and geographic region in the analysis for calibration error across gender groups. The calculation is described in detail in Appendix B. See Fig 2a for the average calibration error for the “male”, “female” and “not reported” subgroups, where “not reported” comprises of authors who did not provide their gender information in their Open Review profile. We do not report statistics for other gender subgroups, which are very small, to preserve respondent privacy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparing authors’ calibration error (Brier score) in prediction of acceptance across different subgroups based on gender and seniority level.

The error bars indicate 95% confidence intervals, obtained via bootstrapping. (a) Gender-based grouping. (b) Seniority-based grouping.

https://doi.org/10.1371/journal.pone.0300710.g002

In many fields of science there is research showing that there exists a confidence gap between female and male participants [44, 45], where men are generally found to overestimate and women underestimate. In NeurIPS 2021, we test for significance of difference in calibration error by male authors and female authors. To test this hypothesis, we consider the test statistic of the difference in calibration errors between female authors and male authors and conduct a two-sided test. We find that there is a statistically significant difference (p = 0.0012) at level 0.05. However, note that the effect size—the difference in the calibration errors between female authors (0.44) and male authors (0.40)—is small (0.04).

5.3 Prediction of acceptance vs. perceived scientific contribution

We investigate the consistency between the predictions by authors about the acceptance of their papers and the scientific contribution (paper quality) of those papers as perceived by the authors. There were a total of 6,024 pairs of papers by the same author where the author provided their responses for both questions for both papers. We break down the responses in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comparing authors’ (relative) predicted acceptance probability and perceived paper quality for any pair of papers authored by them.

This plot is based on 6,024 such responses. In particular, the first two bars enumerate the amount of agreement and disagreement respectively, among responses of any author that had a strict ranking between the two papers for both questions.

https://doi.org/10.1371/journal.pone.0300710.g003

Of particular interest are the first two bars in Fig 3 that comprise responses where the same author provided a strict ranking of two papers they authored in terms of their perceived quality, and also gave distinct probabilities of acceptance for the two papers. Among these responses, we find that there is a significant amount of agreement—the two rankings agree in 92.6% of responses. However, there is a noticeable 7.4% of responses where the authors thought that the peer review process is more likely to reject the better of their two papers.

5.4 Agreements between co-authors, and between authors and peer-review decisions

We first look at author-provided rankings of their perception of the scientific contribution (paper quality) of multiple papers they authored. We compare these rankings with the outcomes (accept or reject) of the peer-review process. We show the results in Fig 4. In particular, observe that among the situations where the decisions for the two papers were different and the author-provided ranking was strict (first two bars of Fig 4), authors’ rankings disagreed with the decision 34% of the time. (An analysis comparing the ranking of papers by authors’ perceived acceptance probabilities and the final decisions yields results very similar to that in Fig 4.)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparing authors’ ranking of their perceived scientific contribution (paper quality) and the decisions from the peer-review process.

This plot is based on 10,171 such responses. In particular, the first two bars enumerate the agreement and disagreement when the author-provided ranking is strict and where one of the papers is accepted and the other is rejected.

https://doi.org/10.1371/journal.pone.0300710.g004

We now compute agreements between co-authors in terms of their perceived scientific contribution (paper quality) of a pair of jointly-authored papers. We show the results in Fig 5. Observe that interestingly, among the pairs where both authors gave a strict ranking, they disagreed 32% of the time—approximately the same level of disagreement as we saw between the authors and reviewers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Comparing co-authors’ rankings of their perceived scientific contribution (paper quality) of a pair of papers that both have authored.

This plot is based on 1,357 such responses. In particular, the first two bars enumerate the agreement and disagreement of co-authors when they both provide strict rankings of their papers.

https://doi.org/10.1371/journal.pone.0300710.g005

This high amount of disagreement between co-authors about the scientific contribution of their jointly authored papers has some implications for research on peer review. Many models of peer review [21, 38, 46–50] assume existence of some “true quality” of each paper. This result raises questions about such an assumption—if there were such a true quality, then it is perhaps the authors who would know them well at least in a relative sense, but as we saw above, authors do not seem to agree. In a recent work, [51] proposes a novel idea of asking each author to submit a ranking of their submitted papers. Under the assumption that this author-reported ranking is a gold standard, [51] then proposes to modify the review scores to align with this reported ranking. However, our observation that co-authors have a high disagreement about this ranking violates the gold standard assumption that underlies this proposal.

5.5 Change of perception

We now analyze the responses to the question posed to authors in the second phase of the experiment on whether the review process changed their perception of their own paper(s). We plot the results in Fig 6. Given significant non-response bias in this phase with respect to acceptance decisions (Section 4), we also separately plot the responses pertaining to accepted and rejected papers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Change in authors’ perceptions of their own papers after seeing the reviews.

The five bars in each plot represent the five options: much more positive (“++”), slightly more positive (“+”), did not change (“0”), slightly more negative (“-”), much more negative (“- -”). The three subfigures depict responses pertaining to all, accepted, and rejected papers, and are based on 4435, 1767, and 2668 such responses respectively. (a) All papers. (b) Accepted papers. (c) Rejected papers.

https://doi.org/10.1371/journal.pone.0300710.g006

We observe that among both accepted and rejected papers, about 50% of the responses indicated a change in their perceived opinion about their own papers. Furthermore, even among rejected papers, over 30% of responses mentioned that the reviews made their perception more positive. While past studies [11, 17–19] document whether the review process helps improve the paper, the results in Fig 6 shows that it also results in a change of perception of authors about their papers about half the time.

A More details about the experiment

In this section, we provide details about the experiment, augmenting the details provided in Section 3. First, we focus on the release timeline of the surveys. Then we provide details about the content of the surveys, including the instructions provided.

Timeline. Phase 1 of the experiment was conducted soon after the paper submission deadline in order to obtain authors’ perceptions of their submitted papers while the papers were still fresh on their minds. The paper submission deadline was on May 28, 2021. The Phase 1 survey was released shortly after, on June 1. Authors were invited to participate in the survey through June 11, after which the survey was closed. To increase participation in the survey, the program chairs sent a reminder email about the experiment on June 9.

Phase 2 of the experiment aimed at understanding the change in authors’ perception of their papers after receiving the initial reviews. The authors received the initial set of reviews on August 3, 2021 and were able to provide their rebuttal (response to the initial reviews) any time until August 10. We invited authors to participate in the Phase 2 survey on August 12. The peer review process was concluded on September 28, 2021, with the announcement of final decisions.

Instructions. In both the Phase 1 and Phase 2 surveys, authors were provided information regarding the privacy and confidentiality of their survey responses. They were informed that during the review process, only the authors themselves could view their responses, in addition to the administrators of OpenReview.net (the conference management platform used by NeurIPS 2021). It was emphasised that authors’ responses could not affect the outcome of the review process and that the responses would not be visible to their co-authors, reviewers, area chairs, or senior area chairs at any point of time. Regarding the analyses and following dissemination of the findings from the experiment, the survey mentioned that, “After the review process, the survey responses will be made available to the NeurIPS 2021 program chairs and Workflow chairs for statistical analyses. Any information shared publicly will be anonymized and only reported in an aggregated manner that protects your identities.” For the purposes of analysis, responses and profiles were accessed algorithmically via the OpenReview api. Further, authors were also told, “To allow authors to freely provide their opinions and keep samples as independent as possible, please do not discuss your answers to these survey questions with other NeurIPS 2021 authors (including your co-authors), or ask others about their responses.”

In Phase 1 of the experiment, we asked authors with multiple submissions to rank their submissions. The instructions for providing ranking were as follows: “Rank your papers in terms of your own perception of their scientific contributions to the NeurIPS community, if published in their current form. Rank 1 indicates the paper with the greatest scientific contribution; ties are allowed, but please use them sparingly. In the table entry for each submission below, there is a pull-down menu called “Paper Ranking.” Please click on it and specify the rank for that submission.”

Finally, among the 6237 authors with multiple submissions, 32 authors (0.5%) provided a ranking for only one of their submissions. We exclude these responses from the analysis of the ranking.

B More details about demographic analysis

In this section, we provide details about the analyses we conduct to test for significant difference in calibration error across demographic groups in Section 5.2. To describe the analysis, we first define some notation. Let n denote the total number of responses obtained in Phase 1 of our experiment. We will use i as an index over responses, where each response pertains to a single author-paper pair. For response i, let p_i ∈ [0, 1] be the acceptance probability indicated by the author. The observed outcome of the associated paper is a binary indicator, denoted by y_i ∈ {0, 1}, where y_i = 1 if the paper is accepted and y_i = 0 if it is rejected. The self-reported gender of the associated author is denoted by g_i ∈ G ≔ {Female, Male, Other, Unspecified}. Note that there are responses where the associated authors did not provide a gender in their Open Review profile. All authors’ seniority is classified into three types based on their reviewing participation, denoted by s_i ∈ S ≔ {Meta-reviewer, Reviewer, Neither}.

Finally, we include the geographical region associated with the author, denoted by r_i. To assign a geographical region to each author, we use the institutional domain of the author’s primary affiliation. We classify the geographical regions using the geographical region division provided by the United Nations Statistics Division (UNSD). Within their division of regions, we further break each region with more than 100 responses in our survey into sub-regions listed by UNSD. This yields the following set of regions denoted by R ≔ {Africa, North America, Latin America and the Caribbean, Central Asia, Eastern Asia, South-eastern Asia, Southern Asia, Western Asia, Eastern Europe, Northern Europe, Southern Europe, Western Europe, Oceania}.

To measure accuracy, we use the Brier score (i.e., squared loss). For response i, the Brier score is given by (y_i − p_i)². With this notation, we define the average calibration error for a gender-based subgroup. To account for confounding by authors’ seniority and geographical region, we bin all responses based on their corresponding seniority and geographical region, and compute their prevalence rate in the population. This gives the weight to be assigned to each response to compute the average calibration error for gender-based subgroups as (1) where is the indicator function. Using this definition of calibration error of a gender subgroup, we derive 95% confidence intervals using bootstrapping [53]. We now move on to our hypothesis comparing miscalibration between male authors and female authors. Formally, in terms of (1), the hypothesis is stated as: To test this hypothesis, we conduct a permutation test to obtain its significance (p-value). In the permutation test, we permute our data within each demographic subgroup of seniority and geographical region. From the permutation test, we obtain a p-value of 0.0006. To account for multiple testing we use the Benjamini-Hochberg procedure [54], which gives a final p-value of 0.0012.

Similarly, we compute the average calibration error for seniority-based subgroups, while accounting for confounding by gender and geographical region. In this analysis, we filter out the responses by authors who did not report their gender. Since the set of authors who did not report their gender may be a heterogeneous set, including this set in the analysis for seniority will violate the exchangeability assumption of the permutation test. Thus, the total number of responses considered in the seniority analysis, denoted by n_g∈G, is given by . With this, the average calibration error corresponding to each seniority level, for s ∈ S, is given by (2) We use bootstrapping to compute 95% confidence intervals. Further, we conduct a permutation test to compare the miscalibration by meta-reviewers (ACs and SACs) and other reviewers. This hypothesis is stated as where M_{meta-reviewer} and M_reviewer are as defined in (2). The permutation test yields a p-value of 0.055. Accounting for multiple testing using the Benjamini-Hochberg procedure does not alter the p-value.