Introduction

Public research grants are among the most institutionalized methods of investing in science [1], but determining who deserves funding is a complex task. While merit is certainly an important criterion [2], economic, social, and political concerns also influence funding decisions. Scientists often value autonomy and believe that their internal rankings are the best predictors of future scientific and societal impact [2, 3]. However, because science frequently relies on public funds, external actors (e.g., funding bodies, science administrators) introduce their own perspectives and agendas. The ways in which scientists and administrators negotiate and reconcile their demands present an intriguing sociological question.

A typical grant competition consists of two stages: first, grant applications are reviewed and scored by subject matter experts; second, selection panels decide whether a project deserves funding. Research identifies many factors influencing positive reviews or grant awards, including scientific merit, evaluative cultures, administrative discretion, previous success, and, unfortunately, gender and race biases [2, 4–9]. However, fewer studies have explicitly addressed the direct interplay between review scores and funding decisions [4, 5].

We contribute to the discussion by analyzing the 2017 grant competition held by the Ministry of Education and Science of Kazakhstan that followed this typical two-stage design. Our data consist of 4,488 grant applications with extensive information on principle investigators’ academic track records, institutional affiliations, and demographics. We specifically analyze the effect that peer review scores had on the final panel decisions across seven possible funding sections.

The present study adds to the literature in two respects. First, it extends the “geography of data.” Most research on grant distribution comes from the US or Western Europe, contexts that arguably share some institutional similarities. A “developing” country, however, may present a comparative case that reveals different “logics” of grant distribution. In Kazakhstan, for example, official state discourse routinely urges scientific communities to meet the “doing science” standards of “advanced” societies. From the official perspective, a hallmark of those standards is commercialization¹. This “catching-up” mindset can create specific institutional pressures on national science funding bodies, mapping of which may offer insights into the global division of labor in knowledge production systems.

The second contribution comes from analyzing the interaction between peer reviews and panelists’ decisions. By design, peer reviewers were predominantly international scholars—subject matter experts commissioned to evaluate the scientific potential of applications (e.g., novelty, feasibility). In contrast, panelists were representatives of local academic and industrial institutions, forming the local academic establishment. The final decision rested with the panels, which considered review scores but had significant discretion. We find that, in some domains, review scores were more predictive of success than in others. We argue that in theoretical disciplines, panelists and experts share a common understanding of what constitutes “good” science. In contrast, panelists are less compliant in areas of applied science, where expertise is more tacit. This discrepancy arises, we argue, from the varying cognitive distance between experts and administrators [10].

The article is organized as follows: in the next section, we review the literature and sketch the grant funding system in Kazakhstan. Then we present the data and basic descriptive statistics. A 3-step data analysis follows: (a) we model peer-review scores using OLS regression, (b) we model the probability of getting funded using logistic regression; and (c) we cross-validate and compare logistic models with some fine-tuned random forest models. In the last section, we articulate the main takeaways and discuss the limitations.

Background and theory

Grant awarding bodies face a selection problem: determining which projects deserve funding [1]. One common, if not normative, solution is to let the community decide. From this perspective, scientists tend to see their peers as suitable juries because internal rankings within a discipline are considered a reliable measure of future scientific output. Peer recognition is seen as a means to ensure valid scientific knowledge because radical openness to peer criticism [11] and “organized skepticism” [12] help filter out unwarranted or dubious claims. Pragmatically, merit-based selection is also appealing to scientists because it implies field autonomy [3, 13]. Indeed, research indicates that selection committees, at least in principle, take meritocracy seriously and believe that it should guide their decisions [2].

One common way to infer scientific merit is peer review, a process in which grant applications are scored by multiple, often anonymous, subject matter experts [1]. The practice has long been institutionalized in sciences and almost all large funding bodies (e.g. NIH, European Research Council) base their decisions on some sort of peer review basis [14].

In the meantime, peer review has a long history of criticism [15, 16]. Some studies show that positive peer reviews do not guarantee measurable scientific impact in publications and citations [17]. Others raise concerns that since one bad review can be enough to block an application, peer reviews prioritize safe, conventional avenues of research at the expense of high-risk, high-reward proposals [18]. Relatedly, commentators point out that peer reviews lack consistency: reviewers often disagree on scientific merit and weight of evaluation criteria, and inter-reviewers consistency is typically low [14, 19–22]. The way the information on projects is presented also seems to sway reviewer scores as the NSF experiment suggests [23].

Partly, peer reviews can appear inconsistent due to factors such as reviewer fatigue, vague evaluation criteria (e.g., significance or novelty), or a lack of reviewing experience [14, 24]. However, disagreements between reviewers also stem from the inherent ambiguity of merit. Scientific communities may agree on what constitutes quality science and who among them comes closest to doing it, yet the literature indicates that consensus is not always reached. Many scientific fields have controversial topics that split communities into what are often called schools of thought [12, 25]. While the degree of (perceived) fragmentation vary across disciplines [25–27], sociological studies of scientific knowledge show that the nature of scientific fact can be, and often is, contested even in the most stable of hard sciences like physics or chemistry [28–31].

The realization that scientists occasionally disagree even on basic principles led some to suggest that peer reviews should be used only to filter out “poor” proposals, while the final decision should be based on a closer, consensual panel discussion. And indeed, many grant funding agencies follow two-step selection procedures where, after the initial anonymous peer review, panels of experts collegially reach final decisions (e.g., NIH). Panels tend to be more diverse than peers: they may contain outsiders to a discipline and supposedly offer a more holistic view of a project’s merit. For example, panelists with industry experience may better understand a project’s economic impact [1].

Empirical studies, however, show that open panels do not eliminate inconsistencies and still face the same basic challenges as peer reviews. Variation in success rates suggests that panelists have cognitive and organizational biases; and studies into the discourse of decision-making show that consensus is often tentative, situational, and guided by heuristics, personal tastes, and strategic considerations besides formal criteria of merit [2].

In part, increasing uncertainty is the obverse of a holistic perspective. Funding bodies often have to go beyond merit alone and reckon with societal, political, and economic considerations. This is visible , for example, in ensuring equal funding chances for women and men, particularly in STEM research [4]. With this in mind, many funding bodies (e.g. NIH) explicitly retain discretion in decision-making, so the administrators can fund a project “out-of-order” if it aligns with their agenda, or compensate for institutional barriers that some social groups may face, as, for example, Bol and colleagues show in the case of the Netherlands’ grant funding system [4].

This brings us to other demographic and institutional factors that may correlate with peer reviews or success probability, net of scientific merit. Studies identify a wide range of such sources: gender, institutional affiliation, educational background, co-publication record, grantsmanship, race, and ethnicity [4–9]. For example, affiliation and academic pedigree can be interpreted by reviewers and panelists as additional indicators of merit or be taken into strategic consideration if, for example, reviewers or panelists share affiliations with the applicant. In the slightly different context of manuscript peer review, Teplitsky and colleagues find that review scores correlate with the closeness in the co-authorship network [25], suggesting that reasons for favorable reviews and decisions can be a result of sharing the same epistemic cultures [32] or belonging to the same school of thought [12].

Moreover, composing a successful application, or grantsmanship, is a separate skill that requires experience and understanding of the selection process. Previous success, then, can easily accumulate, creating the so-called “Matthew effect” [6, 12]. Finally, it is crucial to recognize that non-meritocratic factors, such as an applicant’s gender, ethnicity, or race, can significantly impact the decision-making process. For example, gender often serves as a primary characteristic for homophily in scientific collaborations, meaning that individuals tend to form partnerships and networks with those of the same gender. As a consequence, not only collaboration networks but also citation networks show systematic gender homophily [33, 34]. Research also shows systematic penalties for women in both hiring and grant funding [4, 8]. There is also evidence that minority groups can have inexplicably lower success rates, net of other merit and institutional factors, as Ginther and colleagues showed in the NIH grant distribution for Asian and Black investigators [7].

The upshot of this section of the literature review is that both peer reviews and panel decisions are shaped not only by the scientific validity, significance, or novelty of a proposal, but also by a battery of institutional, cognitive, and even affectual biases of referees, complicated further by administrative discretion. The selection process is asymmetric, and while decision-makers take peer reviews into account, they have considerable leeway in their final decisions. As we mentioned earlier, panels look beyond scientific validity or novelty, so peer reviews, in principle, should account only for a part of variation in success rates [6]. A question, however, remains what can mediate the accord between peer reviews and panel evaluations?

From a normative perspective, peer reviewing and panel evaluation are essentially forward projections of future successes [25]. Both peers and panelists attempt to assess what impact a scientific work will have. The aspects of impact they are scoping, however, differ. Very crudely, peers predict success in the community of scientists (e.g., citations), while panelists predict, among other things, societal impact. For example, when choosing between equally quality-assuring projects in astro- and geophysics, panelists may find the latter easier to justify on the grounds of public benefit. In this regard, a study of the Netherlands’ funding program reveals an interesting, though indirect, insight. Bol and colleagues [6] show that applications “around the funding cutoff” diverge over time in the amount of grant money secured but show no significant difference in the number of citations received. So grants, in that case, begot more grants, but not citations.

The implication for our context is that panelists, when not having clear excellence signals (review scores are about the same), may try to reduce uncertainty by looking at an investigator’s track record. Extending this line of logic, we might argue that any uncertainty makes panelists either infer “grantworthiness” from other proxy measures or take advantage of uncertainty by rewarding themselves (self-service) or colleagues (organizational proximity) [35, 36]. Additionally, uncertainty allows for biases to enter into consideration. For example, Bagues and colleagues argue that gender starts playing a role only in the absence of other merit signals [8].

The situation above applies to a small subset of applications in a narrow window around the “cutoff threshold.” However, a signaling framework allows extending the scope to the whole distribution of scores. We may ask to what degree, in principle, external experts are authoritative for panelists? This may depend on the extent panelists can understand technical evaluation and where they see themselves within a discipline’s internal hierarchy. In other words, how big is the cognitive distance between experts and administrators, and what do administrators think of their own technical expertise?

Scientific disciplines vary in the extent to which they agree on preferred endeavors, theories, methods, or tools. Some disciplines feature institutional structures that impose a higher degree of homogeneity on the education and training required for students to begin their scientific careers [37, 38]. Others disciplines exhibit greater diversity in their epistemic styles [2]. We argue that the level of epistemic diversity may correlate with the stability of evaluation criteria. Specifically, disciplines with more homogenous epistemic styles may tend to have more fixed sets of criteria for distributing resources (e.g., grants, academic positions). This suggests that in fields with low epistemic diversity, science managers may be more inclined to rely on external experts, whose authority helps reduce uncertainty (and potentially shift responsibility). Conversely, disciplines characterized by more diverse epistemic cultures often, by having more flexible evaluation criteria, allow administrators to exercise greater discretion in their decisions.

In our context, it is important to note that all the participating projects were grouped into seven rather broadly defined categories, allowing, for instance, projects in physical cosmology to compete directly with those in differential geometry, or projects in sociology with those in history. The competition occurred in the sense that a single panel of managers was responsible for deciding on funding for all types of projects within one domain. This suggests that panels, by aggregating projects from diverse fields, could be described as epistemically diverse in terms of the range of disciplines represented. However, the degree of epistemic diversity within the panels likely varied depending on how closely related the grouped fields were and how much methodological or theoretical overlap existed between them. For instance, a panel overseeing both physical cosmology and differential geometry might exhibit less epistemic diversity than one spanning sociology and history, despite both being composed of projects from distinct disciplines.

To conclude, we analyze interactions between panelists and reviewers as a signaling problem. Panelists receive signals from reviewers, and when they deem those signals unclear, they infer “grantworthiness” using proxy measures—potentially allowing biases to influence their decisions—or by opportunistically exploiting uncertainty (e.g., for self-reward or organizational proximity). The extent to which panelists comply with reviewer scores may vary depending on the epistemic diversity of the domains. Arguably, in domains comprised of “hard” sciences (e.g., Mathematics, Physics), experts’ opinions may be taken more seriously than in more discursive fields.

In the next section, we will describe the Kazakhstani case and specify how findings from the literature may apply to its context.

Materials and methods

Results

Modeling review scores

As mentioned above, the Kazakhstani grant program includes seven domains. Table 1 summarizes the number of applications, success rates, typical scores, mean and median scores, and other statistics across the domains. Fig 1 displays success rates for different score intervals by domains. Scores, in general, correlate with success rates, yet the strength of association seems to vary across domains.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Success rates by score intervals and domains.

Applications are grouped in review score intervals. Success rates shows what proportion of applications within a given interval received funding.

https://doi.org/10.1371/journal.pone.0318875.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics for review scores and success rates across domains.

https://doi.org/10.1371/journal.pone.0318875.t001

To understand variation in review scores, we run OLS regressions by domains. We exclude data from the Security domain because of the small number of applications (N = 112). Before we proceed, however, we need to address a few methodological issues.

We treat the variable review score here as a continuous one, while in fact it can take on only a finite number of values. More over, conceptually it is more sound to treat score as an ordinal variable, as its values reflect levels of academic fitness assessed by the reviewers (akin to IELTS or TOEFL scores). A more principled approach would be to model reivew score with an ordinal regression treating its values as ordered levels.

This paper, however, mainly analyses the effect of scores on the probability of getting a grant. In other words, review score is an explanatory variable, and our OLS regressions do not explain it, but rather describe it in relation to other variables. So as not to sidetrack too much, we organize the analysis as follows. Below, in Table 2, we present the results of OLS regressions that treat review score as a continuous variable. In the supporting information (S2 Table, S3 Table), we run model diagnostics and alternative models that treat score as a discrete and ordinal variable.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. OLS regression with robust standard errors for review scores by domains.

https://doi.org/10.1371/journal.pone.0318875.t002

First, we may observe that there is no one set of variables that explain variance in scores across all domains. Previous success matters almost everywhere (in Natural RM and Energy significance is lower). H-index also correlates with review scores everywhere, except for Culture. However, it should be noted that the evaluation criteria directly asked reviewers to assess the scientific capacity of the project teams. This part was likely evaluated by looking at PI’s track record, among other things. So, the significance of the h-index could be induced by the evaluation design.

There are factors that do not show any significance in individual domains, for example, gender. When controlling for other academic and institutional variables, the projects led by women and men do not differ significantly in their average review scores. However, the gender gap becomes significant in the pooled data, likely due to the increase in statistical power.

Council members do not score higher than others, and in Science they score even less than the conditional mean by 2.7 points. However, investigators who work with council members score higher in the domains of Energy, Natural RM, Culture, and Life (with low significance though). Also, everywhere, except for Agriculture, affiliation with research institutions (the main component under the other rubric for organizational prestige) is associated with relatively higher scores.

The amount of explained variance changes over domains. In Science and Energy, the model captures around 23% and 20% of the variance, respectively; in Life and Culture 13.5% and 10%; in Natural resource management: about 7%; and in Agriculture: under 3%. These differences may reflect different evaluative conventions in the fields, yet it should be noted that explanatory variables do not vary much in certain domains, limiting models’ capacity to explain variation in the response variable.

Next, we run some regression models on the total dataset. In particular, we examine differences between male and female investigators. A simple baseline model that models score as a function of gender shows a small, yet significant difference of 0.7 points in received scores (Table 3, first column). This difference, however, disappears as more variables are controlled (Table 3, last column).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. OLS regression for the whole data.

https://doi.org/10.1371/journal.pone.0318875.t003

Using Gelbach decomposition [47], we are able to explain 93% of the difference between male and female investigators. In particular, we find that about 28% of the difference can be attributed to h-index (mean male 1.51, female 0.8). H-index itself is heavily skewed, but the degree of skewness is different: 54% and 61% of male and female investigators have 0 h-ndex, yet males have almost 3 times higher probability of having h-index above 5 compared with female investigators. Another major contributor to the score difference is the previous success: 20.8% of the male investigators won grant awards in the previous competition, vs. 12.9% for the females. Previous success accounts for about 20% of the score difference.

Other factors contributing to the difference are presented in Fig 2. Publishing in delisted journals, having a doctor of science degree, and working for a research institution together account for another 28% of the difference. In other words, women in this dataset were more likely to publish in journals delisted from Scopus (31% vs. 21% ), less likely to have a doctor of science degree (31% vs. 44%), and less likely to work in research institutions (38% vs. 43%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Gelbach decomposition.

Variables’ contribution to the gender gap in review scores. To decompose the gap, we used an R implementation by Matthieu Stigler [48].

https://doi.org/10.1371/journal.pone.0318875.g002

Modeling success

We now turn to our main response variable: winning grant funds. We model the probability of getting funded with a set of logistic regressions. Our prime explanatory variables are review scores, closeness to decision-making, and investigators’ gender. These three variables presumably represent the degree to which academic, institutional, and gender concerns influence the council’s decisions.

We start with a model that only accounts for review scores (Table 4, first column). While this model is unlikely to capture real decision-making, it is a useful baseline for comparing with other models. The next model looks into other merit related factors. At the third step, we include both review scores and other merit indicators. Then we specify a model with all available predictors except for the closeness to the decision process. And finally, we include all the available predictors at once. We exclude the closeness to decision making at the fourth step to show that this predictor is important both for overall model fit and the estimation of the effect of other variables.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Logistic regressions for winning a grant (with robust SE).

https://doi.org/10.1371/journal.pone.0318875.t004

First, review scores are robust to various model specifications: one unit increase in score is associated with 1.26 increase in odds of getting a grant regardless of controls. Comparing merit related variables (the third column), we see that score, basically, subsumes other proxies of merit: h-index and publication profile decrease their statistical importance once review scores are taken into account. This squares well with an earlier observation that review scores are likely to be partially produced based on groups’ scientific capacities.

Another thing to note is the decreasing importance of h-index. Besides scores, h-index correlates with domains (the highest being in Science) and membership in the research councils. When these variables are accounted for, h-index loses significance. For example, the council members top other investigators in h-index by more than 3 units (mean 1.12 vs. 4.48, median 0 vs. 2).

We also observe evidence for the “Matthew” effect (accumulated advantage): previous success increases odds by a factor of 1.7 (), and having a doctor of science degree increases odds by 1.5 (). A PhD degree, on the other hand, does not seem to give an edge compared with the reference category (candidate of science). Being a doctor of science works here as an indicator of “old school”, since after 2011 Kazakhstan abolished the title. Doctors of science are likely to be older, and supposedly more authoritative scientists. Besides merit, their project may receive funding out of respect and/or fear of retribution.

Domains vary in success rates. Compared with Culture (the reference category), projects in the Natural Resource Management domain are 1.5 times more likely to be funded, and projects in Life (biomedical research) are almost 3 times more likely to be funded. This apparently reflects a) varying domain priorities and b) different entrance costs: Culture is the most populous domain probably because it is easier to draft a proposal that will qualify for the competition.

Now to address the elephant in the room. The most important predictor of winning is, unsurprisingly, just being a member of a research council. Checking that box increases the odds of receiving funding by a factor of 20. Working with a member also helps, sharing affiliation increases odds by 1.75 when other indicators of prestige and status are controlled.

Finally, we find evidence for gender discrimination: female investigators have, ceteris paribus, 20% less odds of winning a grant than their male counterparts. Here, we observe a situation almost opposite to that described in [4]. There, “gender-equal funding conceals unequal evaluations.” In our case, gender-unequal funding conceals equal evaluations.

We also fit domain subsets separately and show that the aggregate effect of sex discrimination is driven by funding decisions coming from the Natural Resource Management. Table 5 details intradomain effects of the variables (excluding Security). Natural Resource Management is a domain that comes closest to parity between male and female investigators in terms of their sheer numbers, mean review scores, and mean h-index, yet women have less chances of getting money. To be precise, the gender gap in success rates exists in all domains except for Security. Its size varies from 3% (Energy) to 11% (Natural RM). Yet in other domains, this difference can be explained by the difference in review scores and h-index. Only in Natural Resource Management are men and women investigators are similar in their review scores and bibliometric output, yet their success rates still differ.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Logit models for winning a grant by domains.

https://doi.org/10.1371/journal.pone.0318875.t005

Before moving on, we would like to briefly address the issue of model selection. Our models yield many non-significant coefficients, yet we retain them. From a parsimony perspective, this is suboptimal, as it would make more sense to retain only those variables that possess greater explanatory power—and thus simplify the models.

The reason for retaining non-significant coefficients is primarily conceptual. Our explanatory variables were selected based on: (a) theoretical relevance and (b) data availability. Our main goal was to test whether these variables are associated with the probability of funding. When a variable is not (for example, the h-index), it still provides valuable information, as it arguably suggests that the review score already captures what the h-index measures.

However, to demonstrate the robustness of the coefficients, we fit several LASSO regressions [49] and compare their coefficients with those obtained in our final model (excluding interaction effects). The results are presented in the Supporting Information (S2 Figure, S5 Table).

Conclusion

The study uses logistic regression to analyze the likelihood of obtaining a grant. Key explanatory variables include review scores, proximity to decision-making, and investigator gender, which represent the influence of academic, institutional, and gender factors on funding decisions.

The results show that review scores are consistently linked to a higher probability of funding: one unit increase in score increases the odds of winning by approximately 1.26. Other merit indicators lose significance once the review scores are considered. Previous success and holding a Doctor of Science degree also increase the odds of receiving funding.

The most significant predictor of winning a grant is being a member of the research council. Sharing affiliations with members also improves funding chances. Finally, gender bias is apparent, as female investigators have 20 percent lower odds compared with their male counterparts, net of other factors.

Another interesting result is that models, in fact, tend to underestimate the number of winners. For example, logistic regressions generally predict that only approximately 14% of projects would receive funding. And, more than half of the funded projects (56%) had the odds against them: they had low review scores, and their principal investigators did not distinguish themselves in terms of academic achievements, institutional capital, or connections with key decision-makers. And yet, they received funding.

Part of this “unexplained” success rate is likely due to unobserved project characteristics. For example, some projects might be funded simply because they were relatively inexpensive. In other words, council members might take the requested budget into account when deciding for or against a project. Unfortunately, we could not find publicly available data on project budgets.

We, however, argue that, net of other factors, research councils may seek to fulfill a certain number of grants. To a degree, they may use uncertainty to their advantage and distribute grants to themselves and organizationally proximate investigators; also, in some domains, decision makers show gender bias. However, in general, the councils seem to distribute money randomly. This may be due to fear of not using the allocated funds, which could lead to budget cuts in the next competition. This fear, often attributed to bureaucrats, fits into the picture of the bureaucratization of science, other signs of which, for example, focus on quantitative indicators of productivity such as publication count, are anecdotally abundant in local academia.

Based on the findings, several policy recommendations can be made to improve the efficiency of grant allocation. The first recommendation addresses gender inequality. It is crucial to determine whether this inequality is a recurring issue in grant competitions or specific to certain calls or domains. As our data suggest gender bias in the 2017 call was more pronounced in the domain of Natural Resource Management, mostly composed of projects related to oil industry.

If gender inequality proves to be a consistent problem across domains, implementing temporary quotas could help make up for biases. Previous research has shown that, while Kazakhstani academia has achieved quantitative gender parity — evidenced by the growing number of women obtaining scientific degrees — significant gender inequality persists [51]. In this context, such measures could be highly effective.

The second key policy recommendation concerns the members of the National Scientific Council (NSC). As decision-makers, they were also involved in the competition, which created a clear conflict of interest. The Ministry of Science and Higher Education has already taken steps to address this issue. In subsequent competitions, NSC members no longer have the final say in determining which projects receive funding; instead, they score projects alongside external reviewers. However, further in-depth research is needed to clarify the councils role, evaluate their contributions to the final decisions, and open the “black box” of communication between council members during the negotiations. For example, one mechanism that may contribute to gender bias is gender homophily coupled with disproportionate number of men in research councils. In other words, possible interventions may focus not on the outcome of the competition ( for example, gender quotas for winning projects), but on gender composition of the councils (gender quotas for council members).

It would be misguided to assume that the issue with research councils is unique to Kazakhstan. Similar cases have been observed in European contexts, where selection committees sometimes disregard review scores [6] or exhibit self-serving bias [41]. Therefore, it is important not to exoticize the Kazakhstani case but to develop a unifying framework for addressing issues of gender inequality and the balance between external and internal expertise.

Another important discussion point is the extent to which our findings can be extrapolated to other academic contexts. First, it should be noted that one consequence of the 2017 grant call was a redesign of subsequent calls. In this sense, we cannot be certain that our findings are directly applicable to data from, for instance, the 2021 call. On the other hand, the gender bias patterns we observe are surprisingly at odds with findings from other research. For example, as we mentioned earlier, Bol and colleagues, in their study of a grant application call in the Netherlands, found that women received lower review scores than men but had the same probability of success [4]. In contrast, our findings reveal the opposite: there were no significant differences in review scores, but the success rate for men was higher than for women. When we disaggregate the data by domain, this gender gap appears to originate primarily from a specific field—natural resource management, which is dominated by projects related to oil and other extractive industries, arguably associated with masculinity. An implication of this finding is that scientific fields may feature nuanced gender expectations, with the perceived masculinity of a field potentially influencing participants’ chances depending on their gender.

Our study suggests several avenues for future research. First, our analysis relied on regression techniques using observational data without random assignment, which introduces certain limitations. Since the composition of the National Scientific Council changes periodically (with 90% of its members replaced every three years), alternative identification strategies, such as difference-in-differences, could be employed if time-based data were available. Such approaches could address many of the challenges associated with standard regression methods, which often fail to account for the possibility that more frequent awards to members of the National Scientific Council may reflect, for instance, their superior talent or academic excellence. Employing experimental techniques or their replication with observational data would improve the internal validity of causal inferences.

Finally, future research could expand the range of independent variables, including a deeper exploration of the concept of closeness to the decision-making process. In this study, we used organizational proximity—specifically, principal investigators (PIs) sharing affiliations with council members—as a proxy for closeness to decision-making. This served as a basic indicator of potential personal connections to council members. Moving forward, examining co-authorship and citation networks could provide valuable new insights.

Supporting information

Acknowledgments

We thank Alexey Knorre, Sylvan Perlmutter, and Dmitriy Serebrennikov for valuable feedback on earlier drafts of the manuscript. All mistakes are our own.