The evolution of government support for early career scientific training.

The National Institutes of Health (NIH) NRSA program has multiple award mechanisms that fund research training in biomedical science [20–22]. These include institutional training programs (T-series awards) for undergraduates, graduate students, and postdocs, as well as individual fellowships (F-series awards) for graduate students pursuing a PhD or postdoc positions. Since its inception in 1974, these fellowships have been highly competitive. In 2008, the federal government allocated around $751.2 million in funding to directly support biomedical research training for 16,370 undergraduate students, graduate students and postdoctoral researchers via T-series and F-series awards at NIH [23]. Of these awardees or appointees, 1,487 (around 9.1 percent) received an NRSA F32 postdoctoral fellowship award. Our analysis focuses on the most elite competitive award, the F32 postdoctoral fellowship, which provides funding for the transition toward research independence. This award signals to institutions that the early career scientist is independently able to secure funding of unique research ideas. Anecdotally, young scientists claim that high-ranking institutions prefer early career scientists who obtain external signals like elite awards in early career.

Between 1998 and 2008, the number of F-series fellowship awards funded increased by 15.2 percent, as shown in Panel A of Fig 1. While the total number of F-series awards increased, the F-mechanism diversified by funding more positions in other F-series awards, causing the total number of elite NRSA F32 postdoctoral fellowship awards funded in any given year to decrease over the period by 21.9 percent (Fig 1, Panel A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Total NRSA F-series awards and F32 awards funded by NIH by year, 1998–2008.

(A) Awards funded by award type and fiscal year regardless of competitive status. (B) Total newly funded awards by award type and fiscal year [24]. Source: NIH IMPAC II, National Science Foundation Doctoral Record File.

https://doi.org/10.1371/journal.pone.0272230.g001

Even though overall the F-series fellowship awards have experienced growth over the period of our study, they grew much more slowly than the number of PhDs conferred in biological sciences during that time. According to the National Science Foundation, the number of new PhDs in biomedical sciences grew annually from 5,838 in 2000 to 7,797 in 2008, an increase of 33.6 percent [25]. This rapid growth in PhD scientists is a byproduct of the doubling of federal funding for biomedical research at the turn of the 21^st century. The competing needs of the scientific enterprise to increase cheap labor (students and postdocs) to advance science are a conundrum in scientific circles [26]. While competition to receive an award has increased with the number of new biomedical PhDs, it has also increased pressure on funding agencies to understand the value of these programs.

The effect of postdoctoral study on career outcomes.

There are a limited number of studies that have examined the impact of a postdoctoral appointment (postdoc) on later career outcomes. Those that do combine academic fields, which makes it difficult to generalize the results to particular fields of science [27–30]. These limited studies have investigated the impact of a postdoc on subsequent placement in academic careers. Longer duration postdocs are not associated with improved academic placements [27]. One study examined the association between earlier cohorts of federally funded postdocs and their subsequent biomedical careers and found significant gender differences in outcomes; women were more likely to leave scientific research careers than men [28]. Career trajectories of women appeared to be affected by measures related to their mentor’s quality. For female postdocs, the higher the mentor’s h-index, the more likely the postdoc was to receive an NIH grant. That study does note the possibility that results may not be extrapolated to more recent NRSA cohorts [28].

Some studies, however, indicate a negative impact of postdoctoral and graduate appointments. One study found that individuals who took a postdoc in biomedical fields had lower lifetime earnings than those who skipped the postdoc [31]. Although the postdoc was useful for obtaining an academic tenure-track research position, the likelihood of achieving this goal dropped considerably over time. Another study used the Survey of Earned Doctorates to examine whether graduate research assistantships were more effective at launching research careers than graduate student support by federally funded biomedical training grants or fellowships [32]. Graduate students with research assistantships had more successful research outcomes in terms of research-focused jobs although this study did not address postdocs funded by different mechanisms.

The evidence on the impact of postdoctoral and graduate training on later career outcomes in general is mixed, and the evidence on the impact of federally funded biomedical science postdoctoral fellowships using an adequate methodology is limited, prompting this study. That said, studies assessing the impact of federal investments in training the biomedical research workforce, including evaluation studies commissioned by the federal government, show that individuals participating in training programs experience higher rates of remaining in scientific research in comparison to their counterparts who did not receive formal training [14, 15].

This study is most closely related to studies on the impact of NIH postdoctoral training [16] and research grant funding [17] on scientific productivity. A study using a regression discontinuity showed that receiving a postdoctoral fellowship increased publications by 20 percent in the next five years relative to non-awardees [16]. In addition, postdoctoral fellows received higher dollar amounts in subsequent funding compared to non-awardees. Our analysis below indicates that RDD has limitations for estimating the impact of the postdoctoral fellowship awards. A detailed discussion of the application, review and award process discussed below delineates our critique of the RDD as a valid method for analysis of the F32 fellowship.

The independent researcher postdoctoral proposal process.

A thorough understanding of the NRSA application and award process is necessary for the development of our empirical approach. The goal of the award is “to enhance the research training of promising postdoctoral applicants who have the potential to become productive, independent investigators in scientific health-related fields…” In defining a postdoctoral position for the purposes of this paper, we follow the guidance of federal agencies who, in 2007, defined a postdoc as “…an individual who has received a doctoral degree (or equivalent) and is engaged in a temporary and defined period of mentored, advanced training to enhance the professional skills and research independence needed to pursue his or her chosen career path” [33]. One mechanism for accomplishing this goal is through funding a postdoctoral fellowship position in a mentor’s scientific laboratory. Fellowships are allocated based in part on a score assigned during peer review, which considers the merit of an applicant’s research proposal and the quality of proposed training and career development support available through the institution and mentors. In addition, before offering an award, NIH program staff perform a thorough review to ensure the validity of the research idea and the quality of the training environment. Staff also verify whether the research proposed matches well with the NIH institute’s scientific needs and priorities.

Fig 2 shows the number of applications by year. The number of new applications ranged from around 1,500 to 2,500 per year in the mid-2000s. During the NIH Doubling (1998–2003), [34] the number of new applications decreased from around 2,100 in 1998 to a low of approximately 1,500 in 2002. This may reflect the increase in the budget during the doubling that likely increased postdoctoral slots on funded research grants. After the doubling, the number of new applicants once again began to rise and peaked in 2006. In 2008, one award cost, on average, around $48,998.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Total new NRSA F32 applicants, individuals ever awarded, and individuals awarded in same application year.

Data shown for fiscal year of application, 1996–2008. Source: Author calculations based on NIH IMPACII administrative records [35].

https://doi.org/10.1371/journal.pone.0272230.g002

The authors posit that applicants who meet the basic requirements for the award are willing to apply if the benefit of winning is higher than the cost of applying. Applicants are required to have a research doctorate or professional degree by the time of the award, be a citizen or a permanent resident of the United States and have a sponsor (who functions as a mentor), and a sponsoring institution. All these requirements limit the size of the applicant group and make it more homogeneous, limiting the impact of unobserved characteristics on the probability of receiving an award.

Selection to receive a fellowship award depends on multiple measures. First, a scientific review group evaluates each new application for scientific merit. At the NIH, for the most part, the Center for Scientific Review (CSR) develops and runs the group. CSR is an independent body that provides institutes services related to peer review, as well as documenting the evaluation and scoring of applicants. The scientific review groups include accomplished scientists and experts as external reviewers. The evaluation considers observable characteristics of the applicants, A (the applicant’s past academic and research records, publications); observable characteristics of the sponsor, S (the sponsor’s general qualifications); research potential, Pot (references, the applicant’s research goals); research environment, E (training environment); the research proposal, P, plus the perception of the reviewer in that period ε_qt. Each reviewer, r, assigned to an application, i, for institute or center, j, in a particular year, t, and council round, q, gives a preliminary reviewer overall review score, PS_ijrtq, for each application defined as: (1) (2)

Eq 2 gives a final overall review score, PS_ijtq, for each discussed application, i, who applied to an institute or center, j, in year, t, in council round, q, and is simply determined by the mean score of all scientific review members’ preliminary scores. Review panels score proposals from best to worst, where the lowest scores are considered the best. Until 2009, proposal review scores ranged from 100 (high impact) and 500 (low impact). Starting in 2009, a process called Enhancing Peer Review changed the scale of evaluation to 10–90 where the final overall impact scores now range from 10 (high impact) through 90 (low impact) [36].

Prospective awardees are evaluated in council round, q, in a calendar year, t. At the end of the scoring process, the scientific review group reports the review scores to the institute or center’s selection process. At the end of the first selection, those applications with the lowest scores (the best applications) receive a secondary level of review by staff. This step is a key determinant of the funding decision. Program staff examines the applications’ summary statements and review scores and weighs these factors against the needs and priorities of the institute or center. The director (or their delegate) makes the final decision as to whether to offer an award.

Applicants who do not receive an award typically explore multiple options, which include appointment to a position as postdoctoral researcher on a mentor’s research grant, appointment to a training grant if a slot is available, application to other funding sources such as foundation grants, or, if feasible, resubmission of their application. Other scenarios include leaving academic research or exploring other career paths. The true value of elite fellowship awards is their power in developing a skillset to lead and conduct a research project and the freedom it gives the postdoc to engage in independent research under the supervision of their mentor. We are testing whether funding to develop and pursue a high-quality research project early in one’s career has an effect on the probability of receiving future biomedical research funding and establishing an independent research career.

A sharp regression discontinuity design is appropriate if the majority of proposals are funded in the order of the score. Aberrations in the funding of proposals with review scores near the cutoff constitute a fuzzy regression discontinuity design. However, if institutes skip several (competitive) low-scoring proposals and reach for higher scoring proposals that are more consistent with institute priorities or because of other administrative hurdles on low-scoring applications, then an RDD has limitations. This paper now turns to the data before evaluating the appropriate methods for determining the impact of the F32 award on subsequent career outcomes.

An analysis of awarding behavior.

Each institute has discretion in deciding who is offered a fellowship based on the current research priorities of the institute. Institutes and centers vary in their process for awarding a fellowship. For this study, the authors interviewed staff at various institutes and centers who provided a representative view of the variation in procedures across the organization. Through interviews, the authors learned that the process for awarding fellowships is more complex than funding proposals with the best scores.

The review and scoring of fellowship proposals and institute budget realities create scoring thresholds that theoretically would support the use of a regression discontinuity design (RDD) as in a similar study [16]. RDD is appropriate when the forcing mechanism, in this case the review score, is the principal factor determining who gets and does not get funding. The score or budget line is determined based on the availability of funds. In such cases, one can evaluate units around the cut off variable (e.g., budget line) to identify the impact of the program for those who just got in compared to those who just did not. Decisions of who to fund are made at the year-institute-council round level. We demonstrate that using only year and institute to estimate an RDD model as has been previously done [16] is not an optimal method for evaluating training fellowships given the fellowship selection process.

If awards are based solely on the review score allocated from best (lowest) score to worst (highest) score until the institute budget allocations for the fellowship are exhausted, the percent of applicants selected out of order would be zero or close to zero. If this were the case, a sharp RDD would be valid because, in this scenario, the institute follows the guidance of the peer review process for the best (lowest) scores. The pay line determines where all resources have been spent within the institute. Suppose there are 10 applicants arranged in order from 1 (the best) to 10 (the worst) proposal scores. If the budget allows for funding 4 applicants, the pay line is 4 and anyone with a score of 4 or less would be funded. If, however, an institute has funded a majority of applicants with meritorious scores and some discretion is used regarding awards to applications with scores near the pay line, we would observe minimal disorder in funding. In this example, the institute uses discretion to potentially skip some applicants close to the pay line in order to fund applications with slightly worse scores but with a better fit within their scientific priorities and where the institute staff believes the applicant has the best-case scenario for future success. If this were the case, a fuzzy RDD would be appropriate. Alternatively, institutes could just choose to use a significant amount of discretion when selecting proposals for funding in order to meet institutional goals related to scientific priorities or, perhaps, diversity of the workforce. For institutes that engage in this behavior, no real cutoff exists and an RDD is not appropriate. Our prior findings [38] demonstrate both visually and through data analysis that overall discretion in NIH funding of independent research awards overrides decisions made by a cutoff score sufficiently that neither a sharp nor fuzzy RDD model cannot be used as a valid method.

We interviewed program officers from four institutions ranging in topic, size, budget, and programmatic content. The interviews lasted around one hour each and were focused on identifying the administrative steps that lead to funding decisions starting from grantee submissions to review (which is where grants get discussed), scoring, rules for complete applications, budget office interactions with program staff, and final award decision making. Interviews with NIH staff indicated that the funding process is multifaceted. The institute receives an application’s peer review score, which, in most cases, is defined by the study section through coordinated scientific review groups. Once the institute receives the scores, staff assesses the full application, including the summary statement from peer review, the quality of the training institution, and the alignment of the research proposal with the institute’s research priorities. Program officers at each institute then generally participate in a team meeting in which they defend the proposals that best match their defined priorities. Together, the program officers, the training director, and other institute staff make a joint decision for recommendations to the director. Either the director or his/her delegate makes the final decision and signs off on which proposals to fund. Directors vary in terms of direct involvement in the consideration and final approval of proposals. Before a final decision is made and the candidates are informed, the budget office reviews and signs off on the final list of candidates, primarily making sure sufficient funds are available for the recommended awards.

We examined funding decisions within institutes and found a significant amount of discretion. Fig 3 illustrates why RDD is not the appropriate design for evaluating fellowship awards. It demonstrates the range of discretion used in fellowship decision-making for the full sample over the period of our study. In a given council round, anywhere from around 6.9 percent to 20.9 percent of applicants received a decision that was not based on their peer-reviewed ranked score. Either the institute funded the applicant even though other applicants had better scores, or the institute’s final decision was not to fund the applicant even though their application had a better score than others that received funding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Percent of applicants selected out of order by council round, 1996–2008.

Figure excludes council rounds with an N<20. Author calculations using NIH records [35].

https://doi.org/10.1371/journal.pone.0272230.g003

What does this mean for this study? Fig 3 illustrates the level of non-compliance for an RDD method and the strong tendency for discretion in decision-making within each council round. While the level of discretion has decreased in recent years, over the entire sample, around 12.3 percent (1 in 8) of all applications within a given council round received a decision that was not in line with the peer review score. Given the large proportion of discretion used within council rounds, the authors argue that RDD is an invalid method for evaluating the true impact of early career training on later career outcomes as it relates to the NRSA F32 fellowship award.

Matching to identify the causal effect.

As described above, the process for fellowships and research grants generates a peer review score. Given the fact that the groups of individuals applying to the awards are relatively homogenous compared to the total population, Propensity Score Match (PSM) and Nearest Neighbor Match (NNM) methods are considered for this analysis. While it is understood that any unobserved characteristics that are different between funded and unfunded confound the results, the authors argue that matching is a feasible approach for the following reasons. First, they can control for unobserved differences by institute and council round. Second, selection is made at the institute level, and any unobserved differences among applicants are also not observed at the institute level and not likely contributing to the selection process. In short, there is no self-selection of fellowship award offerings. Third, individuals who apply for funding are relatively homogeneous within the institute. They have been encouraged to apply by their mentors (which means their mentors believe they have a chance of getting the award), are committed to biomedical research careers and excel at academic pursuits. Given the unobservable variation that could exist, within this select group of applicants, it is minimal.

The authors use the potential outcomes framework employed in econometric analysis to estimate the causal effect of fellowship awards on subsequent funding outcomes [40]. Let T_i = 1 be the treatment when an individual’s fellowship application is funded and let T_i = 0 if the application is not funded. In the potential outcomes framework, individuals have two potential outcomes of subsequent funding: Y_i(1) if the individual receives the award treatment and Y_i(0) if the individual does not receive the award. The causal effect for individual i of the funding award is defined as the difference in potential outcomes Y_i(1)−Y_i(0). That said, individual i is only observed when they receive the award or they do not, and we must estimate the counterfactual outcome.

Matching methods used in this study assume that treatment is independent of the outcome conditional on covariates, T_i┴(Y_i(0), Y_i(1))|X_i, the unconfoundedness assumption. Unconfoundness means that the treatment is conditionally independent of the outcome after conditioning on observable characteristics. If unconfoundedness holds, we can define the average treatment effect in terms of potential outcomes as the expected value of potential outcomes:

The average treatment effect on the subsample of the treated is defined as:

Two matching methods are used to identify the ATE and ATT. We employ propensity score matching (PSM) and present the PSM results in S1 Appendix The propensity score is the probability of receiving treatment conditional on observed characteristics e(X) = Pr (T_i = 1|X_i = x). In order to implement this method, the propensity scores for the treated and untreated in our sample must overlap such that 0<e(x)<1. The unconfoundedness assumption cannot be tested directly; thus, the authors examine whether the propensity score has a causal effect on a pseudo-outcome that was determined prior to the treatment. Unconfoundedness is not likely to be violated if the estimated effect of the treatment on the pseudo-outcome is not significant [41]. They can evaluate the overlap assumption directly by assessing the balance of the covariates in the treated and untreated groups as well as visually inspecting the overlap in the propensity scores.

Propensity score matching has been widely used in social sciences and economics [41]. However, propensity score estimates break down if the propensity score model fits to the data too well [41, 42]. This means that the authors cannot use the review score to estimate the propensity score related to fellowship funding. We therefore use the coarsened exact matching (CEM) algorithm [43] as our primary preferred method to improve the balance of the data and nearest-neighbor methods to facilitate matching on the review score. Our results presented below use nearest neighbor matching after reducing the data using the CEM algorithm.

Nearest-neighbor matching.

Given sensitivity of the propensity score matching (PSM) to the propensity score specification (see S1 Appendix for details) and recommendations to use alternative methods [42], nearest-neighbor matching is used as our preferred method. Throughout, the Mahalanobis distance for nearest-neighbor matching is used. Let X^T be the set of control variables for those receiving the fellowship award and X^C be the set of control variable for non-awardees. The Mahalanobis distance is given by: and V is the covariance matrix of X. Mahalanobis distance will reduce differences between the treated and control groups by an equal percentage for each covariate in the matrix X.

The authors experiment with a variety of specifications of the covariate list and matching approach for the nearest neighbor matching. First, they match on the review score only since it is a significant determinant of fellowship funding, and it could not be included in the PSM approach. Second, they match on the review score, institute and council round. Third, they match on review score and council round and impose exact matching on institute. Fourth, they match on all of the variables in the second column of Table 3 and impose exact matching on the institute. About 1,000 observations are dropped that do not have an exact match, leaving 13,653 in the analysis sample. This analysis concentrates on the estimates of the ATE in the last column of Table 4 where the number of covariates is the largest. The nearest neighbor matches increase the number of research program grant awards by 0.10, the number of research program grant applications by 0.73, the probability of a research program grant award by 6.3 ppt, the probability of an elite independent research award by 4.6 ppt, and reduces the probability of never applying for subsequent funding by 9.8 ppt. These estimates are somewhat smaller than the PSM estimates in S1 Appendix but still statistically significant.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Nearest neighbor full sample and CEM sample ATE estimates (differ by specification of matching covariates).

https://doi.org/10.1371/journal.pone.0272230.t004

Coarsened exact matching (CEM) for sample selection can improve the balance of the matching estimator by exactly matching on categorical values of the data [43]. The coarsened data have significantly improved balance and balances nonlinearities and interactions in the sample. This analysis begins by using the CEM algorithm to perform exact matches by institute and fiscal year of funding. Next, the authors use nearest-neighbor matching and same covariates to estimate the ATE, and these estimates inherit the balance properties from the CEM procedure. These estimates appear in the bottom of Table 4. Despite the sample size reduction from 13,653 to 8,630, the nearest-neighbor matches after the CEM algorithm are remarkably similar to the results from the top panel in Table 4.