https://orcid.org/0000-0001-7261-929X
Figures
Abstract
Health services, economics, and outcomes research (referred to as health economics research hereinafter) is one of the interdisciplinary sciences that the National Institutes of Health (NIH) supports in order to pursue its overall mission to improve health. In 2015, NIH guidance was published to clarify the type of health economics research that NIH would continue to fund. This analysis aimed to determine if there were changes in the number of health economics applications received and funded by NIH after the release of the guidance. Health economics applications submitted to NIH both before and after publication of the guidance were identified using a machine learning approach with input from subject matter experts. Application and funding trends were examined by fiscal year, method of application (solicited vs. unsolicited), and activity code. This study found that application and funding rates of health economics research were decreasing prior to guidance. Following publication of this guidance, the application and funding rate of health economics applications increased.
Citation: Burgdorf CE, Riley WT, Akbar F, Zuehlke AD, Bibb K, Eshleman HD, et al. (2024) Analysis of the health economics portfolio funded by the National Institutes of Health in response to published guidance. PLoS ONE 19(2): e0284235. https://doi.org/10.1371/journal.pone.0284235
Editor: Masoud Behzadifar, Lorestan University of Medical Sciences, ISLAMIC REPUBLIC OF IRAN
Received: March 26, 2023; Accepted: December 28, 2023; Published: February 14, 2024
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: The dataset the ML algorithm was applied to included all competing applications (both new applications and renewal applications) for research project grants received by NIH between FY2010 and FY2020 (n=623,676), obtained through the Information for Management Planning Analysis and Coordination (IMPAC) II data system. Applications that were reviewed in council year 2009 or included animal subjects were removed. Much of the content used to analyze applications (even funded applications) is not publicly available and contains confidential, proprietary information owned by a third party (extramural academic institution) that is safeguarded by NIH. Furthermore, for unfunded applications, all data — including submission numbers and topics — are confidential as they are owned by a third party (extramural academic institution). The NIH maintains application and grant records as part of a system of records as defined by the Privacy Act of 1974, 5 U.S.C. 552a (as amended), and its implementing regulations (45 CFR Part 5b). You can find more information here: https://grants.nih.gov/grants/policy/nihgps/html5/section_2/2.3.11_availability_and_confidentiality_of_information.htm As NIH employees, the authors have access to the Information for Management Planning Analysis and Coordination (IMPAC) II data system, which is restricted to the public. The data underlying the results presented in the study are available from GrantsPolicy@od.nih.gov.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Health economics is defined by the NIH as “the study of how scarce resources are allocated among alternative uses for the care of sickness and the promotion, maintenance, and improvement of health, including lifestyle patterns and behavioral health” [1]. It includes the study of “how health care and health-related services, their costs, benefits, and consequences, and health itself are distributed among individuals and groups in society by measuring efficiency, effectiveness, value, and impact of health services/resources between individuals, healthcare providers, and clinical settings” [1]. Health economics research offers rigorous approaches that can help NIH pursue its overall mission to improve health and is an integral part of the interdisciplinary sciences that NIH supports.
To emphasize the types of health economics research that are relevant to the NIH mission, NIH published guidance NOT-OD-16-025 to the extramural community in 2015 [1]. This guidance specified research areas that were 1) high priority across NIH, 2) priorities that were Institute- and Center (IC)- specific, and 3) those that were not in alignment with the NIH mission. High-priority research included health economics studies designed to understand how innovations in treatment, diagnosis, prevention, and implementation strategies can be deployed most effectively to improve health and well-being as well as research aimed at designing better interventions with these insights. IC-specific priorities included research specific to an IC’s mission utilizing economic methodologies such as assessments of alternative models of delivering IC-mission-relevant care. Assessments focused on outcomes related to health, including health-related behaviors and healthcare utilization. Areas outside of NIH’s mission included studies that were minimally related to specific health outcomes at the population or individual level.
While the goal of the guidance was to communicate that NIH will continue to accept applications involving health economics research with a focus on improving health outcomes and health-related behaviors, the research community demonstrated mixed responses on the perceived influence of this guidance on funding levels within the field [2, 3]. This study aimed to determine if this guidance had an impact on the number of health economics applications being submitted to the NIH (application rate) or the amount of funding provided to health economics compared to all NIH funding (funding rate). NIH application and funding patterns for research in the field of health economics were analyzed over time, with an emphasis on determining the impact of the 2015 guidance publication on these trends.
Methods
Identification of the NIH health economics research portfolio
A machine learning (ML) model was created to identify the NIH health economics research portfolio using positive and negative training data. The positive training set was created by combining projects curated by subject matter experts in a 2015 internal analysis (n = 198) with health economics research projects identified using keyword searches (n = 36) for a total of 234 projects. The negative training set was a subset of research project grant (RPG) awards classified by Research, Condition, and Disease Category (RCDC) as ‘Translational Research’ that were funded by the NIH from FY2010—FY2020 (n = 3,898).
The machine learning model analyzed features from the text of the title, abstract, specific aims, and research plan of each grant application. Standard text tokenization of the fields was applied including stemming, ignoring case, removing stop words, and collapsing character codes. In addition to using free text, the model also used a controlled vocabulary of health economics terms that was compiled from The British Medical Journal [4], National Library of Medicine [5], Centers for Disease Control and Prevention [6], and the University of York [7] (S1 Table). Gradient Tree Boost and Liblinear algorithms were combined into a single measure using a neural network known as an ensemble algorithm. The model was evaluated for precision (specificity), recall (sensitivity), and F1 score (harmonic mean of precision and recall).
This final ML algorithm was applied to all competing applications (both new applications and renewal applications) for research project grants received by NIH between FY2010 and FY2020 (n = 623,676), obtained through the Information for Management Planning Analysis and Coordination (IMPAC) II data system. Applications that were reviewed in council year 2009 or included animal subjects were removed. Each application was given a discrete prediction score of 1 or 0. A prediction score of 1 was assigned if it was predicted to be health economics while a prediction score of 0 was assigned if it was predicted not to be health economics. Each application was also given a confidence score ranging between 0.5 and 1. The higher the confidence scores, the higher likelihood that the prediction was accurate, such that a prediction score of 1 with a confidence score of 0.95 is very likely to be health economics, and conversely a prediction score of 0 with a confidence score of 0.95 is not. A cutoff confidence score of 0.999 was used to capture the health economics portfolio. Subject matter experts reviewed a sample of results (n = 50 applications) and confirmed that that the machine learning adequately captured health economics research.
Analysis of trends in health economics research funding
Competing RPG applications that were identified using ML to be health economics (n = 23,362), both awarded (n = 3,661) and not awarded (n = 19,701), were compared to competing RPG applications that were not related to health economics (n = 600,314), both awarded (n = 102,649) and not awarded (n = 497,665). Applications and awards within the health economics and non-health economics datasets were analyzed for the number of applications, awards, NIH-designated grant mechanisms and award rates before and after publication of the guidance. As fiscal year and calendar year dates do not align, applications with council review date in January 2016 or before were coded as “pre-guidance” and applications with council review date after January 2016 were coded as “post-guidance”.
An Interrupted Time Series Analysis (ITSA) was used to determine the impact of the release of the 2015 guidance on application and funding rates; an ITSA can evaluate time-dependent outcomes in response to an event by measuring the same outcome parameter at sequential times before and after the event. ITSA was selected as it is the study design of choice for evaluating the effectiveness of interventions with a clearly defined implementation date [8, 9]. The main utility of the ITSA is the ability to determine whether the pattern in the data observed after the intervention is the same as the pattern pre-intervention.
When analyzing the data through ITSA, all regression models used the following formula: Yt = β0 + β1T + β2D + β3P + Ɛ, where:
- Outcome (Yt) (application or funding rate) = baseline (β0) + time since the start of observations (β1T) + treatment/intervention (pre or post) (β2D) + time since intervention (β3P) + random error (Ɛ)
- Time (T): the time elapsed since the start of the observation period (in this case, council rounds), in frequency terms with which observations are taken/utilized.
- Intervention (D): a dummy variable, described in the data as the “treatment variable,” was used to indicate the pre-guidance period or post-guidance time periods.
- Applications with council review date in January 2016 or before were coded as pre-guidance = 0
- Applications with council review date post January 2016 were coded as post-guidance = 1, starting at time = 30 (30th council round)
- Time since intervention (P): the time elapsed since the intervention, in frequency terms, with which observations are taken/utilized (council year and month, as seen above). Time in the pre-intervention period (P) was equal to 0, time since intervention (P) started at P = 1 and incrementally increased for each time point (T) following the intervention.
- Discrete variables: 0 (pre-intervention; 1–44 incremental (post-intervention)
- Treatment began at the Time (T) = 30 (January 2016)
- Outcome (Yt): the outcome variable (at time t), is the aggregated number of health economics applications or aggregated amount of health economics funding, shown as a rate of the total NIH portfolio; see below for specifics on each outcome rate.
- Application outcome: The application rate for each council round was calculated for the application outcome variable using all applications NIH received during FY2010- FY2020 to calculate application rates at different points in time based on year and council round. The recorded rate was calculated as a rate of health economics applications to non-health economics applications: (HealthEconomicsCountofApplications/Non-HealthEconomicsCountofApplications*1000)
- Total cost outcome: health economics funding rates for each council round were calculated using total costs for health economics awards and total costs for non-health economics awards (i.e., at council round August 2015, the number of dollars for both health economics and NIH were recorded to compute the rate: ((HealthEconomicsSumofTotalCost/Non-HealthEconomicsSumofAwdTotal)*1000)
In addition to assessing the health economics portfolio and non-health economics portfolio, applications and awards were further split into two datasets: solicited and unsolicited. The “solicited” dataset contained applications and subsequent awards that were submitted in response to specific funding announcement. The “unsolicited” dataset contained those that were submitted to a general parent announcement or initiated by the investigator.
Results
Machine learning approach
The machine learning algorithm was analyzed for its F1 score, which is a measure that represents the algorithm’s level of specificity and sensitivity. The refined ML model that captured the NIH health economics portfolio had a high F1 score of >0.93.
The machine learning algorithm denoted applications that were predicted to not be health economics (category prediction score of zero) or to be health economics (category prediction score of 1). To avoid potential false negatives, the threshold confidence score for health economics applications was set at 0.999, a value which was determined by analyzing the projects identified as health economics around this threshold. Due to curating projects around the threshold, the algorithm was thought to have captured health economics projects with high confidence, especially since it was applied uniformly to all applications. However, it is possible with an approach such as this that some non-health economics projects are included in the dataset and that some health economics projects were not included in the dataset. All applications receiving a confidence score less than 0.999 were deemed as non-health economics. The number of applications with each category prediction is shown in Table 1.
Health economics portfolio overview
Application and award trends by fiscal year.
Number of applications and awards for health economics research were compared with applications and awards that were not related to health economics in the pre-guidance period as well as the post-guidance period (Table 2). In the pre-guidance period, NIH received an average of 2,155 health economics applications per fiscal year, which decreased (not significantly) to 2,086 applications post-guidance. Despite the trending decrease in application numbers in the post-guidance period, the average health economics award count showed a not significant increase from an average of 317 awards per year to an average of 352 awards per year. These trends resulted in a higher award rate (from 14.7% to 16.9%) post-guidance. For applications that were not related to health economics, there was an increase in the number of applications (from 51,127 to 58,710 yearly average), number of awarded applications (from 8,397 to 10,453 yearly average), and the award rate (16.4% to 17.8%). Award rates by fiscal year are detailed in the appendix (S2 Table; raw numbers S1 Fig; percentages). Despite the decrease in application numbers in the post-guidance period of examination, average HE and non-HE award counts were higher following the release of the HE guidance.
Solicited vs. unsolicited application and award trends.
NIH also releases Notice of Funding Opportunities (NOFOs) to solicit applications that align with NIH goals and objectives in addition to accepting investigator-initiated applications. For research that was not related to health economics, 39% of applications and 36% of awards were solicited from FY2010- FY2020 (Fig 1). However, for research that was related to health economics, 57% of applications and 56% of awards were solicited (Fig 1). This pattern remained consistent across all fiscal years for solicited application rates and respective award rates (see appendix S2 and S3 Figs).
Competing applications and awards for HE and non-HE research were determined for FY2010- FY2020 and then broken out by whether they were solicited or unsolicited. [A] HE applications from FY2010- FY2020. [B] Non-HE applications from FY2010- FY2020. [C] HE awards from FY2010- FY2020. [D] Non-HE awards from FY2010- FY2020.
Application and award trends by activity code.
NIH utilizes a number of grant mechanisms for RPGs that vary in the length and amount of funding, identified by activity codes (e.g. R01, R21, R03). The health economics and non-health economics datasets were analyzed to see if variation existed in the distribution of activity codes for health economics research and the rest of the NIH portfolio (Fig 2). Applications and awards for health economics and non-health economics research from FY2010- FY2020 had similar activity codes with R01s making up the majority of applications and awards for health economics research and non-health economics research, followed by R21s and R03s. Activity code types overlapped in all of the top five categories by application counts and four of the top five in award counts.
Competing applications and awards for HE and non-HE research were determined for FY2010- FY2020 and then broken out by activity code. [A] HE applications from FY2010- FY2020. [B] Non-HE applications from FY2010- FY2020. [C] HE awards from FY2010- FY2020. [D] Non-HE awards from FY2010- FY2020.
Interrupted Time Series Analysis (ITSA)
ITSA is a method used for evaluating intervention studies such as public health interventions, vaccinations, traffic speed zones, policy changes, and health impacts in response to the global financial crisis [8]. Interventions measured using ITSA require a clear differentiation of pre- and post-intervention intervals in addition to sequential measures of the outcome at different times. Time since intervention can also be examined for sustained or ongoing effects of the intervention in the post-policy data. In this study, ITSA was used to retrospectively analyze individual data points for each council round to infer the impact of the guidance on health economics application rates over time both before and after the guidance release.
Application rates.
First, predicted values for the application rate of all health economics applications within each council round were plotted using the ITSA regression (see Methods section and S4 Fig). The slope in the graph accounts for all of the covariates whereas the table shows the coefficient trends independently for each variable (time, treatment, and time since treatment). Regression data was compared before and after guidance publication (dotted red line) (Fig 3A). There was a trending, but not significant, decrease in application rates prior to the guidance and a not significant increase in application rates following the guidance (Fig 3B). Of note, however, analysis of application rates by council round identified a seasonal pattern, with increased numbers of applications each year in the fourth (August) council round. As the August council round is when most of the solicited applications are considered by council, we separated applications by whether they were solicited or unsolicited and analyzed each group independently.
[A] Chart showing changes in application rates in HE applications prior to and following the guidance publication in 2015. [B] Table summarizing the coefficients from the time series analysis.
When examining the application rates of only solicited applications, a similar trend was identified in which there was a trending but not significant decrease in application rate pre-guidance and a trending but not significant increase in application rate post-guidance (S5 Fig). When examining the application rates of only unsolicited applications, there was a trending but not decrease in application rates pre-guidance with no significant difference in application rates post-guidance (Fig 4).
[A] Chart showing changes in application rates in unsolicited HE applications prior to and following the guidance publication in 2015. [B] Table summarizing the coefficients from the time series analysis.
Since the time between the submission date of an application and the notice of grant award date can be up to a year, there was an expected lag of about a year from when the guidance was released to when it might impact application submission rates and subsequent council concurrence. To account for this delay, the application rate of solicited applications was reevaluated using ITSA in which the time designated as the “intervention” was shifted twelve months later (identified in figures as “pseudo guidance publication”). With this shift, the application rates during the first twelve months post-guidance were now included in the “pre-guidance” period. When the delayed effect analysis was performed, this model showed that the application rate was decreasing at a statistically significant level up until one year after publication of the guidance, with no significant change in the application rate one-year post-guidance (Fig 5).
[A] Chart showing changes in application rates in solicited HE applications up to one year after publication of the guidance and starting one year after publication of the guidance. [B] Table summarizing the coefficients from the time series analysis.
A delayed effect analysis of unsolicited applications demonstrated a statistically significant decrease in application rates up to one year after publication of the guidance. There was no change in application rates starting one year after publication of the guidance (Fig 6).
[A] Chart showing changes in application rates in unsolicited HE applications up to one year after publication of the guidance and starting one year after publication of the guidance. [B] Table summarizing the coefficients from the time series analysis.
Funding rates.
The funding rate of health economics applications was determined for each council round by comparing the funding amounts of health economics applications to the funding amounts of non-health economics applications as described in the methods and was analyzed by ITSA. Pre-guidance, there was a decrease in health economics funding rates that was statistically significant while post-guidance funding rates demonstrated a statistically significant increase in health economics funding rates (Fig 7).
[A] Chart showing changes in funding rates in HE applications prior to and following the guidance publication in 2015. [B] Table summarizing the coefficients from the time series analysis.
When the funding rate of solicited applications was analyzed, there was a trending but not statistically significant decrease in funding rates for solicited applications before publication of the guidance; after publication of the guidance, there was a trending but not statistically significant increase in funding rates of solicited applications (Fig 8).
[A] Chart showing changes in funding rates in solicited HE applications prior to and following publication of the guidance in 2015. [B] Table summarizing the coefficients from the time series analysis.
When the funding rate of unsolicited applications was separately analyzed, there was a no difference in funding rates for unsolicited applications before or after publication of the guidance (S6 Fig).
The funding rate of health economics applications was reevaluated using ITSA in which the time designated as the “intervention” was shifted twelve months later. When the delayed effect analysis was performed for all health economics applications, the funding rate was found to be decreasing, to a statistically significant level, up until one year after publication of the guidance, with no significant change in application rate one-year post-guidance (Fig 9).
[A] Chart showing changes in funding rates in HE applications up to one year after publication of the guidance and starting one year after publication of the guidance. [B] Table summarizing the coefficients from the time series analysis.
When analyzing the delayed effect on funding rates of solicited applications, there was a decrease in funding rates up until one year following the guidance publication that was statistically significant (Fig 10). However, one year after publication of the guidance, there was a trending but not statistically significant increase in funding rates for solicited applications (Fig 10).
[A] Chart showing changes in zing rates in solicited HE applications up to one year after publication of the guidance and starting one year after publication of the guidance. [B] Table summarizing the coefficients from the time series analysis.
When analyzing the delayed effect on funding rates of unsolicited applications, there was again no difference in funding rates up until one year following the guidance publication or one year following the guidance publication (S7 Fig).
Discussion
The results from these analyses indicate that while application rates for health economics applications were variable post-guidance, both award rates and funding amounts for health economics applications slightly increased post-guidance. Interestingly, the analysis also demonstrated that pre-guidance, both solicited and unsolicited health economics applications rates and funding amounts were slightly decreasing over time. However, in the council rounds following the guidance, the award rates and funding amounts increased. Of note, award rates for health economics awards increased post-guidance (from 14.7% to 16.9%) but were still less than the post-guidance award rates for applications not related to health economics (17.8%). This suggests that the guidance may have renewed clarity of NIH’s health economics interests for applicants, increasing the chances of applications submitted post-guidance to align with NIH priorities and subsequently to be awarded.
Health economics is a complex field of research with varying definitions based on competing priorities [10]. For example, the Centers for Disease Control and Prevention [11] places a strong emphasis on the use of health economics to determine alternative prevention strategies while the Johns Hopkins Bloomberg School of Public Health [12] uses health economics to promote healthy lifestyle behavior. Due to the inconsistent definitions and lack of a “health economics” classification in the NIH RCDC, the current study used a multipronged approach. The use of an ML approach with input from subject matter experts allowed for a transparent and reproducible identification of health economics applications and awards to assess application and award trends over time. ML provided a consistent approach to identify the corpus of health economics applications over time and ensure that health economics research applications from FY2010 were analyzed using the same methodology as health economics research from FY2020.
It is possible that some health economics applications were not captured using this approach (false negatives) or that some research unrelated to health economics may have been captured at the expense of capturing all health economics research (false positives). This noise, however, is likely proportional across years due to the standardized ML approach. Of note, this analysis was repeated with a lower confidence threshold (0.95), and the results remained nearly identical, indicating similar findings when the labeling of a health economics application was more liberally defined.
To analyze the impact of the guidance on the application and funding rates for NIH health economics, ITSA was used as a quasi-experimental approach for inferring causality following the guidance publication. Because a controlled study of this guidance was not possible, the ITSA model was an appropriate tool for this analysis with the guidance publication acting as a defined time point (unlike most public health interventions) and the pre-guidance data acting as a clear control condition [9]. ITSA also provided further information as to application and funding trends by council round, which identified a difference in application rates between solicited and unsolicited applications and unique trends pre- and post-guidance. This methodological approach has been used in other NIH projects to categorize and analyze complex data [13] and may be a helpful tool to other researchers, especially in the health policy field, for its ability to identify the impact of a specific event on qualifying datasets in a precise and reliable manner.
Conclusion
Cognizant of these limitations, this study shows that the NIH health economics guidance does not appear to have negatively impacted the rate of either health economics applications or awards. Instead, the result of the guidance publication appears to have been an increase in health economics award rates and funding amounts, possibly because health economics applications are now better aligned with NIH mission and interests.
Supporting information
S1 Table. Vocabulary of health economics terms.
Compiled from several authoritative sources and used to build machine learning algorithm.
https://doi.org/10.1371/journal.pone.0284235.s001
(PDF)
S2 Table. Competing RPG applications and awards by fiscal year.
Number of HE and non-HE projects before and following publication of the guidance.
https://doi.org/10.1371/journal.pone.0284235.s002
(TIF)
S1 Fig. Analysis of HE and NIH (excluding HE) award rates for competing RPGs from FY2010- FY2020.
Award counts were divided by the total number of applications for HE and non-HE research for each Fiscal Year.
https://doi.org/10.1371/journal.pone.0284235.s003
(TIF)
S2 Fig. Analysis of competing applications that were solicited for HE and non-HE research from FY2010 to FY2020.
Application counts were divided by the total number of applications for HE and non-HE research for each Fiscal Year.
https://doi.org/10.1371/journal.pone.0284235.s004
(TIF)
S3 Fig. Analysis of competing awards that were solicited for HE and NIH (excluding HE) research from FY2010 to FY2020.
Award counts were divided by the total number of awards for HE and non-HE research for each Fiscal Year.
https://doi.org/10.1371/journal.pone.0284235.s005
(TIF)
S4 Fig. Illustrated explanation of ITSA graph with counterfactual line added.
The red box outlines pre-policy time series points and the purple box, post-policy points. Arrows are pointing to time = 1, 26 and 35; explanations of their predictions are below. Coefficients are otherwise known as independent slopes (per variable) within the regression model.
For the part of the graph outlined above in red, these are the pre-policy time points, therefore, with regard to the regression model, only the baseline constant and time variable are utilized in deriving this point (not intervention or time since intervention). For model 2a above, solicited application rates, the predicted outcome, can be derived using the necessary coefficients and values from the regression table, for example, at time = 1:
Predicted Yt = 73.30
Looking at trends after the policy was put into place, outlined in the purple box on the graph above:
Immediately after the policy change (May 2016), for this model (2a), time = 26, intervention = 1 and time since = 1:
Predicted Yt = 48.16
Looking further out (August 2018), post-policy change, for this model (2a), time = 35, intervention = 1 and time since = 10:
Predicted Yt = 45.91
Had the policy not been put into place, a predicted value (counterfactual) can be calculated. This is visualized in the graph above with the dotted blue line continuing the same pre-policy trend/slope. For the same time = 35 example above:
Predicted Yt = 52.90
There is a slight difference in the slopes of the before and after the policy change. Therefore, at different time points, the predicted outcome will vary and may not always be higher had the policy not been put into place.
https://doi.org/10.1371/journal.pone.0284235.s006
(TIF)
S5 Fig. Application rates in solicited HE applications.
[A] Chart showing changes in application rates in solicited HE applications prior to and following the guidance publication in 2015. [B] Table summarizing the coefficients from the time series analysis.
https://doi.org/10.1371/journal.pone.0284235.s007
(TIF)
S6 Fig. Funding rates in unsolicited HE applications.
[A] Chart showing changes in funding rates in unsolicited HE applications prior to and following the guidance publication in 2015. [B] Table summarizing the coefficients from the time series analysis.
https://doi.org/10.1371/journal.pone.0284235.s008
(TIF)
S7 Fig. Delayed effect in unsolicited HE applications.
[A] Chart showing changes in funding rates in unsolicited HE applications up to one year after publication of the guidance and starting one year after publication of the guidance. [B] Table summarizing the coefficients from the time series analysis.
https://doi.org/10.1371/journal.pone.0284235.s009
(TIF)
Acknowledgments
The authors would like to acknowledge the subject matter experts that validated a sample of the machine learning results for accuracy: Sarah Duffy and Gregory Bloss.
References
- 1.
National Institutes of Health. Clarifying NIH Priorities for Health Economics Research (NOT-OD-16-025). 2015 [posted 2015 Nov 25; cited 2023 March 24]. https://grants.nih.gov/grants/guide/notice-files/not-od-16-025.html
- 2.
Consortium of Social Science Associations. NIH Issues Notice Clarifying its Health Economics Research Priorities. 2015 [posted 2015 Dec 1; cited 2023 March 24]. https://cossa.org/2015/12/01/nih-issues-notice-clarifying-its-health-economics-research-priorities/
- 3.
Wolinetz, CD. NIH Priorities for Health Economics Research. American Society of Health Economists, 2015 [cited 2023 March 24]. https://www.ashecon.org/newsletter/v2015n2i2/
- 4.
BMJ Publishing Group Limited. A glossary of health economics terms. BMJ Best Practice. 2023 [cited 2023 March 24]. https://bestpractice.bmj.com/info/us/toolkit/ebm-tools/a-glossary-of-health-economics-terms/
- 5.
National Library of Medicine. Health Economics Information Resources: A Self-Study Course: Glossary of Frequently Encountered Terms in Health Economics. National Information Center on Health Services Research and Health Care Technology. 2020 [cited 2023 March 24]. https://www.nlm.nih.gov/nichsr/edu/healthecon/glossary.html
- 6.
Centers for Disease Control and Prevention. Economic Evaluation Glossary. 2021 [cited 2023 March 24]. https://www.cdc.gov/policy/polaris/economics/index.html
- 7.
York Health Economics Consortium. Glossary: Health Economic Terms. 2021 [cited 2023 March 24]. https://yhec.co.uk/resources/glossary/
- 8. Bernal JL, Cummins S, and Gasparrini A. Corrigendum to: Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2021; 50(3): 1045. pmid:32879971
- 9. Hudson J, Fielding S and Ramsay CR. Methodology and reporting characteristics of studies using interrupted time series design in healthcare. BMC Medical Research Methodology. 2019; 19(137). pmid:31272382
- 10.
Rebelo, Luis F. The Economics of Health and Health Care: Assessing health determinants and impacts on an aging population. U. Porto FEP. 2011 [cited 2023 March 24]. https://repositorioaberto.up.pt/bitstream/10216/62305/2/TESEFinal.pdf
- 11.
Centers for Disease Control and Prevention. Health Economics and Modeling Unit. 2013 [cited 2023 March 24]. https://www.cdc.gov/ncezid/dpei/hemu/
- 12.
Johns Hopkins Bloomberg School of Public Health. What is Health Economics. 2021 [cited 2023 March 24]. https://publichealth.jhu.edu/academics/academic-programs/masters-degrees/master-of-health-science-in-global-health-economics/what-is-health-economics
- 13. Villani J, Schully SD, Meyer P, Myles R L, Lee JA, Murray DM, and Vargas AJ. A Machine Learning Approach to Identify NIH-Funded Applied Prevention Research. American Journal of Preventive Medicine. 2018; 55(6): 926–931. pmid:30458951