Research funding is an important factor for public science. Funding may affect which research topics get addressed, and what research outputs are produced. However, funding has often been studied simplistically, using top-down or system-led perspectives. Such approaches often restrict analysis to confined national funding landscapes or single funding organizations and instruments in isolation. This overlooks interlinkages, broader funding researchers might access, and trends of growing funding complexity. This paper instead frames a ‘bottom-up’ approach that analytically distinguishes between increasing levels of aggregation of funding instrument co-use. Funding of research outputs is selected as one way to test this approach, with levels traced via funding acknowledgements (FAs) in papers published 2009–18 by researchers affiliated to Denmark, the Netherlands or Norway, in two test research fields (Food Science, Renewable Energy Research). Three funding aggregation levels are delineated: at the bottom, ‘funding configurations’ of funding instruments co-used by individual researchers (from single-authored papers with two or more FAs); a middle, ‘funding amalgamations’ level, of instruments co-used by collaborating researchers (from multi-authored papers with two or more FAs); and a ‘co-funding network’ of instruments co-used across all researchers active in a research field (all papers with two or more FAs). All three levels are found to include heterogenous funding co-use from inside and outside the test countries. There is also co-funding variety in terms of instrument ‘type’ (public, private, university or non-profit) and ‘origin’ (domestic, foreign or supranational). Limitations of the approach are noted, as well as its applicability for future analyses not using paper FAs to address finer details of research funding dynamics.
Citation: Aagaard K, Mongeon P, Ramos-Vielba I, Thomas DA (2021) Getting to the bottom of research funding: Acknowledging the complexity of funding dynamics. PLoS ONE 16(5): e0251488. https://doi.org/10.1371/journal.pone.0251488
Editor: Cassidy R. Sugimoto, Indiana University Bloomington, UNITED STATES
Received: July 29, 2020; Accepted: April 27, 2021; Published: May 12, 2021
Copyright: © 2021 Aagaard et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its two Supporting Information files. Further details of core/expansion approach to produce the publication dataset, i.e. core keywords, journals, and article-level clusters for ‘Renewable Energy Research’ and ‘Food Science’ publications, are in S1. The data used in the paper, in anonymised form so no personal data is included, is in S2.
Funding: This work was funded by the Novo Nordisk Foundation under the project title, 'Promoting the socio-economic impact of research - the role of funding practices (PROSECON)', grant number NNF18OC0034422 (received by authors KA, PM, IRV, DT). The funder website URL is: https://novonordiskfonden.dk/en/. The funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Ways and means to allocate research funding are considered one of the most influential elements in any attempt to govern contemporary public science. Funding is assumed likely to affect which topics get addressed, and the scope, content, direction, outputs and even potential impacts of public research [e.g. 1–4]. Funding secures the livelihood of researchers and is an indispensable prerequisite for almost all research . Yet knowledge of how funding affects research remains fragmented and inconclusive. This stems in part from how research funding has been studied. In this paper we develop and test an approach that attempts to broaden how such studies are framed, by focusing on how researchers co-use funding at multiple different levels of aggregation.
Limitations of existing research funding studies mainly stem from insufficient attention to recent dynamics, i.e. to growing heterogeneity, complexity and related dynamic trends in contemporary research funding [5, 6]. There have been few attempts to acknowledge and identify the wide variety of potentially interlinked sets of funders now operating in scientific fields. Similarly, there has been insufficient exploration of the characteristics of the assorted funding instruments researchers may co-use to do research. Instead, most literature has typically studied single funding instruments and single funding organizations [see e.g. 2, 7] or has analysed strictly-confined national funding landscapes from a top-down/system-led perspective [4, 8–10]. However, the underlying assumptions justifying these approaches may no longer be adequate. They may even prevent deeper understanding of how researchers’ research funding can be composed and co-used, and how it influences research .
Existing approaches are particularly challenged by increasing developments towards more complicated and dynamic, border-crossing funding landscapes, where regional, national and supranational funders have proliferated from public, private and non-profit sectors [3, 5, 11, 12]. Despite recent calls for studies of this emerging research funding reality [5, 13] most scholarly claims remain based on general observations rather than on systematic, empirical studies . An accurate understanding of how funding works, however, is a precondition for more realistic study of potential effects upon research of different kinds and combinations of funding. Improved understanding might also benefit research governance, revealing perhaps unsuspected funding synergies for policy action or discouraging use of possibly problematic research funding designs.
Research funding can also embody signals about needs that government(s), their agencies, industries and societies expect funded research to address . An indicative example is research funded to address Grand Societal Challenges (SCs). For instance, in Europe, dedicated SC-related funding is expected to support research aiming to address pressing global problems (e.g. food security, energy security, public health, impacts of a changing climate). Policy or funder attempts to (re)direct such research to address these challenges is of course mediated by what research funding researchers actually mobilise to do their research. This may now include co-using more than one funding source, potentially leading researchers to face multiple, even conflicting signals from multiple funders.
The funding individual researchers actually encounter and (co-)use therefore is a crucial element. This is especially so because researchers are also an acknowledged obligatory passage point between research funding and research practice . Approaches to study research funding, therefore, must explore the roles of funding from their viewpoint, rather than only from a coarse, system viewpoint. For this reason, our approach takes a researcher-led perspective. This leads to a ‘bottom-up approach’, entering the science system at the grassroots level of individual researchers and their funding, followed by increasing levels of aggregation of co-use of funding of collaborating researchers, and of a field of active researchers. In this paper, this approach is tested for two SC-related fields where multiple funders are known to operate, and are likely to provide varied funding instruments: Food Science and Renewable Energy Research.
For this approach, there are however different points in the research process through which funding could be studied. Funding could be studied as a research process input, as enabling certain research practices, or as associated with particular research outputs. To study funding as an input, for instance, document analysis of funding instrument characteristics could be attempted. To study funding of research practices, researchers could be surveyed or interviewed about their funding uses. Funding of varied forms of research outputs can instead be analysed. To demonstrate the broad applicability of a bottom-up approach, ideally several of these points would be studied together. For the approach testing purposes of this paper, however, we select only to study funding aggregation levels through research outputs in one of these forms: research publications. This selection is advantageous because funding of research outputs is reported in a standardized fashion. Studying non-paper formats of research output could run into difficulties, e.g. variation in funding reporting protocols. By contrast, in publications, for over a decade researchers’ have routinely self-reported certain funding details, as funding acknowledgements (FAs). To provide a manageable scale for the test, FAs in papers are selected only for researchers affiliated to research organizations (ROs) in three countries (i.e. Denmark, Norway, the Netherlands). This test selection gives both sufficient variety, as these are small, similar yet still intensive research funding contexts, and manageable scale.
Overall this paper aims, first, to frame a bottom-up, funding of researchers-led perspective on research funding dynamics. Second, it will empirically test an analysis of research funding using this perspective through the specific example of FAs in researchers’ publications. The approach is based upon three analytical levels:
- ‘Funding configurations’ of funding instruments co-used at the level of individual researchers.
- ‘Funding amalgamations’ of funding instruments co-used at the level of collaborating researchers.
- ‘Co-funding networks’ of funding instruments co-used at the level of all researchers active in the field.
The paper is structured as follows: the next section justifies why adding a bottom-up perspective is needed and could improve understanding of research funding dynamics. This is followed by a section presenting the bottom-up approach. The strengths and weaknesses of FAs as a data source to illustrate the approach are then reviewed, and the added value of using FAs for a multi-level approach to study research funding dynamics is described. The subsequent section describes the test case selection, research field delineations, FA data collection, cleaning and coding. Findings are then presented for funding configurations, funding amalgamations and field co-funding networks based on the FA data. Finally, we reflect on how the bottom-up, researcher-led approach to funding co-use contributes to understanding research funding dynamics, and conclude with implications for science policy and further research directions.
Why add a bottom-up perspective on research funding?
To understand the value of adding a bottom-up perspective, it should be noted that most previous approaches to research funding studies have relied on a top-down or system-led perspective. This means that to observe funding, studies have focused on features of funding aggregated for national or otherwise geographically confined funding landscapes [e.g. 8, 9, 17, 18]. Alternatively, they have examined effects of single funding organizations or single funding instruments within the science system, and attempted to isolate effects for just one funding source [e.g. 10].
There are sound, pragmatic, methodological and conceptual reasons for this situation. Clear research field delineations and geographical boundaries for funders’ assumed spheres of operation and influence, drawn from the top-down, can isolate effects of specific funding organizations or instruments from many other extraneous, mediating factors. They also likely service the interests of funders aiming to document and evaluate effects of their own specific investments. Likewise, a national focus may suit vested interests of national policy makers and stakeholders  in legitimising particular uses of public resources. Such analyses may be valid for specific research objectives, e.g. to delineate the portfolio of a single funding organization. However, additional, complimentary approaches are needed to generate broader understandings of researcher-level funding dynamics.
Traditional positions have also generated important insights. Top-down perspectives–typically based on data from OECD Main Science and Technology Indicators (MSTI), Eurostat, national statistics or large cross-country studies–afford broad overviews of central components of national funding systems. These enable comparative insights into important characteristics at country level, such as volume of R&D funding, balances between institutional funding and project funding, distribution of funding between disciplines, and allocation mechanisms for institutional funding [e.g. 4, 9, 19]. Studies of individual funders and single funding instruments have also provided insight into key characteristics and mechanisms, like how selection procedures for funding instruments might enable support of exceptional research [e.g. 2, 20] or how funding properties, and properties of the research funded, become interdependent for some funding instrument classes [e.g. 7, 21, 22].
However, such studies also build on two key assumptions that may have become increasingly questionable. First, they assume national funding landscapes mapped and studied from the top down provide a relatively complete picture of funding actually mobilised by researchers working within a given national system. Second, they assume individual funders and funding sources mainly operate in isolation–and hence their operations and effects for researchers using their funding can and should be studied in a kind of vacuum. These assumptions may once have been appropriate, but they are now challenged by multiple, interrelated developments in science and science policy.
First, funding has shifted from internal/institutional to external/project-based in most countries [2, 23–26]. Second, the scope of research policy has broadened to direct funding towards more diverse goals, e.g. promoting ‘excellence’ in science, providing solutions to economic and social problems, and fostering technology and innovation . Project funding instruments and funders have thus multiplied and become increasingly differentiated . In addition, public knowledge production has become more integrated into society, creating an ‘extended peer community’ [27, 28] where non-governmental actors are involved as funders, collaborators or stakeholders. Third, increased globalisation of science has made international research collaboration far more widespread , increasing the importance of non-national funding. This includes different types of EU allocations, funding from public, non-profit and private sources, and funding from other countries either directly or indirectly (i.e. through collaboration with international colleagues). All three of these challenges are potentially in play simultaneously. Consequently, the organization of science increasingly ceases to follow disciplinary and/or national borders, and instead features trans-national, trans-sectoral, multi/inter/trans-disciplinary academic communities or fields, and is characterised by both pervasive geographic and research field boundary-crossing [11, 29].
Contemporary research at individual, collaborative and field levels, therefore, also is likely to rely on multiple, heterogeneous funding, and funding co-use. Blends of funding instruments mobilised from the bottom up are a part of this development of increasingly multi-level, multi-actor governance systems, and will differ substantially from traditional perceptions of research funding driven by national, public authorities [3, 5, 6]. For these reasons, studying funding dynamics must now consider not only how national funding systems are designed from the top down, but also how funding is mobilised and even co-used by researchers from the bottom up.
While these challenges are increasingly acknowledged, the majority of literature has largely not addressed them. However, some recent studies have begun to call for a ‘reality check’ [see e.g. 5, 12, 13, 30]. It is becoming recognised that these research funding dynamics require new approaches, and the limitations of traditional approaches are increasingly highlighted [6, 12, 13]. It is argued that research funding might be better treated as sets of interlinked geographical and/or research field spaces of interaction between different layers of research funders and performers . These spaces are being recognised as not exogenously defined by organizational or geographical distinctions. Instead they need to be empirically observed . Additionally, the increasing importance of charities, supranational funding agencies [20, 31] and the emergence of new funding schemes for science  all need to be explicitly considered more. New approaches tailored to capture these more complex funding dynamics are, therefore, seen to be necessary .
Most previous studies are further challenged by the analytical level that is adopted. The typical emphasis on aggregating at national level may mask important field differences even within countries. These can include, e.g. large differences across scientific fields in relation to the balance between institutional and project funding . Similarly, the roles of non-public or non-national funders may vary across fields taken within the same national system. In some fields, public sources may dominate whereas private foundations, patient organizations or companies may in others. Field-specific funding organizations or funding instruments may be marginal for a national picture but nevertheless could play substantial roles at lower levels of aggregation. Similarly, the importance of non-national funding may vary across fields.
National or international research-related statistical organizations are also seldom able to capture the full extent of research fields, to provide usable field delineations for funding studies. Empirically-delineated research fields often will not correspond to the organizational or disciplinary categories used by nation-oriented bodies. Even statistical data at disaggregated levels, if originally framed by national levels, may introduce demarcation problems. Therefore, as science organization and funding have become more complex over time, unsurprisingly the availability of appropriate research funding statistics has struggled to keep pace .
Overall, how science and policy have developed has increasingly challenged traditional top-down, system-led perspectives on research funding. Coupled with field delineation and statistics shortcomings, this generates numerous, problematic assumptions about research funding. The available research funding literature has seemingly not yet fully embraced the implications of these changes. This justifies exploration of a new perspective and approach better suited to tackle these current funding dynamics, and to complement the strengths more traditional approaches can still provide (e.g. regularity, standardization).
A bottom-up approach to research funding dynamics
To begin, the bottom-up approach to research funding dynamics considers funding of the smallest knowledge producer in any research field, i.e. individual researchers, and their research funding instruments. A ‘funding instrument’ is understood as the lowest identifiable level, discrete resource unit provided by any funder (e.g. a specific grant from an internal or external funder). Individual researchers work at a research organization and, considered within a specific window of time, may sometimes co-use a set of funding instruments, i.e. a ‘funding configuration’. An individual researcher’s configuration could fund, e.g. writing a paper, or building equipment, conducting fieldwork, disseminating research results and so on. The next, middle level of aggregation considers funding of knowledge production where researchers collaborate, i.e. funding instruments co-used by collaborating researchers. Co-use may not mean researchers use each other’s funding here, simply that funding has somehow been used to undertake a collaborative research activity. This could include, but is not limited to, co-authoring a research publication. This is a ‘funding amalgamation’. The top level considers funding of knowledge production at level of a research field. This aggregates co-used funding instruments identified in the previous levels, considering now all the researchers active in a research field. This delineates the ‘co-funding network’ of that research field. Scientometrics-based study of co-funding typically only presents this level, and does not frame configurations or amalgamations to study co-use funding by individual or collaborating researchers.
Table 1 contrasts the shift in perspective these three levels of the bottom-up approach provide, relative to the traditional, top-down approach to research funding. Separately, it should also be noted this approach also differs from existing ‘bottom-up’ approaches in scientometrics used to assess research performance [see e.g. 33].
Fig 1 visualises the bottom-up approach’s three levels–i.e. funding configurations, funding amalgamations, and field co-funding networks. The bottom, ‘configurations’ level registers the distinct sets of funding instruments co-used by individual researchers; the middle, ‘amalgamations’ level registers sets of multiple funding instruments co-used, in some way, by collaborating researchers; and the top level is the ‘co-funding network’, registering all funding instrument co-use instances by all researchers active in the field (i.e. all instances when two or more instruments have been co-used to support individual or collaborative research activity, thus aggregating all forms of configurations, and all forms of amalgamations, within the given research field). The network nodes are instruments of different classes, that can be characterised using appropriate codes (e.g. public, private, national, supranational). The thickness of ties between instruments counts the occurrences of those specific linked instrument classes across the field. This is then based on all instances of instrument co-use by all researchers active in the given field (i.e. all configurations of two or more instruments, and all amalgamations of two or more instruments). Particular instrument classes may then be observed as co-used together more than others (e.g. public instrument-public instrument co-use may be the most common form in some fields; public-private-not-profit, in other fields) or instruments can be grouped by the funder that provided them, to study that aspect. The approach overall captures three different levels of funding instrument co-use. It enables study of how different kinds of funding are blended, in various ways, by individual researchers, by collaborating researchers, and across an entire field of active researchers. (To reiterate, this approach does not study isolated uses of single funding instruments by researchers–i.e. not involving co-use. This would be a separate design, and is not undertaken in our multi-level approach, which focuses exclusively on co-use.)
Funding acknowledgements to study funding co-use dynamics
To empirically test our bottom-up approach in the study of funding of researchers at different levels of aggregation, we look at funding of research outputs based on funding acknowledgements (FAs) in publications. We review previous uses of FAs in the literature, and emphasise that the approach developed in this study uses FAs to validate its multi-level framing of funding, and not to study research performance, e.g. from a scientometrics perspective, or to advocate that these funding levels can only be studied via publication FAs. Functional and behavioural-related limitations to using FAs as a data source to test our approach are noted. This is followed by a description of the added value of using FAs to study funding co-use dynamics from the bottom-up.
Previous FA-based studies.
Numerous FA-related studies have been produced since FA data became systematically available in large-scale publication databases from late-2008. Some of these have highlighted co-use dynamics. FAs have previously been used as a data source to demonstrate co-funding patterns [e.g. in nanotechnology; see 34, 35]. Analysis of FAs has also shown multiple funding instruments can be co-acknowledged in the same paper, e.g. a publication can acknowledge support from mixes of both public and private funders . FAs have also been used to map citation scores of research outputs affiliated to particular research organizations within a geographical boundary, e.g. the continent of Africa, to rank relative performance of individual research funders and/or performers .
Even when such studies attempt to delineate apparent ‘funding landscapes’ derived from FAs however, they strictly speaking constitute research performance landscapes, i.e. mapped primarily to assess research performance, and to address which individual research organizations, countries or funders perform or fund the highest cited research. Alternatively, FAs have been used to delineate which research themes and topics are funded across a particular field or geographical area  to inform research prioritisation and policy. In other words, such research typically uses FAs to assess performance or priorities–i.e. for evaluative or descriptive purposes , and does not typically disaggregate multiple levels of funding co-use, which is our approach.
Limitations of using FAs as a data source.
Use of FAs in previous literature has also revealed some limitations to their use as a data source. Reviewing these studies, a first class of limitations can be seen to be functional in nature (Table 2).
A second class of limitations to using FAs as a data source can be considered as caused by researchers’ acknowledging behaviours. They exist because although funders and authors increasingly consider providing FAs in papers to be mandatory or at least good practice, compliance can vary from researcher to researcher, across researcher country and language groups, or across fields. Consequently, FAs can under- or over represent funding data. FAs may also not reveal certain kinds of funding information, or can equate all funding so it seems to be of equal importance for the published research, when in fact it is not (see Table 3).
Strengths of using FAs as a data source.
Despite these limitations, using FAs for a papers-based test of the bottom-up approach usefully enables delineation of funding configurations (funding of outputs produced by individual researchers), funding amalgamations (funding of outputs produced by collaborating researchers) and co-funding networks (funding of outputs by all researchers active in a field). These would otherwise be difficult to map without using FAs [40, 54]. Other methods could have challenges regarding resource intensity, intrusiveness and coverage. For example, instead of using publication FAs, researchers in a field could be surveyed and asked to self-report their funding instruments or configurations. Research groups, teams or networks could be surveyed about funding amalgamations. Such surveys, however, could have issues of memory recall and data completeness, and could be resource-intensive.
Selecting a FA-based method primarily to test the bottom-up approach instead has clear advantages. First, for the test, compared to other methods, e.g. surveys or interviews, FAs are pre-existing paratexts in papers and can be automatically extracted, at scale, from publication databases. This means using FAs is cost-effective. It also enables large-scale, unobtrusive study of funding across all three levels of the bottom-up approach. Second, FAs are funding data self-reported by researchers themselves, rather than via third-party sources, such as funder databases or other repositories. These may not be as direct or comprehensive as FAs. Third, a FA-based approach is scalable and repeatable longitudinally. For studies beyond our current test, they could be used to sample multiple time periods with little marginal cost, if relevant periods are covered by publication datasets. Fourth, as publishers and publication datasets adapt to and advocate their increasing use, FAs may become more available, consistent and stable. As a data source they may become more significant for using a bottom-up approach over time. The primary drawback, however, is that using FAs artificially restricts analysis of funding co-use dynamics to study of funded papers. It excludes studying other co-use instances, which would need separate research, exploring beyond just publication-related research activity.
Using FAs for a bottom-up approach to research funding.
To operationalize use of FAs to test study of research funding dynamics from the bottom up, the starting assumption is that individual researchers can co-use multiple funding instruments, which they acknowledge via FAs, even in single-authored research (i.e. these would then be funding configurations). Funding instruments can become more aggregated within collaborative research activities. These mix–to an unknown greater or lesser extent–some or all the different funding instruments brought in by every researcher (i.e. funding amalgamations). These are evidenced via FAs in multi-authored publications, where co-use includes researchers using separate funding instruments to co-author the paper and/or researchers’ sharing funding instruments. Finally, co-use of funding instruments across a field of researchers becomes highly aggregated. This can be delineated from FA data as a field co-funding network. To produce each level, associated analytical steps are required. These are careful research field delineation, name disambiguation, cleaning of all acknowledged funding instruments, and analytically-informed coding of FAs to classify funding instruments.
To conceptualise this coding, the bottom-up operationalization approaches FAs as traces of complex, adaptive, global contemporary science, potentially involving many funding actors. This acknowledges that paper FAs as a data source report actual funding (co-)used by researchers in a way that is indifferent to boundary/border issues. That is, they naturally accommodate the reality that researchers can and do reach both below and above national levels to obtain funding instruments, and to ‘configure’ or ‘amalgamate’ them together in ways that can transcend geographically-constrained, topic or sector-specific funding patterns . Nevertheless, an ability to study any national system-related patterns can still be retained. This is done by registering the author affiliation(s) of researchers. For this test, this will be to one or more of the three selected countries (Denmark, Netherlands, Norway), taken from reported author affiliation(s) in publications, in the two test research fields (Food Science, Renewable Energy Research).
There are further issues to note for the operationalization. These concern attribution. For funding configurations, FAs in single-authored publications do provide directly attributable data on funding instrument co-use by individual researchers. In a paper with only one author, if multiple instruments are reported, they unequivocally have been co-used. However, funding amalgamations are operationalized via funding of multi-authored publications. FAs usually provide an undifferentiated data string, without funding-author attribution. They may not attribute each instrument (FA) to an author. It cannot be robustly determined if all the acknowledged funding belongs to only one author (e.g. other authors acknowledge or have no funding). Funding amalgamations can therefore vary. They may involve fusion, where all authors producing the paper are supported by one or more shared funding instrument. They may feature juxtaposition, where authors do not mix funding, but use their own, separate instruments. The amalgamation does not aim to describe how acknowledged funding instruments have been mixed; this needs exploration via other methods. It registers only an analytical level of funding co-use–broadly understood–as funding contributing, in some as yet unknown way, to the act of collaborating to produce a paper.
Publication FAs also typically provide only funder and/or funding instrument names. Therefore, the operationalization needs to code funding instruments. This requires selecting instrument characteristics that are analytically meaningful to label. Based on previous literature, ‘type’ and ‘origin’ were selected to be coded for our test (although others could be developed in future). Type distinguishes whether funding instruments are from a public, private, non-profit or university funder. This can classify the often distinctive kinds of funding instruments provided by these bodies [see 5, 10, 55–57]. Deliberately, both internal and external funding ‘types’ are coded. The role of university/internal funding (e.g. institutional/block grant funding of salaried research time, specific internally funded research activities) is quite underexplored [see 30], so should be included
Coding origin follows established practices for FAs [e.g. 36]. It distinguishes domestic from foreign or supranational funding instruments. This enables specific characteristics to be flagged, e.g. domestic (i.e. national or sub-national) funding instruments can differ from foreign funding (non-domestic national or sub-national) and from supranational funding in analytically relevant ways. This may be due to these funders’ geographical scope of operation or focus on particular challenges, economic and/or societal missions (e.g. contrasting the EU, multinational corporations, and international non-governmental organizations).
Selecting the test cases
Our two test cases, the Renewable Energy Research and Food Science research fields, were selected to test the approach. This is because they have varied funding and assorted funders to delineate, due to these fields having a wide breadth of research topic/keyword coverage [e.g. see 58]. They also feature both fundamental science and applied research, and long traditions of involving industry, public authorities, research users and other collaborators. The two fields are also thematically oriented. This means they can engage grand SC themes, e.g. United Nations Sustainable Development Goals (SDGs) and related policies and politics. These test fields do not represent generally the global science system. They are however, appropriate, dynamic and multi-faceted candidates to test the bottom-up approach. They represent a confirmatory case selection . The two fields provide similar heterogeneity, yet sufficient variation to enable testing of the approach.
Researcher author affiliation(s) were selected to be research organizations (ROs) in Denmark, the Netherlands and/or Norway. This retains an ability to explore the role of national contexts, even when adopting a bottom-up, country-crossing perspective. These three selected countries are all small, advanced, Western European economies. They have similar Humboldt-inspired university systems, relatively high shares of institutional funding, and traditional public research councils . At the same time, they have country specific funder/funding variations. E.g. Denmark has relatively clear demarcation between scholarly and societally oriented funders; Norway is more centralized with a broad, unified research council. All three affiliation countries also have active Food Science and Renewable Energy Research sectors.
Field delineations and time horizon selection.
To delineate the two test research fields, Whitley’s notion was adopted of ‘intellectual fields’ as a ‘broader and more general social unit of knowledge production and co-ordination’ than disciplines . Research ‘fields’ were considered as fairly broad units of knowledge production. They can engage activities outside academia, knowledge co-production with industry or other societal collaborators, and still involve public research organization-related social and organizational structures.
To develop the data collection process, all Web of Science (WoS) publications were retrieved with at least one author with any affiliation to a research organization located in Denmark, the Netherlands or Norway. Renewable Energy Research and Food Science publication datasets were created using a core/expansion approach. The core datasets for the two fields included all papers meeting all three criteria of: containing at least one of our search terms; being published in a relevant journal; and being part of a relevant article-level cluster.
The list of search terms used for each field can be found in S1 File. The relevant journals from each field were selected by going through the list of journals included in relevant WoS subject categories (i.e. “agriculture, dairy & animal science”, “agricultural economics & policy”, “agriculture, multidisciplinary”, “agricultural experiment station reports”, “food science & technology” and “agricultural engineering” for Food Science; and “energy & fuels” and “green & sustainable science & technology” for Renewable Energy Research; previous energy research  was also consulted) or by containing one of our search terms in their title. In the Centre for Science and Technology Studies (CWTS, Leiden, Netherlands) database used for this paper (that included high resolution funding data and author-name disambiguation ) articles were clustered based on citation relationships [c.f. 62, 63]. Clusters with at least five publications containing one of our keywords were screened to identify those clearly related to the fields. The relevant journals and article-level clusters are also included in S1 File.
These two core datasets were then expanded by including all papers meeting any three of the following five criteria:
- contained at least one of the search terms;
- published in a relevant journal;
- part of a relevant article-level cluster;
- cited or is cited by a publication in the core set;
- shared an author with a publication in the core set.
Finally, in each field dataset, all papers in the micro-clusters for which more than one third of publications had already been included at that point were also included.
To include publications and their FA data for a reasonable period, the dataset time horizon was selected to be 10 years, i.e. all field publications 2009–18. This had the advantage of starting the datasets soon after WoS began to systematize and make available suitable FA metadata. No specific time criterion was used for author inclusion, i.e. for whether a particular author-researcher had to appear multiple times within the period to be considered ‘active’ in a field. This process produced a publications dataset, covering all six field/affiliation cases, which could be separately studied by filtering fields and/or countries. Gephi software was used for network visualizations.
Disambiguating and coding funding instruments from FA data.
All FA data in this dataset of publications was cleaned and coded. FAs were taken from the ‘funding agencies’ metadata field of WoS records. This provided funder organization names, names of sub-divisions of these funders, and sometimes funding grant numbers or funded project names (i.e. specific details of funding instruments).
Initially, 25,522 distinct funder strings were automatically retrieved from the FAs. Funder strings that occurred at least twice in the dataset were cleaned and disambiguated. Sub-divisions of a funder found in that set were grouped under the main named funder (e.g. diverse funding instruments of the European Commission were grouped). This process merged 19,140 (75%) of the 25,522 original funder strings into a cleaned list of 2,363 unique funding organizations to analyse. Throughout, data errors encountered during the extraction process were addressed, e.g. cleaning funder name variants.
Given the limitations of FA data we have already noted, funders had to be disambiguated and classified so as to code their associated, acknowledged funding instruments. This was done using the selected data labels of type and origin. Four values were used to code type of instruments:
- Public–typically instruments from research councils and public/state authorities like ministries and agencies (but including supranational organizations like the EU, and regional and local authorities).
- Private–predominantly instruments from companies/corporations.
- Non-Profit–instruments from private non-profit organizations, patient organizations, national and supranational non-governmental organizations (NGOs) and related others.
- University–internal funding directly from a university (most likely either institutional funding or competitively awarded internal university funds).
Three values were coded for origin:
- Domestic–instruments originating from Denmark, Norway or Netherlands funders (respective to author affiliation).
- Foreign–instruments originating from all other countries, except the domestic country.
- Supranational–instruments from funders like the EU, OECD, UN agencies.
Table 4 shows the final total number of articles collected in the full dataset, the number and percentage of papers with at least one acknowledged funder (i.e. instrument detail), and the share of FAs able to be coded for type and/or origin (i.e. between 67.9% and 77.8% for type; between 70.7% and 81.2% for origin). There was an average of 2.3 FAs per paper, for the full dataset of papers with FAs (i.e. 55,089 FAs across the ~23,000 papers with FAs). Non-coded FAs were sampled to check for systematic bias, and their characteristics were found to be acceptably similar to the coded proportion.
The sub-sample used to obtain configurations, amalgamations and co-funding networks was derived after exploring the number of acknowledged funders across all publications (see Fig 2). The aim was to exclude articles with no FAs or only one FA. Across the cases, this left 35–45% of the articles from the full dataset with two or more FAs (i.e. instruments). Of these papers, 0.7% were single-authored (funding configurations) and 99.3% were multi-authored (funding amalgamations). Co-funding networks were obtained by including both single and multi-authored papers with multiple FAs.
Findings from testing the approach are now presented. For funding configurations, this presents patterns of co-used (i.e. co-acknowledged) funding instruments at individual researcher level from single-authored publications. For funding amalgamations, patterns of co-used funding instruments, number of FAs, and number of authors for collaborating researchers, are presented to demonstrate dynamics for multi-authored papers. Finally, field co-funding networks are presented for Renewable Energy Research and Food Science, respectively. This combines all possible affiliation countries and co-use funding patterns from both single and multi-authored papers. Country affiliation-specific networks are also presented, to demonstrate the approach can still illustrate nation-related features for study. Across all three levels, funding instrument type (public, private, non-profit, university) and origin (domestic, foreign, supranational) are explored. This presents our approach to co-use of funding by researchers, conceived of as three increasing levels of aggregation–funding co-use by individual researchers, co-use by collaborating researchers, and co-use by a field of researchers. This illustrates co-use of funding by researchers through the specific lens of papers as a research output. As stated earlier, funding co-use could be explored for different research activities, but still aggregated at these three levels. Throughout, it should also be stressed these are illustrative findings to test the approach, not a comprehensive overview of Renewable Energy Research or Food Science.
The bottom level illustration is obtained from single-authored papers with two or more FAs, so presents individual researcher level, funding configurations. Table 5 shows funding configuration patterns. This is in terms of number of funding instruments being configured. From this observation, two findings become evident. First, funding configurations exist in the data. There is acknowledged co-use of funding instruments even in individual researchers’ lone publication outputs (FAs were checked to confirm that each FA did in fact refer to a distinct instrument). Second, in both fields, funding configuration scales vary–even for this small sub-sample. Most configurations featured two instruments (62 out of 90 papers). Others had three, four or even five or more instruments reportedly co-used for a single-authored publication.
This bottom level, for this illustrative test, is only a small sub-sample of the full dataset. It has only 75 authors and 62 single-authored publications, acknowledging two or more funding instruments. It should be stressed this does not demonstrate that some representative portion of researchers in these two fields have funding configurations. Instead, the data verify that it is valid and important to consider, and to focus upon dynamics of co-use of funding, at individual researcher level–i.e. it is valid to study this analytical level, when exploring uses and roles of funding for research.
A limitation of our FA-based test of the approach is that funding configurations are only observable for single-authored papers here. In highly collaborative fields, single-authored papers are likely rare, and multi-authored papers more common. Separate study could use the insight that funding is co-used at individual researcher level, to study individual researchers via a different research activity. This could, e.g. be individual researchers co-using funding for building equipment, for undertaking fieldwork, or to support policy engagement activity. Even with a small sub-sample, there is configuration variety, in terms of funding instrument type and origin. We argue then these findings warrant such additional explorations. Separate qualitative work on single-author researchers in this dataset has also highlighted variety in reasons for, and mobilizations of co-used funding instruments, see .
The main finding here is not that these particular configurations are representative of individual researcher co-use of funding in these specific fields. Rather it suggests there are potentially interesting dynamics at this level that warrant the inclusion of funding configurations in studies of funding moving forward. Most importantly, these dynamics go unexplored if this level of analysis is excluded. However, separate study of research activity–not limited to publications/FAs–would be needed to understand exactly how and why individual researchers co-use funding instruments, for various research purposes. Similarly, exploring which patterns of funding configurations are prevalent in any given field would require alternative methods. This is because FAs cannot study individual researcher co-use dynamics that happens anywhere other than in single-authored papers.
Within these limitations, and for our specific FA-based findings, we see configurations of ‘public’ instruments are the most common co-use by individual researchers, in terms of funding instrument ‘type’. However, there are also ‘public-private’ and ‘public-university’ configurations, and ‘non-profit’ instruments configured with all three other types by individual researchers. Similarly, there are configurations with mixed ‘origins’. The most common is ‘domestic-domestic’. This seems an interesting indication of even single-authored research needing to draw upon multiple domestic instruments (e.g. multiple national funders). However, there are also ‘domestic-foreign’, ‘domestic-supranational’ and ‘foreign-supranational’ configuration varieties.
Overall, the sub-sample shows funding configurations exist, and that they can vary in terms of type and origin. This is even when only exploring a small number of cases. Taken together, this suggests configurations are a valid analytical level to consider in studies of funding dynamics in a broader sense (and this would eventually be broader than only FA/publication-based data collection).
The middle level in our bottom-up approach presents amalgamations of instruments co-acknowledged in multi-authored papers (with two or more authors, and two or more FAs). These funding amalgamations indicate that collaborating researchers co-use a wide range of different instruments. This is seemingly broader than how co-use occurs in funding configurations of individual researchers. As we have stated, it cannot be determined–through FAs alone–whether these amalgamations are fused or juxtaposed instrument co-use. However, these illustrative findings are promising to validate the approach, and an emphasis upon this level of funding aggregation. They suggest more funding variety can be seen in funding amalgamations than in funding configurations. This tentatively offers interesting insights by including this level of funding aggregation in studies. It also indicates potentially important funding dynamics may arise from co-use of instruments at the level of collaborating researchers (here addressed only for multiple authors co-authoring a paper).
Patterns of funding amalgamations are presented by type of instrument in Table 6. The most prevalent amalgamation type is different public funders being co-acknowledged. In Renewable Energy Research for Netherlands- and Denmark-affiliated publications, this ‘public-public’ funding amalgamation type accounted for around 60% of FA cases (60.9% and 58.8%, respectively); for Norway-affiliations, it represented 43.4%. For Food Science, across the same three affiliations, the pattern was different: 44.9% for Netherlands-affiliated, 44.2% for Denmark, and 50.4% for Norway. Such patterns might suggest certain co-influences of these types of funders for these collaborative research outputs, and that influences may vary by field.
Many ‘public-public’ amalgamations included the EU (coded as a supranational, public instrument) plus a national public instrument. Public-private (i.e. companies) and public-university amalgamations were also present. Norway-affiliated articles had the highest prevalence of these; Netherlands-affiliations had the lowest. Amalgamations with no public funding instruments were rare. For Denmark-affiliations these accounted for less than 10 percent of amalgamations; for Netherlands and Norway-affiliations, less than 20 percent.
Patterns of funding amalgamations by instrument origin are presented in Table 7. Across all three affiliations, in both fields, the most prevalent amalgamation was ‘domestic-domestic’ (32.5% for Renewable Energy Research; 34.2% for Food Science). There were amalgamations with no domestic funding acknowledged. This indicates papers funded exclusively by amalgamations of foreign and/or supranational funders. Amalgamations in papers with Netherlands-affiliated authors reported the fewest domestic funders.
There were also ‘domestic-foreign’ and ‘domestic-supranational’ amalgamations. These are country-crossing co-funded publications, funded by funders operating in different countries or at differing geographic scales. Further research could determine if this was intentional action, e.g. funder-to-funder coordination efforts or international research projects, or due to separate dynamics such as researcher collaborations not attributable to the acknowledged funding.
Co-funding networks for the two fields present dynamics that stretch beyond national landscapes. They differ from what might be visualised via nationally-supplied research funding statistics or perspectives. They present an overall view of co-use of funding at the level of researchers active in a field, built upon funding co-use by discrete researcher collaborations (amalgamations) and co-use by individual researchers (configurations). At this level, co-used instruments can also be grouped according to the funders providing them, in addition to labelling of instrument type and origin.
Fig 3 presents the Renewable Energy Research co-funding network across all author country affiliations; Fig 4 shows the Food Science co-funding network for all affiliations. Both networks have many instances of multiple, rather than simple, ties between funding instrument types (indicated by node shape; with funder names also shown) and origins (shown by node colour).
Origin colours: Blue = affiliated-Denmark; green = affiliated-Netherlands; orange = affiliated-Norway; purple = affiliated other countries; red = supranational. Type shapes: Triangle = non-profit; square = public; circle = private; pentagon = university. Node size = number of FAs mentioning that funding instrument; ties = instrument co-occurrences within the same article’s FAs.
Origin colours: Green = affiliated-Denmark; orange = affiliated-Netherlands; blue = affiliated-Norway; purple = affiliated other countries; red = supranational. Other labels as per Fig 3.
Diverse associations emerge from these visual representations. For example, for Fig 3, a first noticeable feature is the centrality of the EU within the co-funding field network. Such a focal position is emphasized by its node size, indicating the number of FAs mentioning any EU funding instrument, and by multiple links converging at this particular node. The distribution of the most acknowledged funders, to some extent, is also displayed in the highly interconnected sub-network areas. These are populated by funder names from each of the three specific study countries. Among them, particular funding agencies, such as the Danish Council for Strategic Research, seem to play a significant role. Research funding from other countries (e.g. China, USA, Sweden or Spain) may stem from internationalization patterns in publication co-authorships (this would require further study). Collaborative knowledge production is reflected in the co-use of funding instruments from different types and origins, to generate scientific outputs in a field. Varied and mixed combinations (e.g. public-public/supranational-foreign features) are also evident, similarly in Fig 4. This reveals a fine-grained view of funding at the top level of our approach. Separate methods could in future be used to determine the degree to which the network positions and apparent roles of each funder here are strategic, i.e. a conscious result of some funder actions or deliberate funding instrument designs. The co-funding network could be used to begin such an exploration, e.g. interviews with funders–perhaps also combined with insights from the observed funding amalgamations and configurations.
Similarly, assorted type-based ties are present in both Figs 3 and 4. Specific private companies and non-profit organizations are identifiable among the variety of research funding acknowledged across all publications in both fields. For instance, in Fig 3 we see energy-related organizations, and in Fig 4, food-related ones. Within the Fig 3 network, as an example, numerous university type instruments (internal funding) are linked together, as well as simultaneously to multiple varieties of public, private and non-profit instruments (e.g. Technical University of Denmark instruments linked to those of Aarhus University, the Novo Nordisk and Carlsberg foundations, Danish or Norwegian public agencies). In other words, researchers here are engaged in heterogeneous funding co-use (similarly to their configurations and amalgamations, but now at a higher level).
The full dataset can be filtered to present other related funding dynamics to study, i.e. to show field co-funding networks filtered by author country affiliation. These country-filtered co-funding networks still present highly heterogeneous funding dynamics, due to how they have been built from configurations and amalgamations, rather than top down. For our cases, traditional public funding sources, like the EU, remain prevalent once the networks are filtered by affiliation countries, but are clearly operating as parts of large networks of interlinked funding instruments, and not in isolation.
Specific features are also present when the networks are filtered by affiliations. For instance, when filtered for Norway-affiliated researchers, the Norwegian Research Council is shown to have a ‘hub’ function. This may be expected for a centralised public funder, but here can be visualized as occurring from researchers actually co-using their funding, in some way, to collaborate to produce research outputs (irrespective of whether such activity was strategically intended by funders). However, even when filtered this way, ties between more traditional national public funders and non-public, foreign or supranational funders are also present. This emphasizes the interconnected nature of the funding. Field-specific funders stand out for certain cases, e.g. oil companies in the Norway-affiliated Renewable Energy Research case. Additionally, for all six cases, filtered co-funding networks show smaller/niche funders that might be overlooked in a top-down perspective.
This filtering approach underlines that the bottom-up approach does not have to discard entirely the notion of countries. National funding remains present and can still be studied. National funders appear important across all cases, even with the noted emergence generally of multi-level, multi-actor ‘global science’ [c.f. 11]. The bottom-up approach can also provide more nuance here, and show that traditional funders nevertheless do however operate in heterogeneous, variously interlinked ways with other funders’ instruments, due to multiple levels of co-use by researchers.
Overall, the bottom-up approach enables study of potentially useful insights into seemingly highly complex funding situations. Future research using the same approach could go further, e.g. to explore field-nexus contexts, e.g. a two field, energy-and-food co-funding network. This would make sense primarily in instances where research topics overlap between fields. For such cases, the approach could reveal unexpected linkages. Similarly, the co-funding networks could be filtered according to author affiliation(s). This would enable study of ranked prevalence and variety of named funders/funding instruments, relevant to country affiliation related aspects.
Discussion and conclusion
Overall, the findings presented above support the key assumption in this paper: taking a researcher-led perspective to get to the ‘bottom’ of research funding can reveal important, varied funding co-use dynamics. There are also several other implications from testing this bottom-up approach.
First, for the test fields of Renewable Energy Research and Food Science, the configuration, amalgamation and co-funding network levels present funding co-use dynamics one would expect from globalisation of science or ‘global science’ , but add more nuance and granularity. The bottom-up perspective, as tested here, verifies recent assertions that knowledge production is increasingly no longer (if indeed it ever was) the ‘product’ of any particular, single country or national science system. This assertion holds across co-use of funding by individual researchers (configurations), collaborating researchers (amalgamations), and fields of collaborating researchers (co-funding networks). It implies not only individual researchers and research organizations navigate an increasingly complex globalised science funding system, but also that funders need to understand and adapt to this complicated operational context, featuring highly interlinked, multi-level, country border-crossing and seemingly interdependent funding co-uses [see 12]. Further research using the bottom-up approach for more fields, and for other collaborative research activities not just limited to research in the form of publications (and funding registering via FAs), could generate more general insights about funding dynamics, throughout more of the global science system.
Second, the findings support the assertion that a bottom-up, researcher-led perspective on research funding is valid and necessary. The approach provides useful analytical units that aggregate funding at meaningful levels (researchers, collaborators, fields). These add nuance and help to understand more about the nature of contemporary, interlinked research funding realities in various fields and contexts. They have been tested here using FAs and focusing on publications, but need not be limited to this. At the same time, the framing and test may bring new opportunities for further exploration of the possible complementarities that bottom-up findings could contribute to top-down/system-led studies.
Third, the test presented a high heterogeneity of funding co-use. Domestic, foreign and supranational funders from public, private, university and non-profit sectors were configured, amalgamated and networked in various ways. The implication for understanding funding is that researchers (either individual, collaborating or within a field) can co-use assorted funding instruments to do research. Certain national and sectoral specificities remain present, but geographic boundary-spanning dynamics seem now inherent. This implies future studies of research funding not only need to assign continued importance to national funding but also address funding co-use that can be present right from the bottom up, for individual researchers and beyond. In other words, this bottom-up approach is not simply intended to be a descriptive analysis of funding/funders, similar to what is already possible with scientometrics approaches. It is an attempted re-framing, aiming to stress that there are additional, important levels at which of co-use funding by researchers needs to be considered, because of increasing complexity in contemporary funding dynamics.
Fourth, the bottom-up approach could be used to generate intelligence for funders, policymakers, university research managers and so on. The approach is grounded in funding as it is actually co-used. This is independent of whether such use has been strategically coordinated, intended or expected by various stakeholders [c.f. 35, 47, 48, 65]. This perspective could potentially enable funders to look more inclusively and pragmatically at their funding portfolios, to assess and adjust their bi- or multi-lateral coordination efforts, and perhaps to understand better their potential co-influence throughout the science system [c.f. 39, 44, 52].
At the same time, two limitations need to be highlighted. First, the selected fields to test the approach are highly applied and interdisciplinary. These cases may have limited generalisability. Specifically, they might have idiosyncratic knowledge production and collaborations that are not relevant elsewhere (e.g. in mono-disciplinary fields). Their similarity was useful in this paper to supply funding variety to test the multi-level funding co-use approach. For wider applicability, the approach should be tested not only with more but also with different fields. Similarly, the configuration, amalgamation and co-funding network patterns presented in this paper are obviously specific to the two illustrative fields and for Denmark, Netherlands or Norway affiliated researchers only (i.e. these are researchers in small but open economies that already have strong international ties). Therefore, these cases likely affect how complex and country boundary-spanning co-use appears here.
A second limitation concerns the selected collaborative research activity of research outputs/publications, and use of FAs as the data source for the test and operationalization of our approach. Publication FAs do provide relevant data. However, they do not afford fine details to unpack funding dynamics. Improving guidelines around author use of FAs could potentially contribute to this end, e.g. journals could mandate clear, robust author-funding attribution. The bottom-up approach could then better study the co-use dynamics of funding amalgamations, using this additional funding-author attribution data. Nevertheless, it would remain challenging to attribute particular discrete research content or insights within any given article to particular funding, even with direct author-funding attribution. Further research would still be needed to provide understanding of whether collaborating authors used their own funding in isolation (amalgamations juxtaposing independent funding instruments) or collectively (amalgamations fusing interdependent instruments).
The three levels of the bottom-up analysis could also be differently explored. Future research could use additional methods to study, for instance, what funding configurations exist in particular fields and contexts–or how different types and origins of funding instruments are co-used within configurations. As we have stated, this aggregation level of funding co-use has been largely overlooked in previous studies. And yet, as we have illustrated, even for a small sub-sample, configurations can exist as a distinct level of funding co-use. They can also be varied, warranting further research attention. This clearly requires use of a different method than tracing funding via FAs, and a sole focus on papers. This can result in insufficient numbers of configurations to study (i.e. single-authored papers), particularly in fields where publication is primarily collaborative (multi-authored). Additionally, for funding amalgamations, their composition and how instruments are co-used are other potential research interests. In particular, the inability of a FA-based empirical exploration to distinguish between fused or juxtaposed amalgamations, invites further research using additional methods. Likewise, variation in patterns of co-funding networks in different field types, in field-spanning contexts, or in emerging research areas located in-between existing fields, could also be investigated. For instance, here it could be explored whether network patterns correlate with deliberate funder strategies to achieve synergy through funding co-use within a given field. There could also be separate but related questions to explore, including whether and how researchers enact agency to pursue strategies to (co-)shape the availability and characteristics of funding in a field, which researchers later assemble into funding configurations, amalgamations, and co-funding networks. This could explore if researchers attempt to influence funders to affect funding provision that they need to co-use to do their research in particular ways, in their field [c.f. 5, 14].
Expanding the approach could include additional methods like interviews, case studies, bibliometrics, altmetrics, surveys and/or network analysis. The approach presented in this paper clearly remains instrumental. It has been an initial means to these kinds of analytical ends, not a final goal in itself. Next steps could add further funding variables or characteristics to move beyond the descriptive, technical classifications of type and origin. Further research could attempt to determine what is distinctive about funding instruments apart from these labels, in a given context, which was beyond the scope of this paper. Examples could be features of autonomy and flexibility of instruments, their duration, amount, or focus on impact/users–and whether these matter more in some fields than others. Additionally, whether and how researchers respond to such characteristics of funding provision through configuration and amalgamation-related behaviours could also be explored.
For this current paper, the bottom-up, co-use of funding by researchers approach has provided both a ‘proof-of-concept’ of the perspective, and a foundation for such future studies. For the approach to provide more insights, and for it to be generalisable across the science system, it should ideally comprise part of a broader, multi-faceted framework and mixed methods toolkit. This could then facilitate detailed study and better understanding of a wide range of the effects of contemporary research funding dynamics upon researchers and their research.
S1 File. Dataset core/expansion further details.
List of core keywords, relevant journals, and article-level clusters for the dataset of Renewable Energy Research and Food Science publications.
We thank three anonymous reviewers for their helpful comments, and are grateful for insights and comments on an earlier version of this work from Carter Bloch, Arlette Jappe, Maria Nedeva and Corina Balaban.
- 1. Edqvist O. Layered science and science policies. Minerva. 2003;41(3):207–21.
- 2. Heinze T. How to sponsor ground-breaking research: a comparison of funding schemes. Science and public policy. 2008;35(5):302–18.
- 3. Sörlin S. Funding diversity: performance-based funding regimes as drivers of differentiation in higher education systems. Higher Education Policy. 2007;20(4):413–40.
- 4. Aagaard K. The Evolution of a National Research Funding System: Transformative Change Through Layering and Displacement. Minerva. 2017;55:279–97.
- 5. Gläser J, Serrano Velarde K. Changing Funding Arrangements and the Production of Scientific Knowledge: Introduction to the Special Issue. Minerva. 2018;56(1):1–10.
- 6. Lepori B, Reale E. The changing governance of research systems. Agencification and organizational differentiation in research funding organizations. Handbook on Science and Public Policy: Edward Elgar Publishing; 2019.
- 7. Franssen T, Scholten W, Hessels LK, de Rijcke S. The Drawbacks of Project Funding for Epistemic Innovation: Comparing Institutional Affordances and Constraints of Different Types of Research Funding. Minerva. 2018;56(1):11–33. pmid:29449745
- 8. Auranen O, Nieminen M. University research funding and publication performance—An international comparison. Research Policy. 2010;39(6):822–34.
- 9. Lepori B. Public research funding and research policy: a long-term analysis for the Swiss case. Science and Public Policy. 2006;33(4):205–16.
- 10. Reale E. Analysis of National Public Research Funding-PREF. JRC- European Commission, 2017.
- 11. Wagner CS. Global science for global challenges. In: Simon D, Kuhlmann S, Stamm J, Canzler W, editors. Handbook on Science and Public Policy. Cheltenham, UK: Edward Elgar Publishing Limited; 2019. p. 92–103.
- 12. Lepori B. Coordination modes in public funding systems. Research Policy. 2011;40:355–67.
- 13. Cocos M, Lepori B. What we know about research policy mix. Science and Public Policy. 2020;47(2):235–45.
- 14. Gläser J. How can governance change research content? Linking science policy studies to the sociology of science. In: Simon D, Kuhlmann S, Stamm J, Canzler W, editors. Handbook on Science and Public Policy. Cheltenham, UK: Edward Elgar Publishing; 2019. p. 419–47.
- 15. Veletanlić E, Sá C. Implementing the Innovation Agenda: A Study of Change at a Research Funding Agency. Minerva. 2020:1–23.
- 16. Laudel G, Gläser J. Beyond breakthrough research: Epistemic properties of research and their consequences for research funding. Research Policy. 2014;43(7):1204–16.
- 17. Öquist G, Benner M. Why are some nations more successful than others in research impact? A comparison between Denmark and Sweden. Incentives and performance: Springer; 2015. p. 241–57.
- 18. Aagaard K, Schneider JW. Research funding and national academic performance: Examination of a Danish success story. Science and Public Policy. 2016;43(4):518–31.
- 19. Aagaard K. Kampen om basismidlerne: Historisk institutionel analyse af basisbevillingsmodellens udvikling på universitetsområdet i Danmark: Dansk Center for Forskninganalyse; 2011. Danish.
- 20. Luukkonen T. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation. 2012;21(1):48–60.
- 21. Lal B, Hughes ME, Shipp S, Lee EC, Richards AM, Zhu A. Outcome Evaluation of the National Institutes of Health (NIH) Director’s Pioneer Award (NDPA), FY 2004–2005 Final Report. 2011.
- 22. Wagner CS, Alexander J. Evaluating transformative research programmes: A case study of the NSF Small Grants for Exploratory Research programme. Research Evaluation. 2013;22(3):187–97.
- 23. Bourke P, Butler L. The efficacy of different modes of funding research: perspectives from Australian data on the biological sciences. Research Policy. 1999;28(5):489–99.
- 24. Geuna A, Martin BR. University Research Evaluation and Funding: An International Comparison. Minerva. 2003;41:227–304.
- 25. Langfeldt L. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science. 2001;31(6):820–41.
- 26. Laudel G. The art of getting funded: how scientists adapt to their funding conditions. Science and Public Policy. 2006;33(7):489–504.
- 27. Nowotny H, Scott PB, Gibbons MT. Re-thinking science: Knowledge and the public in an age of uncertainty. Cambridge, UK: Polity Press; 2001.
- 28. Ravetz I. What is post-normal science. Futures-the Journal of Forecasting Planning and Policy. 1999;31(7):647–54.
- 29. Klein JT. Crossing boundaries: Knowledge, disciplinarities, and interdisciplinarities: University of Virginia Press; 1996.
- 30. OECD. Effective operation of competitive research funding systems. OECD Science, Technology and Industry Policy Papers. Paris: OECD Publishing; 2018.
- 31. Nedeva M. Between the global and the national: Organising European science. Research Policy. 2013;42(1):220–30.
- 32. Lepori B, Van den Besselaar P, Dinges M, Van der Meulen B, Potì B, Reale E, et al. Indicators for comparative analysis of public project funding: concepts, implementation and evaluation. Research Evaluation. 2007;16(4):243–55.
- 33. Van Leeuwen TN. Modelling of bibliometric approaches and importance of output verification in research performance assessment. Research Evaluation. 2007;16(2):93–105.
- 34. Shapira P, Wang J. Follow the money. Nature. 2010;468(7324):627–8. pmid:21124430
- 35. Wang J, Shapira P. Funding acknowledgement analysis: an enhanced tool to investigate research sponsorship impacts: the case of nanotechnology. Scientometrics. 2011;87(3):563–86.
- 36. Álvarez-Bornstein B, Díaz-Faes AA, Bordons M. What characterises funded biomedical research? Evidence from a basic and a clinical domain. Scientometrics. 2019;119(2):805–25.
- 37. Kozma C, Calero Medina C, Costa R. Research funding landscapes in Africa. In: Beaudry C, Mouton J, Prozesky H, editors. The next generation of scientists in Africa. Cape Town, South Africa: African Minds; 2018. p. 26–42.
- 38. Institute RoRI, Waltman L, Rafols I, van Eck NJ, Yegros Yegros A. Supporting priority setting in science using research funding landscapes. 2019. Available from: https://rori.figshare.com/articles/Supporting_priority_setting_in_science_using_research_funding_landscapes/9917825/1.
- 39. Álvarez-Bornstein B, Morillo F, Bordons M. Funding acknowledgments in the Web of Science: completeness and accuracy of collected data. Scientometrics. 2017;112(3):1793–812.
- 40. Costas R, van Leeuwen TN. Approaching the “reward triangle”: General analysis of the presence of funding acknowledgments and “peer interactive communication” in scientific publications. Journal of the American Society for Information Science and Technology. 2012;63(8):1647–61.
- 41. Cronin B, Shaw D. Citation, funding acknowledgement and author nationality relationships in four information science journals. Journal of Documentation. 1999;55(4):402.8.
- 42. Díaz‐Faes A, Bordons M. Acknowledgments in scientific publications: Presence in Spanish science and text patterns across disciplines. Journal of the Association for Information Science and Technology. 2014;65(9):1834–49.
- 43. Giles CL, Councill IG. Who gets acknowledged: Measuring scientific contributions through automatic acknowledgment indexing. Proceedings of the National Academy of Sciences. 2004;101(51):17599–604. pmid:15601767
- 44. Grassano N, Rotolo D, Hutton J, Lang F, Hopkins MM. Funding data from publication acknowledgments: Coverage, uses, and limitations. Journal of the Association for Information Science and Technology. 2017;68(4):999–1017.
- 45. Morillo F, Álvarez-Bornstein B. How to automatically identify major research sponsors selecting keywords from the WoS Funding Agency field. Scientometrics. 2018;117(3):1755–70.
- 46. Paul-Hus A, Diaz-Faes AA, Sainte-Marie M, Desrochers N, Costas R, Larivière V. Beyond funding: Acknowledgement patterns in biomedical, natural and social sciences. PlOS ONE. 2017;12(10). pmid:28976996
- 47. Rigby J. Systematic grant and funding body acknowledgement data for publications: new dimensions and new controversies for research policy and evaluation. Research Evaluation. 2011;20(5):365–75.
- 48. Rigby J, Julian K. On the horns of a dilemma: does more funding for research lead to more research or a waste of resources that calls for optimization of researcher portfolios? An analysis using funding acknowledgement data. Scientometrics. 2014;101(2):1067–75.
- 49. Sirtes D. Funding acknowledgements for the German research foundation (DFG): the dirty data of the Web of Science database and how to clean it up. 2013. Available from: https://www.wissenschaftsmanagement-online.de/beitrag/funding-acknowledgements-german-research-foundation-dfg-dirty-data-web-science-database-and.
- 50. Tang L, Hu G, Liu W. Funding acknowledgment analysis: Queries and caveats. Journal of the Association for Information Science and Technology. 2017;68(3):790–4.
- 51. Gök A, Rigby J, Shapira P. The impact of research funding on scientific outputs: Evidence from six smaller E uropean countries. Journal of the Association for Information Science and Technology. 2016;67(3):715–30.
- 52. Kundu O, Matthews NE. The role of charitable funding in university research. Science and Public Policy. 2019;46(4):611–9.
- 53. Rigby J. Looking for the impact of peer review: does count of funding acknowledgements really predict research impact? Scientometrics. 2013;94(1):57–73. pmid:23704798
- 54. Schmutz A, Salignat C, Plotkina D, Devouassoux A, Lee T, Arnold M, et al. Mapping the global cancer research funding landscape. JNCI Cancer Spectrum. 2019;3(4):1–7. pmid:32337488
- 55. Bozeman B, Gaughan M. Impacts of grants and contracts on academic researchers’ interactions with industry. Research Policy. 2007;36(5):694–707.
- 56. Braun D. The role of funding agencies in the cognitive development of science. Research Policy. 1998;27(8):807–21.
- 57. Lepori B. Analysis of national public research funding (PREF) Handbook for data collection and indicators production. Luxembourg: Publications Office of the European Union, 2017.
- 58. Rizzi F, van Eck NJ, Frey M. The production of scientific knowledge on renewable energies: Worldwide trends, dynamics and challenges and implications for management. Renewable Energy. 2014;62:657–71.
- 59. Seawright J, Gerring J. Case selection techniques in case study research: A menu of qualitative and quantitative options. Political research quarterly. 2008;61(2):294–308.
- 60. Whitley R. The intellectual and social organization of the sciences: Oxford University Press on Demand; 2000.
- 61. Caron E, van Eck NJ, editors. Large scale author name disambiguation using rule-based scoring and clustering. Proceedings of the 19th international conference on science and technology indicators; 2014: CWTS-Leiden University Leiden.
- 62. Traag VA, Waltman L, van Eck NJ. From Louvain to Leiden: guaranteeing well-connected communities. Scientific reports. 2019;9(1):1–12. pmid:30626917
- 63. Waltman L, Van Eck NJ. A new methodology for constructing a publication‐level classification system of science. Journal of the American Society for Information Science and Technology. 2012;63(12):2378–92.
- 64. Thomas DA, Ramos-Vielba I, Aagaard K. Tracing researcher ’funding configurations’: Some initial approaches and challenges. SocArXiv. 2020:[1–33 pp.]. Available from: https://osf.io/preprints/socarxiv/d4tr8/.
- 65. Paul-Hus A, Desrochers N, Costas R. Characterization, description, and considerations for the use of funding acknowledgement data in Web of Science. Scientometrics. 2016;108(1):167–82.