3. Data and methods

In this section, I first describe the selection criteria for inclusion of studies, as well as the search strategies used to identify such studies, and discuss generalizability in light of this sampling strategy. Second, I present the coding scheme and procedures applied to extract methodological information from each individual study. In line with the theoretical framework, practices of bibliometric research evaluation are examined for an extended historical time period. The study focuses on work products of professional actors in the form of published evaluation reports (grey literature) or journal articles. In most cases, these evaluation studies have been designed by bibliometric experts for their individual clients, i.e. decision makers from research organizations or funding agencies, and the professional diagnostics proposed by the bibliometricians have been accepted by the respective clients in a contract relationship. Therefore, assessment reports document an important segment of professional work. Other, less codified applications of bibliometric indicators were excluded, in particular performance assessment by private firms and research laboratories, software tools such as SciVal and Incites, and unpublished or ad-hoc use for decision-making by research organizations or funding agencies. While these uncodified applications also seem relevant for the question of de-facto standards, there is currently no systematic information on their usage and diffusion. The selection criteria for professional evaluation studies were as follows:

1. Each evaluation must include a publication and citation analysis. The sample includes studies relying exclusively on bibliometric data, as well as multi-dimensional evaluations that combine bibliometric data with other information, such as peer evaluations, financial and staff data, or case studies [34, 35]. In either scenario, my analysis focused only on bibliometric analyses.
2. The objects of evaluation are either research organizations or funding instruments. Research organizations are typically universities and/or their departments or faculties, or extra-university public research institutes. Funding instruments are typically active in supporting research projects or individual researchers at public research organizations, sometimes with the involvement of private firms, and sometimes supporting long-term investments, such as excellence schemes.
3. Evaluation objects (research organizations and funding instruments) must be located in Europe.
4. The evaluation had to be conducted with the stated purpose of informing decision making on behalf of the respective research organizations or funding instrument. Purely academic studies of bibliometric data were excluded from the study sample.
5. All analysed evaluations, including grey literature (project reports) and journal articles, were published between 2005 and 2019.

Sampling started in 2015, resulting in 85 evaluation studies for the period 2005–2014. In November/December 2019, sampling was updated, resulting in 53 additional studies for the period 2015–2019. I combined the following search strategies in order to identify a maximum number of studies from diverse sources:

1. Expert organizations and individual experts having a central position in the academic research area of evaluative citation analysis were identified by prior research [19]. Requests for non-confidential evaluation studies were sent in early 2015 to experts from 35 research organizations in 13 European countries. I received responses from 18 organizations (51%), of which 11 (31%) shared evaluation reports or information about published studies. A total of 16 studies (12% of the eventual sample) were identified in this way.
2. Evaluation studies were extracted from a set of WoS publications identified as “follow-up research on citation impact indicators” [19]. This set comprised all publications in WoS that cite any of 169 specified citation impact indicators, including a total of 2757 publications from 2005–2014 and 2776 publications from 2015–2019. Relevant studies from this sample were identified by analysing keywords and journal titles. The most frequent keywords related to research evaluation were “performance”, “universities”, “departments”, and “faculty”. I identified ten relevant journals within this publication sample, including “Research Evaluation”, “Research Policy”, “Research in Higher Education”, “Evaluation Review”, “Zeitschrift für Evaluation”, and “Higher Education”. From an initial set of 898 publications that were identified as potentially relevant (315 for 2005–2014; 583 for 2015–2019), 33 evaluation studies were retained (24% of the study sample). This search strategy proved particularly valuable in that it retrieved a total of eleven assessments published in medical journals, including for example the British Journal of Neurosurgery or the European Journal of Cancer, and four other disciplinary and foreign language journals, that would have escaped a more conventional search strategy based on a pre-defined set of core journals in scientometrics, science policy, and research evaluation.
3. I searched for bibliometric studies in the “Science and Innovation Policy Evaluation Repository” (SIPER), which was created under the 7^th framework program of the European Union. This publicly accessible database contains meta-data, as well as original documents of evaluation studies [36]. A search for citation analysis retrieved 32 potentially relevant documents, six of which were retained (4% of study sample).
4. I used material from [32], including 24 evaluation reports by CWTS, and three Dutch evaluations by other bibliometric authors (2%) for the period 2005–2014. For the period 2015–2019, additional CWTS reports were coded at a visit at CWTS Leiden, plus some published in the internet, resulting in 37 reports for CWTS in the total sample (27% of study sample).
5. I included reports from the Italian national evaluation agency ANVUR which has a legal mandate to evaluate the quality of activities performed by all research organizations receiving public money, and by funding instruments focused on research and innovation [37]. Reports are included from the first and second evaluation rounds. VQR I covers the period 2004–2010, and included nine of the 14 disciplinary areas in the Italian system, while VQR II covers 2011–2014 with 11 disciplinary areas. Since each disciplinary committee has the mandate to determine the appropriate evaluation criteria within its field(s) of research, the reports of the different sectors were treated as individual bibliometric exercises in this meta-evaluation, although I found that bibliometric methods in VQR II were more streamlined than in VQR I. Thus, the VQR assessments were treated as 20 individual studies to my study sample (14% of study sample).
6. Finally, I searched the worldwide web for evaluation reports by funding agencies. Some countries and agencies follow high standards of transparency with regards to public research evaluation, including the Swedish Council for Science (VR), the Swedish Environmental Protection Agency (SEPA), the Danish Council for Strategic Research, and the British Wellcome Trust, among others. These web searches identified a total of 23 relevant and publicly available evaluation reports (17% of study sample).

The generalizability of the descriptive findings on professional practices hinges on the quality of the sampling process. Randomization was not possible due to the exploratory nature of the study. Relative to the size of their national research systems, some countries are strongly represented while others appear underrepresented. I am confident that this variation is to a large extent due to different levels of bibliometric activity, since some countries have a tradition of quantitative research assessment while others do not [26, 27, 32]. Relatedly, some professional actors have published a large share of the study sample while others have authored only one or two reports. Since my purpose is to describe the professional field, I have to deal with the fact that this field is dominated by certain actors holding a large “market share”. In order to deal with this unequal distribution and to compare different segments of professional practice, I decided to analyse the bibliometric methods by the five largest “dedicated” organizations (CWTS, NIFU, TR/ Clarivate, VQR, CSIC) separately from the rest of the “other” bibliometric experts.

The final study sample includes 138 distinct bibliometric studies, of which 102 (74%) evaluate research organizations and 36 (26%) evaluate funding instruments. The Italian VQR was the largest evaluation exercise within the sample in terms of number of researchers and publications under assessment. Since discretion on impact metrics was given to disciplinary committees, I treated the VQR as two rounds of parallel assessments to examine the methods used in each case (n = 20, S1 Table). The largest share of studies was produced by the CWTS, a contract research institute at Leiden University specializing in bibliometric assessment services (n = 37, S2 Table). NIFU is a non-university institute conducting research on the Norwegian and neighbouring Nordic science and innovation systems (n = 12, S3 Table). The assessment service by former Thomson Reuters Evidence Ltd, today part of the Web of Science Group owned by Clarivate Analytics, and abbreviated here as TR/ Clarivate, is represented with studies on funding instruments in the UK and EU (n = 7, S4 Table). The Spanish CSIC is an extra-university research organization that has its own internal evaluation unit (n = 5, S5 Table). Since studies from the same organization used identical citation impact metrics (CWTS, NIFU, TR/ Clarivate, CSIC), or at least shared important characteristics (VQR), I separately analysed the respective subsets with regards to some dimensions. The remaining 57 studies are labelled as studies by “other bibliometric experts” (S6 and S7 Tables for evaluations of research organizations and funding instrument). In this way, I can also observe if the methods by the “other” bibliometric experts are similar or different from those employed by prominent organizations, which is relevant for the question if the latter function as role models of methodological know-how. While the meta-evaluation was not designed as a country comparison, some national differences in assessment methodology are readily apparent as a result of this clustering.

Some uncertainty remains, since the coverage of individual nations does not only reflect the diffusion of bibliometric methods, but may also result from different national transparency policies. For example, evaluations of the German Max Planck Society are usually kept confidential. In contrast, publication is mandatory under transparency rules in Sweden, such that evaluation reports are generally available on the internet. In line with Abbott´s theory, the meta-evaluation approach should be considered in tandem with case studies of national jurisdictions, such as [32], to elucidate how science and higher education policies interact with the application and acceptance of bibliometric research assessment. There is no prima facie reason to assume that bibliometric techniques systematically differ between confidential sources and published reports, except with regards to the reported level of aggregation. For example, published reports by the Italian VQR and CWTS do not contain individual data although the reports state that the same methods were also applied at the level of individuals (VQR) or teams (CWTS). On the other hand, systematic methodological variation between countries with more transparency in contrast to countries with more confidential policies cannot be excluded. Consequently, the results of this study refer only to the available studies and cannot simply be generalized to unpublished work.

Analyses are presented cross-sectionally over the whole study period 2005–2019 in the main text. Longitudinal analyses comparing three five-year periods (2005–2009; 2010–2014; 2015–2019) are documented in supplementary materials (S8–S13 Tables). Bibliometric research assessment has expanded considerably during the period investigated from 2005–2019 and was fiercely debated among academics and evaluating agencies in several European countries. Also since the mid-2000s, academic publications on citation indicators soared, as documented in [19]. A comparison across time-periods mainly shows that bibliometric assessments spread to a larger number of countries within Europe, while an increasing influence of certain professional standards could not be asserted. In other words, while the present study covers fifteen years of expansion of bibliometric research assessment in Europe, institutional obstacles to bibliometric professionalism should be analysed in more detail as a next step [38].

Before answering the question if there are certain de-facto standards in terms of indicators used, I need to ask to what extent there is agreement in evaluation objectives. To a large extent, variation in study objectives is limited by the study selection criteria. The choice of citation indicators usually implies some sort of performance comparison between different units under study. It is important to understand that similar methods of performance assessment can be and are employed for widely different policy and management purposes [39]. For example, [21] argued that bibliometric evaluation was used by Dutch research organizations with the aim to identify promising young researchers that had not yet been fully established in terms of disciplinary reputation and networks, but also to put underperforming research areas on display in order to justify change action by university management. Quite differently, the Italian assessment exercise VQR was designed to inform central decision-making concerning the redistribution of national block funding between universities and academic departments. The important policy objectives behind the choice of metrics are not necessarily well documented in the bibliometric reports, and may even fluctuate during the policy process, since informal goals may be as influential as stated formal goals. Therefore, the meta-evaluation is confined to analysing formally stated goals, the frame of reference, and the main dimensions of comparison in each study for judging similarity in evaluation objectives.

Methodologically, this meta-evaluation is based on structured content analysis. I analysed the bibliometric design of each individual study using a scheme of 37 coding questions (S1 Appendix) in the following ten topical areas: (1) meta- information regarding the individual study, (2) the professional framework, (3) the object of evaluation, (4) the citation databases, (5) quality enhancement of the bibliometric raw data, (6) sampling strategy and data collection, (7) research fields under evaluation, (8) definition of citation data, (9) citation impact indicators, and (10) utilized statistical methods. Most items involved a nominal level of measurement, i.e. non-ordered qualitative characteristics. Five items were formulated as open questions, enabling raters to record more detailed information. This coding scheme was developed via an iterative procedure beginning with a partial sample. To test interrater reliability, two raters applied the initial coding scheme to an initial sample of 20 different studies. When coding differences became apparent, they were discussed among the two raters and the items were improved to reduce their ambiguity.

This remainder of this section comments on methodological choices involved in the design of the coding scheme (S1 Appendix). The topical area (2) “professional framework” was derived from sociological theory, distinguishing bibliometric experts working as external contractors from those who are employed as staff of the organization to be evaluated. Most other coding items were defined following the methodological literature on citation analysis, particularly [6, 8, 21].

Regarding (3) “the object of evaluation”, I distinguish research organizations and funding instruments, which pose different challenges for evaluation design. In general, publications are linked to authors and authors are linked to research organizations via their institutional affiliations. In this way, research organizations are treated as author aggregates, with large variations in scale. It is more difficult to attribute publications to individual funding instruments because authors, and author teams in particular, typically receive funding from diverse sources and there is a variable time lag between funding input and publication output. Only during the most recent years, funding acknowledgements were in a few cases applied for funding instruments evaluations, and are therefore not included in this analysis.

(4) “Citation databases” includes major multidisciplinary databases WoS and Scopus along with Google Scholar and other specialized disciplinary databases, such as PubMed for the medical sciences or MathSCInet for mathematics. These databases are commodities storing abstract knowledge according to [18].

(5) “Quality enhancement of bibliometric raw data” refers to the fact that quality of citation data as offered in conventional licences by WoS and Scopus is insufficient for most bibliometric purposes. It is at least necessary to disambiguate author names and institutional affiliations, but also journal names. Data quality also includes controls of the accordance between the actual research fields of an evaluation object and their operationalization for assessment purposes, which can be regarded as checks for external validity.

The topical areas (6) “sampling strategy and data collection”, (7) “research fields under evaluation”, (8) “definition of citation data” refer to methodical details of data collection, for example time period, treatment of self-citations, and citation windows of a given analysis.

(9) “Citation impact indicators” belongs to the core of the meta-evaluation. Since a large array of citation impact metrics have been proposed in the literature ([19] collected 169 indicator variants published 1972–2016), the question is how detailed the analysis should be in order to meaningfully describe convergence or divergence in terms of indicators. Based on widely used bibliometric reviews [6, 8, 40–42], I distinguish among six groups of metrics: (a) journal impact metrics, (b) field-normalized arithmetic mean, (c) other field-based percentiles, (d) h-index and h-type indices, (e) source-normalized metrics, (f) indirect impact metrics, and (g) other metrics as rest category, and to record combinations of these. An open question was provided in order to include the formula and methodological details of respective indicators. These six categories describe divergent measurement concepts on an aggregated level, but it is possible to examine variation within categories through the open question. The use of normalization was encoded separately including the underlying classification of scientific fields.

(10) “Utilized statistical methods” refers to information on the significance of performance differences across study units, but also includes an open item recording special features such as innovative tools for analysis or software.

4.1 Bibliometric research assessment is most frequently used in the Nordic countries, followed by Italy, the Netherlands, and the United Kingdom

I found instances of bibliometric evaluation in many European countries, but the most regular use of bibliometric assessments during the observation period was concentrated in a few countries. Overall, my sample includes studies from 21 countries plus Framework Programs of the European Union. Approximately 26% of the studies, including four cross-country evaluations, were performed by the four Nordic countries Sweden, Norway, Finland, and Denmark, followed by Italy, the Netherlands, and the United Kingdom (Table 1). The Netherlands and the Nordic countries are medium-size research systems that show strong performance in international comparison. Among the larger public research systems in Europe, Italy is the only country that performs national-scale bibliometric assessment. The UK Research Excellence Framework only uses bibliometric data to inform peer review [9, 25], but the sample includes evaluations from several important British funding agencies, including the Medical Research Council and the Wellcome Trust. Germany has no national framework for research evaluation [43], and my search strategy did not yield any bibliometric assessments from France. A longitudinal analysis reveals that bibliometric assessment only recently spread to Eastern European countries, including Romania, Lithuania, Serbia, and Slovakia.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Frequencies of studies across countries and evaluation objects.

https://doi.org/10.1371/journal.pone.0231735.t001

4.2 The Web of Science (WoS) is the dominant database for public research assessment in Europe

The bibliometric evaluation of public research in Europe during the observation period was largely based on the citation indices contained in the WoS. Of the total sample set, 87% of studies relied on WoS, while 21% used Scopus or a combination of WoS and Scopus (Table 2). The share of WoS decreased somewhat from 95% in 2005–2009 to 87% in 2015–2019, but the use of ‘WoS improved versions’ remained stable or increased slightly (S9 Table). The use of Scopus still derives in large part from the Italian VQR. Some studies employed designated databases, such as PubMed or MathSCInet, but these alternative citation databases exist for only a few disciplines. In other studies, citation data were complemented by national databases, which are more comprehensive in terms of research products but do not contain original citation data [44]. For example, the Norwegian Current Research Information System (CRISTIN) includes a larger array of document types, such as books and book chapters [27]; and the Italian VQR includes all types of research outputs, e.g. software, patents, maps, and artworks [37]. While 7% of studies relied on the search engine Google Scholar to cover a larger variety of sources, the TR/ Clarivate book citation index BKCI was not used at all within this sample [45].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Databases used for bibliometric research assessment.

https://doi.org/10.1371/journal.pone.0231735.t002

One important issue in bibliometric assessment is the extent to which bibliometric databases actually cover the investigated research fields [3, 21, 46]. Expert organizations have applied different methods to address this question of external validity. VQR and NIFU used their respective national publication databases to determine the coverage in international citation databases (external coverage) [47]. Using another approach, CWTS analysed the database coverage of the references cited within the studied publication sample (internal coverage) [21]. Among the studies by other bibliometric experts, only 18% investigated database coverage.

4.3 Expert organizations invest in the improvement of WoS citation data and set technical standards for data quality

Raw citation data, as provided by WoS or Scopus, require considerable processing before they are adequate for the assessment of authors and research organizations [48]. The main issues are the ambiguity of author names and institutional addresses, and the unambiguous assignment of authors to research institutions (Table 3). The correct disambiguation of institutional name variants requires detailed knowledge of national research systems. These and other technical problems can further lead to certain proportions of false citation linkages in the raw data [49]. Expert organizations—including CWTS, NIFU, the Italian Institute for System Analysis and Computer Science (IASI), the German Max Planck Society, and the German Competence Centre Bibliometrics—currently deal with this situation by buying raw data from database providers (Clarivate Analytics, formerly Thomson Reuters, sometimes complemented by Scopus Elsevier) and constructing in-house databases with improved data quality. Access to WoS citation data of improved quality was available to 48% of the total study sample, including studies directly by TR/ Clarivate, but only to 19% of studies by other bibliometric experts. Consequently, studies by the other bibliometric experts also frequently mention the effort required for disambiguation of institutional addresses (51%) and author names (47%). Not surprisingly, one of the main practical arguments for the h-index is its greater robustness with regards to incomplete publication and citation data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Enhanced data quality.

https://doi.org/10.1371/journal.pone.0231735.t003

A related issue is the need to verify the completeness of publication records. If a completed analysis were later found to be missing individual highly cited papers, this could seriously jeopardize assessment credibility in the eyes of stakeholders. Italy and Norway impose a mandatory requirement for each scientist to register a certain number of publications (Italy) or all publications (Norway). These national publication records provide the basis for author searches of citation databases for evaluation purposes. CWTS uses a different approach, collecting internal publication records from research organizations, and sometimes sending these records to the authors for personal verification and completion. Personal verification by authors was also used by 21% of the studies by other bibliometric experts.

4.4 Citation impact is most frequently assessed with reference to international scientific fields

As stated in the method section, the formal evaluation objectives are broadly homogeneous in that they all involve a comparative assessment of research performance. This section further examines the frame of reference, i.e. how the relevant comparison for performance assessment was construed in each case. It is striking that the analysed publication samples vary in size by orders of magnitude. For research organizations, the modal size category is 1,000–10,000 publications. In contrast, for funding instruments, the modal category is 10,000–100,000 publications. There are a total of eight studies with sample sizes over 100,000 publications, five of which are in the area of medical sciences, and three investigate multiple research organizations. (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Sample sizes in bibliometric evaluation studies.

https://doi.org/10.1371/journal.pone.0231735.t004

To analyse the frame of reference in greater detail, research organizations were differentiated according to scale (number of institutes or universities) and scope (mono-disciplinary vs. multi-disciplinary), while funding instruments were distinguished according to type of unit funded (research projects, scientists, research organizations, or portfolio review) (rows in Table 5). Concerning performance measurement, I distinguished between those with “international field comparison”, “national rankings”, and “other” (columns in Table 5). International field comparison refers to the assessment of observed citation rates with reference to the expected citation rates for the same research field and time period (often also for the same document type) throughout the entire database [6, 50]. This type of measurement was used in 64% of the study sample, including 55 studies by CWTS, NIFU, and TR/ Clarivate (except B6). In contrast, with national rankings, the relative national position defines research performance. This type of measurement was used in 25% of studies, including all 20 studies by the VQR I-II [37], but also 14 studies by other bibliometric experts. While international field comparisons occurred across all categories of evaluation objects (rows in Table 5), national rankings were used only for the comparison of departments and institutes, often by h-index/ h-type indices within single fields (rows 1.3; 1.4). The remaining category “other” contains mainly inter-group comparisons, including some quasi-experimental designs (funded vs. non-funded scientists) in studies on funding instruments. Dedicated organizations used international field comparisons more frequently (70%) than other bibliometric experts (56%). Notably, I identified the divergence of the Italian VQR from the dominant approach in professional evaluation practice.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Frame of reference for research assessment.

https://doi.org/10.1371/journal.pone.0231735.t005

The predominance of international field comparisons is affected by three dedicated organizations: CWTS, NIFU, and TR/ Clarivate. A closer look at the subset of 32 studies by other bibliometric experts with international field comparisons reveals two important sources of influence on their choice of methods: at least ten studies explicitly use indicators by and refer to authors from CWTS in their method section (F1; F6; F7; F8; F20; F27; F29; G5; G8c; G12), documenting the status of CWTS as a leading expert organization. Second, in eight cases, expected citation rates were purchased directly from Thomson Reuters (F12), based on Essential Science Indicators (F3, G15) or Incites (F19; F20; F27; G18; G19), in addition to the seven bibliometric assessments commissioned directly from TR/ Clarivate. Fewer studies used Scopus (F16; F18; F26; F28; F30) or Google Scholar (F18; G5; G14) as alternative or additional sources. These findings show that while there remains considerable variation in the details of the calculation of international field averages during the period observed, a leading expert organization and a single database provider are important sources of conceptual and methodological convergence.

4.5 The WoS classification of science fields functions as a de facto reference standard for research performance assessment

Dedicated organizations generally used field-normalized indicators for impact assessment, most frequently either field-normalized top-percentiles, field-normalized arithmetic mean, or both (Table 6), while a more diverse picture emerges for other bibliometric experts. Some individual experts adhered to the same professional framework as defined by CWTS, NIFU, and TR/Clarivate, based on international field comparison and field-normalization (39%). Examples include bibliometricians at the Swedish Science Council, the German Max-Planck Society, or studies by the Canadian expert organization Science Metrix. But there are also experts more remote from the core of academic bibliometrics, who tend to choose different methods. First, there are authors from other disciplines, mostly medicine, but also economics, conducting bibliometric assessments within their own specialties (medicine: F1; F11; F12; F24; F31; F37; F38; economics: F14; F33). Second, several studies are from countries that were for a long time peripheral to the European science system, and where research evaluation started only recently, including Greece (F13; F21; F22; F23; F36), Romania (F25a,b; F34), Lithuania (G18), and Slovakia (F33). Third, there are bibliometric studies questioning the cost-effectiveness and usefulness of comprehensive peer-review based evaluation schemes in Italy, United Kingdom, and Romania (F4a,b; F25a,b; F26a,b; F34). These three groups of studies have in common that they used other bibliometric indicators than the dedicated organizations, in particular CWTS, NIFU, and TR/ Clarivate. They often employ the h-index (sometimes other h-type indices) (46%), journal impact factor (46%), and often do not have access to citation data of highest quality. By contrast, dedicated organizations only rarely used h-type indices (4%) and more often use impact-level categories of journals (16%) for scientific fields with insufficient coverage in citation databases (Table 6, 2.4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Types of impact metrics used in evaluation studies.

https://doi.org/10.1371/journal.pone.0231735.t006

In principle, field normalization is applicable to different types of citation metrics [6], including arithmetic mean, highly cited percentiles, h-type indices, and indirect citation metrics, and also journal impact. To avoid issues regarding field normalization, exclusively source-normalized impact metrics have been construed as a methodological alternative [41]. However, only some of the possible combinations are actually found in my sample. Eight studies used JIF-Quartiles, a variant of field-normalized journal impact (included in row 2.2), but no study uses a field-normalized h-index. No study calculated either indirect citation (prestige) indicators or source normalized indicators for observed citations. Instead, citation databases provided indirect and source normalized journal impact metrics including Eigenfactor metrics (TR/ Clarivate) and SNIP (Scopus) in recent years. In general, the overview in Table 6 reveals that research assessment practice during the study period included few of the methodological inventions that have recently been proposed in the academic debate on impact metrics [8, 19, 51].

Journal impact was used quite frequently (49%), despite repeated criticism that the substitution of a journal’s impact for the actual number of citations lacks validity [52, 53]. While 91% of studies report observed citation data, nevertheless journal impact was often used to substitute missing data, either because publications are so recent that actual citations are not yet available (e.g. VQR), or because articles are published in journals that are not covered by the database (e.g. NIFU and VQR). Notably, journal impact metrics are easily accessible via Journal Citation Reports, which is especially relevant for bibliometricians lacking fully licensed access to citation databases. Sometimes journal impact was used very pragmatically, for example simply for distinguishing between two broad levels of journal quality (Table 6, 2.4).

Overall, 78% of all studies used field normalization, including the field-normalized arithmetic mean, other field-related percentiles, and field-normalized journal impact (Table 7). Among these 107 studies, 83% relied on the WoS classification of science fields (WoS subject categories SCs), and an additional 5% used the related Essential Science Indicators classification by TR/ Clarivate. The Scopus science classification was mainly used in the Italian VQR. Of all field normalizations, 17% were based on self-defined journal sets, sometimes combined with keywords. Both, the alternative journal-based classifications proposed in the academic literature, and the publication-based clusters by CWTS, have been developed to overcome methodological problems associated with the WoS classification of science fields. CWTS actually replaced the WoS classification by a taxonomy of approximately 4000 clusters since about 2016 [54]. Yet while alternative journal-based taxonomies have had little influence to date, and have not been used at all by dedicated organisations, the publication-based clusters, on the other hand, are proprietary and thus not available for use by other professionals. It can be concluded that thus far the WoS classification of science fields has retained the status of a de facto reference standard.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Classification of science fields used for field normalization.

https://doi.org/10.1371/journal.pone.0231735.t007

There has been some debate in the literature regarding the adequacy of WoS SCs as the basis for field normalization. One point of concern is the lack of transparency regarding the methodology used to construct and update the categories [56, 58]. One study investigated the adequacy of the WoS classification, and empirically demonstrated that very few journals in the WoS were miscategorized in terms of their direct citation relations [58]. These authors found that some journals in the WoS do not display strong citation relations with their present category, but these journals generally appeared to be located between fields rather than to have been suboptimally assigned. Overall, WoS performs significantly better than Scopus with regards to the adequacy of journal categorization, which is related to the fact that journals belong to fewer categories in WoS than in Scopus.

Notably, [58] do not address the more fundamental question of how journal citation clusters are distributed across WoS SCs. Random walk models demonstrate that the total inter-journal citation network is characterized by densely connected regions and areas of much lower citation traffic [59]. The WoS SCs substantially vary in size; thus, it seems likely that some categories contain several clusters while others may comprise only one or even no cluster at all. [60] demonstrated that the WoS category “Library and information sciences” includes two clearly distinguishable journal citation clusters, and that eight journals publishing research in “Science and technology studies” belong to eleven different WoS SCs. Concerns have been raised that the internal heterogeneity of SCs with regards to research topics and citation densities may pose serious problems for field normalization [61]. One possible way to address this problem are more fine-grained publication-based clusters [54].