Background

The wider impact of research is becoming an important concern across higher education in the UK, where it is being seen as a key component of universities’ societal and economic role [1]. In particular, the increasing need for publicly funded research to be more accountable to the taxpayer and treasury has led to a large-scale attempt to capture the value and benefits of research and to build this into the national research evaluation and funding process via the Research Excellence Framework (REF) [2].

To assess the quality of research at universities, the Higher Education Funding Councils (e.g.for England: HEFCE) introduced the REF in 2014 [3] to replace the previous Research Assessment Exercise (RAE) of 2008 that had been on-going since 1986. REF evaluates three elements with different weightings: 1. Originality, significance and rigour of research outputs (65%); 2. Reach and significance of impact (20%); and 3. Vitality and sustainability of research environment (15%). The impact (2.) measure was new for 2014 and is a major distinction between the REF and its predecessor, the RAE [4].

For the first REF assessment in 2014, institutions made submissions to 36 distinct units of assessment (UoAs) based on areas of research. The submissions were assessed by an expert sub-panel for each UoA, working under the guidance of four main panels, A-D, based on broad areas: Panel A focused on the life sciences, Panel B on the engineering and physical sciences, Panel C on the social sciences and Panel D on the arts and humanities (see S1 Table for list of panels and units of assessment within the panels).

Evidence of impact was provided to the REF exercise in the form of impact statements or case studies, which followed guidelines and a template provided by the REF, including descriptions of the research and the process by which impact was achieved beyond academia. The comprising sections were: 1. Summary of the impact, 2. Underpinning research, 3. References to the research, 4. Details of the impact, and 5. Sources to corroborate the impact. These case studies were then evaluated by expert sub-panels. The REF process advised (through guidelines and people associated with the process) that all types of impact beyond academia were admissible, but that the impact had to benefit a particular sector and include beneficiaries outside of the academe as well as link directly back to published academic work of reasonably high international standard (‘two-star’ minimum) [5] (p.29, para 160b). Furthermore, each of the REF panels provided criteria by which submissions would be assessed [6]. These criteria included examples of impact.

This new inclusion of impact has triggered academic debates around the impact of research in much too broad terms to cover here substantially. However, we would like to mention, in particular, the discussions around the moral imperative to ensure impact [6]; the potential for enhanced partnerships, co-production or boundary work between practice and academia [7,8,9]; ensuring equitable, socially accountable impacts of knowledge [6,7,8,9]; balancing the cost and gains from impact assessment [10,8,11]; whether it is creating pressures to change the character of knowledge [6,7,12,2]; and, possible threats/gains to academic freedom and intellectual autonomy [7,13,11,9,14]. Further topics of discussion include (but are not limited to) the challenge of attribution of impact [10], also in part stemming from the time lag between publication of research results and implementation through policy or practice [7,12,2] and the different definitions and dynamic nature of impact itself [10,12].

Definition of impact: it varies

The concept of research impact is not new. In general terms, research impact is clearly important: it is commonly recognised as an inherent and essential part of research, and an important way in which publicly funded research is accounted for [15,16,17,18,19,20]. Impact implies a relationship through which researchers gain funding in return for contributing to society, through activities such as knowledge sharing via public or stakeholder participation or commercial exploitation of a new finding or technique, for example.

The word ‘impact’, however, has slightly different connotations in different contexts, including within the recent REF. Currently used definitions of impact largely originate from individual UK research council guidelines for funding applications and HEFCE guidelines for REF2014 [3]. Oancea [21] previously described how understanding of impact was subject specific and, in keeping with that, within the REF2014, each assessment panel (A-D) had slightly different definitions embedded in their guidelines. For example, while examples such as impacts on society, culture and economic prosperity were common to the guidelines for all panels, Panel A had a specific example around ‘impact on production’, while Panel D mentioned ‘Education’. [6]

In addition, individuals, research communities and research institutions have their own conception of impact as it pertains to their specific context: impact to a mathematician is different from that of a medical researcher or sociologist. We therefore continue the discourse started by Oancea [21] on subject-specific understanding of impact.

It is however important to note that the types of impacts presented in the REF can only be a rough proxy for how researchers view impact. Impact, and indeed the impact agenda, is much broader than REF impact alone. For the purposes of this research however, we deal with 'impact' as an actor's category, based on the definition and guidance given by the REF process. Indeed it is part of our objective here, to understand more fully how actors (researchers and the REF) interpret and validate the term 'impact'.

Assessing impact

Studies looking at the procedures and methodologies to investigate individual, programme and institute-wide impact of research are becoming more common [22,23,24,25,26]. Already the REF pilot exercise [27,28] highlighted that gathering evidence on research impact is a new endeavour with little institutional infrastructure available to support the process, and a heavy reliance on the personal knowledge of senior academics. Given the broad ways in which impact is interpreted, it is not surprising that the possibility of reviewer bias towards personal experience over ‘untested guidelines’ has also been raised [14,29,30,31,32].

The increased importance of impact and its new relationship with research funding allocation has meant that impact can be a controversial topic in the academic research community. This is particularly so in research areas that are not traditionally viewed as having obvious or immediate impact–for instance in the less applied fields of research such as fundamental natural science and certain humanities branches. Other fields do, however, have strong traditions of impact, such as the bio-medical ‘bench to bedside’ translational research, and the social sciences via economic and social policy. We address this by reviewing impact outcomes of the REF2014 by discipline (UoA) as well as by institution.

To date, three previous studies have examined the REF impact data: Firstly, RAND Europe was commissioned by HEFCE [33] to examine the process of assessment of submissions by panels of academics and research users. Looking at the strengths and weaknesses of the assessment process and whether or not it delivered reliable and fair outcomes, the study concluded that a large majority of the academics and research users on the panels felt that the process enabled them to assess impact in a fair, reliable and robust way.

Secondly, RAND Europe also evaluated universities’ preparation of submissions describing impact [34]. Looking at a sample of 21 HEIs and three stakeholder groups that included HEI leadership teams, HEI case study authors and research users, the study set out to understand their experience with the impact submission process. They found that while the process helped HEIs to develop skills in understanding and identifying impact, which they argued was starting to move towards a greater focus on impact in HEI research, complying with the impact component of the REF was found to be burdensome and resource intensive.

Thirdly, Digital Science, a division of Macmillan Science & Education in conjunction with the policy institute at King’s College, London [34] was also commissioned by HEFCE to look at the nature, scale and beneficiaries of research impact (hereafter the ‘King’s study’). They used a text mining approach (MALLET) to look at the 6,679 non-redacted impact case studies in order to identify general patterns and thematic structures in research impact across the sector. They found that research impact was wide and diverse, identifying 60 unique ‘impact topics’. They also found 3709 unique ‘pathways to research’, which the authors argued demonstrated how the sources of impact were also diverse. And while this might not have been the main objective of the research, some of the topics identified in this study however were difficult to understand as ‘types’ of impact and seemingly represented orthogonal categorizations: alongside topics such as ‘technology commercialization’ and ‘informing government policy’ were other topics such as ‘climate change’, ‘crime and justice’, ‘Asia’ and ‘cancer’, with limited exploration of the classification ontologies these terms represent. Similarly, the analysis also did not relate the topics to one another, making it difficult to draw many conclusions beyond acknowledging the diversity of impact, leaving questions such as whether particular interpretations of impact were more highly rewarded than others unanswered by this analysis.

This paper aims to build on the insights gained from these previous studies and to contribute further to critical reflection. Firstly, we look beyond the REF assessors’ views of impact (RAND Study [33]), to examine researchers’ and institutions’ interpretations of impact–both across the whole of the REF and within sample universities. Secondly, by taking an alternative text mining approach, we extend the understanding gained in the King’s study [34], to build a sharper overview of the impact landscape. In particular, we set out to understand the relationship between types of impact and reward, as well as between impact and research excellence.

Measuring impact

In this paper, we compare institutional REF2014 outcomes to the entire database, and examine one institution in detail, namely UCL. We believe UCL is a good case study for looking at impact for a number of reasons: Firstly, as internal researchers, we were able to access some of UCL’s redacted material and as such have been able to look at and comment upon the whole range of material submitted to the REF not just that which is publicly available. Secondly, UCL covers a diverse range of research disciplines and submitted 91% of its eligible staff to the REF process (some Universities submitted as few as 7% of eligible staff). This means the UCL results are representative of the institution as a whole.

Data

We used five sources of data for the research presented in this paper. The data for all institutions’ impact case studies was obtained from the REF2014 results portal [35] where the 6637 impact case study submissions into the REF evaluation process are accessible. Secondly, we looked at seven of the institutions that the Times Higher Education Supplement listed as top-ranking institutions in terms of their impact outcome (namely Cardiff University, Institute of Cancer Research, Imperial College London, King’s College London, London School of Hygiene and Tropical Medicine (LSHTM), St George’s and UCL). These were chosen to reflect a range of specialist/versus broad focused universities, those with and without medical schools and those in London and outside London, in order to check that these factors were not having a disproportionate effect on the results. Thirdly we looked at the 283 UCL impact case studies available from the REF online database. Fourthly, the outcomes of the peer evaluation (results) for UCL and others (also available on the portal). Finally, the guidelines for submissions to REF, available at http://www.ref.ac.uk/pubs/2012-01/ [35,3].

To prepare the data for the Iramuteq analysis described below, it was necessary to format it for input. The code used to do this has been made freely available online at https://github.com/martinaustwick/REF_analysis_text_preparation/blob/master/REF_Impact_cleaner_inc_UCL.ipynb, created using the Python programming language (and using the Pandas library for data manipulation). This is presented in a Jupyter Notebook (previously iPython) and includes annotations to provide context to the code. In brief, the process begins by iterating through each UoA. For each UoA, the code iterates through every entry, gathering the data for each in the JSON format (JavaScript Object Notation) from a REF API (Application Programming Interface) call, and structuring it into a Pandas dataframe. This is a programming framework that allows our Python code to request a specific dataset from the REF database, and we are using UoA as our filter, to reduce the quantity of data interrogated at each step. Every entry is then processed, creating asterisked references for title, UKPRN, UoA, ID, and panel, which is required for the main analysis stage. The text associated with the fields “Impact Summary” and “Impact Details” are then pre-processed. This pre-processing consists of removing a series of characters that interfere with Iramuteq parsing ("*", '(', ')', '\'', '\‴, '[', ']', '{', and '}'), changing currency values to words (e.g. “$” is converted to “dollars”), removing URLs, and converting all words to lower case. Processed text for Summary and Details are concatenated with the asterisked identifiers (see above) with appropriate whitespace. The processed text of each of these entries (with identifiers) is combined and saved into one large (46MB) text file that forms the basis for further analysis. Conditional filtering on UKPRN allows us to easily select or exclude individual institutions for special attention prior to saving to text file; thus institution-specific text files can easily be generated.

Method

To look at how researchers interpreted impact in their case studies, a computational text mining technique was used to examine the discourses within the impact case studies submitted. As described above, sections 1. ‘Summary of impact’ and 4. ‘Details of impact’ were selected for analysis. The analysis was carried out on three levels: 1) the entire impact case study database; 2) the UCL case studies and a selection of six other top-ranking institutions separately, chosen to reflect varying geographies and institutional focuses; and 3) the UCL submissions to each REF panel (A-D) separately. Altogether we aimed for these different depths of analysis to inform how the interpretation of impact differed by e.g. discipline, size of database or location (central London vs. rest of the country).

The IRAMUTEQ (Interface de R pour les Analyses Multidimensionnelles de Textes et de Questionnaires) software package was used to analyse the corpus of text derived from the impact case studies [36]. The tool produces a statistical map of a corpus, based on the pattern of words within the corpus: the resulting simplified pattern, or ‘map’ is based on the Word Space Model [37], which is a computational model that derives the meaningful relationships and associations between words from the way in which they are distributed and situated [38]. The map reflects significant clustering of language, as well as meaningful links from individual words to clusters. IRAMUTEQ selects the ‘meaningful’ words in the corpus by reducing them to their root forms (lemmatizing), then dividing them into function words (articles, prepositions and pronouns) and ‘content’ words (nouns, adjectives, verbs and adverbs) by grammatical filtering. Only the content words are used in the analysis, although the function words are retained for context (e.g. sample sentences).

The software then breaks the corpus into text segments of fixed length, mimicking sentences within the text. The presence or absence of a particular word in each text segment is then mapped into a contingency table and this contingency table is used to group the words in the corpus into classes according to their distribution in the text, grouping together words which are used in similar ways. Descending Hierarchical Classification (DHC) is used to create these classes. Beginning with all of the words in one class, DHC iteratively splits the full word list into two classes, then splits the biggest of these two into another two, and so forth. Each split is made by considering all of the possible ways that the classes could be cut into two and accepting the division that produces two classes which are the most dissimilar to each other, according to a chi-squared criterion. This continues until ten (default) rounds of splitting has been undergone, or until no further splits can be made (a setting of eight, ten and twelve rounds was tested with sample data, and ten was not found to be a limiting factor).

Finally, the classes produced are cross-tabulated with the words in the corpus as part of a correspondence analysis, which presents the contingency table as a graph showing the frequency and relationship of each word to the relevant classes. At this point, any metadata used to tag the corpus (in this case unit of assessment; panel; institution) is mapped onto the classes, so it is possible to see which UoA etc. are most closely associated with each class. Further details of the software and statistical analyses involved are provided in the references [39,40,41,42,43, 44].

The differences between the algorithms that IRAMUTEQ (analogous to ALCESTE) and MALLET (used in the King’s study) are based upon are explained in detail in Bholat et al [45]. In summary, MALLET uses Latent Dirichlet Allocation (LDA) which is a mixed-membership model in which words and documents are assigned probabilities related to multiple topics. For each word in each document a topic assignment is made and the probability of that word appearing in that topic calculated. The algorithm therefore looks for similarity and gives a weighting for each topic, ultimately identifying prevalence of topics. IRAMUTEQ in contrast uses Descending Hierarchical Classification (DHC) which tries to discover stable classes of terms that are maximally associated within classes while being minimally associated with other classes–effectively searching for difference and creating classes that are as different to each other as possible. While the user specifies the maximum number of divisions to take place, the process will stop before that if there are no further statistically significant divisions possible. Overall, this approach provides an alternative lens to look at the data. In particular, this descending approach allows the relationship between classes to be understood. Table 1 provides a brief comparison of the King’s method to ours.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of the text mining methods used in the King’s [34] vs. UCL analysis.

https://doi.org/10.1371/journal.pone.0168533.t001

Interpretation of impact and quality

Once the vocabulary was grouped into classes via IRAMUTEQ, the salient themes of each class were identified. To do this, each co-author (representative of a wide range of disciplines) individually reviewed the sources of information produced by the software (classes, correspondence analysis etc.) to determine a (possible) interpretation for each of the classes, which was negotiated to agree on the most plausible interpretation for each class. A consensus was reached on the choice of ‘titles’ for each class. Finally, the resulting topics were cross-referenced twice independently with the language present in the REF2014 guidance documents on criteria and working methods by panel [3,35].

To understand whether impact affected research quality, the institution-specific UoA impact scores (Impact GPA) were investigated in relation to the research quality scores (Output GPA) and any correlations identified. For this analysis the entire results database of 154 institutions and their UoA submissions was used comprising altogether 1911 data points: detailed breakdowns of sector performance has been made available by unit, panel and Higher Education Institution (HEI) by REF; each unit at each university is listed, with the proportion of submissions achieving 4,3,2 and 1 ‘star’ ratings. From this a grade point average was produced via a simple weighted sum of these percentages: (1) where p_i is the proportion of submissions being rated at i star quality, and runs from 1 to 4.

Rewarding impact

To understand which (if any) of the interpretations of impact or impact classes scored most highly, we again focused on the UCL case studies. The impact case studies were categorised according to the class that their unit of assessment was most closely associated with (see Fig 1) and the impact score mapped onto them (Fig 1). Most submissions were strongly associated with one class only. Some submissions however contributed to more than one class–in other words covered more than one interpretation of impact. In order to check that these less focused interpretations of impact didn’t affect the impact score, we placed these cases into a separate category. From Fig 1 it is possible to see that impact scores were spread across the range of UoAs (and therefore panels), with no one UoA-specific interpretation of impact standing out as gaining on average a higher score than another–even when separating out those which cross more than one interpretation of impact. We therefore argue that there does not appear to be any particular interpretation(s) of impact that has been more highly rewarded than another, and submissions which evince more than one form of impact are not advantaged (or disadvantaged).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Impact GPA by Unit of Assessment vs. impact Class.

https://doi.org/10.1371/journal.pone.0168533.g001

Association of classes by panel

The King’s study suggested that different types of impact are common in different disciplines. To test this hypothesis and to explore this link further, we carried out the same text analysis but looking at the UCL data submitted to each of the REF panels separately.

We found that while most panel submissions are dominated by one or two impact classes, they all produced classes relating to policy, evidence and management. Importantly, the classes produced appear to correlate to the subject focus of the panels themselves, see Tables 4–7. For instance, classes on education, arts and museums were absent from the analyses of Panels A and B, despite ‘culture and creativity’ being an example given in the impact guidelines for both these panels. Similarly, Panels C and D produced no classes relating to health or clinical practice, even though Panel C giving ‘impact on health and welfare’ as an example of impact in their guidelines. And although impact on the economy was given as an example in all of the panels, only Panel B exhibited strong impact in the area of enterprise / business. This suggests that impact was being interpreted by the researchers and their institutions and not just followed from the guidelines.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Panel A–Life Sciences

https://doi.org/10.1371/journal.pone.0168533.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Panel B–Physical Sciences

https://doi.org/10.1371/journal.pone.0168533.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Panel C–Social Sciences

https://doi.org/10.1371/journal.pone.0168533.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Panel D–Humanities

https://doi.org/10.1371/journal.pone.0168533.t007

Impact and research quality

To understand the relationship between impact and research quality, we plotted the impact score (Impact GPA) against the research quality score (Output GPA) for UCL (black) and for the entire database, shown in Fig 2. In terms of the output scores, we compared centres, departments, and research units that submitted to a particular UoA. Plotting the GPA for Impact against the GPA for Output as well as Overall score (not shown) for each UoA in each institution illustrates possible trends and correlations across the sector, with any bias or unusual features in the UCL data (none were found).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Impact GPA as a function of Output GPA for each Unit of Assessment in REF2014.

Sector data is shown in blue, UCL contributions are shown in black. Marginal distributions show the positive skew of the data (Output Median = 2.82, Impact Median = 3.1).

https://doi.org/10.1371/journal.pone.0168533.g002

From this, it is possible to see that those units of assessment which scored highly in research quality (output GPA) also scored highly in impact. HEFCE guidelines require that research submitted for consideration of Impact be of at least 2* quality; the majority of data is indeed above the 2* level for research outputs (although not all, suggesting that universities are not perfect predictors of their own performance). Indeed Wooding et al (2015) [46] have also found that UK research submitted to the RAE and REF was of better quality than worldwide research on average, arguing that while there is little evidence to support the level of increased quality evident in the REF submissions. This, they argue, suggests that as compared to the previous RAE, the REF results implied a lower citation threshold for declaring a 4*. However, the data show sufficient variation in both Output GPA (1.1–3.68) and Impact GPA (0.1–4.0) for us to argue that there is also evidence of a correlation between output and impact scores (Pearson Product Moment Coefficient = 0.65). A correlation between impact and quality is therefore evident in these results, although we must temper this claim with the previous finding from Derrick and Samuel, which identified a firm belief in an intrinsic link between high ‘societal’ impact and “excellent research” [30] (assumed here to be based on e.g. ranking of journal in which the research evidence is published) in the minds of several REF evaluators prior to the execution of the exercise. The data also exhibits a clustering in the top-right corner of the graph (i.e. an over-representation of higher GPAs). This is likely to be due to the selection process universities employ to present their research in the best possible light. The median and interquartile range (within which 50% of data fall) for Output are 2.8 and 2.6–3.0, and for Impact these figures are 3.1 and 2.6–3.5—demonstrating a high degree of “bunching” around these already high median values.

Given that the data points (Fig 2) are based on Units of Assessment rather than individuals or papers however, it is possible that within an institution there are individuals producing high quality research with limited impact, and others producing research with mediocre academic quality but with high impact. In fact, the REF2014 process did not require the same individuals to be submitted for assessment on quality as those submitted for impact, which has potential consequences for academic career structures and rewards. Nevertheless, by looking at the data in aggregate as Units of Assessment, the overarching picture, we argue that is that there is evidence of a correlation between research quality and impact, which we discuss further below.