A new resource for identifying and assessing the impacts of research

In March, we published a report that characterised the 6,679 non-redacted case studies submitted to REF 2014 [5], of which 1,594 (24 %) were in Panel A, which included biomedicine, health, and clinical disciplines. To trawl through the large amount of text available in the case studies (there were more than 6 million words in ‘details of the impact’ sections alone) text mining techniques were used in three ways: i) topic modelling was used to uncover hidden thematic structures or ‘topics’ that occurred in the documents, ii) keyword searches were used to look for specific instances of impact (e.g., Quality Adjusted Life Years (QALYs)) or specific organisations (e.g., NICE), and iii) information extraction was used to match third party information with the case studies, typically around proper nouns such as countries, cities, and institutions.

The analysis of the case studies led to the identification of 60 impact topics or areas where research influences society, such as medical ethics, climate change, clinical guidance, and women, gender, and minorities. In supplementary ‘deep mines’, we read more than 1,000 case studies to provide a deeper picture of the data – and looked at specific questions such as ‘what is the impact and value of research on clinical practice and health gain?’ and ‘what has been the impact of research on BRIC countries?’

One of the most striking observations from the analysis is the diverse range of contributions that UK HEIs have made to society, ranging from improvements in access to care for tuberculosis patients in African countries to the development of a super-repellent surface invented by UK researchers now used in products worldwide such as mobile phones and hearing aids. Parallel to this is the largely multidisciplinary nature of the research that underpins the impacts identified. The analysis has shown that a very large proportion of the case studies draw on underpinning research from diverse disciplines [6]. To illustrate this, 156 fields of research within 36 Units of Assessment were linked to the 60 impact topics resulting in 3,709 unique ‘pathways to impact’.

Beyond the general themes and patterns identified in the data, any further quantitative analysis is challenging to automate from the case studies; in the ‘details of the impact’ sections of the case studies, approximately 70,000 pieces of numerical information, excluding dates, were identified. The possibility of monetising the health gain from UK HEI research by qualitatively extracting information in the case studies was explored. One approach for calculating the health gain from an intervention is to use QALYs, where one QALY is a measure of the health gain from a treatment equivalent to 1 year of perfect health. A keyword search on the term “QALY” identified 23 case studies (all but one from Panel A). Reading these case studies determined that for 12 of them, the QALY was being used to illustrate the cost-effectiveness of an intervention, but for the remaining 11, researchers evidenced and monetise the actual or potential health gain arising from the underpinning research in the case studies. These latter case studies were used to crudely estimate a potential net total gain of around £2 billion in the impact period 2008 to 2012, although we stress this is an indicative estimate.

To obtain this figure, the monetary benefit from the data given in the case studies was calculated, but data had to be supplemented and manipulated in various ways. For example, the information presented in the case studies was neither consistent nor standardised, with some case studies presenting the QALY gain for an individual patient, whereas others for a patient population. Some case studies provided an estimate of the net monetary benefit, but different figures for the value of a QALY (ranging from £25,000 to £40,000) were used. In addition, for some of the case studies, additional information had to be sourced from the cited material and, in one case, an external source was referred to that was not cited in the case study (as it was not published until after the case study had been submitted).

There are a number of key lessons arising from this research. Whilst an invaluable resource, there are limitations to using the case studies for analytical purposes. The case studies were written for assessment purposes, which meant there was often a universally positive sentiment in the language used, researchers could carefully select which case studies were included, and there was a small number of identical or near-identical submissions. Furthermore, while some quantitative data is available in the case study narratives, there was no requirement or mechanism for standardised reporting of metrics or impact outputs. As shown in the health gain example, producing return on investment type figures requires further work in matching the data available to external data sources. The key message here is that it is possible to extract valuable information and patterns on impact arising from UK HEI research. In order to optimise this process, however, these case studies would need to be re-purposed from an advocacy tool to an analytical one. By improving how we analyse and report non-academic impact of HEI research, we can begin to understand what works in research funding, research translation, and the broader research system.