Commentary | Open | Published:
Research impact: neither quick nor easy
BMC Medicinevolume 13, Article number: 265 (2015)
Greenhalgh and Fahy’s paper about the 2014 Research Excellence Framework provides insights into the challenges of assessing research impact. Future research assessment exercises should consider how best to include measurement of indirect and non-linear impact and whether efforts in knowledge transfer and co-production should be explicitly recognised. Greenhalgh and Fahy’s findings also demonstrate that the structure of the assessment exercise can privilege certain kinds of research and may therefore miss some research that has a high impact on policy and practice. There are a growing number of courses, tools, and funding models to assist researchers in making an impact, although as yet there is little evidence about whether these approaches work in practice.
Please see related article: http://www.biomedcentral.com/1741-7015/13/232.
The potential of research evidence to improve health outcomes and optimise resource use is widely recognised by governments [1, 2]. There is a burgeoning research literature about the factors that influence research use by policymakers [3–5] and in clinical practice . Arguably, there has been relatively less attention paid to understanding the kinds of research and researchers that have influence.
Measuring research impact
Greenhalgh and Fahy’s paper  will stimulate discussion about the ways in which research has impact and what this means for assessment processes such as the 2014 Research Excellence Framework (REF2014). They point out that the REF2014 encourages reporting of direct and relatively short-term research impacts. While these are important to capture, there is a widely accepted view that indirect impacts driven through non-linear mechanisms should also be considered [8, 9]. Weiss, for example, in her seminal paper , describes a common process by which research enters policy as ‘enlightenment’, in which ‘concepts and theoretical perspectives’ accumulate to influence policy. Indeed, there is growing analysis of approaches to assessing research impact, most of which draw attention to the need for nuanced thinking. For example, commentators have noted the need to pay attention to interactions and feedback loops , relative contribution [12, 13], the policy context [14, 15], and the differences between research uptake, use, and impact .
As experience with expert panel-based research assessment exercises accrues, it will be valuable to better understand how these panels make decisions, whether the decisions are replicable, and whether researchers and policymakers make similar assessments. In considering methods of assessment, a distinction can also be made between the impact of a piece of research and the impact of a researcher over his or her lifetime. This lifetime impact may prove to be a more dependable assessment of researcher contribution.
Greenhalgh and Fahy’s paper  also raises questions about the kinds of research that influence policy and practice. They report that most of the case studies included in the REF2014 used quantitative methods (randomised trials, systematic reviews, longitudinal cohort studies, and modelling studies) and conclude that the format of the REF2014 privileges certain types of research. However, policymakers must consider local applicability, scalability, generalisability, cost, and unintended side effects. Consequently, they will be interested in questions focused towards what works, for whom, in what circumstances, in what respects, how, and why. How can the program be adapted to help it work better? How should elements in the wider system be modified to help [8, 16]? Inevitably, these questions will require mixed methods research and likely the development of new methodologies for working in more applied ways. Impact assessment exercises should be designed to capture these kinds of research.
The role of researchers in knowledge translation
This analysis of REF2014 also raises interesting questions about co-production of knowledge and how researchers think about their role in relation to knowledge translation. Greenhalgh and Fahy report that, of the 162 case studies, 82 ‘described strong and ongoing linkages with policymakers, but only 38 described targeted knowledge translation activities. In 40 case studies, no active efforts to achieve impact were described’. They call for clearer reporting of processes and activities oriented to achieving an impact.
Researchers are ambivalent about their role in knowledge translation. In a study exploring the strategies that thirty-six ‘influential’ researchers used to influence public health policy  many described high levels of engagement and co-production, but others felt that ‘the independence of research is compromised when policymakers are involved in its development’ and were more comfortable with the archetype of the disinterested scientist who sees ‘the accumulation of institutionally certified knowledge as an end in itself’ . This study , like many others, notes the considerable cost associated with knowledge translation and co-production (e.g. [19, 20]).
However, policymakers place considerable value on what might be called partnering skills in determining whether to work with researchers, citing, pragmatism, understanding government, authenticity, and collaboration and communication skills as among the criteria they used to assess trustworthy researchers . There is also evidence that researchers are increasingly using knowledge translation strategies and that these appear to pay off. For example, Newson et al.  examined a sample of 50 intervention studies funded by the Australian National Health and Medical Research Council and found that dissemination actions by researchers, particularly trying to engage with policymakers or decision makers, and translational inputs such as protocols, treatment manuals, and training materials, were important in influencing whether the research had an impact.
Supporting researchers in knowledge translation
Despite these complexities, assessment of impact in exercises like the REF2014 is valuable because it demonstrates to governments and the community the value of investing in research; it can also encourage researchers and their institutions to think about the end use of their research and to get better at maximising its impact. More could be done to support researchers in these efforts. For example, in recent years, a number of courses have been established to help researchers build skills in working with policy agencies, although to date there is only limited evidence of their value. A Nigerian course evaluation suggested improvements in participants’ understanding of the policy process and self-reported capacity to adapt research for policy (significance not tested) . A multicomponent evaluation of the Canadian Summer Institute’s course  is underway; however, process data and reports from satisfied students suggest a successful model [25, 26]. A brief report on the first two rounds of the Public Health Insight (Australia) group’s one-day Knowledge Translation course suggested the course was considered relevant and useful by participants and that gains were made in understanding and confidence in regards to the skills taught . Policymakers report that the extent to which researchers understand the policy environment is important ; therefore, opportunities for researchers to work in the policy setting and vice versa may be particularly valuable (e.g. ) and placements are sometimes included in training programs.
Tools and resources might also be valuable in speeding up the rate of learning about how to work effectively with policy agencies. A growing number are available [29–32], although again, there is little evidence about their use or value.
There is also a role for different models of funding research and knowledge translation that offer support for mutual knowledge exchange and co-production, such as the NIHR CLARHRCs in the UK, the Canadian Knowledge to Action funding program, and the NHMRC partnership grants in Australia . Again, there is as yet relatively little investigation of whether these kinds of funding increase impact; an Australian study examined impact of a policy-driven research funding program run by a state health department – while no comparative data were provided, the level of impact was high and arguably higher than one might find in an investigator-initiated scheme .
The formal assessment of research impact is in its infancy. Greenhalgh and Fahy’s paper  will contribute to discussions about how to improve assessment exercises in the UK and internationally. We must be sure that in the process of attempting to measure impact we retain a sophisticated and contextualised perspective, and that we support researchers to work effectively with policy and practice agencies.
UK Cabinet Office. What works: evidence centres for social policy. London: HM Government; 2013.
World Health Organization. World report on knowledge for better health: strengthening health systems. Geneva: WHO; 2004.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
Moore G, Redman S, Haines M, Todd A. What works to increase the use of research in population health policy and programmes: a review. Evidence Policy. 2011;7:277–305.
Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88:444–83.
Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.
Greenhalgh T, Fahy N. Research impact in the community based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework. BMC Med. 2015;13:232.
Nutley SM, Walter IC, Davies HTO. Using evidence: how research can inform public services. Bristol: University of Bristol, The Policy Press; 2007.
Bornmann L. What is societal impact of research and how can it be assessed? A literature survey. J Am Soc Inf Sci Technol. 2013;64:217–33.
Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979;39:426–31.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.
Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10:21.
Morton S. Progressing research impact assessment: A ‘contributions’ approach. Res Eval. 2015. Ahead of print.
Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8:165–70.
Moat KA, Lavis JN, Abelson J. How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis. Milbank Q. 2013;91:604–48.
Pawson R, Tilley N. Realist evaluation. London: Sage Publications; 2004.
Haynes A, Derrick GE, Chapman S, Redman S, Hall WD, Gillespie J, et al. From “our world” to the “real world”: Exploring the views and behaviour of policy-influential Australian public health researchers. Soc Sci Med. 2011;72:1047–55.
Ernø-Kjølhede E. Scientific norms as (dis) integrators of scientists? No 14/2000, Working Papers. 2000. http://openarchive.cbs.dk/handle/10398/6394. Accessed 28 Sep 2015.
Mitton C, Adair CE, McKenzie E, Patten SB, Waye PB. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85:729–68.
Kothari A, Wathen CN. A critical second look at integrated knowledge translation. Health Policy. 2013;109:187–91.
Haynes A, Derrick GE, Redman S, Hall WD, Gillespie JA, Chapman S, et al. Identifying trustworthy experts: How do policymakers find and assess public health researchers worth consulting or collaborating with? PLoS One. 2012;7:e32665.
Newson R, King L, Rychetnik L, Bauman AE, Redman S, Milat AJ, et al. A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers. BMJ Open. 2015;5.
Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers. Glob Public Health. 2012;7:750–65.
Straus SE, Brouwers M, Johnson D, Lavis JN, Légaré F, Majumdar SR, et al. Core competencies in the science and practice of knowledge translation: description of a Canadian strategic training initiative. Implement Sci. 2011;6:127.
Kho ME, Estey EA, DeForge RT, Mak L, Bell BL. Riding the knowledge translation roundabout: lessons learned from the Canadian Institutes of Health Research Summer Institute in knowledge translation. Implement Sci. 2009;4:33.
Leung B, Catallo C, Riediger ND, Cahill NE, Kastner M. The trainees’ perspective on developing an end-of-grant knowledge translation plan. Implement Sci. 2010;5:78.
Jones K, Armstrong R, Pettman T, Waters E. Knowledge Translation for researchers: developing training to support public health researchers KTE efforts. J Public Health. 2015;37:364–6.
Marshall M, Pagel C, French C, Utley M, Allwood D, Fulop N, et al. Moving improvement research closer to practice: the Researcher-in-Residence model. BMJ Qual Saf. 2014;23:801–5.
Bennett LM, Gadlin H, Levine-Finley S. Collaboration and team science: a field guide. 2010. Center for Cooperative Resolution. https://ombudsman.nih.gov/collaborationTS.html. Accessed 28 Sep 2015.
Gadlin HNO, Jessar KNAO. Preempting discord: Prenuptial agreements for scientists. 2011. US Department of Health and Human Services. The Office of Research Integrity. https://ori.hhs.gov/preempting-discord-prenuptial-agreements-scientists. Accessed 28 Sep 2015.
Young J, Mendizabal E. Helping researchers become policy entrepreneurs. Briefing paper. ODI Research and Policy Development. 2009. http://www.odi.org/publications/1127-become-policy-entrepreneur-roma. Accessed 28 Sep 2015.
Laegran AS, Thomlinson K, Berry C. Pathways to impact: developing your knowledge exchange strategy. University of Edinburgh. 2014. http://www.ed.ac.uk/schools-departments/humanities-soc-sci/research-ke/support-for-staff/knowledge-exchange/pathways-impact. Accessed 28 Sep 2015.
Anderson WP, Papadakis EM. Research to improve health practice and policy. Med J Aust. 2009;191:646–7.
Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst. 2013;11:5.
The authors are grateful for the support of the NSW Ministry of Health to the Sax Institute and for an NHMRC Centre of Research Excellence GNT1001436.
The Sax Institute receives funds from the NSW Ministry of Health and from other government, non-government, philanthropic, and competitive research funding agencies. AW and AH are funded through an NHMRC Centre of Research Excellence GNT1001436.
All authors contributed to the concepts. SR drafted the overall manuscript; AH and AW drafted sections of the manuscript. All authors read and approved the final manuscript.
SR is CEO of the Sax Institute (https://www.saxinstitute.org.au), an independent not-for-profit organisation that aims to improve health and wellbeing by driving the use of research in policies, programs, and services; primarily by acting as a bridge between researchers and health decision makers, giving each the tools to work more closely together. AW is the Director of an NHMRC Centre of Research Excellence, the CIPHER (the Centre for Informing Policy in Health with Evidence from Research), which is investigating tools, skills and systems that could contribute to an increased use of research in policy. AH is a Senior Research Officer with CIPHER and is currently leading the process evaluation of a trial designed to enhance the capacity of policy agencies to use research.