Table 1 Summary of literature reviews
Reference ID number | Author | Year | Location | Main aims | Search strategy (1 = yes, 0 = no) | Search Methods | Keywords list (1 = yes, 0 = no) | PRISMA (1 = yes, 0 = no) | Timeline and language | Discipline of research/perspective | Methods of data collection (1 = included, 0 = not included) | Categories (1 = included, 0 = not included) | Frameworks (1 = included, 0 = not included) | Table or diagram of empirical impact assessments (1 = included, 0 = not included) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Hanney et al. | 2003 | UK | To review how health research is used in policy-making and the approaches to assess the policy impact of research | 0 | NA | 0 | 0 | Unknown | Healthcare | 1 | 0 | 1 | 1 |
2 | Buxton et al. | 2004 | UK | To identify key studies that have estimated the economic value of the impact of health research to society | 1 | Databases (11 in total) and grey literature | 1 | 0 | Unknown/English only | Healthcare | 1 | 0 | 0 | 1 |
3 | Coryn et al. | 2007 | USA | To describe, classify, and comparatively evaluate national models and mechanisms used to evaluate research in 16 countries | 0 | NA | 0 | 0 | Unknown | General | 0 | 0 | 1 | 0 |
4 | Hanney et al. (Chapter 2) | 2007 | UK | To review the literature describing the evaluation of the impact of programmes of health research to inform an evaluation of the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003 | 1 | Database (13 in total), citation analysis, expert consultation, advisory group consultation | 1 | 0 | 1990–2005/unknown | Healthcare | 1 | 0 | 1 | 1 |
5 | Brutsher et al. (Part 1 of report) | 2008 | UK | To present and discuss five key elements of research evaluation | 0 | NA | 0 | 0 | Unknown | General | 1 | 1 | 1 | 0 |
6 | Buxton et al. (Chapter 2) | 2008 | UK | To review current practices of assessing the economic benefits of health research | 0 | NA | 0 | 0 | Unknown | Medical research | 1 | 0 | 0 | 0 |
7 | Boaz et al. | 2009 | UK | A literature review to explore methods for evaluating the impact of research on policy outcomes that might be appropriate to the Sustainable Waste and Resource Management and the Sustainable Consumption and Production research programmes | 1 | Databases (10 in total), web searches of 30 organisation websites, citation tracking, expert contacts | 1 | 0 | 1987–2007/English only | Cross-sector but written for the Environment, food and rural affairs department of the UK government | 1 | 0 | 1 | 0 |
8 | Marjanovic et al. | 2009 | UK | To give a historical overview of landmark studies in the research evaluation field. To use this historical account to reflect on the methodological developments and recurrent themes in research evaluation studies | 0 | NA | 0 | 0 | Unknown | Biomedical and health research | 1 | 0 | 1 | 0 |
9 | Yazdizadeh et al. | 2010 | Iran | A systematic review to identify the methods used to assess the economic impact of healthcare research, and the outcomes | 1 | Databases (8 in total), 21 relevant websites | 1 | 0 | Unknown | Healthcare | 1 | 0 | 1 | 0 |
10 | Banzi et al. | 2011 | Italy | To identify the most common approaches to research impact assessment, categories of impact and their respective indicators | 1 | Databases (2 in total), research funding and charity's foundations websites cited in the studies, citation screening | 0 | 0 | 1990–2009/English, French, Spanish, Italian | General | 1 | 1 | 1 | 0 |
11 | Hanney et al. | 2011 | UK | To review studies that have attempted to assess economic impacts from health research in the field of nursing health research | 1 | Databases (2 in total), reviews of retrospective studies already known to the authors | 0 | 0 | Unknown | Healthcare | 1 | 0 | 0 | 1 |
12 | Patel et al. | 2011 | UK | A systematic review to identify the indicators that have been used to measure healthcare research performance | 1 | Databases (4 in total), citation screening | 1 | 1 | 1950–2010/No restrictions | Healthcare | 1 | 0 | 1 | 0 |
13 | Ruscio et al. | 2012 | USA | To evaluate 22 scholarly impact metrics, including conventional measures, the h index, and many variations on the h theme | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 0 | 1 |
14 | Bornmann et al. | 2013 | Germany | To present existing research on and practices employed in the assessment of societal impact in the form of a literature survey | 1 | Databases (2 mentioned, total not clear), internet search engines, citation screening | 0 | 0 | Unknown | General | 1 | 0 | 1 | 1 |
15 | Guthrie et al. | 2013 | UK | To identify and review frameworks in use for research evaluation, to identify the research evaluation tools applied to those frameworks to provide a guide to developing a research evaluation framework that can be used in a range of circumstances | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 1 | 0 |
16 | Smith et al. | 2013 | UK | To review the methods of assessing research impact that are relevant to academic promotion | 1 | Databases (5 in total), internet search engine, citation screening | 1 | 0 | Unknown | General | 1 | 0 | 1 | 0 |
17 | Carpenter et al. | 2014 | USA | To provide a broad overview of widely available measures of academic productivity and impact using publication data and to highlight the uses of these metrics for various purposes | 0 | NA | 0 | 0 | Unknown | Science | 1 | 0 | 0 | 0 |
18 | Penfield et al. | 2014 | UK | To explore what is understood by the term research impact, and to provide a comprehensive overview of the literature to understand which methods and frameworks of impact assessment could be used for UK impact assessment | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 1 | 0 |
19 | Milat et al. | 2015 | Australia | To synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research | 1 | Databases (6 in total) | 1 | 1 | 1990–2013/English only | Healthcare | 1 | 0 | 1 | 0 |
20 | Moed et al. | 2015 | The Netherlands | To provide a broad overview of the wide array of metrics to assess research impact currently in use in academia and research | 0 | NA | 0 | 0 | Unknown | General | 1 | 1 | 1 | 0 |
21 | Pollitt et al. (Appendix 2) | 2015 | UK | To identify a wide range of potential impacts of research, investigate different ways of classifying impacts, produce a possible long list of types of impact and domains for impact | 1 | Search "Largely covered grey literature as well as some academic literature and focused initially on a limited set of key sources known to the project team and advisory board." | 0 | 0 | Unknown | General | 0 | 1 | 1 | 0 |
22 | Thonon et al. | 2015 | France | To identify and critique indicators of impact that could be used to measure the output and outcome of medical research | 1 | Databases Snowballing | 1 | 1 | Unknown/French or English | Biomedical research | 1 | 1 | 0 | 0 |
23 | Wouters et al. | 2015 | UK | A literature review of academic research looking a range of impact indicators that may be useful in research evaluations, including the next Research Excellence Framework (REF) | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 0 | 0 |
24 | Agarwal et al. | 2016 | USA | To provide a broad overview of the wide array of evaluation metrics currently in use in academia and research | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 0 | 0 |
25 | Chikoore | 2016 | UK | To explore the meaning of research impact, the issues regarding how it can be evaluated, the methods used in evaluating this research impact and identifying the challenges of these evaluations | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 1 | 0 |
26 | Raftery et al. | 2016 | UK | To review published research studies on tools and approaches to assessing the impact of programmes of health research and, specifically, to update the previous 2007 systematic review funded by the HTA programme | 1 | Databases (8 in total), hand searching selected journals, citation screening, literature known to the research team already (snowballing), bibliographic searches of other reviews and references identified in relevant literature | 1 | 1 | 2005–2014/Unknown | Healthcare | 1 | 1 | 1 | 1 |
27 | Greenhalgh et al. | 2016 | UK | To review the strengths and limitations of six established approaches of measuring both the outcomes of research and the processes and activities through which this is achieved | 0 | NA | 0 | 0 | Search strategy based on Raftery et al. | General | 1 | 0 | 1 | 1 |
28 | Wimmer et al. | 2016 | USA | To review both traditional and more novel impact evaluation tools, the impact metrics they calculate, and why the tools are particularly relevant to the field of nursing | 0 | NA | 0 | 0 | Unknown | General | 1 | 0 | 0 | 0 |
29 | Bornmann et al. | 2017 | Germany | To review how impact is measured within science and beyond, the effects that impact measurements have on the science system and which problems are associated with impact measurement | 1 | Databases (3 in total) and other literature reviews | 1 | 0 | Unknown | Science | 1 | 0 | 0 | 0 |
30 | Cruz Rivera et al. | 2017 | UK | To identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix | 1 | Databases (4 in total), internet search engines including images, communication with experts | 1 | 1 | Not specified (Appendix 1 containing full search strategy also checked) | Healthcare | 0 | 1 | 1 | 0 |
31 | Deeming et al. | 2017 | Australia | To list the stated objectives for research impact assessment frameworks, to identify existing frameworks and to evaluate whether the identified frameworks possessed the capabilities necessary to address the specified objectives | 1 | Database (Scopus) and grey literature | 1 | 1 | 2005–2015/English only | Healthcare/Medical | 1 | 0 | 1 | 0 |
32 | Peter et al. | 2017 | Canada | To identify the approaches that have been used to understand the impacts of health research, to identify ways that research impacts have been defined and measured and to provide recommendations for occupational science | 1 | Traditional databases, author search from the included articles, assessment tools that were mentioned in the included reviews used as keywords to search traditional databases, reverse citation analysis and a forward citation search in the Scopus database | 1 | 1 | No restrictions | Occupational science | 1 | 0 | 1 | 0 |
33 | Reale et al. | 2017 | Italy | To understand how the impact assessment methods used in social sciences and humanities and how far these approaches attempt to apply methods and instruments that take into account the distinctive features of this discipline | 1 | Journal articles/databases Books Reports Working papers CORDIC database EU FP17 Flash-it project Web searchers Guidelines for applicants and evaluators Grey literature Snowballing | 0 | 0 | 2006–2012. Mostly English | Social sciences and humanities | 1 | 1 | 1 | 0 |
34 | Newson et al. | 2018 | Australia | To review the extent and nature of studies measuring the impacts of health research on policy and compare forward and backward tracing approaches for assessment | 1 | Electronic databases (5) References of included studies | 1 | 1 | 1995–2016. English only | General | 1 | 0 | 1 | 1 |
35 | Pedrini et al. | 2018 | Italy | To analyse the approaches to the assessment of healthcare research's social impact with a focus on different stakeholders | 1 | Databases (3 in total) | 1 | 0 | 2000–2016/Unknown | Healthcare | 1 | 0 | 0 | 1 |
36 | Weisshuhn et al. | 2018 | Germany | To conduct a literature review to analyse how impacts of agricultural research are assessed | 1 | Databases | 0 | 0 | 2008–2016/Unknown | Agricultural | 1 | 1 | 0 | 0 |
37 | Williams et al. | 2018 | UK | To provide a systematic review of the evolution of research impact assessment approaches in Australia and the UK | 1 | Public policy documents Newspaper commentary Academic literature | 0 | 0 | Unknown | General | 0 | 0 | 1 | 0 |
38 | Braithwaite et al. | 2019 | Australia | To identify what is known about methods for assessing researchers’ achievements for the purposes of producing a new assessment model | 1 | Databases (all Web of Science databases) | 1 | 1 | 2007–2017. English only | General | 1 | 0 | 0 | 0 |
39 | Gomes et al | .2019 | UK | To review empirical impact evaluations to understand the impact generated by publicly and charity-funded health research in the UK | 1 | Databases Citation tracking Reference searching of including articles Hand searching of specific journals | 1 | 1 | 2006–2017. English only | Health research | 1 | 0 | 1 | 1 |
40 | Heyeres et al | 2019 | Australia | To perform a systematic review of studies that used a case study approach to assess research impact | 1 | Databases (11) Reference list of impact case studies identified | 1 | 1 | 2000–2018. English only | General | 1 | 0 | 1 | 1 |