Engagement and Impact Assessment 2018-19

In December 2015, as part of its National Innovation and Science Agenda (NISA), the Government announced the development of an Engagement and Impact (EI) assessment. The EI assessment examines how universities are translating their research into economic, environmental, social, cultural and other benefits (see Appendix 1 for a list of institutions eligible to participate).

EI 2018 aims to encourage greater collaboration between universities and research end-users, such as industry, by assessing engagement and impact. The EI 2018 assessment is a companion exercise to Excellence in Research for Australia (ERA) 2018, and data collected for ERA 2018 forms part of the EI 2018 assessment.

The objectives of the EI assessment are to:

• provide clarity to the Government and the Australian public about how their investments in university research translate into tangible benefits beyond academia
• identify institutional processes and infrastructure that enable research engagement
• promote greater support for the translation of research impact within institutions for the benefit of Australia beyond academia
• identify the ways in which institutions currently translate research into impact.

For the purposes of the EI 2018 submission and assessment the following definitions were applicable.

Research

Research is the creation of new knowledge and/or the use of existing knowledge in a new and creative way to generate new concepts, methodologies, inventions and understandings. This could include the synthesis and analysis of previous research to the extent that it is new and creative.

This is the same definition used for ERA. It is consistent with a broad notion of research and experimental development comprising "creative and systematic work undertaken in order to increase the stock of knowledge—including knowledge of humankind, culture and society—and to devise new applications of available knowledge" as defined in the ARC funding rules.

Aboriginal and Torres Strait Islander research

Aboriginal and Torres Strait Islander research Aboriginal and Torres Strait Islander research means that the research (as defined above) significantly:

• relates to Aboriginal and Torres Strait Islander peoples, nations, communities, language, place, culture or knowledges, and/or
• is undertaken with Aboriginal and Torres Strait Islander peoples, nations, or communities.

Engagement

Research engagement is the interaction between researchers and research end-users outside of academia, for the mutually beneficial transfer of knowledge, technologies, methods or resources.

Impact

Research impact is the contribution that research makes to the economy, society, environment or culture, beyond the contribution to academic research.

Research end-user

A research end-user is an individual, community or organisation external to academia that will directly use or directly benefit from the output, outcome or result of the research.

Examples of research end-users include governments, businesses, non-governmental organisations, communities and community organisations.

Specific exclusions of research end-users are:

• publicly funded research organisations (CSIRO, AIMS, ANSTO, NMI, DSTO etc.)
• other higher education providers (including international universities)
• organisations that are affiliates, controlled entities or subsidiaries (such as Medical Research Institutes) of a higher education provider
• equivalents (international or domestic) of the above exclusions.

EI assesses the engagement and impact arising from research undertaken in Australian institutions by Unit of Assessment (UoA). EI defines the UoA as the two-digit Fields of Research (FoRs), as set out in the Australian and New Zealand Standard Research Classification (ANZSRC), at each institution.

The ANZSRC is a hierarchical classification system with three levels. The highest level is the two-digit Fields of Research (FoR), followed by the four-digit and then six-digit codes. The 22 divisions (the two-digit FoRs) are:

• 01 Mathematical Sciences
• 02 Physical Sciences
• 03 Chemical Sciences
• 04 Earth Sciences
• 05 Environmental Sciences
• 06 Biological Sciences
• 07 Agricultural and Veterinary Sciences
• 08 Information and Computing Sciences
• 09 Engineering
• 10 Technology
• 11 Medical and Health Sciences
• 12 Built Environment and Design
• 13 Education
• 14 Economics
• 15 Commerce, Management, Tourism and Services
• 16 Studies in Human Society
• 17 Psychology and Cognitive Sciences
• 18 Law and Legal Studies
• 19 Studies in Creative Arts and Writing
• 20 Language, Communication and Culture
• 21 History and Archaeology
• 22 Philosophy and Religious Studies

For more information on the ANZSRC, refer to Appendix 2.

There are three types of UoA:

• two-digit FoR
• Aboriginal and Torres Strait Islander research (impact only, optional)
• interdisciplinary (impact only, optional)

Note: FoR 11 Medical and Health Sciences is divided into two groups—the Biomedical and Clinical Sciences (BCS), and the Public and Allied Health Sciences (PHS)—due to the diversity and high volume of research covered by this FoR. The allocation of the four-digit FoRs to BCS and PHS is at Appendix 3.

The UoAs coded to two-digit FoRs do not necessarily correspond to departments or research groups within an institution.

Aboriginal and Torres Strait Islander research

ANZSRC does not classify Aboriginal and Torres Strait Islander research at the two-digit FoR level. For EI 2018, Aboriginal and Torres Strait Islander research was treated as its own discipline and a definition was created for the assessment. Institutions were also able to identify (flag) Aboriginal and Torres Strait Islander content in two-digit FoR or interdisciplinary impact studies.

The purpose of the Aboriginal and Torres Strait Islander research impact study was to enable the reporting of impact studies that demonstrated ways in which Aboriginal and Torres Strait Islander leadership and governance arrangements were integrated into the activities and processes throughout all stages; from the initial research, through translation, to the impact itself. A key element of this is demonstrating Indigenous-led principals which embody the right to self-determination as articulated in the United Nations Declaration on the Rights of Indigenous Peoples (of particular note, articles 3, 4, 19 and 31).

Interdisciplinary

Institutions could choose to submit an interdisciplinary impact study. The purpose of the interdisciplinary impact study was to enable the submission of impact studies where the impact was so broad it could not reasonably fit within one two-digit FoR.

In general, EI assessed two-digit FoR UoAs where there were meaningful levels of data for assessment. For this reason, a low-volume threshold was applied.

In EI 2018, the low-volume threshold was 150 weighted apportioned outputs (1 book counted as 5) based on an institution's relevant two-digit FoR submission to ERA 2018. EI 2018 assessed UoAs that met the low-volume threshold for both engagement and impact. If an institution considered that a UoA falling below the low-volume threshold had sufficient evidence for assessment of engagement or impact, it could choose to opt-in to assessments of either or both.

The ARC acknowledged that for some UoAs there might be no impact, or insufficient impact, to report. If an institution met the low-volume threshold in a UoA but

• the majority of the research outputs were primarily basic or fundamental research OR
• the research area at the institution was too new
  then an institution could request that the UoA not be assessed for impact.

There was no option to request not to be assessed for engagement.

The low-volume threshold did not apply to the interdisciplinary nor the Aboriginal and Torres Strait Islander research impact studies. Institutions could opt-in to either or both.

For more information, please refer to the EI 2018 Submission Guidelines.

In EI 2018, each UoA that met the low-volume threshold, or opted in, was assessed for engagement, impact and approach to impact. The assessments comprised the following:

• Engagement—Panels assessed research engagement activity based on an engagement narrative, a small suite of quantitative indicators, and an engagement indicator explanatory statement. The engagement narrative explained the interaction between researchers and research end-users outside of academia for the mutually beneficial transfer of knowledge, technologies, methods or resources.
• Impact Studies—Panels assessed research impact and the institution’s approach to impact based on qualitative impact studies that detailed the impact, the research associated with the impact, and the approach to impact for each UoA. Each UoA received two ratings—one for impact and one for approach to impact.

Panels assessed research impact based on the information within the impact studies which outlined the impact, explained the relationship between the impact described and the associated research, and identified the contribution the research made beyond academia.

Panels assessed approach to impact based on the information within the impact studies which summarised institutional approaches to facilitating the impacts described in the impact studies 

EI used a three-point rating scale—high, medium and low. Each point of the rating scale used to assess the UoA had a specific descriptor for engagement, impact and approach to impact.

In EI 2018, five assessment panels comprising researchers and research end-users were responsible for assessing UoAs. The assessments consisted of one rating for each of engagement, impact and approach to impact. Panels assessed the UoA based on a three-point rating scale—high, medium and low. The allocation of two-digit FoRs to assessment panels is at Appendix 2.

Engagement

Panels assessed research engagement activity based on an engagement narrative, a small suite of quantitative indicators, and an indicator explanatory statement. Each UoA received a rating for engagement.

Impact studies

Panels assessed research impact and the institution's approach to impact based on qualitative impact studies that detailed the impact, the research associated with the impact, and the approach to impact for each UoA. Each UoA received two impact ratings—one for the impact and one for approach to impact.

There are several documents that provide more information on the EI 2018 assessment. These include:

• EI 2018 Framework
• EI 2018 Submission Guidelines provided guidance to institutions on how to complete submissions.
• EI 2018 Assessment Handbook provided detailed information about EI 2018 assessment approach and process.

More information about EI is available on the ARC website.

The EI 2018 National Report presents ratings and analysis of the data and information submitted as part of the assessment of the research engagement, research impact and approach to impact by discipline within Australia's higher education institutions.

Coverage

EI retrospectively assessed research engagement, impact and approach to impact which occurred during specific reference periods. As EI 2018 reference periods ended on 31 December 2016, research engagement, impact, and approach to impact activities, may have changed since that time.

Comparison across data items

UoAs are assessed for engagement, impact and approach to impact against the EI rating scales for research engagement, research impact and approach to impact. EI ratings cannot be used as ranking devices. Further, as each EI rating point might include a range of performances, and the gap between rating points is not defined, it is not appropriate to average ratings even within disciplines.

EI has been designed to provide flexibility for, and recognition of, discipline-specific research behaviours rather than comparison between disciplines. EI 2018 was conducted by discipline and research end-user experts assessing the information in each submission in the context of their own expert knowledge of the discipline, research end-user engagement and research impact.