Part of KE's work on Open Science aims to enhance the evaluation of research and researchers. This currently does not cover recognition of non-academic contributions to make Open Science work, such as activities to open up and curate data for re-use, or making research results findable and available. Our approach in this activity is to raise more awareness on the lack of recognition in current evaluation practice and work towards a possible solution, through the development of an 'Openness Profile'.
The KE Research Evaluation task & finish group have worked to improve awareness, listing all academic and non-academic contributions that are essential to Open Scholarship and should be recognised when evaluating research.The group also works on the 'Openness Profile', a tool that is meant to allow evaluation of currently ignored contributions that are essential for Open Science.
Interim report - Openness Profile: Defining the Concepts
We documented our ongoing investigation into the need for and value of new evaluation approaches, in the report 'Openness Profile: Defining the Concepts'.
The report provides an extensive overview of strategies, barriers, and community needs regarding openness and explores what contributions an Openness Profile,as introduced in the report, can make to enable desired openness and fairer assessment in research.
It is based on interviews conducted with 19 research contributors, both research conductors as well as research supporting professionals. The focus areas of the interviews were strategies, mandates, skills, community norms, appraisals/evaluations, non-individual profiles, barriers, incentives, and feedback on the OP concept itself.
The study showed that:
- The Openness Profile could serve the purposes of being part of their annual review, informing decision making or creating incentives/metrics at their organization.
- There is a frustration with current incentive structures and cultural inertia is very common, which translated into a desire for systemic change in how contributions to scholarship are valued and who is credited.
- It was frequently found that openness is either not currently discussed in detail in interviewees' personal evaluations, or that interviewees had only informal evaluations or none at all.
Final Report - Openness Profile: Modelling research evaluation for open scholarship
This report, published in March 2021, describes mechanisms and approaches to improve and incentivize the recording, evaluation and recognition of contributions to Open Scholarship practice. As a follow-up on the report ‘Openness Profile: Defining the concepts' (January 2020), it presents how the Openness Profile can help address existing gaps in the assessment of open science.
The Openness Profile is a digital resource, a portfolio of a research contributor's outputs and activities, accessible in a single place. Academic and non-academic open scholarship activities become visible and more easily recognised.
Over 80 individual stakeholders from 48 different organisations provided input to this report on research assessment and open scholarship. The work and writing were done by consultants Fiona Murphy and Phill Jones of MoreBrains Cooperative, together with the KE Task & Finish group on Research Evaluation.
The Openness Profile is modelling how research evaluation in an open science context can be improved. Expected benefits are highlighted and requirements listed. Recommendations are provided to various stakeholders around how to establish the Openness Profile as routine research evaluation.
Members of the KE Open Scholarship Research Evaluation task & finish group
- Clifford Tatum (lead expert) - CWTS, Leiden University
- Heidi Laine - CSC - IT Center for Science
- Verena Weigert - Jisc
- Frédéric Hélein - Institut de Mathématiques de Jussieu-Paris Rive Gauche
- Rachel Bruce - Jisc
- Lorna Wildgaard - The Royal Danish Library (ORCID ID orcid.org/0000-0002-3900-5058)
- Daniel Beucke - University of Goettingen
- Joonas Nikkanen - CSC - IT Center for Science
- Serge Bauin - The French National Centre for Scientific Research (CNRS)
- Josefine Nordling (KE lead) - CSC - IT Center for Science
- Jean-Francois Nominé (KE co-lead) - The French National Centre for Scientific Research (CNRS)
Consultants working on this KE activity:
- Fiona Murphy - MoreBrains Cooperative
- Phill Jones - MoreBrains Cooperative
