Reproducibility of research ultimately will make research better: reproducing research and repeating analyses can confirm that the original results are indeed correct (or not) and can therefore more reliably be built upon, stimulating and accelerating research.
The FAIR principles play an important role to make research data reproducible. But how should FAIR be applied to the complexities around code and software, workflows and documentation, methodology and technology? And what else is required to move reproducible research from innovative exceptions to a commonly accepted way of doing science? The challenges for researchers to successfully carry out research in a reproducible way go far beyond technology and principles, it is a cultural change.
Knowledge Exchange (KE) will investigate approaches by researchers within the scope of Artificial Intelligence (AI) methodology, on what the minimal requirements are for researchers to develop reproducible research practice as the norm. How can we increase uptake of reproducible approaches by the large group of early adapters, now that the pioneers have paved the way?
To add to the perspective of reproducibility, the project will explore the ambitions, perspectives and considerations of institutional management to stimulate reproducible research. What are their incentives and what role can they play to make reproducible research standard practice? An in-depth examination of the considerations of institutional management to enable reproducible research practice will help understand what is needed and from this, recommendations can be formed.
Building on the previous work carried out by KE, Publishing Reproducible Research Outcomes, the FDSR Task and Finish Group drafted an Activity Outline document, FAIR Data and Software supporting Reproducible Research, which explains the challenges and objectives of work in this area in more detail. Work will also build on key findings from other reports such as Assessing the reproducibility of research results in EU Framework Programmes for Research, which was issued by the EC.
Call for bids is now closed and consultants Michelle Barker and Neil Chue Hong selected to undertake the work. Full details of the work and its requirements are included in the Call for proposals document.
Survey: Practices that support the scaling of research reproducibility
The next step was a survey to help us understand what types of practices assist individual researchers and managers to scale up practices that improve research reproducibility. This survey focused on researchers and managers in European research performing organisations, for example universities and research laboratories. It was aimed at personnel whose role potentially includes the practice and/or support of research reproducibility.
The next stage of the work will be the publication of a report.
A full list of participants of the Task and Finish group who are contributing to this activity is available at the end of this page.
Report: Approaches to scaling up reproducibility in research organisations
This report delves into how reproducibility can be enhanced in research organisations and the stakeholders involved, providing a framework for progressing reproducibility practices.
The report showcases the results of a literature review, survey, focus groups and community engagement carried out to understand what different stakeholder groups, from researchers to managers, need to increase uptake of good reproducibility practices within their institution. Some main findings from this research are:
- Organisations will benefit from a method in which they can assess areas where scaling up of reproducibility is already happening, and where further efforts will prove beneficial.
- Collaboration within research institutions is vital in broadening reproducibility practices. Dialogue between different stakeholder groups, at all levels, enables collaborative working. This leads to viable solutions for a wider uptake of reproducible research practices.
- Feedback from community and focus groups gave insight into how using relevant examples and case studies can help research organisations understand where they are with their reproducibility practices and how to enhance them. The report therefore identifies enablers for scaling up reproducibility, providing inspirational case study examples of how organisations have achieved this.
How the Knowledge Exchange framework can help
The main output of this work is a framework for research institutions to understand how to support and enhance reproducibility practices internally. It can be used as a gap analysis tool and/or to stimulate conversation. It allows you to do three things:
- Talk about reproducibility at the different organisational levels.
- Identify enablers for scaling up reproducibility.
- Complete an assessment spreadsheet that will help you determine your institution's ability to support reproducibility practices, and act as an opener for discussions around maintaining or improving these.
Read the full report here.
The assessment worksheet and guidance on how to use it are here .
