By Lucia Loffreda and Andrea Chiarelli, 04/11/2021
The KE activity "Publishing Reproducible Research Output" has published its final report "The art of publishing reproducible research outputs - Supporting emerging practices through cultural and technological innovation." The report represents the culmination of a 12-month project aiming to investigate current practices and barriers related to publishing reproducible research outputs and to determine how infrastructure (technical and social) can support progress in this area.
The final report, delivered by Research Consulting in partnership with the Knowledge Exchange Task and Finish group, includes a literature review of over 130 sources and engagement with over 50 individuals from 12 different countries through focus groups and interviews. To practise what we preach, we have sought to work in a reproducible way ourselves: you can find all project outputs in our Zenodo Community!
In this post, we present five key findings that emerged from our work, which we hope will help frame future discussions around research reproducibility and support the ongoing processes of culture change and technological development that are needed to both enable and sustain this.
#1 Reproducibility is part of the long-term vision for open science
In this project, we defined research reproducibility as cases, where data and procedures shared by the authors of a study are used by others to try to obtain the same results as in the original work. This may require, for example, a detailed description of the methods used to process and analyse the data, access to any relevant datasets and an ability to obtain and run computer code, where appropriate.
However, defining reproducibility across the board is no easy task. Through our literature review and interviews, we found that reproducibility is related to other concepts such as replication, robustness, and the generalisation of research findings. The most significant differences in vocabularies occur between disciplines, which is often due to varying research methods and practices. It is therefore difficult to find coherent terminology that speaks consistently to all stakeholder groups.
Nevertheless, the fact that disciplinary communities are discussing reproducibility is a clear positive: growing pockets of interest around this topic are emerging across the research landscape (albeit with some variations in terminology and pace of progress), and are working towards the same long-term vision increasing the openness and transparency of research.
#2 Disciplinary requirements for reproducible publications need to emerge from the bottom up
Since disciplinary communities apply different approaches and standards when it comes to conducting research, a ‘bottom-up' approach is needed to shape future policies in support of reproducible publication practices. In practice, this means that disciplines have to run through at least some fundamental steps in the policy development cycle (see Figure 1) before their needs can be appropriately understood, operationalised and supported by research performing organisations, funders, publishers and service providers.
A symptom of this is that, today, most research funders (with Dutch funder NWO being a notable exception) do not play a prominent role with regard to reproducibility, even if they are keen to get involved. The case is very similar for publishers, and some fear that adding strict reproducibility requirements (see Figure 2) to their journals might lead to lower submission rates. Regardless of this, several publishers are now experimenting, including, for example, the American Economic Association's reproducibility requirements and eLife's partnership with Stencila to deliver executable research articles.
#3 Reproducibility efforts should be "baked into" the research process, but incentives are needed
In an ideal world, "reproducibility labour would be baked into the research process, so that the effort it takes is indistinguishable from the research effort itself". However, we found that practices conducive to reproducible research are currently seen as additional, unrewarded efforts by researchers. There are a number of reasons for this, but the most prominent are competing incentives (particularly around publication numbers and impact factors) and the ever-increasing time pressures on researchers (see Figure 2).
To help ease some of these challenges, research performing organisations can provide valuable training for researchers and complement their skills where lacking. With the appropriate funding, and broader institutional commitments to open research practices, this type of support could take the shape of new institutional roles such as data stewards, data curators and subject or reproducibility librarians.
At the moment, a wide range of training and support pathways are emerging in institutions and communities worldwide, but more structured and dedicated approaches, including incentives, could help in making reproducible publication practices a more widespread reality.
#4 Good data management practices are a necessary condition for reproducible publication
The only way to easily publish reproducible research outputs is to develop them as such from the very beginning of a research project. Good data management practices from a project's inception can help individual researchers and their organisations achieve findings that are reliable, well-documented and easy to share. The FAIR data principles Findability, Accessibility, Interoperability and Reusability are a helpful place to start in terms of both everyday practices and understanding the research infrastructure needed for reproducible publication.
As we noted above, the significant disciplinary variation in terms of research data management practices is a barrier, not least because of the different language used.
#5 Although digital tools and infrastructures are available, interoperability remains a gap
The number of digital tools and infrastructures available in today's research landscape is significant and growing. While most researchers are aware of these tools, and how they can potentially be used to enable reproducible publication practices, we found that a lack of interoperability (the ‘I' of the FAIR data principles) between digital infrastructures is a practical obstacle to reproducible research workflows.
As reproducibility should be considered across all phases of the research lifecycle, service providers have an important role to play in continuing technological innovation in this direction. Indeed, there are emerging examples supporting reproducible publication practices, including OSF's Projects feature or the Renku platform (see our report for many other examples). However, we are still far from the finishing line as many disciplines still have to communicate their requirements to the tools and digital infrastructures that serve them.
What happens next?
As the reproducibility discourse continues to evolve, there is a risk for policies and their enforcement to leave little room for nuance. For some epistemic cultures, for instance, reproducibility will be harder to understand and implement, or perhaps is not even the goal; in others, reproducibility may not be seen as the key quality hallmark, but just as an option among many. It will therefore be necessary to consider diversity as we rethink research practices to preserve and boost trust in science.
Overall, we believe that our findings paint a cautious yet optimistic picture: by engaging with the obstacles we have identified and leveraging the opportunities available, there is significant room for reproducible publication practices to play a prominent role in the ongoing shift to open science.
If you'd like to find out more about Knowledge Exchange's efforts around open research practices stay tuned via @knowexchange or sign up to our newsletter at the bottom of this page!
