Transparent Definitions and Community Signals: Growth in the Open Science Community

May 4th, 2018, Ian Sullivan

green bottles

There is an active discussion in the open science community about what "preregistration" means in practice and what researchers should conclude about research that has been preregistered.  These two intertwined challenges, one about knowledge production and one about knowledge consumption, are useful lenses through which to look at broader challenges in the growing open science community. Addressing these challenges will involve new roles for individuals as well as changes to our collective systems for sharing and evaluating knowledge, changes that are only possible through community support.  Whether or not you have heard of preregistration before, these discussions are worthy of consideration.

I run workshops for COS with researchers around the country to introduce open science practices like preregistration and I always start with a definition. I tell researchers that preregistration means putting your research plan in a public repository before you run the study and that the benefits of preregistration depend directly on how much information you put into it. In a recent blog post titled "Preregistration is a Hot Mess. And You Should Probably Do It.” Professor Alison Ledgerwood from UC Davis uses this wonderful chart to lay out the main potential benefits from preregistration and what information is needed to achieve each one:

ledgerwood transparency table

Shortly thereafter a presentation (p.24) by Stanford's John Ioannidis outlined five different "levels" of preregistration and the information required for each.  

Both ideas underscore the variability in content and goals that may be present in any individual preregistration.  This variability presents two challenges for researchers: 1) How to make sure they include the right information in their own preregistrations to achieve the benefits they wish and 2) What, if anything, the existence of a preregistration tells them about other people's research.  For me the line between these two challenges shows the limitation of what we can achieve with education efforts alone. 

We can, and do, work to directly help interested researchers learn how to preregister their own work to get whichever benefits of preregistration that they wish (email if you want some specific advice!).  Yet, even if every researcher were looking for the same benefits from preregistration, the mere existence of a preregistration only tells you about that researcher's intentions.  Knowing these intentions may be useful in evaluating the research that followed but the biggest benefit of preregistration comes not from this signal of researcher intent but from the actual content of the preregistration and the additional transparency it brings to the research process. Evaluating this kind of content is beyond what education, or even the efforts of individual researchers, can accomplish alone and requires community action.

Transparent practices like preregistration and sharing data, materials, and analysis code from research enable others in the field to understand a particular study by increasing the information about that study available for review.  This increase in information can also act to reduce the number of studies that any one researchers can practically review during the time they have available. Part of the discussion about signals of transparency and the definition of preregistration is a response to this increase in the amount of information produced; we are looking for signals about research quality and researcher practice to help them navigate this growing volume of information.  This is a good problem to have because it means that "open science" is becoming just "science" but sorting through this kind of information boom can be a real problem and one that will only increase as these open science practices continue to become more normal.

Building community tools to address this need is one of the great areas of growth and innovation in the open science community right now.  Those tools take many forms, from departments outlining open science goals for their research, journals that verify the computational reproducibility of published articles (1, 2, 3, 4, 5), community-moderated preprint services, or journals that published Registered Report, in which the alignment between the preregistration and the final results are verified.  Each of these efforts help produce useful signals for the community about how research is conducted and what information is available about that research.  How to organize these systems and what kind of information they should present to the community is still evolving, see for instance the recent preprint by Tom Hardwicke and John Ioannidis exploring the Registered Reports landscape and ways in which it may not line up with expectations about preregistrations more generally.  The continued growth and evolution of these tools is vital to the growth of open science.

It can be easy, especially if you are new to the field, to get lost in the discussions of individual open science practices like preregistration and conclude that open science is just about changing what researchers publish to the world.  While that work is important, what this individualistic focus misses is the vital work that we all do together to help spread open science practices and to better process all the new information that open science practices produce. There are a host of ways you can make a difference in open science, whether that means modifying your own research practices, advocating for policies at the department level, or participating in community comment and review processes at the preprint or journal level, there is always room for growth and improvement.

Recent Blogs

The Content of Open Science

What Second Graders Can Teach Us About Open Science

What's Going on With Reproducibility?

Open Science and the Marketplace of Ideas

3 Things Societies Can Do to Promote Research Integrity

How to Manage and Share Your Open Data

Interview with Prereg Challenge Award Winner Dr. Allison Skinner

Next Steps for Promoting Transparency in Science

Public Goods Infrastructure for Preprints and Innovation in Scholarly Communication

A How-To Guide to Improving the Clarity and Continuity of Your Preregistration

Building a Central Service for Preprints

Three More Reasons to Take the Preregistration Challenge

The Center for Open Science is a Culture Change Technology Company

Preregistration: A Plan, Not a Prison

How can we improve diversity and inclusion in the open science movement?

OSF Fedora Integration, Aussie style!

Replicating a challenging study: it's all about sharing the details.

How Preregistration Helped Improve Our Research: An Interview with Preregistration Challenge Awardees

Some Examples of Publishing the Research That Actually Happened

Are reproducibility and open science starting to matter in tenure and promotion review?

The IRIS Replication Award and Collaboration in the Second Language Research Community

We Should Redefine Statistical Significance

Some Cool New OSF Features

How Open Source Research Tools Can Help Institutions Keep it Simple

OSF Add-ons Help You Maximize Research Data Storage and Accessibility

10 Tips for Making a Great Preregistration

Community-Driven Science: An Interview With EarthArXiv Founders Chris Jackson, Tom Narock and Bruce Caron

A Preregistration Coaching Network

Why are we working so hard to open up science? A personal story.

One Preregistration to Rule Them All?

Using the wiki just got better.

Transparent Definitions and Community Signals: Growth in the Open Science Community

We're Committed to GDPR. Here's How.

Preprints: The What, The Why, The How.

The Prereg Challenge Is Ending. What's Next?

We are Now Registering Preprint DOIs with Crossref

Using OSF in the Lab

Psychology's New Normal

How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

The Landscape of Open Data Policies

Open Science is a Behavior.

Why pre-registration might be better for your career and well-being

Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

Towards minimal reporting standards for life scientists

Looking Back on the Prereg Challenge and Forward To More Credible Research

OSF: Origin, growth, and what’s next

A Critique of the Many Labs Projects

The Rise of Open Science in Psychology, A Preliminary Report

Strategy for Culture Change

New OSF Registries Enhancements Improve Efficiency and Quality of Registrations

Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.