One Preregistration to Rule Them All?

February 5th, 2018, Alexander DeHaven, David Mellor


ring

One of the most frequently asked questions we receive about preregistration is “how do I preregister this complex study that will result in many different publications?” This question is particularly challenging if there is only a single data collection effort that will be used to support distinct parts of the larger study and subsequent papers.

First, let's describe where this is not problematic. Many papers describe a series of studies, each with a concise objective. If a round of data collection precedes each study, a researcher can preregister prior to each*, and report the results of confirmed findings and unexpected discoveries separately for each study. Nothing out of the ordinary there.

However, many large projects, particularly interdisciplinary collaborations, will collect data from a population of study participants. Then, each researcher will conduct a set of analyses, while collaborating to a greater or lesser extent with their peers. This is problematic. 

Why? Because complete reporting of proposed analyses is necessary in order to properly interpret the results. Let's imagine an extreme example to demonstrate the possible problem: we preregister 100 hypothesis tests but only report the results of the 5 that have a p-value of less than .05, claiming that other results will appear in subsequent publications. Even though we are sincere, when the results are disconnected from each other the reader is unable to easily interpret the true credibility of the reported findings. Five out of five significant results are very different than five out of 100.

The best solution to this problem is the same solution that underlies any preregistration: planning. Clearly state ahead of time which subset of results will be reported together. This guarantees that our results can be comprehensively reported. Even if we never get all the publications out of the project that we want, each individual set of results will be reported based on the grouping determine ahead of time, before the results could bias that decision.

Alternatively, it could also be beneficial to make more than one preregistration, even if the data collection effort is shared. If each preregistration is connected to the related ones (for example, they could be part of the same OSF project), this will help clarify the relationship between them. Then, each study simply needs to report the results of the analyses specified in the corresponding preregistration. Disclosing the existence of related research efforts is also helpful for providing the full context of the preregistration. For example, if we preregister work that analyzes a common dataset with two collaborators, note that in the preregistration and link to any other preregistration; this enhances the discoverability of all related analyses.

Finally, a reminder about the three rules for reporting the results of preregistered work. First, include a link to the preregistration in any resulting article. Second, report the results of all preregistered analyses (or if the preregistration specifies which results will be reported separately, note that). Third, any unexpected trends or discoveries that you make that were not part of your preregistration should be reported and described as preliminary findings that deserve additional confirmation.

*Commonly, just the final study will be preregistered, though we encourage preregistration prior to any data collection effort.

Recent Blogs

The Content of Open Science

What Second Graders Can Teach Us About Open Science

What's Going on With Reproducibility?

Open Science and the Marketplace of Ideas

3 Things Societies Can Do to Promote Research Integrity

How to Manage and Share Your Open Data

Interview with Prereg Challenge Award Winner Dr. Allison Skinner

Next Steps for Promoting Transparency in Science

Public Goods Infrastructure for Preprints and Innovation in Scholarly Communication

A How-To Guide to Improving the Clarity and Continuity of Your Preregistration

Building a Central Service for Preprints

Three More Reasons to Take the Preregistration Challenge

The Center for Open Science is a Culture Change Technology Company

Preregistration: A Plan, Not a Prison

How can we improve diversity and inclusion in the open science movement?

OSF Fedora Integration, Aussie style!

Replicating a challenging study: it's all about sharing the details.

How Preregistration Helped Improve Our Research: An Interview with Preregistration Challenge Awardees

Some Examples of Publishing the Research That Actually Happened

Are reproducibility and open science starting to matter in tenure and promotion review?

The IRIS Replication Award and Collaboration in the Second Language Research Community

We Should Redefine Statistical Significance

Some Cool New OSF Features

How Open Source Research Tools Can Help Institutions Keep it Simple

OSF Add-ons Help You Maximize Research Data Storage and Accessibility

10 Tips for Making a Great Preregistration

Community-Driven Science: An Interview With EarthArXiv Founders Chris Jackson, Tom Narock and Bruce Caron

A Preregistration Coaching Network

Why are we working so hard to open up science? A personal story.

One Preregistration to Rule Them All?

Using the wiki just got better.

Transparent Definitions and Community Signals: Growth in the Open Science Community

We're Committed to GDPR. Here's How.

Preprints: The What, The Why, The How.

The Prereg Challenge Is Ending. What's Next?

We are Now Registering Preprint DOIs with Crossref

Using OSF in the Lab

Psychology's New Normal

How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

The Landscape of Open Data Policies

Open Science is a Behavior.

Why pre-registration might be better for your career and well-being

Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

Towards minimal reporting standards for life scientists

Looking Back on the Prereg Challenge and Forward To More Credible Research

OSF: Origin, growth, and what’s next

A Critique of the Many Labs Projects

The Rise of Open Science in Psychology, A Preliminary Report

Strategy for Culture Change

New OSF Registries Enhancements Improve Efficiency and Quality of Registrations

Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.