Candice C. Morey and Loukia Tzavella

Registered Reports are a type of journal article in which the journal evaluates a proposal for a hypothetical research project. The Stage 1 manuscript includes an introductory literature review, details of the proposed method, and a plan for analyzing the data. Reviewers consider this package, without of course seeing the results, which don’t exist yet. The idea is that expert reviewers should be able to judge whether a study’s design is sound and whether the results would be of interest without actually seeing the results. The novelty, significance, or agreement of the results with reviewers’ biases therefore cannot influence the reviews. Producing these sorts of papers will ensure that robust null results or findings that contradict a popular theory that may otherwise be dismissed become part of the published record.

There’s a lot to love about Registered Reports. I always have more plans than resources, so it would be nice to have peer review determine which plans to invest in. It would be great to know when I start data collection that the data are already destined for publication. For students especially, it sounds great to have certainty that the project you are working so hard on has a clear path to publication. Students understandably worry that after much hard work, they may end up with unpublishable null results. A Registered Report can seem like good insurance against this outcome.

However, just because Registered Reports are becoming widely available and great for some purposes does not mean that all of your projects should be Registered Reports. Registered Reports are not to be undertaken lightly. Unexpected things happen in research: the process of learning something from a paradigm can be winding and messy. Journals entertaining proposals for Registered Reports wisely have standards in place to ensure that research they green-light is robust enough to take seriously. These standards are meant to prevent journals from being obliged to publish papers that ultimately don’t advance knowledge because the quality of the results is dubious. For instance, Registered Reports usually require that you commit to achieving very high power (perhaps 0.95 power) or commit to continue collecting data until your Bayes factor is at least 10 in favor of one hypothesis or the other. If you over-estimate your anticipated effect size (and there’s reason to think you have), you could have inadvertently committed to collecting samples much larger than you imagined. Would that be feasible? Would it be worth the effort? Journals also typically expect that a Stage 1 Registered Report will clearly specify some checks on the data’s validity to ensure that data going into the analysis plausibly reflects task performance, or is interpretable as intended. In recognition memory tasks, we may intersperse very easy trials throughout a session, so easy that a reasonable responder could not fail to respond correctly, to check whether each participant was sincerely attempting the task. Someone who responded incorrectly to these easy trials would be presumed to be randomly responding, or being inattentive in general. In some dual-task situations, we set minimum performance thresholds for one task so that we can be sure that participants were truly doing two tasks at once, not focusing on one task to the exclusion of the other. Do you know your Registered design well enough to be sure that your checks catch the behaviors they are meant to, and that you can reasonably estimate how much of your sample will be affected by loss due to the checks? Do you have enough experience with the task to design it in ways that minimize data loss?

The point here is that you do not want to jump into a Registered Report unprepared. Even if you manage to persuade the journal to green-light your idea, you may end up having to request changes to your plan (which can affect in-principle acceptance), or having to re-start data collection after discovering a problem in the design, or being saddled with much more data collection than you bargained for.

Ideally, when you decide to go for a Registered Report, you’ve worked with the paradigm already. You have a strong basis for estimating your effect size, and you know how to design the task to minimize extraneous noise and the motivate participants to perform the task as intended. If you have a great idea for an experiment that involves using a task that’s novel, you should probably not go straight for the Registered Report. Instead, you should do some preliminary work with the task so that you understand it thoroughly. You could eventually include preliminary experiments as a lead-in to the ultimate, registered protocol you are planning. A Registered Report does not need to contain one and only one study.

Also, you must have time to commit to the project, even if it ends up taking longer than anticipated. As with any peer review, it could be a few months before you have comments, and the verdict may be a request for revisions. So it could be many months before your Stage 1 proposal is granted in-principle acceptance. This makes using the RR format problematic for students on limited contracts, unless their institution can grant them an extension or their supervisor is happy to facilitate and monitor data collection on their behalf if the project goes over-time.

In an effort to practice what I regularly preach, I’ve been working the last few months on preparing two Stage 1 Registered Reports. One is a replication, which you might think would be pretty straightforward. Even with a replication, we found that many decisions needed to be made surrounding converting a procedure that was originally done without computers to one in which stimulus presentation was computer-controlled. So many design choices are typically made by whoever is programming, but when we must consider whether each small choice could change the anticipated outcome of the study (or whether any reviewer of the Stage 1 proposal could think so), my colleagues and I spent a lot more time considering seemingly minor design choices than we ordinarily would. I think that has been time well-spent, but that’s debatable. Before I tried preparing a Stage 1 Registered Report myself, I was very enthusiastic about encouraging my PhD students to attempt Registered Reports, but now that I’ve experienced the process, I feel a lot more hesitant. On 3-year UK PhD degrees, I wonder whether there is sufficient time to learn a paradigm well enough to plan and carry out a Registered Report. I now think a more reasonable expectation might be that a PhD student works on a collaborative Registered Report during their studies – perhaps implementing one that was conceived before they arrive on campus, or writing a Stage 1 manuscript that will be implemented by lab colleagues after they finish and move on. I’m curious to hear from researchers who have tried organizing Registered Reports while on time-limited contracts, or supervisors who encouraged students to go for it. Did it work as intended, or were there unforeseen logistical difficulties?