Imagine you are a criminology researcher. You’ve just received your data, and you’re excited to dive in. You test your hypothesis, but the results don’t seem quite right. So, you tweak the model a bit, try again, and – voila! You’ve got a statistically significant result. Success! Or is it?

Let’s break it down. A statistically significant result means there’s a low probability (typically less than 5%) that the pattern you found happened by chance. But here’s the catch – if you run 20 different tests, pure probability suggests that at least one of them will turn up significant just by luck, even if no real effect exists. This is called a false positive – a result that looks meaningful but isn’t actually real.

Now, imagine you don’t just test one model but try multiple variations – adding or removing variables, changing the way data is grouped, or re-running the analysis until something “works”. This practice, known as p-hacking, increases the chances of stumbling upon a false positive. On the one hand, you’ve found a result that seems exciting and might help you publish your research. On the other hand, it might be nothing more than a statistical illusion – one that won’t hold up when someone else tries to replicate your study.

Picture a different scenario. Before receiving your data, you carefully plan your hypothesis, analysis, and how you’ll handle surprises. You write this plan down and share it online to hold yourself accountable. Once the data arrives, you stick to the plan and report all results, even if they aren’t exactly what you hoped for. Congratulations – you’ve just followed a preregistration plan!

The case for preregistration in criminology

Preregistration is a growing practice in science where researchers create a detailed plan for their study before collecting or analyzing data. In preregistration, the researcher doesn’t only commit to the methodology in advance, but also to their hypotheses and analysis plan. This practice can help prevent (unintentional) bias, and increase transparency and credibility in research.

Research in social sciences, including criminology, faces significant challenges like publication bias and the pressure to produce significant results. These pressures can lead to questionable research practices (QRPs), such as p-hacking – rerunning variations of an analysis until something statistically significant appears – selective reporting, or hypothesizing after results are known (HARKing), pretending like you predicted the results all along.

Consider this: 90% of published studies in social sciences report statistically significant results¹. This sounds impressive—until you realize that only 45% of large-scale replication attempts succeed. In criminology specifically, a 2020 survey of over 1,500 published researchers revealed troubling numbers²:

• 53% admitted to not reporting all analyses conducted.
• 45% omitted null findings.
• 39% acknowledged p-hacking.
• 29% HARKed.
• 24% excluded data selectively.

These practices undermine the reliability of our findings and hinder scientific progress. In criminology, where research directly influences policy, policing practices, and the justice system, these issues are critical. If we don’t improve the reliability of our findings, we risk making decisions based on flawed or incomplete evidence. The consequences of misinformed policies can be far-reaching.

A big part of the problem lies in the way research is rewarded.  Publications are the currency of academic success, and studies with statistically significant results – often called positive results – are much more likely to be published, creating a publication bias where studies with non-significant or “boring” findings are buried and flashy, positive results dominate the literature. But many of these positive findings could be false positives, contributing to the replication crisis.

Why preregistration is a game-changer

One thing that we can do to battle this is preregister our research. How can it help?

1. It forces careful planning: Before the study starts, researchers think critically about their hypotheses, methods, and potential challenges. This ensures that studies are well-designed and resilient to unforeseen issues.
2. It keeps us honest: Preregistration prevents cherry-picking analyses or outcomes, making it harder to engage in questionable research practices.
3. It builds trust: Sharing a preregistered plan with your audience builds trust, showing your audience that you value transparency and accountability.
4. It saves time and resources: When results are more reliable, we don’t waste time chasing dead ends or false positives.

To take things a step further, you can test your entire workflow. You can even conduct your planned analysis on simulated or made-up data to ensure that your code works as intended and your approach is robust. This added layer of preparation not only minimizes errors but also ensures you’re ready to hit the ground running once the real data arrives. Once it does, you only hit “play” and run your preregistered analysis to get immediate results!

Concerns and misconceptions

Preregistration isn’t without its critics. Some argue that it’s time-consuming or impractical, especially in criminology, where human behavior can be unpredictable. What if you encounter an unexpected pattern in your data? Should you ignore it because it wasn’t in your preregistration?

Not at all. Preregistration is a floor, not a ceiling, as statistician Andrew Gelman³ aptly puts it. If you discover something novel or need to deviate from your plan, you can – but it’s important to be transparent about it.

How to Get Started with Preregistration

Ready to give it a try? Platforms like the Open Science Framework (OSF)⁴ make it easy. They offer simple templates to guide you through the process⁵, from writing your hypothesis to outlining your analysis plan. Once you’re done, you can share the preregistration link in your paper to show your readers what you planned ahead of time.

The choice is yours

So, think back to that moment when you were tweaking your analysis, searching for a significant result. What if, instead, you had a clear path from the start? You reduce the risk of bias and increase the credibility of your results. Wouldn’t that be a stronger foundation for meaningful research?

Preregistration isn’t a magic bullet, and it won’t solve every problem in criminological research, but it can help us move toward a more transparent and trustworthy criminology. As researchers embrace this practice, we can create more trustworthy research, reduce wasted resources, and strengthen evidence-based policies. While debates around its role in exploratory research or peer review processes persist (see registered reports⁶), preregistration remains a powerful tool to elevate the standards of criminology.

So, next time you start a study, ask yourself: Do I want findings that look good—or findings I can truly trust?

---

This blog is written by Danielle Stibbe, PhD Candidate at NSCR.

---

Read further:

1. Scheel, A. M., Schijen, M. R., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2). DOI: https://doi.org/10.1177/25152459211007467.
2. Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2023). Questionable research practices and open science in quantitative criminology. Journal of Quantitative Criminology, 39(1), 21-51. DOI: https://doi.org/10.1007/s10940-021-09525-6.
3. https://statmodeling.stat.columbia.edu/2024/03/17/preregistration-is-a-floor-not-a-ceiling/
4. https://www.cos.io/initiatives/prereg
5. https://help.osf.io/article/158-create-a-preregistration
6. https://www.aje.com/arc/pre-registration-vs-registered-reports/