Skip to main content

Scientific Reproducibility and its Predictability


September 20, 2016 

I study the reproducibility of results from lab experiments in social sciences and how well these can be predicted.

Reproducibility

In Camerer et. al. (2016), we replicate 18 studies in experimental economics published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. We follow a carefully designed procedure and find a significant effect in the right direction in 11 of the replications. See the article and our website for more information.

In a new project, we plan to replicate 22 experimental studies in the social sciences published in Nature and Science in 2010-2015. We plan to conduct the replications between September 2016 and September 2017. See our website for more information.

Predictability

Replications are very expensive, the work is hard and often not very rewarding. Therefor we also study how well reproducibility can be predicted. If a predictive mechanism is accurate enough, we can use it to evaluate which studies to actually replicate and which results we can trust.

In another project (Munafo et. al., 2015), we study how the outcome of the 2014 Research Excellence Framework (REF) evaluation of UK Chemistry departments could be predicted in a market where the traders were faculty members at the participating schools. We show that prediction markets can be a useful tool to complement costly large-scale quality evaluations.

Currently, I am developing the methods we used in the papers above to study how to improve prediction of reproducibility. I use techniques from machine learning to explore which features are important for replication and to estimate the accuracy of the best predictive model.

References

  • Camerer, Colin F., Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, Emma Heikensten, Felix Holzmeister, Taisuke Imai, Siri Isaksson, Gideon Nave, Thomas Pfeiffer, Michael Razen, and Hang Wu. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.” Science. American Association for the Advancement of Science. doi:10.1126/science.aaf0918.
  • Munafo, Marcus R., Thomas Pfeiffer, Adam Altmejd, Emma Heikensten, Johan Almenberg, Alexander Bird, Yiling Chen, Brad Wilson, Magnus Johannesson, and Anna Dreber. 2015. “Using Prediction Markets to Forecast Research Evaluations.” Royal Society Open Science 2 (10). The Royal Society. doi:10.1098/rsos.150287.