Variation and Covariation in Large-scale Replication Projects: An Evaluation of Replicability
Over the last decade, large-scale replication projects across the biomedical and social sciences have reported relatively low replication rates. In these large-scale replication projects, replication has typically been evaluated based on a single replication study of some original study and dichotomously as successful or failed. However, evaluations of replicability based on a single study and as dichotomous are inadequate, and evaluations of replicability should instead be based on multiple studies, be continuous, and be multi-faceted. Further, such evaluations are in fact possible due to two features shared by many large-scale replication projects. In this paper, we provide such an evaluation for two prominent large-scale replication projects, one which replicated a phenomenon from cognitive psychology and another which replicated 13 phenomena from social psychology and behavioral economics. Our results indicate a very high degree of replicability in the former (e.g., variation ranging from 15 to 17 milliseconds across variates and 2 to 3 milliseconds across effects at the lab level) and a medium to low degree of replicability in the latter (e.g., variation ranging from 0.11 to 0.69 σd across variates and 0.09 to 0.79 σd across effects at the lab level). They also suggest an unidentified covariate in each, namely ocular dominance in the former and political ideology in the latter, that is theoretically pertinent. They finally have implications for evaluations of replicability at large, recommendations for future large-scale replication projects, and design-based model generalization.
