Meta-research for evaluating replicability in psychological science: a roadmap to a crowdsourcing project

Methodological transparency is one of the key elements for research replicability. Our project aims to assess a large number of published studies to evaluate their transparency and investigate how reporting of scientific methodologies has changed over the last decade. After conducting a pilot, we are currently setting up an extended crowdsourcing study to widen the scope of our research. In the pilot, we evaluate 180 experimental studies published in five social and general psychology journals, and related methodological transparency to basic publication characteristics (i.e., area, year, and open access policy). Results indicate that transparency has increased over the years and tends to be greater in articles from journal with a strong accessibility policy. Our plan is now to involve a large network of collaborators to analyse correlational as well as experimental studies from a broader range of psychological areas and to consider additional publication and journal features. We will discuss the challenges we have encountered in setting up our first crowdsourced research, the solutions we found and some issues yet to solve.