The ReplicationWiki – an important resource for economists

The ReplicationWiki gives information about empirical studies in the social sciences. It offers a database containing empirical research and shows where to find program codes and data, which methods were used and which software. One can search for keywords, authors, databases or journals. In particular it lists so far 515 replications, 20 corrections, and 10 retractions; including information about the results of replications. 3,388 studies are already covered. The pages have been accessed more than 5.9 million times.

It helps researchers see which results have already been revisited. For instructors it helps to identify practical examples for which data and code are available in a software format that is accessible to their students. In our experience it helps to motivate students to study quantitative methodology and to approach published results with healthy scientific scepticism.

The ReplicationWiki also offers information on literature on the topic of replication, on journal policies on data availability, on data and software repositories and on projects that employ replication in the different social sciences.

Originally the Wiki was founded for economics. By now a number of studies from other social sciences were included, in particular political sciences and sociology. It has been cited as a project to follow also from fields like empirical law and empirical archaeology. Contributors are welcome! As the project is set up as a wiki website, any researcher can participate. After registration with one’s real name and an institutional email address one can for example add replications or studies that should be replicated, announce relevant events, discuss suggestions how to improve the project, set up one’s own user page and vote on which studies should be replicated. Registrations just to signal support for the project are also welcome. The ReplicationWiki is meant for an international audience and is therefore set up in English. Adding studies in other languages to its database is however welcome, and users can write in their own language on their user page to help others find them.

A detailed description how to add further studies can be found here: for replications and replicated studies and for studies that have not yet been replicated.

Further reading: ReplicationWiki – Improving Transparency in the Social Sciences, Jan H. Höffler, D-Lib Magazine, March/April 2017, Volume 23, Number 3/4, doi: 10.1045/march2017-hoeffler.

Picture: Image by Jan Vašek from Pixabay . License: click

New paper highlights replication studies in economics

Four researchers associated with the former project EDaWaX have recently published a new research article entitled “Replication studies in economics – How many and which papers are chosen for replication, and why?” The article, written by Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff and Gert G. Wagner, sheds light on replication practice in empirical economics. The article can be found here (published in Research Policy under a Creative Commons license). Here, they provide a brief overview of the paper:

Read the rest of this entry »

Replications in the social sciences: New study confirms ongoing challenges

Much has been said on the importance of replications. Recently, nature has published another comment that deals with this question. Paul Gertler, Sebastian Galiani and Mauricio Romero (GGR) conducted a survey, in which they focussed on the fields of economics, political science, sociology and psychology. They conclude that ‘the current system makes original authors and replicators antagonists.’

They found, that in the top-tier economics journals only few articles are replications – and all of those refute the original results. That said, GGR also asked 35 editors and co-editors of these economics journals about their perceptions towards publishing replications. While all editors who responded would publish a study that refutes the original findings, only a fourth can image to publish a study which confirms the original results. Read the rest of this entry »

KNAW recommends making replication studies a normal and essential part of science

Over the past few years several systematic series of replication studies have been unable to reproduce many important scientific findings in a whole slew of disciplines. This has led to a debate within the scientific community about the way science is currently being conducted and the role of replication studies.  Our blog also published many posts which deal with this question.

Now, KNAW (the Royal Netherlands Academy of Arts and Sciences) has published a report, which analyses the causes of non-reproducibility, assesses the desirability of replication studies and also offers recommendations for improving reproducibility and for conducting replication studies. Read the rest of this entry »

The replication crisis in economics – and how we might answer it

On September 8, 2017 the ZBW Leibniz Information Center for Economics hosted the workshop “Replications in Empirical Economics – Ways out of the Crisis” at the Annual Conference of the Verein für Socialpolitik in Vienna, Austria. Thirty participants and four speakers engaged in lively and stimulating discussions about replications and the publication of replications in Economics.

This is a cross-post of the weblog of the replication network. Read the rest of this entry »

Against the replication crisis: New international journal encourages replication studies

Read the rest of this entry »

American Economic Review publishes AEA’s Session Papers on Replication

Two weeks ago, the American Economic Review published the ‘Papers and Proceedings‘ of the 129th annual meeting of the American Economic Association (AEA) held in January, 2017.

At this year’s meeting, one session was dedicated to the topic of ‘Replication in Microeconomics‘ while another focussed on ‘Replication and Ethics in Economics: Thirty Years after Dewald, Thursby, and Anderson“.

In both sessions, very interesting and excellent papers were presented.

Below, I list all presentations of these sessions and the corresponding links to the papers (if available): Read the rest of this entry »

New Working Paper published: ‘The Economics of Replication’

Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff and Gert G. Wagner have published a new working paper on the ‘economics of replication’.

In the abstract of the paper, the researchers state:

“Replication studies are considered a hallmark of good scientific practice. Yet they are treated among researchers as an ideal to be professed but not practiced. To provide incentives and favorable boundary conditions for replication practice, the main stakeholders need to be aware of what drives replication. Here we investigate how often replication studies are published in empirical economics and what types of journal articles are replicated. We find that from 1974 to 2014 less than 0.1% of publications in the top-50 economics journals were replications. We do not find empirical support that mandatory data disclosure policies or the availability of data or code have a significant effect on the incidence of replication. The mere provision of data repositories may be ineffective, unless accompanied by appropriate incentives. However, we find that higher-impact articles and articles by authors from leading institutions are more likely to be subject of published replication studies whereas the replication probability is lower for articles published in higher-ranked journals.”

The paper is available here.

Read the rest of this entry »

Dutch research funder grants 3 million Euros for replication studies

The Netherlands Organisation for Scientific Research (NWO), one of the biggest science-funding bodies in the Netherlands, is making 3 million euros available for a Replication Studies pilot programme. In this programme, scientists will be able to repeat research that has been carried out by others. The pilot focuses on replicating studies that have a large impact on science, government policy or the public debate – namely on the area of the social sciences and health research and healthcare innovation.

The pilot programme Replication Studies focuses on two types of research: On the one hand it focusses on replications with existing data: the datasets from the original study are reanalysed. On the other hand, it focusses on reproductions (with new data): a data collection is put together, which is subjected to the same research protocol as in the original study (read more about the different definitions of reproduction and replication). Read the rest of this entry »

New Working Paper: “Perceptions and Practices of Replication by Social and Behavioral Scientists”

One of our project partners has just released a publication that deals with the replication crises in economics and the social sciences.

In the abstract the three autors state:

We live in a time of increasing publication rates and specialization of scientific disciplines. More and more, the research community is facing the challenge of assuring the quality of research and maintaining trust in the scientific enterprise. Replication studies are necessary to detect erroneous research. Thus, the replicability of research is considered a hallmark of good scientific practice and it has lately become a key concern for research communities and science policy makers alike.

In their discussion paper Fecher, Fräßdorf and Wagner analyze perceptions and practices regarding replication studies in the social and behavioral sciences. Their analyses are based on a survey of almost 300 researchers.

Read the rest of this entry »

Contemporary, useful and subject-based: The replication network

Today I would like to introduce the replication network (TRN) to our readers, a network whose purpose is “to encourage economists and their journals to publish replications.” This is all along in line with the purpose of our own project.

The website of the replication network serves as a channel of communication to both update scholars about the state of replications in economics and to establish a network for the sharing  of information and ideas among economists. It offers important information on the possibility to publish replication studies in economics journals and provides lists of publications dealing with the topic of replications in economic research. Also a list of published replication studies is available. Read the rest of this entry »

Goettingen University: Project fosters Replication in Economics

There are many good reasons why we should replicate scientific findings. In his article “Open Access Economics Journals and the Market for Reproducible Economic Research“, the economist B.D. McCullough (2009) lists some of the reasons why replicable research is crucial for science:

„[…]replication ensures that the method used to produce the results is known. Whether the results are correct or not is another matter, but unless everyone knows how the results were produced, their correctness cannot be assessed. Replicable research is subject to the scientific principle of verification; non-replicable research cannot be verified. Second, and more importantly, replicable research speeds scientific progress. We are all familiar with Newton’s quote, ‘If I have seen a little further it is by standing on the shoulders of giants.’ […] Third, researchers will have an incentive to avoid sloppiness. […] Fourth, the incidence of fraud will decrease.“ (p.118)

More recently, the case of the US-economists Rogoff and Reinhart clearly illustrated the need for replications to be much more common in science and scientific education. But… Read the rest of this entry »

Why Replications & Data Availability matter: The Case of Rogoff and Reinhart

Chart by Reinhart/Rogoff

Yesterday several blogs, journals and news agencies spread the news: The US star economists Kenneth Rogoff and Carmen Reinhart are in trouble, because a paper of them -which had and has a considerable influence on worldwide economic policy- contains at least one serious error.

Their paper “Growth in a Time of Debt” was published in the flagship of economic research – the American Economic Review (AER) with also has a data availability policy that was evaluated in the course of our research project.
Important political assumptions and decisions were based on the research of Rogoff and Reinhart:  Olli Rehm, European Commissioner for Economic and Monetary Affairs and the Euro and vice president of the European Commission and also the US-Republican Paul Ryan argued with the findings of Rogoff and Reinhart to justify austerity economics. Read the rest of this entry »

Requirements for Data Availability Policies to enable Replications

In our analyses for work package 2 we collected some criteria to evaluate the quality of the data policies we found in our sample.

It was important to identify some core requirements that aim to ensure the replicability of economic research.  This was not an easy task, because we had to find some criteria that are suitable for many fields of research in economics.

Therefore we consulted several research papers and used the recommendations we found in the papers as a basis for analysing and assessing the suitability of data availability policies of economics journals in our study.

We’d like to discuss these criteria with our readers. Feel free to submit comments or send me an e-mail.

Read the rest of this entry »

Nature Magazine Special Issue: Data Replication & Repoducibility

The nature magazine has just published a special issue about data replication and reproducibility.

In their introduction the authors are claiming that replication is considered the scientific gold standard. To give a broader view about replication, the journal explores some of the issues associated with the replication of results in different scientific disciplines as for example primate cognition and behaviour research, computer sciences, biology and climate change studies.

Worth reading is the Editorial by J. Crocker and L. Cooper that is dealing with the fraud of Diederik Stapel and raises the question “what could be done to protect science and the public from fraud in the future?”. The answer of the authors, both psychologists, is:

“Greater transparency with data, including depositing data in repositories where they can be accessed by other scientists […], might have sped up detection of this fraud, and it would certainly make researchers more careful about the analyses that they publish.”

Read the rest of this entry »