New paper highlights replication studies in economics

Posted: October 23rd, 2018 | Author: | Filed under: Data Sharing, EDaWaX | Tags: | No Comments »

Four researchers associated with the former project EDaWaX have recently published a new research article entitled “Replication studies in economics – How many and which papers are chosen for replication, and why?” The article, written by Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff and Gert G. Wagner, sheds light on replication practice in empirical economics. The article can be found here (published in Research Policy under a Creative Commons license). Here, they provide a brief overview of the paper:

Read the rest of this entry »


Replications in the social sciences: New study confirms ongoing challenges

Posted: February 26th, 2018 | Author: | Filed under: Data Policy, Data Sharing | Tags: | Comments Off on Replications in the social sciences: New study confirms ongoing challenges

Much has been said on the importance of replications. Recently, nature has published another comment that deals with this question. Paul Gertler, Sebastian Galiani and Mauricio Romero (GGR) conducted a survey, in which they focussed on the fields of economics, political science, sociology and psychology. They conclude that ‘the current system makes original authors and replicators antagonists.’

They found, that in the top-tier economics journals only few articles are replications – and all of those refute the original results. That said, GGR also asked 35 editors and co-editors of these economics journals about their perceptions towards publishing replications. While all editors who responded would publish a study that refutes the original findings, only a fourth can image to publish a study which confirms the original results. Read the rest of this entry »


KNAW recommends making replication studies a normal and essential part of science

Posted: January 22nd, 2018 | Author: | Filed under: Data Sharing, found on the net | Tags: , , | Comments Off on KNAW recommends making replication studies a normal and essential part of science

Over the past few years several systematic series of replication studies have been unable to reproduce many important scientific findings in a whole slew of disciplines. This has led to a debate within the scientific community about the way science is currently being conducted and the role of replication studies.  Our blog also published many posts which deal with this question.

Now, KNAW (the Royal Netherlands Academy of Arts and Sciences) has published a report, which analyses the causes of non-reproducibility, assesses the desirability of replication studies and also offers recommendations for improving reproducibility and for conducting replication studies. Read the rest of this entry »


The replication crisis in economics – and how we might answer it

Posted: September 21st, 2017 | Author: | Filed under: Conference, Workshop | Tags: , | Comments Off on The replication crisis in economics – and how we might answer it

On September 8, 2017 the ZBW Leibniz Information Center for Economics hosted the workshop “Replications in Empirical Economics – Ways out of the Crisis” at the Annual Conference of the Verein für Socialpolitik in Vienna, Austria. Thirty participants and four speakers engaged in lively and stimulating discussions about replications and the publication of replications in Economics.

This is a cross-post of the weblog of the replication network. Read the rest of this entry »


Against the replication crisis: New international journal encourages replication studies

Posted: August 1st, 2017 | Author: | Filed under: Data Sharing, journals, Projects | Tags: , , | Comments Off on Against the replication crisis: New international journal encourages replication studies
 Replications are pivotal for the credibility of empirical economics. Only findings which are robust and replicable can be generalized and can serve as evidence based advice to economic policy. But, already in 1983 Edward Leamer stated (p. 37):

Read the rest of this entry »


American Economic Review publishes AEA’s Session Papers on Replication

Posted: May 16th, 2017 | Author: | Filed under: Conference | Tags: , , | Comments Off on American Economic Review publishes AEA’s Session Papers on Replication

Two weeks ago, the American Economic Review published the ‘Papers and Proceedings‘ of the 129th annual meeting of the American Economic Association (AEA) held in January, 2017.

At this year’s meeting, one session was dedicated to the topic of ‘Replication in Microeconomics‘ while another focussed on ‘Replication and Ethics in Economics: Thirty Years after Dewald, Thursby, and Anderson“.

In both sessions, very interesting and excellent papers were presented.

Below, I list all presentations of these sessions and the corresponding links to the papers (if available): Read the rest of this entry »


New Working Paper published: ‘The Economics of Replication’

Posted: February 14th, 2017 | Author: | Filed under: Data Sharing, EDaWaX, journals | Tags: , | Comments Off on New Working Paper published: ‘The Economics of Replication’

Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff and Gert G. Wagner have published a new working paper on the ‘economics of replication’.

In the abstract of the paper, the researchers state:

“Replication studies are considered a hallmark of good scientific practice. Yet they are treated among researchers as an ideal to be professed but not practiced. To provide incentives and favorable boundary conditions for replication practice, the main stakeholders need to be aware of what drives replication. Here we investigate how often replication studies are published in empirical economics and what types of journal articles are replicated. We find that from 1974 to 2014 less than 0.1% of publications in the top-50 economics journals were replications. We do not find empirical support that mandatory data disclosure policies or the availability of data or code have a significant effect on the incidence of replication. The mere provision of data repositories may be ineffective, unless accompanied by appropriate incentives. However, we find that higher-impact articles and articles by authors from leading institutions are more likely to be subject of published replication studies whereas the replication probability is lower for articles published in higher-ranked journals.”

The paper is available here.

Read the rest of this entry »


Dutch research funder grants 3 million Euros for replication studies

Posted: August 2nd, 2016 | Author: | Filed under: Data Sharing, found on the net | Tags: , | Comments Off on Dutch research funder grants 3 million Euros for replication studies

350_incentive_6912846727_48f07c8899_zThe Netherlands Organisation for Scientific Research (NWO), one of the biggest science-funding bodies in the Netherlands, is making 3 million euros available for a Replication Studies pilot programme. In this programme, scientists will be able to repeat research that has been carried out by others. The pilot focuses on replicating studies that have a large impact on science, government policy or the public debate – namely on the area of the social sciences and health research and healthcare innovation.

The pilot programme Replication Studies focuses on two types of research: On the one hand it focusses on replications with existing data: the datasets from the original study are reanalysed. On the other hand, it focusses on reproductions (with new data): a data collection is put together, which is subjected to the same research protocol as in the original study (read more about the different definitions of reproduction and replication). Read the rest of this entry »


New Working Paper: “Perceptions and Practices of Replication by Social and Behavioral Scientists”

Posted: April 21st, 2016 | Author: | Filed under: Data Sharing, Report | Tags: , , | Comments Off on New Working Paper: “Perceptions and Practices of Replication by Social and Behavioral Scientists”

300 CoverOne of our project partners has just released a publication that deals with the replication crises in economics and the social sciences.

In the abstract the three autors state:

We live in a time of increasing publication rates and specialization of scientific disciplines. More and more, the research community is facing the challenge of assuring the quality of research and maintaining trust in the scientific enterprise. Replication studies are necessary to detect erroneous research. Thus, the replicability of research is considered a hallmark of good scientific practice and it has lately become a key concern for research communities and science policy makers alike.

In their discussion paper Fecher, Fräßdorf and Wagner analyze perceptions and practices regarding replication studies in the social and behavioral sciences. Their analyses are based on a survey of almost 300 researchers.

Read the rest of this entry »


Contemporary, useful and subject-based: The replication network

Posted: October 23rd, 2015 | Author: | Filed under: found on the net | Tags: , | Comments Off on Contemporary, useful and subject-based: The replication network

3d social network_by_stockmonkeys.com_cc_by_3.0

Today I would like to introduce the replication network (TRN) to our readers, a network whose purpose is “to encourage economists and their journals to publish replications.” This is all along in line with the purpose of our own project.

The website of the replication network serves as a channel of communication to both update scholars about the state of replications in economics and to establish a network for the sharing  of information and ideas among economists. It offers important information on the possibility to publish replication studies in economics journals and provides lists of publications dealing with the topic of replications in economic research. Also a list of published replication studies is available. Read the rest of this entry »


Goettingen University: Project fosters Replication in Economics

Posted: June 3rd, 2014 | Author: | Filed under: Projects | Tags: , | 1 Comment »

300_med_miracleThere are many good reasons why we should replicate scientific findings. In his article “Open Access Economics Journals and the Market for Reproducible Economic Research“, the economist B.D. McCullough (2009) lists some of the reasons why replicable research is crucial for science:

„[…]replication ensures that the method used to produce the results is known. Whether the results are correct or not is another matter, but unless everyone knows how the results were produced, their correctness cannot be assessed. Replicable research is subject to the scientific principle of verification; non-replicable research cannot be verified. Second, and more importantly, replicable research speeds scientific progress. We are all familiar with Newton’s quote, ‘If I have seen a little further it is by standing on the shoulders of giants.’ […] Third, researchers will have an incentive to avoid sloppiness. […] Fourth, the incidence of fraud will decrease.“ (p.118)

More recently, the case of the US-economists Rogoff and Reinhart clearly illustrated the need for replications to be much more common in science and scientific education. But… Read the rest of this entry »


Why Replications & Data Availability matter: The Case of Rogoff and Reinhart

Posted: April 18th, 2013 | Author: | Filed under: found on the net | Tags: , , | 1 Comment »

Yesterday several blogs, journals and news agencies spread the news: The US star economists Kenneth Rogoff and Carmen Reinhart are in trouble, because a paper of them -which had and has a considerable influence on worldwide economic policy- contains at least one serious error.

Their paper “Growth in a Time of Debt” was published in the flagship of economic research – the American Economic Review (AER) with also has a data availability policy that was evaluated in the course of our research project.
Important political assumptions and decisions were based on the research of Rogoff and Reinhart:  Olli Rehm, European Commissioner for Economic and Monetary Affairs and the Euro and vice president of the European Commission and also the US-Republican Paul Ryan argued with the findings of Rogoff and Reinhart to justify austerity economics. Read the rest of this entry »


Requirements for Data Availability Policies to enable Replications

Posted: June 21st, 2012 | Author: | Filed under: Data Policy, EDaWaX | Tags: , , | 2 Comments »

In our analyses for work package 2 we collected some criteria to evaluate the quality of the data policies we found in our sample.

It was important to identify some core requirements that aim to ensure the replicability of economic research.  This was not an easy task, because we had to find some criteria that are suitable for many fields of research in economics.

Therefore we consulted several research papers and used the recommendations we found in the papers as a basis for analysing and assessing the suitability of data availability policies of economics journals in our study.

We’d like to discuss these criteria with our readers. Feel free to submit comments or send me an e-mail.

Read the rest of this entry »


Nature Magazine Special Issue: Data Replication & Repoducibility

Posted: December 22nd, 2011 | Author: | Filed under: Data Sharing | Tags: , | Comments Off on Nature Magazine Special Issue: Data Replication & Repoducibility

The nature magazine has just published a special issue about data replication and reproducibility.

In their introduction the authors are claiming that replication is considered the scientific gold standard. To give a broader view about replication, the journal explores some of the issues associated with the replication of results in different scientific disciplines as for example primate cognition and behaviour research, computer sciences, biology and climate change studies.

Worth reading is the Editorial by J. Crocker and L. Cooper that is dealing with the fraud of Diederik Stapel and raises the question “what could be done to protect science and the public from fraud in the future?”. The answer of the authors, both psychologists, is:

“Greater transparency with data, including depositing data in repositories where they can be accessed by other scientists […], might have sped up detection of this fraud, and it would certainly make researchers more careful about the analyses that they publish.”

Read the rest of this entry »


New Paper asks: “Do journals journals enforce their data policies?”

Posted: April 16th, 2018 | Author: | Filed under: EDaWaX, journals | Tags: , , | Comments Off on New Paper asks: “Do journals journals enforce their data policies?”

I am happy to announce that our paper “Journals in Economic Sciences: Paying Lip Service to Reproducible Research?” finally has been released.

The article, published in the latest issue of the IASSIST Quarterly, regards the practice of 37 journals in economics and business administration when it comes to the enforcement of their data policies.

For this purpose, we checked 599 articles which have been published in two issues of these journals. We chose the issues 1/2013 and 1/2014.

First, we estimated the share of articles that fall under a data policy, because replication data is needed to verify the published results. Afterwards, we checked the journal data archives and supplemental information section of each of these articles for the availability of replication files.

For a reduced sub-sample of 245 data-based articles, we checked in depth whether the replication files we found are compliant with the requirements of the journal’s respective data policy. Thereby, we are able to determine how much journals in economic sciences enforce their data policies and to calculate the ‘compliance rate’ for each journal in our sample. Read the rest of this entry »