Open Access to Research Data: Strategic Delay and the Ambiguous Welfare Effects of Mandatory Data Disclosure

Patrick Andreoli-Versbach and Frank Mueller-Langer (two economists from the Max Planck Institute for Innovation and Competition in Munich) have published a paper on strategic delay and the ambiguous welfare effects of mandatory data disclosure (2014).

By setting up a model describing the incentives of a researcher to share self-generated data with the research community in the context of a published article, they found that these incentives can be distorted by a policy they call “First Paper Policy” (requires authors to share their data immediately after the first publication): Read the rest of this entry »

Biomedical Sciences: Journals unite to forster reproducible research

When it comes to the topic of replication, it always is a good idea to consult the webpages of the nature journal. Yesterday, for instance, the journal reported that a group of editors representing more than 30 major journals, representatives from funding agencies as well as scientific leaders discussed principles and guidelines for preclinical biomedical research in June 2014.

The gathering was convened by the US National Institutes of Health, Nature and Science.

The attendees agreed on a common set of principles and guidelines in reporting preclinical research that list proposed journal policies and author reporting requirements in order to promote transparency and reproducibility. Read the rest of this entry »

Related Literature

[Related Literature]

Luckily, around the globe, there are many activities to link data and publications or to foster replication of scientific research. Below we list some papers in English and a few in German that are dealing with the topic of reproducible research in economics and beyond, data policies in journals and other aspects. We list only a few papers published on these topics.

in English:

• Alsheikh-Ali, AA / Qureshi, W / Al-Mallah, MH & Ioannidis, JPA (2011). Public Availability of Published Research Data in High-Impact Journals. In: PLoS ONE 6(9) 2011: e24357. doi:10.1371/journal.pone.0024357 [Link]
• Anderson, Richard / Greene, William H. / McCullough, B. D. & Vinod, H. D. (2008). The Role of Data/Code Archives in the Future of Economic Research. Journal of Economic Methodology, 15(1), 99-119 [Download]
• Anderson, Richard & Areerat Kichkha (2017). Replication, Meta-analysis, and Research Synthesis in Economics. American Economic Review, 107 (5): 56-59. doi:10.1257/aer.p20171033
• Bryan, Nelson (2009). Empty Archives. Nature, 461, 160-163, doi:10.1038/461160a [Download]
• Chang, Andrew C. & Li, Phillip (2015). Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ”Usually Not”, Finance and Economics Discussion
  Series 2015-083. Washington: Board of Governors of the Federal Reserve System. http://dx.doi.org/10.17016/FEDS.2015.083 [Download]
• Dewald, William G. / Thursby, Jerry G.  & Anderson, Richard G. (1986). Replication in Empirical Economics: The Journal of Money, Credit and Banking Project. In: The American Economic Review, 76(4), 587-603 [Download]
• Duvendack, M. / Palmer-Jones, R. W. & Reed, W. R. (2015). Replications in Economics: A Progress Report. Econ Journal Watch, 12(2), 164–191. [Download]
• Duvendack, M. / Palmer-Jones, R. W. & Reed, W. R. (2016). What is Meant by ‘Replication’ and Why Does It Encounter Resistance in Economics? [Download]
• Fecher, B. /Fräßdorf M. & Wagner G.G (2016). Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful. DIW Discussion Paper No. 1572  [Download]
• German Data Forum (2011). Building on Progress. Expanding the Research Infrastructure for the Social, Economic and Behavioral Sciences, Opladen & Farmington Hills [Table of Contents | Volume I | Volume II ]
• Gherghina, S. & Katsanidou A. (2013). Data Availability in Political Science Journals, European Political Science 12, 333–349; doi: 10.1057/eps.2013.8
• Glandon, Philip (2010). Report on the American Economic Review Data Availability – Compliance Project, Vanderbilt University [Download]
• Hamermesh, Daniel S. (2007). Viewpoint: Replication in economics. Canadian Journal of Economics  / Revue canadienne d’Economique, 40(3), 715-732 [Download]
• Höffler, Jan H. (2017). Replication and Economics Journal Policies, American Economic Review, 107(5), 52-55. doi: 10.1257/aer.p20171032
• Huschka, Denis (2013). Why should we share our data, how can it be organized, and what are the challenges ahead?, RatSWD Working Paper Series, No. 216  [Download]
• Huschka, Denis & Wagner, Gert G. (2012). Data accessibility is not sufficient for making replication studies a matter of course,  RatSWD Working Paper Series No. 194  [Download]
• Linek, S. B., Fecher, B., Friesike, S. & Hebing, M. (2017). Data sharing as social dilemma: influence of the researcher’s personality. PLoS ONE 12(8): e0183216. https://doi.org/10.1371/journal.pone.0183216
• McCullough, B.D. / McGeary, Kerry Anne & Harrison, Teresa D. (2008). Do economics journal archives promote replicable research? Canadian Journal of Economics, 41(4), 1406-1420 [Download]
• McCullough, B.D. / McGeary, Kerry Anne & Harrison, Teresa D. (2006). Lessons from the JMCB Archive. Journal of Money, Credit, and Banking, 38(4), 1093-1107 [Download]
• McCullough, B.D. (2007). Got Replicability? The Journal of Money, Credit and Banking Archive. Econ Journal Watch, 4(3), 326-337 [Download]
• McCullough, B.D. & Vinod, H.D. (2003). Verifying the Solution from a Nonlinear Solver: A Case Study. American Economic Review, 93(3), 873-892 [Download]
• McCullough, B.D. (2009). Open Access Economics Journals and the Market for Reproducible Economic Research. Economic Analysis and Policy, 39(1),117-126 [Download]
• Pinfield, S. / Cox, AM. & Smith, J. (2014). Research Data Management and Libraries: Relationships, Activities, Drivers and Influences. PLoS ONE, 9(12): e114734. doi:10.1371/journal.pone.0114734
• Reed, W. B. (2017). Replication in Labor Economics. Working Paper No. 6/2017, Department of Economics and Finance School of Business, University of Canterbury, New Zealand [Download]
• Sturges, P./ Bamkin, M. / Anders, J./ Hubbard, B. / Hussain, A. & Heeley M. (2014). Research Data Sharing: Developing a Stakeholder-Driven Model for Journal Policies [Download]
• Stodden, Victoria / Hurlin, Christophe & Prignon, Christophe (2012). RunMyCode.Org: A Novel Dissemination and Collaboration Platform for Executing Published Computational Results. Working Paper [Download]
• Vines, Timothy H. et al. (2013). Mandated data archiving greatly improves access to research data. The FASEB Journal – Journal article fj.12-218164, Published online January 8, 2013. [Article]

in German:

• Altenhöner R. / Oellers, C. (2013). Langzeitarchivierung von Forschungsdaten. Standards und disziplinspezifische Lösungen [Download]
• Altenhöner, Reinhard / Wagner, Gert G. (2013). Herausforderungen der Archivierung sozial-, verhaltens- und wirtschaftswissenschaftlicher Datenbestände, RatSWD Working Paper Series No. 213, [Download]
• Jensen, Uwe (Ed.): GESIS-Technical Reports 2012|07. Leitlinien zum Management von Forschungsdaten. Sozialwissenschaftliche Umfragedaten. [Download]
• Siegert, Olaf / Toepfer, Ralf: Von der Sammlung gedruckter Statistiken zum Datenrepository. Wie sich das Forschungsdatenmanagement in der ZBW verändert und welche Rolle die Wissenschaft dabei spielt, RatSWD Working Paper Series, No. 172 (2011) [Download]
• Wagner, Gert G. / Huschka, D: Datenverfügbarkeit reicht nicht, um Replikationsstudien zur Routine zu machen, RatSWD Working Paper Series, No. 194 (2012) [Download]

Image: “and read all over” by Jonathan Cohen on flickr.com. License: CC-BY-NC 2.0

About the 1st project phase

[About]

A brief description of the EDaWaX-Project:

Empirical studies to verify and refine theoretical models are increasingly important in economics. This is reflected in a rising number of empirical contributions to journals in which authors have collected their own research data or used external datasets. However, so far there have been few effective means to replicate the results of economic research within the framework of the corresponding article, to verify the results, to make the research data that was used in a paper available for repurposing or to use it to support the scholarly debate.

The DFG-funded project EDaWaX is intended to meet these challenges. One of the main objectives is the development of a publication-related data archive for journals using an integral approach.

During the first phase of the project, a subject-specific analysis will systematically redact the current challenges as manifested mainly by the lack of incentive structures for the replication of research results and the multiple use of data. At the same time, currently existing technical and organisational solutions within the context of economics journals and data archives will be analysed and investigated with regard to their suitability for the requirements of the EDaWaX project and with particular attention to the legal framework.

Based on the results of this analysis, a metadata schema for the description and labelling of the data will be developed during the second phase of the project. A set of rules for the schema-compliant transformation of the descriptions and data will also be established. The knowledge gained will lead to the third phase where a pilot project for innovative publication-related data archives will be developed using the renowned journal “Schmollers Jahrbuch / Journal of Applied Social Science” as an exemplar.

EDaWaX is a joint project of RatSWD (German Data Forum), INNO-tec Institute at LMU Munich and  ZBW. RatSWD acts as editor of Schmollers Jahrbuch and coordinates the work package “legal framework” while the INNO-tec Institute in cooperation with IMPRS-CI investigates the incentive structures of so-called data sharing. ZBW undertakes the remaining analytical work packages, designs and develops the publication-related data archive and acts as project coordinator.

Project Output: Funding Phase I

On this page you will find all our journal articles, working papers, presentations as well as our poster and some miscellaneous content.

Though we are moving on to project phase 2, we assume that several parts of project phase 1 are still of interest to a broader audience.

The project output of EDaWaX’s second funding phase is grouped into the different sorts of publications, presentations and other outouts.

Journal Articles and Conference Proceedings:

in English:

• Paper: Andreoli-Versbach, Patrick /Mueller-Langer, Frank: “Open access to data: An ideal professed but not practised”.  Research Policy, Vol. 43, Issue 7, 2014, http://dx.doi.org/10.1016/j.respol.2014.04.008.
• Paper: “Data Management in Scholarly Journals and possible Roles for Libraries – Some Insights from EDaWaX.”  LIBER Quarterly, Vol.23, Issue 1, 2013 [html | pdf]
• Paper: “On the role of research data centres in the management of publication-related research data.” LIBER Quarterly Vol.23, Issue 4, 2014. [html | pdf]

in German:

• Paper: “Welche Rolle spielen Forschungsdaten eigentlich für Fachzeitschriften? Eine Analyse mit Fokus auf die Wirtschaftswissenschaften.” In: Mittermaier, Bernhard (Ed.): Vernetztes Wissen – Daten, Menschen, Systeme. 6. Konferenz der Zentralbibliothek, Forschungszentrum Jülich, 5. – 7. November 2012, Proceedingsband, Schriften des Forschungszentrums Jülich, Reihe Bibliothek / Library Band / Volume 21, pp. 29 – 44.
• Paper: Vlaeminck, S. / Wagner, G.G / Wagner, J. / Harhoff D. / Siegert, O.: “Replizierbare Forschung in den Wirtschaftswissenschaften erhöhen – eine Herausforderung für wissenschaftliche Infrastrukturdienstleister.” LIBREAS. Library Ideas, 23 (2013). (October 2013)

Working Paper:

in English:

• Paper: Andreoli-Versbach, Patrick /Mueller-Langer, Frank: Open Access to Data: An Ideal Professed but Not Practised (February 21, 2013). RatSWD Working Paper Series No. 215.
• Paper: ”On the role of research data centres in the management of publication-related research data.” RatSWD Working Paper Series (226)

in German:

• Summary: Max Planck Institute for Innovation and Competition: Information infrastructure for research data: The project European Data Watch Extended. [main parts in German]
• Paper: “Welche Rolle spielen Forschungsdaten eigentlich für Fachzeitschriften? Eine Analyse mit Fokus auf die Wirtschaftswissenschaften.” RatSWD Working Paper Series, No. 210. (Updated version)
• Paper: “Zur Rolle von Forschungsdatenzentren beim Management von publikationsbezogenen Forschungsdaten. Ergebnisse einer Befragung von wissenschaftlichen Infrastrukturdienstleistern.” RatSWD Working Paper Series (225).

Presentations:

in English:

• Presentation: “Data Management in Scholarly Journals and possible Roles for Libraries – Some Insights from EDaWaX.” Talk given at the 41st Annual Conference of LIBER | June 27, 2012 | Tartu | Estonia.
• Presentation: “Facilitating Replications of Research Results in Economics.” Talk given at the 1. Open Economics International Workshop | December 16th 2012 | Cambridge | UK.
• Presentation: “Research Data Management in Economics Journals – Data Policies and Data Description as prerequisites of reproducible Research.” Presentation given at the IASSIST-Conference | May 31, 2013 | Cologne | Germany.

in German:

• Presentation: “„Linking Data and Publications“ – Anforderungen und Vorgehensweise am Beispiel des Aufbaus eines publikationsbezogenen Datenarchivs.” | Presentation given at a plenary meeting of the Standing Committee Research Data Infrastructure (FDI) | March 2013 | Berlin.
• Presentation: “Wirtschaftswissenschaftliche Forschung replizierbar machen“. Project presentation and first findings. Talk given at the 101st German Bibliothekartag | June 2012 | Hamburg | Germany.
• Presentation: “Anforderungen für den Aufbau eines publikationsbezogenen Datenarchivs in den Wirtschaftswissenschaften”. Talk given at the 6th Open Access Day | September 26, 2012 | Vienna | Austria.
• Presentation: “Welche Rolle spielen Forschungsdaten eigentlich für Fachzeitschriften? Eine Analyse mit Fokus auf die Wirtschaftswissenschaften.” Talk given at the 6th WissKom Conference | November 5, 2012 | Jülich | Germany.
• Presentation: “Forschungsdaten und Forschungsdateninfrastrukturen in den Wirtschaftswissenschaften” | Talk given at the 7th Open Access Days | October 2013 | Hamburg | Germany.
• Presentation: “Forschungsdatenmanagement bei wirtschaftswissenschaftlichen Zeitschriften – Ergebnisse des Projekts EDaWaX “ | DFG-Workshop „Vernetzung – Nachnutzung – Austausch“ | December 16/17 2013 | Bonn.
• Presentation: “Wirtschaftswissenschaftliche Ergebnisse replizierbarer machen. Das Projekt EDaWaX und seine Ergebnisse.” Talk given at the PubMan Days 2013 | October 21, 2013 | Munich | Germany.
• Presentation: “Forschungsdatenmanagement bei wirtschaftswissenschaftlichen Zeitschriften – Einblicke aus dem DFG-Projekt EDaWaX .” Talk given at the 6. Conference for Social- and Economic Data (KSWD)| February 21, 2014 | Berlin | Germany.

Poster:

• Poster for a Knowledge Exchange (KE) Workshop on Research Data | November 2011 | Bonn | Germany.

Miscellaneous:

• Dataset(s) of our analyses in work package 2 “Research Data Management in Economics Journals.”
• Full text of all data policies we found in our sample of 141 economics journals.
• Results of the EDaWaX online survey on hosting options for publication-related research data | German version | English version
• Questionnaires sent to editors of economics journals to examine whether in a journal editor’s opinion a data availability policy is useful. We were also interested in the experiences the editorial offices made with such policies.
  

Image: “Trinity belt pack in/output” by Larry Jacobsen / ljguitar on flickr.com. License: CC-BY 2.0
All icons (except the last one) made by Freepik from www.flaticon.com  License:Flaticon Basic License.
Icon (‘miscellaneous’) last shown:”Folder couple one black other white” by SimpleIcon from www.flaticon.com. License: CC-BY 3.0

OJS-Dataverse Integration Plugin released!

In one of my previous blog posts I introduced the PKP/IQSS OJS-Dataverse integration project. After a really short developmental period the project now is happy to announce that the first version has been released! Congrats!

The plugin has been developed by PKP (Public Knowledge Project) in collaboration with Harvard’s Institute for Quantitative Social Science (IQSS).  Funded by a $1 million Alfred P. Sloan Foundation grant, the OJS-DVN project developed the plugin for journals that are using the Open Journal System (OJS). a journal management and publishing system .

Read the rest of this entry »

Adapting CKAN for Open Research Data

One aim of EDaWaX is to develop and implement a web-platform prototype for a publication-related research data archive. We’ve chosen CKAN –an open source data portal platform– as basis for this prototype.

This post describes the reasons for this decision and tries to give some insights in CKAN, its features and technology. We’ll also discuss these features both in regard to our special use case and to the suitability for research data management in general.

Read the rest of this entry »

Endorse the Open Economics Principles!

As I mentioned in one of my previous posts, the Open Economics Working Group has just finalised the Open Economic Principles. Now it is also possible to endorse these principles online.  In an e-mail the working group announces:

The Open Economics Working Group would like to introduce the Open Economics Principles, a Statement on Openness of Economic Data and Code. A year and a half ago the Open Economics project began with a mission of becoming central point of reference and support for those interested in open economic data. In the process of identifying examples and ongoing barriers for opening up data and code for the economics profession, we saw the need to present a statement on the guiding principles of transparency and accountability in economics that would enable replication and scholarly debate as well as access to knowledge as a public good. Read the rest of this entry »

Metadata for publication-related Data Archives: As much as necessary and as little as posssible

In the course of our project we had to deal with a work package (WP5) in which we had to develop (or at least to choose) a metadata schema capable of describing and labelling publication-related metadata. Today I would like to summarize our approach and some of our thoughts for choosing an appropriate metadata schema.

I already characterised our approach in my talk at the IASSIST 2013 conference, but I would like to describe it more in-depth to start a discussion with researchers and the community, so feel free to comment.

In a first move we evaluated the data and further materials we found in the data archives of economics journals. These types of data we found within the journals’ data archives were seen as functional requirements, because the metadata schema should be capable to describe these data.

We focussed on a schema that should be capable to describe quantitative data – mainly because qualitative approaches are rare in Economics. Read the rest of this entry »

Introducing the OJS-Dataverse Integration Project

Today I want to introduce the PKP- Open Journal System / Dataverse Integration Project to our readers, an approach that is in some parts quite similar to our own approach in the field of economics.

Funded by a $1 million Alfred P. Sloan Foundation grant, the OJS-DVN project intends to develop a plugin for journals that are using the Open Journal System (OJS), a journal management and publishing system that has been developed by the Public Knowledge Project (PKP), to expand and improve access to research. Read the rest of this entry »

EDaWaX: New Article published

Last year our project won the LIBER-Award for Library Innovation 2012. Now our paper was published in LIBER-Quarterly, the peer-reviewed journal of the Association of European Research Libaries.

Photo: “Tippen” by Jorma Bork / pixelio.de. License:>click<

New Nature Special: Challenges in irreproducible Research

nature has published a new special issue on challenges in irreproducible research. The journal addresses the challenges and barriers of reproducibel research:

No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process. In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies. Nature has published a series of articles about the worrying extent to which research results have been found wanting in this respect. The editors of Nature and the Nature life sciences research journals have also taken substantive steps to put our own houses in order, in improving the transparency and robustness of what we publish. Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility. We hope that the articles contained in this collection will help.

All articles within this issue are available free of charge. The table of contents is available here.

Graphic: pasukaru76, www.flickr.com

Data Sharing: Poor Status Quo in Economics #Update

In the context of our research project EDaWaX a new research paper has been published by Patrick Andreoli-Versbach (International Max Planck Research School for Competition and Innovation (IMPRS-CI), LMU Munich, Munich Center for Innovation and Entrepreneurship Research (MCIER)) and Frank Mueller-Langer (Max Planck Institute for Intellectual Property and Competition Law, IMPRS-CI, MCIER).

The paper analyzes the data sharing behavior of 488 randomly chosen empirical economists. More specifically, the researchers under study were chosen uniformly across the top 100 economics departments and the top 50 business schools and randomly within the respective institution. Economics departments were chosen using the Shanghai Ranking 2011 in Economics and Business and business schools were chosen using the Financial Times Global MBA Ranking 2011.

Read the rest of this entry »

Research Data Management in Economic Journals (Part I)

This article is cross-posted from the blog of the Open Economics Working Group

Background

In Economics, as in many other research disciplines, there is a continuous increase in the number of papers where authors have collected their own research data or used external datasets. However, so far there have been few effective means of replicating the results of economic research within the framework of the corresponding article, of verifying them and making them available for repurposing or using in the support of the scholarly debate.

In the light of these findings B.D. McCullough pointed out: “Results published in economic journals are accepted at face value and rarely subjected to the independent verification that is the cornerstone of the scientific method. Most results published in economics journals cannot be subjected to verification, even in principle, because authors typically are not required to make their data and code available for verification.” (McCullough/McGeary/Harrison: “Lessons from the JMCB Archive”, 2006)

Harvard Professor Gary King also asked: “[I]f the empirical basis for an article or book cannot be reproduced, of what use to the discipline are its conclusions? What purpose does an article like this serve?” (King: “Replication, Replication” 1995). Therefore, the management of research data should be considered an important aspect of the economic profession.

Read the rest of this entry »

EDaWaX: First paper published! #Update

I am happy to announce that some of the results we obtained in our analyses have been published in the proceedings of the WissKom2012 conference (pp.29 ff).

In the research paper for Wisskom2012 Olaf Siegert and I summarized some of the findings of the EDaWaX project.  Unfortunately the paper is available in German only, but I promise to publish some more of our major findings in English – eighter on this blog or in a seperate paper. Some results of our analyses are already published here: ( |1 | 2 | 3 | 4 | )

The presentation I gave in Jülich is available in the download section of the blog.

I also wrote a paper in English that has already been accepted by a peer-reviewed journal. I hope to find time to rework the article and to integrate the very helpful notes and comments of the reviewers anytime soon.

I’ll come back to you, when I’ve got news.

# Update 2012-11-14 #

A slightly updated version of the paper is also available now. It has been published in Working Paper Series of the German Data Forum (RatSWD), No. 210. The RatSWD Working Papers series was launched at the end of 2007. Since 2009, the series has been publishing exclusively conceptual and historical works dealing with the organization of the German statistical infrastructure and research infrastructure in the social, behavioral, and economic sciences.