After doing desktop research of more than 150 national and international economic scholarly journals regarding the implementation of data policies, we’re just starting phase 2 of our analysis for work package 2 of the project.

With the two questionnaires we would like to examine whether in the journal editor’s opinion a data availability policy is useful and how it should be designed. Answering the questionnaire takes between 5 and 10 minutes – depending on the fact whether a journal already owns a data policy or not.

As our readers might know, we are presently evaluating these data policies – a report summing up our preliminary results will be published in the next weeks.

Furthermore, we are interested in the technical infrastructure in use as well as in models for providing research data. Legal and technical aspects also are of interest.

Economic scholarly journals handle their guidelines regarding the provision of data sets and other materials quite differently. We would like to acquire knowledge about the practical implementation of such policies in these journals as well as about the experiences the editors made with their policies.

Regarding the conception of a publication related data archive (and that is a major aim of our project) as well as for the analysis of data policies, the opinion of the editors is highly important to us.

Of course, responses are aggregated for analysis and made anonymous. Neither the IP address of the editors nor any personal data will be imparted to or evaluated by us. Editors interested in the outcome of the survey might provide us with their e-mail address. After completing the survey a summary of the results are send to them.

picture: Thorben Wengert / pixelio.de

Beside a new look the catalogue has been overhauled to focus on software and services that directly perform curation and management tasks.
The catalogue splits these resources into five major categories, based on who the intended users are (e.g. researchers or curators) and what stage of the data lifecycle they will be most useful in (deposit and ingest / archiving and preserving / managing and administering / managing active research data / sharing output and tracking input).

Sub-categories contain tables for quick comparison of tools against others that perform similar functions, linked to in-depth descriptions of how the resource can help.

…in my opinion a very helpful collection of tools.

picture: Gerd Altmann / pixelio.de

Monash University is active in RDM already for many years – milestones since 2006 include:

•  the establishment of governance structures that provide leadership and coordination

• development and approval of a research data management policy and associated procedures
• the creation of ongoing and project-based advisory and technical roles dedicated to research data management, and inclusion of data management responsibilities in a wide range of other roles (e.g. subject librarians)
• active programs of professional development that have increased the knowledge and skills of researchers (including higher degree by research students) and professional staff
• establishment of a multi-petabyte research data store, available to all Monash researchers
• deployment and development of a range of discipline-specific and versatile solutions for the management of research data and associated metadata and documentation.

The Research Data Management Strategy outlines a number of goals and initiatives, which will be prioritised and scoped in consultation with internal and external stakeholders. Many smaller initiatives are already underway, while larger and more complex areas of work would require more detailed plans and costings.

Cathrine Harboe-Ree (University Library) emphasised the availability of the strategy to other organisations, which could be interested and use it as a basis for their own strategies.

“We are committed to improving research data management at our own institution, and also across the research sector as a whole. Most research is now undertaken collaboratively, and Australia is leading the world in terms of developing a national approach through the Australian National Data Service. We hope that the public release of the Monash University strategy under an open licence will prompt other organisations to develop holistic and sustainable institutional approaches to research data management. We also look forward to working with other organisations to address collaboratively many of the broader national and international issues such as research data skills development and emerging requirements for measuring the social and economic value of data re-use.”

 

Graphic by Gerd Altmann / www.pixelio.de

“If you’re a psychologist, the news has to make you a little nervous…”

The source of his fears is the Reproducibility Project  – a group of researchers that aim to replicate every study within the three journals Psychological Science, the Journal of Personality and Social Psychology and the Journal of Experimental Psychology: Learning, Memory, and Cognition published in the year 2008.

The project is part of Open Science Framework (OSF), a group that is interested in increasing the alignment between scientific values and scientific practices. Despite developing some tools and infrastructure projects its stated mission is to “estimate the reproducibility of published psychological science.”

For Bartlett the works of the Reproducibility Project  is a more polite way of saying “We want to see how much of what gets published turns out to be bunk” – and I think he’s right with his interpretation.

But in general Bartlett dosn’t find it useful, that OSF is replicating the published results: ” It’s time-consuming and doesn’t do much for your career to replicate other researchers’ findings. Journal editors aren’t exactly jazzed about publishing replications. And potentially undermining someone else’s research is not a good way to make friends.”

In my opinion these aren’t valid arguments at all. If science is a principle, researchers have to take care that scientific findings aren’t nonsense or even fabricated. What if fabricated or fraudulent results shape therapies and the way how humans are medicated? Would the replication of results then still be a waste of time? Not a good way to make friends?  I don’t think so – in the contrary I think it is a useful approach to replicate the results of a scientific disciplines in a few journals from time to time. And it would be very interesting to see how well the peer-review process in the journals performs.

In general it would be a step in the right direction: Scientists have to think about the worth of reproducible research and how they might support it. Incentives for the fabrication of results as well as for sloppiness are minimized and a whole profession gets reminded that the fundamental principles of science are more than just a annoying part in discussions.

In my opinion verifying the results of research is a fundamental goal of science and should not only be the mission of a single replication project. I think it would be a much better model if scientific organisations or professions are checking the results of published research themselves – because fraud, fabrication and bad documentation of research undermines research much more, than every replication group might do.

Readers interested in discussing this topic should also read the large comments section at the end of Bartletts article.

Photo: Benjamin-Thorn / pixelio.de

Commissioner Neelie Kroes, responsible for the digital agenda for Europe, has launched this consultation in July 2011 for seeking views on access to and preservation of digital scientific information – to be more precisely, the survey broached the issues of Open Access for scientific publications, accessibility of research data and digital long term preservation.

The purpose of the consultation was to gather information from as many sources as possible and receive important input for the future development of policy options in the area of scientific information in the digital age.

1.140 persons from 42 countries responded the online-survey. The biggest parts of the respondents were from Germany (422 answers), followed by people from France (129) and the UK (127). The majority of respondents were individual researchers (37.6 %). The remainder were citizens (27.5 %), university/ research institutes (8.4 %), libraries (7.3 %), publishers (6.4 %), international organisations (4.3 %), research funding organisations (1 %) and national, regional or local governments (0.8 %).

Regarding the most relevant field for us – access to research data – the report stated that

“…the vast majority of respondents (87 %) disagreed or disagreed strongly with the statement that there is no access problem for research data in Europe.”

The major barriers to access research data are:

• lack of funding to develop and maintain the necessary infrastructures (80 %),
• the insufficient credit given to researchers for making research data available (80 %),
• and insufficient national/regional strategies/policies (79 %).

Almost 90 % of responses had the opinion that research data that is publicly available and results from public funding have to be, as a matter of principle, available for reuse and free of charge on the Internet.

Already when starting the survey, Mrs. Kroes had a clear vision regarding accessibility and availability of publicly funded research:

“The results of publicly funded research should be circulated as widely as possible as a matter of principle. The broad dissemination of knowledge, within the European Research Area and beyond, is a key driver of progress in research and innovation, and thus for jobs and growth in Europe. Our vision is Open Access to scientific information so that all of us benefit as much as possible from investments in science. To accelerate scientific progress, but also for education, for innovation and for other creative re-use. For the same reason we must preserve scientific records for future generations”.

With this vision, Europe seems to be on a good way.

Many thanks to Heinz Pampel (blog post in German) for spreading the news!

Graphics: European Commission

More and more universities and research centres are starting to build research data repositories allowing permanent access to data sets in a trustworthy environment. Due to disciplinary requirements, the landscape of data repositories is very heterogeneous. Thus it is difficult for researchers, funding bodies, publishers and scholarly institutions to select appropriate repositories for storage and search of research data.

The goal of re3data.org is to create a global registry of research data repositories.

The registry will cover research data repositories from different academic disciplines. re3data.org will present repositories for the permanent storage and access of data sets to researchers, funding bodies, publishers and scholarly institutions. In the course of this mission re3data.org aims to promote a culture of sharing, a better understanding of data management as well as increased access and better visibility of research data. In the first phase of the project the following tasks are prioritized:

• the conception and construction of a web-based registry of research data repositories,
• the definition of selection criteria of research data repositories,
• the formulation of a metadata schema to describe research data repositories.

re3data.org cooperates with DataCite by recently signing a Memorandum of Understanding. With this Memorandum of Understanding DataCite and re3data.org define their efforts to enhance accessibility and visibility of data sets.

According to the schedule the registry will go live in autumn 2012.

re3data.org is funded by the German Research Foundation DFG from 2012 to 2014. re3data.org is a joint project of the Berlin School of Library and Information Science at the Humboldt-Universität zu Berlin, the German Research Centre for Geosciences (GFZ) and the Karlsruhe Institute of Technology (KIT). The partners are actively involved in the German Initiative for Network Information (DINI) and current research data management activities.

In this context I read some interesting news: As part of the commitment to reproducible research and transparency, BioMed Central has now partnered with LabArchives to work together for the shared goal of making datasets supporting peer-reviewed publications available and permanently linked to online publications – and available under terms which permit reuse freely, as Open Data.

A growing number of repositories for scientific datasets – which with persistent identification can be cited in and linked from published articles – are available but many fields still lack an obvious repository – as for example this is the case in Economics.

There has been debate about whether institutional or subject-specific repositories (and journals) are the best solution for data archiving and publishing. But what is clear is that, for data one size does not fit all – literally and metaphorically.

When available, data repositories are usually the best place for larger datasets which cannot be included as additional files. But some scientists who are willing to share data may be reluctant to deposit data in a repository with which they are not familiar, or which cannot guarantee permanence, or perhaps has suboptimal or ambiguous licensing terms.

LabArchives enables individual scientists to manage, share and publish data files, providing an accessible platform for sharing and publication which is controlled by authors themselves. LabArchives is web-based, or may also be installed on a local server, enabling a user to access their laboratory documents, protocols, notes and data from any location.

As part of this partnership, all BioMed Central authors are entitled to an enhanced free version of LabArchives. This ‘BioMed Central Edition’ of the software offers additional storage capacity compared to the standard free edition, integrated manuscript submission to BioMed Central journals, along with important open data publishing features.

graphic by  Gerd Altmann  / pixelio.de

Our abstract:  “Data Management in Scholarly Journals and Possible Roles for Libraries – Some Insights from EDaWaX” was among the top three papers that were submitted to the conference.

In our paper for LIBER we are pointing out the results of our study in work package 2:  We will evaluate how many journals are equipped with a data policy, how these policies are structured and what kind of materials (e.g. data; code; programs) authors need to submit beside their manuscripts.

Our paper will also show what criteria we chose to group the data policies regarding their suitability for replication purposes. Based on the results we present best practices and guidelines for data policies in the field of economics.

In the conclusion of our paper we will present our thoughts abouthte role of libraries in this process. The linkage between publications and their underlying research data is an interesting role, libraries could fulfil in the future. Replication is a cornerstone of the scientific method – and libraries might be well advised to support these processes in
cooperation with data centres, editors, publishers, and researchers.

We are really delighted winning this prize – it shows the importance of our work and strongly motivates us for the future challenges in the project.

The LIBER Award for Library Innovation was instituted by LIBER with support from Elsevier. It honours the three abstracts which best describe innovative work in the overall theme of the Annual Conference “Mobilising the knowledge economy for Europe”.

Specific criteria for the Award selection were:

• Level of innovation
• Impact on the wider library community
• Scalability of the idea
• Expected return-on-investment

Beside our contribution two other papers won the award:

Justine Haeberli, Daniel Beucke
Open Access Statistics – Services and Standards for Internationally Comparable Usage Statistics

Kathleen Shearer, Birgit Schmidt, Norbert Lossau
Licensing Revisited: Open Access clauses in practice

Congratulations to these colleagues, too! I’m looking forward to meeting you in Tartu!

photo: Robert Köhn / pixelio.de

As mentioned in one of my previous blog posts, part of our mission in WP2 is the analysis of data availability policies of scholarly journals in Economics. Therefore we analysed almost 200 journals – regarding whether they own such a policy or not. If we found a policy we wanted to know what the aims of this policy are, how the policy is structured and what kind of material authors have to submit to comply with the policy. Of course it was even very interesting so see whether a policy is mandatory or not – as well as to examine who carries the responsibility to achieve that data and additional material are provided as mentioned within the data policy.

For our sample who chose a mixed approach: On the one hand we used the sample, the German Economists Bräuninger, Haucap and Muck (paper is available German only, sorry) took for examining relevance and reputation of scholarly journals in Economics. This sample is primarily interesting because it compares international top journals and journals that are published in the German speaking area. Supplementary we added a few (four) more journals of which we explicitly knew that they own a data availability policy.

Not only because we’re funded by the German Research Foundation (DFG) this is almost a perfect sample – the sample furthermore offers the possibility to get in contact with the editors of economic scholarly journals publishing in Germany, Austria or Switzerland. An important approach, because the editors are very important for addressing changes in the way academic publishing and especially publishing and linking data and publications takes place.

So in total we had 154 journals in our primary sample. 13 of them were excluded for analysis, because these journals have a non empirical / non econometrical focus and were oriented on economic policies or theory only. Of course in some cases we were not really sure about a journal’s focus respectively whether we can be sure that this journal is not publishing any of the above mentioned articles. In these cases we did not exclude them from further analysis.

I have to say that the sample doesn’t typify Economics in general – it’s higher ranked than average journals in Economics and by adding a few journals explicitly having a data policy the percentage of Journals owning a data policy also is higher than we might expect on average.

Publishers
In our sample of the remaining 141 journals the publishers Elsevier, Wiley-Blackwell and Springer dominated. I think this represents more or less the typical segmentation of economics scholarly journals.

Journals published by professional associations or by university publishers comprise only a small amount within this sample. With 8 journals, Oxford University Press is the biggest of these publishers in our sample, followed by Cambridge University Press (3), University of Chicago Press (3) and the American Economic Association (3).

Subscription Journals and OA-Journals
Regarding the form of publication the overwhelming majority of journals we investigated were traditional subscription journals. Only one of 141 journals was an OA-journal.
With so few OA-journals in the analysis it wouldn’t be possible to state any well-founded results for these journals. Therefore we analysed another 54 OA-Journals in a special attempt. We did not merge this investigation with the other journals, because it wouldn’t be possible to check some of our hypothesis, if we had integrated these journals. So this is a special chapter and some kind of an excursus.

Inclusion of Journals in SSCI
32 out of 138 journals (three were excluded, because they are cross disciplinary journals) were not listed in the SSCI (Thompson Reuters Social Sciences Citation Index) what representats 23,2%. So the remaining 76,8% are listed within the SSCI. This high percentage shows that our sample is manifestly higher ranked than average journals in Economics. On average the journals achieved rank number 118 (median: 110) what also underlines the statement above.

Despite the fact that this journal is neither part of our research sample for EDaWaX’s work package 2 nor in the scientific field of Economics it is worth to investigate the motivation for banning the supplements.
And of course it is interesting to notice what the journal proposes to do instead of hosting and peer-reviewing supplementary material. In general the Journal of Neuroscience does not think that supplementary material is needless or unessential. These are good news.

The major motivation for removing supplementary material from their websites is comprehensible. John Maunsell, Editor-in-Chief, announces two reasons for changing this practice:

1) The amount of material associated with a typical article has grown dramatically. Maunsell has published a chart that shows the growth of data for articles as well as for supplementary materials in the past few years. While the size of articles has grown gradually over the past decade, the supplemental material associated with an article grew exponentially.
This fact is not the problem on its own as Maunsell stated:

“Making more information available is a good goal, and the financial costs of storing extra material electronically are small.”

But the problem starts at the point where a journal is not only peer reviewing the article but also the supplementary material – what is of course a necessary and useful attempt:

“…like most journals, [the Journal of Neuroscience] currently peer reviews supplemental material, the depth of that review is questionable. Most well qualified reviewers are overburdened with requests to review manuscripts, and many feel that it is too much to ask them to also evaluate supplemental material that can be as extensive as the article itself. It is obvious to editors that most reviewers put far less effort (often no effort) into examining supplemental material. Nevertheless, we certify the supplemental material as having passed peer review.”

That’s the fly in the ointment – and these are arguments are very coherent. But does this really justify the abandonment of all supplementary material? Is this the only possible consequence?! Let’s take a look at a second argument:

2) Another troubling problem associated with this context is that the extensive use of supplementary material in journals encourage reviewers to demand even more material and details. Reviewers increasingly insist that authors have to add further analysis, proofs, experiments etc. – even if these additions were subordinate or tangential. For the authors, the Editor-in-Chief stated, it is real work and sometimes unjustified burden. Additionally reviewer’s demands delay publication.

Hm. So far, so good…but…at first glance argument 1 and argument 2 doesn’t really seem to fit to each other: If reviewers have to much workload they demand authors to get even more of it?!

But maybe that’s only seemingly a contradiction because even if reviewers have a lot to do they are maybe wanting that the “scientific standard” of the journal is met. And even if they know that they will not able to check if these standards are really fulfilled, they might demand authors to submit even more material.

And the Journal’s contribution for solving these challenges?

“We believe that this is best accomplished by removing the supplemental material from the peer review process and requiring that each submission be evaluated and approved as a complete, self-contained scientific report. By allowing the authors to include a link to supplemental material on their own site, readers will continue to have access to any amount of additional material that the authors consider interesting, but with the clear warning that the material has not gone through peer review.”

In my opinion the journal chose an (too) easy way for “solving” this problem – they just got rid of it. Now, when authors are publishing an article he or she will be allowed to include a footnote with a URL that points to supplementary material on a site the authors support and maintain.

This is I think one of the worst offenders possible. Of course the journal might state now the sentence “the material is not peer-reviewed and the responsibility for these material is in the hands of the corresponding author”, but this doesn’t solve the challenges. Having the responsibility for supplementary material in the hands of the authors is definitely not a way to ensure that data and other supportive materials will be provided to other researchers or to the public nor is it a sustainable way to ensure that data and other material are usable, findable and that these data is preserved .

In contrast to the claim of Maunsell I think that authors will NOT maintain their sites “for as long as they consider their supplemental material to be valuable and important.”

In the scientific community publications are the currency for researchers – not supplementary material. There are almost no incentives for authors to do the extra job, if there is no gratification for it – or in contrast,  if there is no policy that regulates the publication of data and other supplementary material.

In my opinion it is a bad joke that the Journal of Neuroscience claims that “removing supplemental material from articles might motivate more scientific communities to create repositories for specific types of structured data, which are vastly superior to supplemental material as a mechanism for disseminating data.” In fact these repositories are strongly needed, but to state “we are banning supplements – and that will help you to implement repositories for your scientific field” is almost cynical and changes nothing for the better. I think it would have been a more useful and clear-sighted way to discuss the problem with stakeholders within the research community first and to ensure that a repository or another solution is build up second, before banning supplementary material.

I want to conclude with one last quote of Maunsell’s Editorial: “We should remember that neuroscience thrived for generations without any online supplemental material.” …wow – that’s an oath of disclosure! …the world rotates as it always did, right? Internet? Online Access? Digital science? Forget it! Paper and Pencil – that thrived for generations, too!

photo: Niko Korte  / pixelio.de