Scientific journals need to make access to research data a prerequisite for reviewing

Data sets on which our publications are based need to be made available, for obvious reasons of verifiability, replicability, and cumulative progress in research. This point is made eloquently in many publications. For instance:

“Openness is one of the central values of science. Open scientific practices such as sharing data, materials and analysis scripts alongside published articles have many benefits, including easier replication and extension studies, increased availability of data for theory-building and meta-analysis, and increased possibility of review and collaboration even after a paper has been published. Although modern information technology makes sharing easier than ever before, uptake of open practices has been slow.” 

(Morey, Richard D., Christopher D. Chambers, Peter J. Etchells, Christine R. Harris, Rink Hoekstra, Daniël Lakens, Stephan Lewandowsky, et al. 2016. The Peer Reviewers’ Openness Initiative: incentivizing open research practices through peer reviewRoyal Society Open Science 3(1).)

Since I first wrote down my fundamental beliefs on this topic back in 2002, the field (linguistics, and specifically, phonetics) has not changed as it should have: disposable data are still the norm. This post takes the example of three small data sets that I deposited this year (data associated to publications from the early 2000s) to illustrate the current situation of lack of support for the conservation of research data. The conclusion is that changes to the mainstream model of publication and evaluation should be made now: scientific journals need to make access to research data a prerequisite for publication. And for these changes to happen, we need to take action. 

1. Data from my first phonetics paper

My first phonetics paper was published in 2004:  “Final consonants and glottalization: new perspectives from Hanoi Vietnamese”. Phonetica 61(2–3). 119–146. It is based on audio and electroglottographic recordings. At the time, neither the reviewers nor the editor asked for access to the research data. Various pieces of software carry out citation counts to estimate ‘impact’ (this paper was cited 1 time in PubMed, 33 times in Web of Science, 120 times in Google Scholar, etc.) and analyze ‘traffic’ (how many times the PDF was downloaded, when, by whom, etc.). On the other hand, even though the corresponding data (multimedia recordings) have been online for more than a decade, no tool that I know of ever looked for connections between the study and the data (and software) on which it is based.

The recordings were first made available online in 2005/2006 through a “Databases” project at Université Paris 3 – Sorbonne Nouvelle that aimed to make linguists’ data available online in good order, as multimedia files accompanied by metadata and logically structured transcriptions and annotations. The metadata (here) are sufficiently clear that a web crawler could identify the language of the recordings as Vietnamese: the language is encoded in the metadata as x-sil-VIE, and while this code is not very widely used, a key to the language codes used in the database is available (here), clarifying that this is a code from the Summer Institute of Linguistics’ Ethnologue database and that it refers to Vietnamese. The deposit is also associated with a PDF description (here) that is not hard to ‘mine’ for information, not to mention that high-tech tools could conduct automatic language recognition from the audio files. 
Software built by Google Scholar engineers is pretty good at identifying references despite the bewildering diversity of citation formats. Similar results could be attained in locating data sets: allowing for multifaceted search across databases, tracking data set usage, and so on. Linking data, publications and software into an Open Science environment is not a technical impossibility: the reason why it is not happening is that Open Science is clearly not a priority of the big players (the commercial publishers and service providers). The assessment of research output (for finding a job in a research institution and keeping it) is based on publications, with little if any recognition of the time spent on the empirical basis of science. And paradoxical as it may seem, the evaluation of a submitted manuscript in phonetics/phonology is often conducted without even a sample of the data on which the results, discussion and conclusions are based.

This year, the data set of my 2004 study was made available in the Pangloss Collection, with time-aligned transcripts and translations. These data not only constitute an indispensible complement to the research article: they also have potential for an open-ended list of uses. (The materials are currently at bottom of list here.) But for want of institutional recognition and support, such efforts for making data available along with publications remain an uphill journey. 

2. Sample data from a 2003 book chapter

In 2003, I contributed a chapter to a book about focus marking:

“Focus et tons lexicaux: de la variété des paramètres prosodiques”. In A. Lacheret-Dujour & J. François (eds.), Fonction et moyens d’expression de la focalisation à travers les langues (Mémoires de La Société de Linguistique de Paris, nouvelle série, tome XIII), 13–47. Louvain: Peeters.

This was my first contribution to a book and I was keen to make a good job of it. In view of the relevance of speech prosody to focus marking, I suggested to the editors to have multimedia materials published alongside the printed volume. I am delighted to report that they readily agreed to this, and not a few of the contributors to the volume chose to make audio and video recordings available on a CD-ROM. The printed volume, with accompanying CD-ROM, is available from a dozen of university libraries in the (French) network of academic libraries, SUDOC. The catalogue entry mentions the CD-ROM, but cannot go into any detail on its contents, which limits possibilities for locating these data samples.

When I deposited the preprint version of my chapter in a preprint archive about four years later, I forgot to upload the supplementary files. I have now done so, and the audio files (WAV files) are now available as supplementary files associated to the publication (here), also with a PDF providing explanations about these data. I feel good about doing so, but it is, again, a lonely journey. I frequently get automated e-mail asking me to confirm authorship of a paper, to review my list of publications in a database (there are quite a number of such databases), but I never got e-mail asking for this kind of assistance (‘academic crowd-sourcing’) in relation to data or software. This is because, as recapitulated above, the importance of primary data is oddly underestimated in the current editorial environment (‘publication ecosystem’?) in linguistics.

3. Sample data from a 2003 conference presentation

In the same year that I wrote my first book chapter (in French), I wrote a paper (in English) for a conference about speech prosody. ‘Deaf and dumb’ papers about prosody sound like a strange paradox, so I asked the organizers whether they would find it useful to have multimedia files accompanying the research papers. Like the editors of the 2003 volume published by the Société de Linguistique de Paris, the editors of the “Prosodic Interfaces” conference (Amina Mettouchi and Gaëlle Ferré) found the idea relevant and forward-looking. Participants were encouraged to submit audio files for inclusion on the CD-ROM containing the conference proceedings. These files are still accessible (as of 2018), here. This year, I added the files associated to my conference paper to the deposit in the preprint archive, here. When doing so, I felt proud about these excerpts, which (in my perfectly objective opinion as data collector and author of the paper!) add a lot to the publication. Cited examples can be listened to, and I also included audio recordings of explanations provided in Naxi (the target language) by language consultants expressing their intuitions about subtle stylistic nuances associated with two variants of an utterance (one with a full form and the other with a reduced form). This allowed a colleague who is a native speaker of Naxi to spot a beautiful example of syllable reduction (which I had not noticed) inside the very sentence in which the consultant described processes of syllable reduction. This lovely case of mise en abyme constitutes a further piece of evidence that syllable reduction processes are pervasive in the Naxi language.

But contrary to what I expected, such initiatives failed to gain much traction among linguists. Despite possibilities for embedding sound files of cited examples into PDFs (as exemplified in this paper), it has not yet become common practice to offer at least a sample of data alongside scientific papers. 

Conclusion

“An ounce of practice is worth more than tons of preaching”, as the saying goes: looking it up, I realize that it’s from none less than Mahatma Gandhi. But it’s now over 20 years since I made documentation and conservation of spoken language data one of my lifelong priorities, and judging from my experience, making small steps on one’s own will not get us where we want (any more than sporadic, individual attempts at consuming less resources and creating less pollution can avert environmental disaster). To advance towards open science, we do not only need to get the principles right and put them to good practice. We also need to do advocacy and change the rules.

The field needs to take a turn towards Open Science, and this is not going to happen without us investing some of our time into it: taking a strong stand, and voicing concerns that are key to our discipline’s future, every time that we have opportunities to speak out as authors, as members of editorial teams, as evaluators, and so on. If scientific journals make access to research data a prerequisite for reviewing (as they should really have started to do two decades ago), also insisting on open access to the data (with an embargo of several years or even decades where necessary), practices will change, and the Open Data Excuse Bingo will be a thing of the past.

Open Data Excuse Bingo. (Click here for source)

Alexis Michaud

I'm a linguist, focusing on un(der-)documented languages, in particular Na (Mosuo), a Tibeto-Burman language.

More Posts - Website