The IRIS Digital Repository recently announced an award for replication studies. We caught up with members of the IRIS team, Sophie Thompson, Luke Plonsky, and Emma Marsden, to talk about the repository, their community, and why they value replications.
COS: Can you tell us a bit about IRIS: What is its purpose, who are its users, and what community does it serve?
IRIS is a repository of research materials and data for use in second language research. IRIS helps researchers to share materials, to increase the impact and exposure of their work, and to replicate and build on the work of their colleagues. IRIS aims to assist the field in overcoming the lack of replication studies on language learning and teaching research. The resource is completely free and currently contains more than 1,400 materials. IRIS is used by teachers, researchers and students to help investigate a wide range of topics and theoretical domains and is methodologically very inclusive.
COS: What impact is IRIS having in the community that is serves?
Since its launch IRIS has had 18,278 downloads and 34,308 hits. We also have a Facebook page which is used to share new material uploads, IRIS news, and news about open science in general. Nearly all top journals in the field also encourage their authors to share their materials on IRIS to increase their impact and availability. Materials from 161 different journals are held on IRIS from over 1,200 authors. These numbers show that IRIS has a big impact on its target community of second language researchers, students and teachers.
COS: What do you see as a vision for the future of IRIS? How would you like to see it evolve in the future and how will researchers interact with it?
Over the last year IRIS has developed special collections holding all available examples of a specific type of research instrument. Currently the collections held are self-paced reading instruments and grammaticality/acceptability judgement tests. We plan to develop more special collections and would encourage researchers to consider holding special collections on IRIS.
IRIS has also recently started to hold data. Whilst materials sharing is now more widespread in the field, data sharing is not currently common practice. We aim to change this, to increase transparency and to facilitate meta-analyses as well as re-analysis and other forms of secondary analysis. IRIS is also the only repository in our area that qualifies for journals to award COS badges to researchers. We will continue to encourage journals to ask their authors to share through IRIS and further the open science message with the help of COS.
Philp & Iwashita 2013 https://www.iris-database.org/iris/app/home/detail?id=york%3A854602&ref=search
COS: You recently announced the IRIS Replication Award. Can you briefly tell us about that and what you will be awarding and how a researcher would win one of these awards?
The IRIS Replication Award aims to encourage and reward researchers carrying out rigorous replication research in the field. We will award £300 annually to a published (or accepted) self-labelled replication using materials held on IRIS. The first round is now open and will close on 30th November 2018. Submissions will be welcome from any area of second language research. All methodological approaches and theoretical perspectives are welcome. To choose a winner we will be considering the strength of the justification for the replication, the soundness and transparency of methods and analysis, and how well the discussion and conclusions are supported by the methods, data and analysis. We will announce the first award by 1 January 2019.
COS: Why do you value replications and why give an award to those who publish them?
Replication increases the external validity and reliability of research, and helps test theories, constructs and measurements. Yet, we’ve found that in the field of second language research the rate of self-labelled replication published in journals is about 0.26, about one in every four hundred articles. And even among these, the replications were largely conceptual, involving numerous changes between the original and the replication studies, making comparisons difficult. We’d like to encourage researchers to carry out closer replications, and to feel confident in self-labelling them as such. This helps the field track research agendas and their theoretical and methodological precedents. There has been some stigma attached to replication, even though it is so important for making progress in understanding the variables that affect language learning and the effects of instruction.
So, the emphasis to date has been on carrying out one-shot studies, but this is beginning to shift. There is a realisation that we need more intensive and extensive research, so we can arrive at studies that clarify the size, stability, and generalizability of previous findings. This change is one we seek to facilitate and encourage. It is for this reason that we have developed the IRIS Replication Award, to encourage and reward those researchers who are carrying out rigorous replications in order to move the field forward to greater accuracy and generalisability of findings.