Tuesday 18 January 2022

Is there a replication crisis in ELT?

In a 2012 post I talked about the famous Pygmalion effect in education which shows that teachers expectations for students alone were enough to influence actual outcomes. It’s a truly amazing finding and minimizes the influence of home life, peer group and IQRosenthal and Jacobson showed that, as I noted, "teachers' attitudes towards their students can affect students in quite remarkable ways". The authors were lauded and their paper has been cited over 12,000 times but there was just one problem, it probably wasn't true. 

As I noted in that original post, there were some failed attempts to replicate the results. What I didn't realise at the time is just how shaky the original research was (Thorndike wrote that the study “is so defective technically that one can only regret that it ever got beyond the eyes of the original investigators!”) and just what a controversial subject it has become (there is a nice overview here). 

The Pygmalion effect is not, however, an anomaly. In fact a host of famous psychological studies have come under scrutiny in recent years. The Stanford prison guard experiment, for example, has, to date, not been successfully replicated. Nor has the claim that holding a pencil in your teeth will make you feel happier by creating a "smile".  Did you hear the one about how priming people with adjectives relating to old age can make them walk like old people? That didn't replicate. Neither did the one in which thinking about professors makes you smarter. How about the one about how standing in a power pose makes you actually more powerful? You guessed it. 

In one incredible article with over 270 authors that attempts to replicate 100 studies in top ranking psychology journals, researchers found that only around one third to a half replicated. This finding is part of the replication crisis in science, which has involved fields from medicine to cognitive psychology. One paper notes that "Data on how much of the scientific literature is reproducible are rare and generally bleak." 

Bad news for science, but what's the situation like in ELT? 

The simple answer is that no one really knows as replication is vanishingly rare in SLA and applied linguistics. One of the people pushing for replication in ELT is Emma Marsden, co founder of the oasis database. In a comprehensive 2018 paper (which stretches to 70 pages) Marsden and colleagues investigate the state of replication in the field. They note that at present "little is known about replication in second language (L2) research". 

In the paper Marsden et al surveyed the literature for replications, finding 67, a number they describe as "very low". The authors estimate that for every 400 papers published, 1 gets replicated. Of the 67 papers they found around a third of the replications they looked at did not produce the same result as the original study. At a rough estimate then 30% of the research out there is wrong.  (Edit: Dan Isbell correctly points out that failure to replicate does not mean the original study was "wrong". As he notes "a single replication is not a final verdict.")

Replication can provide important insights for teachers. A replication of nations work on 432 for instance, showed that merely repeating the task, without shortening the time, led to the same increases in fluency as practice + time reduction. That means 4/4/4 is as effective as 4/3/2. Replication can also help by testing claims in non-"weirdpopulations (particularly useful for ELT)The lack of replication may explain the surprisingly low level of retraction in ELT

There are efforts to change the situation and increase the amount of replication. For instance, Language Learning is publishing a special "replication" edition.  The publication of this project is set of for June of this year and will make interesting reading. But as things stand the lack of replication in SLA should seriously worry those trying to argue for the importance of research in supporting teaching. 

No comments:

Post a Comment