Monday, 3 December 2012

When nothing is better than something

Dr. Duncan MacDougall found that after humans die their weight changes by exactly 21 grams. He carried out his research on terminal patients and weighed them before and after death. He also carried out the same tests on dogs and found no weight change. No one can explain this strange phenomenon and the religiously minded as well as the New York Times wasted no time attributing it to the weight of the human soul.

An incredible and disturbing finding, were it true. 

Which it isn't. 

While it's completely true that what this experiment was carried out and that those were its findings, it's equally true that it was a naff experiment. It's easy to think any research is better than no research but bad research is often pretty useless; it tells us nothing and worse, sometimes it can even be dangerous. Dr. MacDdougall's experiments were conducted on 6 people which is a horribly small number to warrant such extravagant claims. And that's not all, quoting here from the blog "rationally speaking" the research had a number of other problems:

Not only was the experiment never repeated (by either MaDougall or anyone else), but his own notes (published in American Medicine in March 1907) show that of the six data points, two had to be discarded as “of no value”; two recorded a weight drop, followed by additional losses later on (was the soul leaving bit by bit?); one showed a reversal of the loss, then another loss (the soul couldn't make up its mind, leaving, re-entering, then leaving for good); and only one case actually constitutes the basis of the legendary estimate of ¾ of an ounce. With data like these, it's a miracle the paper got published in the first place.

Second, as was pointed out immediately by Dr. Augustus P. Clarke in a rebuttal also published in American Medicine, MacDougall failed to consider another obvious hypothesis: that the weight loss (assuming it was real) was due to evaporation caused by the sudden rise in body temperature that occurs when the blood circulation stops and the blood can no longer be air-cooled by the lungs. This also elegantly explains why the dogs showed no weight loss: as is well known, they cool themselves by panting, not sweating like humans do.

Ah, you may say, this was a long time ago before we had proper research. Well, while it's true that scientific techniques improve all the time, research now, like research then is carried out by humans. So could a badly designed study get newspaper headlines these days? Over to Andrew Wakefield who 'discovered' a link between vaccinations and autism, research which led to, and continues to lead to parents not vaccinating their kids and thus the return of previously controlled diseases, such as mumps and measles as well as occasional deaths. Wakefield, who was eventually struck off the medical register, conducted his research on exactly 12 children so, twice as many as MacDougall's study.  This didn't stop theDaily Mail and other papers creating huge panic with this information.
Now small studies are not always problematic, but newspapers tend to have an undue influence on what people think and a story which might not get much (or any) attention in academia because of problems, such as its sample size, could have considerable influence if published in a newspaper. I wrote here about Memrise the amazing new technology which mean you can learn a new language in only 22 hours (disclaimer, for "language" read "some words" , for "22 hours" read "three months" and for "amazing new technology" read "flashcards and mnemonics"). Memrise is a good example of how the media can create excitement about something that really isn't all that exciting
Recently there was a TEFL article in the guardian making the claim that the "argument was over, the facts were in" and that explicit grammar teaching was a must for EFL. Catherine Walker's bold claims were marred by a couple of issues. Firstly the article wasn't in a peer reviewed journal, it was in the Guardian (though even journals can get it wrong, and do, regularly, and spectacularly) and journalists are not experts and are therefore much more likely to let things slide that academics would probably pick up. Being a newspaper Walker didn't have to provide any evidence for her claims, but when prodded by commentators listed the following:

Norris, J. M. & L. Ortega. 2000. Effectiveness of L2 instruction: a research synthesis and quantitative meta-analysis. Language Learning 50/3: 417-528.
Gass, S. & L. Selinker. 2008. Second Language Acquisition: an Introductory Course (Third Edition). New York: Routledge/Taylor.
Spada, N. & Y. Tomita. 2010. Interactions between type of instruction and type of
language feature: a meta-analysis. Language Learning 60/2: 1-46.
Spada, N. & P. M. Lightbown. 2008. Form-focused instruction: isolated or integrated?
TESOL Quarterly 42: 181-207.

To be fair to Walker, meta-analysis are the creme de la creme of research and are positioned at the very top of of the hierarchy of evidence pyramid. However there are still some problems with the piece. The first flaw is that the headline for the article is misleading. Articles from 2010, 2008 and 2000 can't be run in 2012 with the headline "the evidence is finally in", without stretching the "finally" beyond recognition. Secondly the headline makes the claim that evidence shows that grammar teaching is effective, yet later in the article this is watered down to:

However, evidence trumps argument, and the evidence is now in. Rigorously conducted meta-analyses of a wide range of studies have shown that, within a generally communicative approach, explicit teaching of grammar rules leads to better learning and to unconscious knowledge, and this knowledge lasts over time.

That is, if you are using the Communicative Approach, it is better to have grammar taught explicitly. So this is not so much a debate about the value of grammar teaching but a debate about the value of teaching grammar explicitly within a certain method. The title of the article may have had us all rushing back to the oft mocked (but pretty widely used) Grrammar Translation method.  Another possible problem is the conflict of interests. Walker has a written a number of grammar textbooks and while this doesn't mean she's wrong, the possibility of bias is there; "shock! grammar teacher claims teaching grammar works!". 

So language articles are often annoying because they get tweeted and retweeted when the findings may be problematic or in some cases nonexistent, like a story doing the rounds  at the moment. Apparently, English is not a Germanic language but a Scandinavian one. For years linguistics have been wrong and this new research shows conclusively that English comes from Scandinavian not from Old English. Except it doesn't because, as far as I can discover, there is no research. Yes, there are researchers and yes there are news articles and yes there is even some evidence of conference talks but I can't seem to find a paper published in a major peer-reviewed journal (please someone link to it if you can find it).

Now, I'm out of my depth with the argument as to whether English is or isn't a Scandinavian language perhaps it really is but what I can say is that a massive claim like this, if correct, would make the careers of both the authors. I can also safely state that a claim which has such large potential, needs a equally large amount of evidence to back it up. To use the Sagan Standard "Extraordinary claims require extraordinary evidence". Judging by the comments on some of the blogs that have reported this, I'm going to tentatively suggest that that evidence will not be forthcoming.

The king of getting media attention with little research is of course, Chomsky. Slayer of the evil behaviorists, discoverer of the mysterious UG, Noam wins the prize by virtue of having done exactly 0 research to test his theories. His ideas, which have held sway over linguistics for 60 years, were thought up by him, and then left for others to argue about.  People who actually took the trouble to look into and test Chomsky's claims found him to be wrong, wrong and more wrong.*  Geoffry Sampson writes:
Hang on a minute,’ I hear the reader say. ‘You seem to be telling us that this man [Chomsky] who is by common consent the world’s leading living intellectual, according to Cambridge University a second Plato, is basing his radical reassessment of human nature largely on the claim that a certain thing never happens; he tells us that it strains his credulity to think that this might happen, but he has never looked, and people who have looked find that it happens a lot.’

Yes, that’s about the size of it. Funny old world, isn’t it! (2005:47)


Sampson, Geoffrey. 2005. The Language Instinct Debate. London & New York: Continuum

*I'm really really out of my depth on Chomsky but if someone wants to come and put me right I'll be happy to listen.


  1. hey there
    another solid post.
    though i think the last time chomsky got media attention due to his language work was sometime ago no? ;)

    1. thanks for the comment! In a way that makes it worse doesn't it? He got famous for his linguistics stuff and then just moved off into poitics!

  2. Excellent stuff!
    I am getting addicted to you blog, and winter, which is normally very hectic for me, is not a good time to be addicted! Argh! As regards Catherine Walter's article I was so infuriated that I wrote a reply. If it doesn't get published, I'll put it on my blog.
    Thanks for this enjoyable and, as usual, thought-provoking piece.

    1. Thanks for the comment!

      Did you write a reply for the Guardian? I hope they do publish it but I don't have that much faith in them. Please let me know if they do.