Sunday 16 December 2012

Why we need evidence part 3: expert opinion

What do Rod Ellis, Michael Swan, Jeremy Harmer, Scott Thornbury and Jim Scrivener all have in common?  Yes, they're all white, native speakers and they're all men in a profession which is largely female, non-white and non-native speaker. But that's not what I was aiming for, no they are all TEFL "authorities", and probably for whatever reason, the most recognisable names in TEFL/applied linguistics.

This isn't a post attacking them. I happen to like all of them (though Swan is my fav!) but it's more to question the authority that is afforded to them. Harmer for example is a teacher and a teacher trainer. Is he better or more capable than any other teacher trainer? -I have no idea and I have no idea how you would even work that out. Like Thornbury, Harmer has written a very influential book which means he's certainly dedicated and driven, but is what he says any more reliable than a teacher who hasn't written a book?

I have taken a lot of useful advice from Harmer's practice of English teaching, as I'm sure many people reading this have. But I've also seen him recommend teachers test students to find out their learner styles in spite of the evidence he himself quotes against it. I've also seen him promote NLP, without a single word of criticism. though in this regard he's no different from the British Council which seems to have no problem promoting either of these things.

Thornbury scores more highly in both these areas. He's also refreshingly honest about some of ELT's more entrenched practices which have dubious credibility. But one has to wonder about Dogme, criticised, -somewhat ironically perhaps, -by Harmer here.

I like both Harmer and Thornbury's books, but neither men, as far as I can tell, are researchers. Their words should have just as much (or just as little) weight as anyone else's. Even if they were researchers and leading researchers in their field, like Rod Ellis for example, it wouldn't mean that their opinion on a given issue is necessarily the right one. For example Swan takes issue with Ellis' whole approach to language teaching, -so which expert are we to believe?


In the world of evidence based research, opinion albeit "expert" opinion ranks dead last, (and sometimes doesn't appear at all) -and for good reason. The whole purpose of research and the power behind the scientific method derives from the fact that people are often wrong, and often wrong about being wrong. The Nobel Prize winning Scientist Linus Pauling serves as an important cautionary tale:
After becoming convinced of its worth, Pauling took 3 grams of vitamin C every day to prevent colds. Excited by his own perceived results, he researched the clinical literature and published Vitamin C and the Common Cold in 1970. He began a long clinical collaboration with the British cancer surgeon Ewan Cameron in 1971 on the use of intravenous and oral vitamin C as cancer therapy for terminal patients (wikipedia)

I've highlighted a very important sentence in bold here. Pauling believed that vitamin C really really worked and thus ignored his scientific training. Other people may take this idea seriously because such a famous researcher said it. They may even use an authority figure's opinion as evidence of a point they are making, -and I have documented cases of that here. This is often called the "argument from authority" for obvious reasons.

But how does someone become an authority? Well, you need to get people to listen to you. Imagine tomorrow I announced that I would be starting a new teaching method called "langology". I laid out all the precepts and techniques and wrote a passionate call for teachers to use it. How many people would? I'm guessing the number would be zero. Yet there are others who could affect the way thousands of teachers teach, just by doing this, get books published off the back of it and even stir up controversy in the TEFL world. Yet the people who suggest new theories or give advice about best practice are really only giving you their opinions.

Of course, opinions may come in a range of probabilities depending on the claims being made, but we should never forget that they are, no matter how accurate they sound, just opinions. And as such, the opinion of say Dr. James Ascher (the Dr. appears on all the TPR stuff) that TPR is good for kinesthetic students, is exactly and entirely as valid as my saying that "langology" will not only make you fluent in any language in a matter of weeks but it will also make you more successful, handsome and probably slightly taller. And importantly should be given exactly that much weight. This is unlikely to happen though, but why?

I can't really answer that question but I would guess that it has something to do with the Halo effect.  This is what makes us think a celebrity giving us insurance advice is worth listening to or what leads to better looking students getting higher marks for an essay. Also perhaps it relates to how easily we are convinced by authority figures. Either way, it's something we should be on our guard against. The next time someone at work, round the water cooler, mentions that "Harmer is big on drilling!" or that Ellis doesn't think contrastive analysis is useful,  remember that in the absence of evidence, this is just an opinion.

12 comments:

  1. Hear hear! From the ideas to the links, this was an insightful post.

    I particularly like the pyramid image that accurately sums up the quantity/quality relationship between experiential 'expert' opinion and actual systematic review.

    I'm sure Ellis would agree that we need evidence to support our opinions.

    ReplyDelete
    Replies
    1. Thanks for your comment! I'm glad you liked it.

      Delete
  2. Hello Sir,

    Nice post!

    I certainly agree with Jeroen Root that this was an insightful post. Thanks so much for sharing it. I do, unsurprisingly perhaps at this point, have to step back and question some things here. Thanks in advance for permitting me to do so and thanks again for getting me thinking.
    Apologies in advance for sounding much ruder and more direct than I probably intend to!
    My first very simple thought was that (from my understanding at least) part of being a researcher is specializing in a specific area. I would be more likely to listen to Swan’s view on teaching grammar (or even better aspects of grammar but perhaps not the teaching of it) than I would to his views on sociocultural theory or modern dance.

    The analogy of a general practitioner (family doctor) vs. a specialist comes to mind.

    I am not sure that Harmer’s “being” a researcher** would make much of a difference when he writes books like “The Practice of Language Teaching.” I think the idea with a book like that is breadth rather than depth, no?

    It seems that you make some pretty broad claims about the power of researchers/research. You show that systematic review holds the highest place on the pyramid but I am left asking where is the systematic review to show this superiority?

    You mention that Harmer is a teacher and a teacher trainer. To my mind this is a good start and is better than some dude in a lab wearing a white coat. (sorry for that overgeneralization as well as the potentially false dichotomy). You ask, “Is he better or more capable than any other teacher trainer?” I don’t that is really the question. I think the question is something more like, “Is he capable and knowledgeable enough to write a general book on English teaching methodology (that will hopefully be both accessible and informative). *2

    To take the medical doctor analogy a bit further, perhaps NLP is something like leeches that we as a field will slowly grow out of but it still manages to appear in the handbooks of the day. I think I will stop using that analogy before things get really crazy.

    Ok back what you wrote, about Thornbury and Harmer you wrote, “Their words should have just as much (or just as little) weight as anyone else's” Hmm. Doesn’t writing a bunch of books that sold a bunch of copies and were edited by a bunch of people matter? Am I being too simplistic or capitalistic? I know that Thornbury is the series editor for the Cambridge Handbooks for Teachers series. To me, at least, this speaks to the breadth that I was speaking about before. Just after googling Harmer I see that he has written some coursebooks as well.

    I have felt many times in reading the work of Thornbury that he has an uncanny ability to take complicated thoughts (often from research) and boil them down for the practicing teacher thus making current research (in my view) much more accessible to the teacher on the ground.

    Finally, I am very interesting in this langualogy stuff you are talking about. I would love to hear more.

    STAY TUNED FOR PART II

    ReplyDelete
    Replies
    1. Hi Mike,

      Thanks for replying and thanks for your continued patronage of this blog -and happy new year!

      Quite a lot to get through here. Firstly though I should say clearly that the number of books sold, editorship of whatever series accounts for very little in terms of evidence. The example of Pauling in the blog I hope shows that. If we're talking about whether Swan's opinion about grammar is better than my opinion about grammar then I think there might be a case for that. Still, at the end of the day it's opinion and nothing else.

      Systematic review is better because it collects the results of a number of tests and combines them. Logically for example a case study would trump an opinion. A series of case studies would trump one. A large double blind study would trump that and a large number of those would trump one. This isn't controversial I hope?

      Back to Harmer and Thornbury. Well simplistically NLP authors sell thousands of books. Paul Mckenna and gillian McKeith sell thousands of books. It really only means they can sell books. Harmer and Thornbury can be wrong and are at times. Harmer is almost certainly wrong about learning styles and NLP. So how do you weigh that against the stuff he gets right? also how do you know what he's getting right?

      Quite often people read something and if they agree with it then they think "Oh yeah, Harmer is on the money here" if they don't they might frown and say "Hmmm, I'm not sure I'd agree with that".

      A good question perhaps is...if Harmer hadn't written his book, would his opinion still be as important as it is now? If not then what in the act of writting the book gave his oinion more weight?

      Langualogy? It's going to be all the rage in 2013, -mark my words!

      Delete
  3. PART II

    I just realized something as I labored away and reached the 600 word barrier on this highly insightful and not at all scrambled comment: We largely agree on the lack of credibility that we should be giving experts. I think that teachers (through reading and whatever else too, sure) *should be making their own decisions based on their own contexts and their own sense of plausibility and not following the decrees of experts. You seem to be saying that we need to be way of “expert opinion” and I largely agree. I can’t believe it took me so long and some impassioned defenses of others to get here.

    I wrote briefly and ineloquently about my thoughts on researching certain things here:
    http://eltrantsreviewsreflections.wordpress.com/researching-designer-elt-methods/ and I can see some connections between this and thoughts on the primacy of research (or at least the related difficulties)


    Thanks as always.


    Yours in skepticism,
    @michaelegriffin



    *1 I am not entirely what it means to “be a researcher.”
    (How much research does one need to do? How many published papers? How many hours? What specific job title? Then, is being a teacher (teacher trainer, materials writer, whatever) mutually exclusive?

    *2 I don’t really want to reveal myself as capitalist swine here but I think that it is worth considering the selling of books. What goes into it? Name value, sure. Desire and ability to go touring around the world advertising and the like, of course. There is also the obvious factor of writing

    ReplyDelete
    Replies
    1. interestingly I think we agree but for different reasons. You agree because of the ESiD philosophy. I agree for different reasons and prbably disagree with the ESid idea to some extent. Obviously ESiD is true and kids in Palestine will be different from business men in China. I do though think that they all possess exactly the same brains. I reckon that all languges are learnt in the same way -I'm not talking about approaches or preferences here, but rather how eventaully the language is stored and produced in the brain of the indvidual.

      I remember reading your nice article on methods in the summer and it struck a chord. I started writing a reply but abandoned it after finding I didn't have the evidence to make the point I wanted to make.

      I think I use the word "researcher" to imply to (possibly false) dichotomy that exists in EFL between the chalk face teachers and the ivory tower researchers. I wouldn't give (as i think I say) a researcher/professor and more credit for an opinion than any other expert talking about their field. I would give their research (depending on it's quality) a lot of credit though.

      I'd like to hear more about the selling books things. I'm not sure I'm getting your point. Is selling books to your mind an indicator of ...say...trustworthiness? or usefulness?

      Delete
  4. This comment has been removed by a blog administrator.

    ReplyDelete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. This comment has been removed by a blog administrator.

    ReplyDelete
  7. Whoops...sorry for spamming your site with my long reply in various lengths!

    ReplyDelete
  8. Hi EBEFL,

    Thanks for the response and a very happy and evidence based new year to you. Sorry for the exhaustive comments, this round will be much shorter.

    I think I got my point crossed up here. I didn't mean to suggest that selling a lot of books makes on an expert. You wrote, "... the number of books sold, editorship of whatever series accounts for very little in terms of evidence." Sure. I was more trying to answer your question on how some gets "authority status."

    So, no, I don't think that selling books = trustworthiness...just a way to be put in the position to write (and sell) more books.

    I suppose your central point was about how "experts" are not be followed as they are on the lowest rung of the pyramid. I guess I went in some different directions. :)

    As for systamatic review you wrote, "logically for example a case study would trump an opinion. A series of case studies would trump one. A large double blind study would trump that and a large number of those would trump one" and hoped it was not controversial. I guess I am just not completely sure what can be seen clearly from such studies but that might be a different and old point.


    You wrote, "Harmer is almost certainly wrong about learning styles and NLP." I smiled and mostly agreed. You then wondered, "So how do you weigh that against the stuff he gets right? also how do you know what he's getting right?"Great questions. I have no idea. I guess my thought is that with a book like that, designed to be an introduction,

    Quite often people read something and if they agree with it then they think "Oh yeah, Harmer is on the money here" if they don't they might frown and say "Hmmm, I'm not sure I'd agree with that".

    A good question perhaps is...if Harmer hadn't written his book, would his opinion still be as important as it is now? If not then what in the act of writing the book gave his opinion more weight? Very good question. As I said above, I think I got away from the point of expert opinion being the bottom rung. But surely Harmer's opinion is more valuable than my cousin's (who have never taught or read anything in the field). Absurd example, I know.

    Thanks for your clarifications (on terms and such).
    (And also good point about the potentially false dichotomy between chalk face teacher and ivory tower researcher)

    ***suggestion alert***
    I wonder if you would consider detailing some research that you think is is effective/clear/helpful/suitable/high quality in a future blog post. Or similarly, something you think about teaching and how it has been guided by research. Just throwing it out there. It might make things easier to follow for agnostics like myself and might also lend some more credence to your treatment of things like NLP.

    thanks again for the exchange and for getting me thinking.

    Cheers,
    Mike

    ReplyDelete
    Replies
    1. thanks as always Mike.

      That's an excellent suggestion...now if I can just find some research like that!!!!

      Delete