Sunday 20 December 2015

Seeking Nirvana

“Nobody made a greater mistake than he who did nothing because he could do only a little.” – Edmund Burke

How do you know that smoking causes cancer?

Easy, right? scientists said so and they did lots of research to prove it. But what research did they actually do and how did they do it? If you're anything like me, you probably have absolutely no idea.

In the 1950s two British doctors carried out a cohort study. This is when you look at a large group of people (40,000+ doctors in this case) over a period of time and study which conditions they suffer from and then try to match those conditions with other factors. For example those in the group getting lung cancer seemed to also overwhelmingly be the ones who smoked. Bingo, we have a correlation. 

I often wonder what would have happened if this were education research posted on twitter nowadays? My feeling is that as soon as it had been tweeted out countless blogs would have popped up to discredit it. 

Firstly someone would point out that correlation doesn't always mean causation. Next we would read that doctors shouldn't be trusted because 'remember what happened with Thalidomide'. Then, someone else would casually note that there must be hundreds of other factors which could influence these people, like diet and lifestyle. They would then pull out the classic educational trump card that 'every smoker is different' and that what affects one wouldn't necessarily affect another. Next someone would ask for the authors to define exactly what they meant by 'smoking' are we talking pipes or roll ups? And just how many cigarettes makes one a smoker? Finally the coup-de-grace would be delivered with the comment that 'my grandfather smoked 40 a day and lived till he was 100'.

Once the cloud of doubt was thick enough, everyone could go back smoking, safe in the knowledge that the imperfections in this research would protect them from cancer. 

The reasons used to dismiss research in education also exist in medical research and psychological research and somehow they seem to manage. 

Take a human beings for example. Each has their own unique genetic code. The differences are so extreme that some people can drink a little alcohol and suffer quite high levels of liver damage while others drink lots and are fine. Other can smoke their whole lives without getting lung cancer. Other people can die if given penicillin.  



Yes despite these differences when I buy a packet of painkillers it says "take one per day for adults" with no warnings about "unless you're a middle-aged woman weighing between X and Y". Somehow we can all just take one a day and 'it works!' But in education context is king and attempts to move the field forward can often be dismissed out of hand by this kind of low level niggling. 


The Nirvana fallacy is where 'good' is rejected because it isn't 'perfect'. It's the enemy of 'good enough' or just 'better than before'. And in education these kinds of improvements are exactly what we should be aiming for. There will never be a perfect method, but we should be asking are there ways of doing things that are a little better than how we're doing them now. 

The Nirvana fallacy is not only apparent in criticisms of research, it also makes an appearance in two other areas of TEFL; textbooks and testing. Textbooks often don't represent real language use, have contrived levels and use 'old fashioned' teaching methodology. They are often bland and designed by companies seeking to make a profit

None of this is controversial and there is plenty of research to back this up. But new textbooks come out all the time and are often better than the ones that precede them. Yet here again 'better than before' is not seen as good enough and instead there are many who seem to feel they should be thrown out altogether unless they are perfect. Of course 'perfect' here means applicable to every individual student's needs regardless of the context, first language, learning preferences and cultural beliefs. They would also use the teaching methodology preferred by whichever teacher was using them and contain language appropriate and authentic for every knowable context. 


Tests too fall victim to the nirvana fallacy. In all areas of education it seems anti-test sentiment is high. Certainly tests can be powerful and life changing and bad tests are disastrous but again is that a reason to stop testing students or is it an argument for better tests? 

Testing is one of the most well-researched and evidence driven fields in education. The test 'form' a person sits is the very tip of a complex and expensive test writing process which has been refined for decades. Tests also give us information on what a students is capable of, how well they've progressed and what they need to work on. Test writers and theorists go to incredible lengths to ensure tests are fair for students and yet I know of hardly any teachers who have positive views about testing. 



Bad research, bad textbooks and bad tests are all arguments for better research, better textbooks and better tests. It's absolutely right that teachers should be critical of things that don't work, and I will be there with them, pointing out sloppy research, crappy textbooks and poorly written tests. But should we dismiss the whole endeavour because it's not perfect? Would we make similar arguments about other fields? charity for instance; 'sure this well may supply clean drinking water but the hospitals are still in a terrible state and the government is unstable so why bother?'


We can still aim for improvements while admitting that things are not perfect. As Michael Long notes
The responsibility of professionals in any field is not to know the right answer, but to be able to defend recommendations in light of what is thought to be the right answer or the likeliest right answer (best practice), given what is known or thought to be known at the time. What is irresponsible is to throw up one’s hands and declare that no proposals should be made and defended until everything is known for sure (which will never happen). 




12 comments:

  1. It is hardly the main thrust of your post -which I found thought-provoking and somewhat humbling...- but I wanted to comment that Seeking Nirvana is probably exactly what we SHOULD be doing.

    "Nirvana" is a Sanskrit word which means "blown out" (as in what happens to candles). What we are trying to do is to blow out the flames of concepts such as "good" and "bad" and "real" and "perfect". Instead, we see things as they really are. This, it seems to me, is more in keeping with evidence-based action.

    In any event, it's a small point and easily counter-balanced by the introduction to the Voltaire quote "The perfect is the enemy of the good" and simply knowing that there is a word for all of this...a couple of weeks ago, a teacher was trying to dissuade me from improving the morale of the staffroom: "What's the point? Anytime you have a load of women together, this is going to happen. There's nothing you can do." I sighed and said that we should be looking at it from a different persepctive: "Anytime you have a bunch of professionals together, this should never happen."

    ReplyDelete
    Replies
    1. Thanks for your reply :)

      I see you have a different take on the word 'Nirvana'. I wonder if the fallacy could be renamed? the 'perfect solution fallacy'?

      I'm curious did a woman say 'What's the point? Anytime you have a load of women together, this is going to happen. There's nothing you can do'???

      Delete
    2. It was a woman who said such a thing. I probed a little bit and it would appear that she believes that when women are together, they are incapable of behaving professionally or of respecting the terms and conditions of employment. There must be something in their genetic make-up that pushes them towards bitching and one-up-woman-ship. To avoid being misunderstood, I must say that I could not have disagreed with her more!

      Interestingly, Steve Peters (Chimp Paradox author) claimed that females are much more hierarchical than males. He said that males typically have a very flat hierarchy - leader, deputy, everyone else. Females, he claimed, have a much more layered hierarchy. He said this was true in the chimpanzee world and true in the human world. A quick Google search reveals that if you study a handful of young people at university, you can get the opposite results (http://bit.ly/1YAp4Jn). Steve Peters often emphasises his scientific credentials as a psychiatrist rather than as a hippy psychologist, so I wonder if he bases his assertion on the findings mentioned in this Harvard Gazette story (http://bit.ly/1mzwvod).

      Delete
  2. I very much enjoyed this post. Thanks for sharing. Your smoking reference reminds me of the reaction to recent links between processed meat and cancer and how that research was criticised.

    A lot of the argument about pedagogical approaches (e.g. the coursebook debate) seems to be either revolution or evolution. That something is either completely awful or completely wonderful. For instance, I can relate to a lot of the arguments about testing and it's negative effects. But as a learner, I adored tests. So that prejudice (or preference if I don't want to be too hard on myself) informs my teaching, in the same way that being compelled to use a particular coursebook would inform a teacher's opinion of coursebooks. So there seems to be too much going on, between teachers preferences and students preferences/abilities, for simple dichotomies to really represent what is going on in the classroom. Like any other profession, we have to work with what we have. Teaching is a series of compromises: lack of equipment, skewed levels in the class, prescriptive overlords, lack of time. We have to work with all of these various factors that mean we are not able to teach to the best of our ability. We are only able to teach to the best of those circumstances. I think how people respond to that determines their stance on issues like coursebooks. If students are learning even within compromised circumstances then improving those circumstances seems like a good option. But if you believe students are not learning or are being held back by those circumstances, then I imagine overhauling the system would seem preferable.

    Again, thanks for sharing.

    Stephen

    ReplyDelete
    Replies
    1. Hi Stephen,

      Thanks for reading and commenting. glad you liked the post. I also am a fan of tests I must say, but I've never had to do a really life shattering one so maybe it's different for me. some of our Kurdish students, if they fail, have to return home where they are fired, lose their government home and have to pay back all the fees that have been spent on them so far. Hell of a risk!

      Delete
    2. Good Lord. That's a horrific scenario. It is hard to imagine how anyone in such a scenario could be expected to do well. I don't think I've ever heard such high stakes before. The students I work with are also under huge pressure. The problem there seems to be coming from whoever is wielding the power. For instance here in Ireland if you don't pass your driving test you can still drive on your provisional licence and you get the chance for limitless resits. Everyone expects that you should have to pass a driving test (or at least I imagine everyone does) but at the same time it is also acknowledged that tests are tricky and not everyone gets them first time. So simply because you fail your driving test one time, you are not precluded forever from driving. In that situation you mention the stakes are ridiculously high. Why? And why are those stakes accepted by the student? It must be because they have little choice? In which case, the test is hardly the problem and as a teacher I wouldn't feel I was being helpful if I wasted any of my energy railing against testing instead of spending it helping the student to avoid that awful scenario and treating them with as much compassion as I was capable of.

      Delete
  3. Hello Russ,

    Thanks for the interesting, thoughtful and thought provoking post. There is a lot to take away from it. I hope you will forgive me for targeting just one aspect, though.

    Regarding textbooks you seem to suggest that they are good enough and our target should be striving to make them better.

    I am not sure I agree.

    I think my central question might be something like, "What if textbooks are a dead end and destined not to be a good thing?"

    (I am intentionally vague here on what a "good thing" might mean because I think context does matter and people might define "good" in various ways)

    You said you'd be there criticizing crappy textbooks but what if the majority of them are crappy?

    You linked to articles on textbooks often not representing real language use, having contrived levels and using 'old fashioned' teaching methodology as well as being bland and designed by companies seeking to make a profit. I would think these are real concerns and it might lead me to think that textbooks are not worth trying to improve incrementally if they are going to be generally crappy anyway.

    Surely some things are dead ends and not worth improving right?

    My question for myself is something like "What sort of research in favor of textbooks would you find compelling?"

    ReplyDelete
    Replies
    1. Thanks Unknown (you don't need to be),

      You make a good point and one that left me contemplating whether to publish this post for about 6 months. I know there are a lot of good arguments against textbooks (and I've tried to list them) but what is the alternative? Of course, some will point to Dogme and I must say that I don't really know enough about Dogme to comment. I also don't think Dogme is widely used (certainly not in the places I've taught).

      I don't think that I suggest as you say that coursebooks are 'good enough'. I think they can be improved. My argument is more that doing away with them altogether doesn't seem to me to be the solution. Particularly, as Mike Swan notes and as I have seen first hand, when the replacement to coursebooks are teacher created materials. (check Swan in cook and wei page 131)

      Are all textbooks crappy? No, I don't think so. I do think, like tests, they are a really easy target for teachers. Of the conferences I've been to I've noticed it's easy to get a cheer from the crowd by giving either a good kicking. It seems a little simplistic to me. Certainly I've seen some textbooks which are better than others.

      I'm aware that we're in danger of getting into quite a evidence-free zone with this discussion. There is no real objective measure of textbook 'goodness', is there? One teacher will say 'oh headway!' and roll their eyes while another would have viewed headway as revolutionary and wonderful compared to what came before it.

      I guess we disagree on whether or not the whole project is worth abandoning but unfortunately (and I think you hint at this with your last line) I'm not sure if we could find any research which would say which of us was right.

      I guess my question to you would be, -what would you propose instead?

      Delete
    2. While I am not Unknown (how conceited must I be?!), I hope they will not mind if I chip in here.

      You are quite right to say that we can't objectively state whether or not coursebooks are any good. It all depends what one expects from them. In my case, I expect long(ish) texts that are interesting and varied. Coursebooks tend to fail on this criterion. I also look for materials that direct learners to various aspects of grammar, textual organisation, lexical flourishes, useful language, etc within each text. Again, coursebooks fail at this. They tend to use texts as vehicles for one particular grammatical point, or one set of lexis. I have yet to come across any that draw students' attention to how a text is constructed. Coursebooks also dictate the grammatical syllabus that is to be followed (although teachers are obviously free to ignore this). I am not convinced that this grammatical syllabus is the secret to becoming a better language learner.

      Of course, we are far from knowing this particular secret. There are all sorts of factors that influence language learning. And yet there are not all sorts of coursebooks. So, another gripe is that publishers are restricting the options available to teachers and students (albeit for perfectly sound commercial reasons).

      What would be the alternative? I imagine a world where writers put together their own coursebooks, free from the restrictions imposed on them by publishing houses. People could then buy either digital copies or print copies produced on a much smaller scale. I imagine that this would result in a wealth of substandard publications, but market forces might be expected to contribute to the forces of natural selection.

      I imagine a world where Scott Thornbury and Luke Meddings could put out the Dogme coursebook; or where Hugh Dellar and Andrew Walkley could produce the lexical coursebook. Those who feel that we have responsibilities under the banner of Critical Pedagogy could produce their coursebook looking at global issues and those who believe that English is an international language could produce a book that moves away from the confines of The Queen's English or Murican English.

      This, it seems to me (in my unadulterated ignorance), is quite a feasible alternative. Less feasible would be that we train teachers to produce materials that are directed at their learners and their working contracts allow them proper paid time for preparation. Or that we abandon the idea that foreign language learning is an academic pursuit that needs to be done in institutions (rather than a skill that is perfected when working alongside a mentor or a trainer).

      Delete
  4. Hi Russ,

    I'm not familiar with the Swan and Wei article. However, despite my posts railing against rubbish books I might have mellowed at least a bit. I still have massive reservations though because tailoring material use to learners is hampered by publishers marketing books as 'total' or 'complete' courses and the fact that few books are presented as non-linear. Too many have units 1A-3B followed by a review, hinting that the 'correct' progression is turning from page 1 to page 2 onward. So are we trying to fit a linear method/approach to a non-linear process?

    I'd be happy to see an improvement in books but I can't see it happening. What I can see is more of the same with extra DVD-ROM (seriously!) or internet bells and whistles still prescribing narrative tenses then modals then… you get the picture.

    Luke Meddings gave a really good webinar on testing (and/or not testing) for ToBELTA this summer, basically advocating low-stakes testing. I like this idea a lot and I think you are right. The Japanese Proficiency Test is low-stakes for most takers this provides motivation rather than a ton of stress. IELTS for a visa? Probably a bit too much stress, in my opinion.

    ReplyDelete
    Replies
    1. Hi Marc,

      thanks for commenting. I have a couple of questions, I hope you don't mind.

      1) Testing is essentially a way to divide finite resources up. Not everyone can go to Harvard, not everyone can drive a car (some people are too unsafe) not everyone can get a government paid house etc. It is just impossible. These are all pretty high stakes things so how do we decide who gets what without some form of testing? How do we decide which students we should let study for a BA/BSC at an English speaking uni without tests?

      2) You say you can't see improvements in books happening but I'm sure you'll agree that it has happened before, right? That is, the textbooks of nowadays are better than the ones used in the 60s, etc. Perhaps I'm assuming too much here, but I'll assume you agree.

      Given that (and I share your reservations for DVD ROM etc etc) is it not possible they could get better?
      TBs get a lot of stick but they are products so half of the problem is the audience (teachers) demanding books like that. It's like TV and movie. You and I might think a lot of what is produced is poor but while people continue to buy it transformers 5 will get made. So perhaps teachings need a better understanding of what they should be teaching?

      3) I share your massive reservations about books. I'm curious what you suggest as the alternative though. Teacher written materials?

      4) You note that 'tailoring material use to learners is hampered by publishers' but I'm curious as to whether there isn't too much emphasis placed on tailoring things to individual students. I understand the appeal but generally speaking they all have the same aims (to learn some form of English) and they share a class so there is only so much individualisation that can be done. This kind of rhetoric is what started all the learning styles stuff in the first place so I'm a little wary of it. I also don't see why certain individualised parts can't be supplemented.

      Delete
  5. Hi again.

    The testing: high stakes tests for high stakes is, of course, necessary. However, I am alarmed by standardised test creep. I see a place for IELTS in academia. I don't see how TOEIC relates to business communication very much.

    Tailoring: I think you can tailor according to needs that you analyse. You are right that you can't individualise things completely but to a group, I think it's possible.

    I do like the idea of resource books or task sequences based on needs or likely situations, this being a chance for rehearsal leading to reflection. This does depend on situations, etc. but generally, it's what I'd favour.

    ReplyDelete