Q1. Have you ever illegally downloaded movies from the Internet or engaged in other forms of copyright infringement? Yes/ No
One of my favourite parts of teaching postgraduate students is helping them when they start planning their research. They almost always plump for questionnaires and then ask their friends to fill them in. Like many students, they seem to think this is the easy option. But actually questionnaires are somewhat like temples in Indiana Jones films. One false move and suddenly you're running in a blind panic as a large bolder tumbles after you, and a pit full of snakes opens at your feet. Metaphorically speaking.
Many of my students look glum when I tell them that questionnaires can often have very poor response rates even dropping as low as 10%. This isn't really surprising, just ask yourself how often you willingly click on an online questionnaires (sure thing website, I'm happy to sit here and fill out your survey! Really! I've nothing better to do)
When numbers get low there is a danger of non-response bias which means the non-responders outnumber the responders and therefore your result may not be representative. Even when people do respond their answer are subject to response bias. The question above, for example, asks about 'illegal downloading' and so the the likelihood people who see themselves as generally law-abiding, but yet who download movies may not answer truthfully. This is known as social desirability bias, and it's not the only response bias.
Biases aside, question writing is a minefield. The question at the top is no good for a few reasons. firstly, it asks two questions which can be confusing. Secondly, it uses language which some respondents might not understand which may lead to them abandoning the whole thing. They might also do this if the survey is too long and boring, or if the answer options don't allow them to give the answer they want. It's also not a good idea to have leading questions, irritating questions, questions that use negatives (or double negatives) or even too many open ended questions.
I read a lot about surveys recently since I've been trying to write one with Nicola Prentis over the course the last 3 months and it only has about 10 questions. It had several pilots (and we still found flaws when we released it). One question in particular had endless rewrites -can you guess which one? Hopefully all this has whet your appetite! Go and take our survey here!