On online surveys

This has little to do with teaching and learning, but a great deal to do with the credibility of on-line surveys (and hence with research methods).

I recently signed up with a survey company in order to manage a questionnaire to former students (which proved to be more problematic than I had anticipated, but that's another story). The signing up process included asking me whether I would like to earn money by responding to on-line surveys. I was a little intrigued, so I said yes.

Today I received the first request to participate. It proved to be a market "research" survey about chocolate and other snacks. I answered honestly, only to get to the end to get a message (sorry, I should have copied and pasted the wording) which said in effect that I was not the kind of person they wanted answers from! (So I would not get any credits for answering.)

From a methodological point of view, sampling from that small subset of consumers who sign up to respond to such surveys is very dubious in the first place (given that they in turn are a small subset of net-savvy people, who are in turn a small subset of the population --however defined--at large). But effectively throwing away answers which do not suit is the unforgivable methodological sin.

OK—they were only interested in regular consumers of chocolate bars and crisps and their preferences, andbut I buy them vary rarely. (I always buy a multi-pack of crisps at Christmas for some reason and then throw most of them away the next Christmas because I've just bought another pack... why?) But how legitimate is sampling from net-savvy, money-motivated geeks? Only if it can be demonstrated that those "qualities" are independently correlated with product consumption. Has that test ever been done? Who knows? I do know that I read the results of some very strange polls in .net magazine; I don't believe a word (or statistic) of them, but some credulous marketing executive might.

So what? That's their problem. I want valid and reliable research; trust me, I'm an academic!