Wednesday, June 01, 2011


Lord Ashcroft has published his findings for Project Blueprint (how to get a Conservative majority in the next election) what makes this interesting is the publication of some of the raw data gleaned from the general public.What makes this more interesting is how relatively useless this is.

As an explanation I'll go step by step through what a poll is and how it might not always work.

Simply put a poll is where a question is asked of a selection of people and an answer returned. It can be as specific as asking a group of your friends if they want Chinese or Indian tonight or as general as asking random members of the public what they think of Chinese food in general. These answers are tabulated and we get headlines such as 'British love Chinese food' or 'British hate Chinese food'. In past entries I've tried to point out that such statements are almost worthless unless we know the methodology used to produce them, but why is that? Because they're flawed - they all exhibit bias.

I'll start with the largest most major bias that affects every single voluntary poll ever undertaken - the person answering the questions is a person willing to answer the questions. That sounds pretty obvious, however consider that if ten random people were asked if they had time to answer a few questions how many would decline? Let's say 9 out of 10 are too busy or don't want to. If this were representative that means the sample size (the people asked) only reveals the views of one-tenth of the population.

From that if a result is gained that 75% of those asked liked Chinese food how can a headline be produced that 'Britain loves Chinese food'? It can't it's a lie, or it may not be; there's not enough data to tell. Beyond compulsory polling (such as the Census) all such polls share this problem and there's little that can be done to address it. It gets worse as there are more additional main biases that can affect a result.

The first is simple - how are the questions phrased? Asking "Should we all live in a Godless anarchic society?" is likely to produce a different result to asking "Is there still a place for God in our society?". Reputable polling firms understand this and tailor the questions to be as neutral as possible; don't expect the mass media to be as unbiased.

The next bias is a little more complicated - demographics. Stick a polling agent in a town centre at 10am and they'll be dealing with a different set of people then they would be at 4pm. Put them in a city centre and both times may produce a different cross-section of the public compared to the town. Put them outside the Poundshop or Waitrose, the bus station or the train station. Each time completely different results may be produced.

Another bias is the personal touch. Are the questions being asked directly by the agent or have have they been posted, handed over or put online? The person asking the questions can alter the results just by their presence either through body language cues or just because the questioned person is trying to give an answer to impress the agent - "Yes young attractive lady I have read both Plato's Republic and Spinoza's Ethics; perhaps we could discuss this further over coffee?"

Now given these known biases polling firms still try to derive results (it's what they're paid for) and one method used for this is weighting. In its simplest form if I knew the population of this country voted 50% "Yes" and 50% "No" on some subject if I conducted a poll which asked the same question first and produced a 60/40 split then I can weight the results so that all the 40% numbers are multiplied by 1.25 and the 60% figures by 0.83 to bring them in line with my known figures. As I'm manipulating the figures I should inform any readers of what I'm doing and why.

From all this a plan can be derived of examining polls:

  • Who was asked?
  • Where were they asked?
  • How were they asked?
  • What were they asked?
  • How were the results weighted and why?
With that in mind I'll turn to the Ashcroft general poll.

Who was asked? The general public, various ages.
Where were they asked? Online.
How were they asked? Unknown.
What were they asked? Questions listed.
How were the results weighted and why?  Unknown.

Taking those answers flaws can be probed for.

Although the general public were polled it's restricted to those online reducing the sample size. No statement is made as to how these people were asked - was it a random email shot; a link from a website or something else? In any case that could both bias (only those who regularly visited the site would know about it) and reduce the sample size even further. From the report we know the results were weighted, but no indication as to how or why is given.

Final verdict - Incomplete; treat results with suspicion.

[Update - in light of the Wyre Forest questionnaire as detailed by Tav here. I'll add add another bias - were those asked present presented with a set of fixed answers?

This can either be presented as a set of answers or answering on a scale of 1 to 10. In the case of the former it may well be that the person questioned doesn't agree with any of the answers; yet feels 'forced' to pick one that's closest to their ideals or simply not to pick one at all. In the case of scales there is a known bias that people gravitate around the average (5 on a scale from 1 to 10) unless something is truly exceptional or diabolical.

As such these are pretty unreliable guides to anything. So why do they keep being used? Because it's easy to tabulate set answers into a report. If someone says they like chips, but another says they only like chips when served with fish; is that two people liking chips or only one? How many other questions get answered with "Don't know"? Do you leave them out of the results or add them in as a separate point? How reliable is a guide to intent when the majority answer "Don't know"?

Far easier to just limit the answers that can be given.]