“Below the margin of error” an inexcusable error

Referring to “below the margin of error” is a common error made when talking about polls. It’s an inexcusable error for media like One News and 3 News. It’s impossible to be below the margin of error.

Gavin White from UMR slams margin of error ignorance in a blog post:

In passing I’d like to express my irritation with a comment made about this on TVNZ Breakfast, along the lines of “the Internet Party polled 0.4%, below the margin of error for this poll’. That is a stupid statement that displays the journalist’s lack of knowledge of how polls work.

I heard similar, repeated several times, on 3 News’ Firstline on Monday morning. This is a common mistake and shows a basic lack of understanding of what the margin of sample error is.

Gavin explains:

The margin of error given for surveys is almost always ‘for a 50% figure at the 95% confidence level’.

The important thing to note is that the margin of error shrinks as the percentages expressed in the survey move away from 50%.

If a survey of n=1000 New Zealanders has National on 50%, for example, then probability theory indicates that we can be 95% confident that their actual vote lies between 46.9% and 53.1%.

If the same survey has NZ First on 5%, then we can be 95% confident that their actual vote lies between 3.6% and 6.4%. Why? Because the margin of error for that 5% figure is only 1.4%.

By the same token with the Internet Party on 0.4%, we can be 95% confident that their actual vote lies between 0.0% and 0.8% – the fact that they are below the margin of error for the survey as a whole is about as relevant to the discussion as the fact that the British Conservative Party got 36% of the vote at their 2010 General Election.

This week’s 3 News poll report doesn’t explain it well, they only say:

The maximum sampling error for a simple random sample of 1000 eligible voters is +/- 3.1 percent at the 95 percent confidence level.

One News has a much better explanation on their poll report from Colmar Brunton:

SAMPLING ERROR: The maximum sampling error is approximately ±3.1%-points at the 95% confidence level. This is the sampling error for a result around 50%. Results higher and lower than 50% have a smaller sampling error. For example, results around 10% and 5% have sampling errors of approximately ±1.9%-points and ±1.4%-points, respectively, at the 95% confidence level.
These sampling errors assume a simple random sample of 1,000 eligible voters.

Reporting of polls by media needs improving.

Poll ‘margin of error’ explained

The “margin of error” quoted in polls is not well understood – particularly, it seems, by journalists reporting on polls. It is more correctly called Margin of Sampling Error (MOSE)* – see note at bottom.

It’s usual to have a “margin of error” of 3.1% on a poll of 1000 people with a confidence of 95%. This means in simple terms:

A poll result of 50% has a 95% confidence a full population election/poll would be between 46.9 and 53.1 (or a 1 in 20 chance it is outside that range).

The margin of error depends on two important things as well as the % confidence – sample size and result size.

Sample size

With a poll result of 50% in a sample size of:

  • 100 the margin of error is 9.8%
  • 384 the margin of error is 5.0%
  • 600 the margin of error is 4.0%
  • 786 the margin of error is 3.5%
  • 1,000 the margin of error is 3.1%
  • 1,068 the margin of error is 3.0%
  • 2,400 the margin of error is 2.0%

Polls are commonly of 800-1,000 people. Above that you have to increase the sample size substantially to get a little extra accuracy.

Result size

The size of the result makes a difference. The quoted margin of error applies with a poll result of 50%. It reduces as the result approaches 100% or 0%.

In a poll of 1,000 with a result of:

  • 50% the margin of error is 3.1%
  • 25% or 75% the margin of error is 2.7%
  • 10% or 90% the margin of error is 1.9%
  • 5% or 95% the margin of error is 1.4%
  • 1% or 99% the margin of error is 0.6%

Low results for small parties is proportionally less accurate.

Putting that into a real poll

Using this in last week’s 3 News/Reid Research poll we get:

Poll Result Minimum
(at 95%)
Maximum
(at 95%)
National 44.5% 41.4% 47.6%
Labour 33.5% 30.6% 36.4%
Greens 12.4% 10.4% 14.4%
NZ First 5.7% 4.3% 7.1%
Maori 1.0% 0.4% 1.6%
Mana 0.3% 0.0% 0.6%
UnitedFuture 0.0% 0.0% 0.0%
Act 0.0% 0.0% 0.0%
Conservative 2.1% 1.2% 3.0%
Using opposite extremes of these ranges:
  • the difference between National (41.4) and Labour (36.4)  is 5%
  • the difference between National (47.6) and Labour (30.6)  is 17%

Either are a less likely result than middle-ish (11%) but are possible within a confidence of 95%.

Patrick Gower lead the news with this poll result stating “Winston Peters is up and over the 5 percent mark. He gets into Parliament on that, and it simply changes everything”.

Apart from the fact that a poll taken in January doesn’t tell us what people may think in the election this coming October (probably) NZ First is still not certain of making the 5% threshold based on this one poll, which ranges from 4.3% to 7.1%.

And the Roy Morgan poll a few days later had NZ First at 4.5% (3.2%-5.8% at 95% confidence).

Sub samples

The margin of error also increases when talking about sub samples because you asre working with a small sample size.

If a poll is of 1,000 people of which 45% (+/-3.1%) support National, then if quoting a preference of National supporters their sample size is 450 so the margin of error is 4.6% on a result of 50%, or 2.8% on a result of 10%.

* Margin of error versus margin of sampling error

Margin of sampling error is an actual quantity we can measure.  There is no such thing as a measurable margin of error for a poll.

Poll results are subject to lots of sources of errors ranging from how well the questions were designed and asked to how well the interview was conducted to how well the sample design was implemented.  Good pollsters and researchers do everything in their power to minimize these other possible sources of errors, but they are non-measurable in any case, and one can never know the precise amount of error associated with any poll finding.

Ref: http://www.aapor.org/Margin_of_Sampling_Error1.htm

Also ref: How to Calculate the Margin of Error for a Sample Proportion

Prompted by discussions on the margin of error at Pundit.

Any clarifications or corrections welcome.