Poll ‘margin of error’ explained

The “margin of error” quoted in polls is not well understood – particularly, it seems, by journalists reporting on polls. It is more correctly called Margin of Sampling Error (MOSE)* – see note at bottom.

It’s usual to have a “margin of error” of 3.1% on a poll of 1000 people with a confidence of 95%. This means in simple terms:

A poll result of 50% has a 95% confidence a full population election/poll would be between 46.9 and 53.1 (or a 1 in 20 chance it is outside that range).

The margin of error depends on two important things as well as the % confidence – sample size and result size.

Sample size

With a poll result of 50% in a sample size of:

  • 100 the margin of error is 9.8%
  • 384 the margin of error is 5.0%
  • 600 the margin of error is 4.0%
  • 786 the margin of error is 3.5%
  • 1,000 the margin of error is 3.1%
  • 1,068 the margin of error is 3.0%
  • 2,400 the margin of error is 2.0%

Polls are commonly of 800-1,000 people. Above that you have to increase the sample size substantially to get a little extra accuracy.

Result size

The size of the result makes a difference. The quoted margin of error applies with a poll result of 50%. It reduces as the result approaches 100% or 0%.

In a poll of 1,000 with a result of:

  • 50% the margin of error is 3.1%
  • 25% or 75% the margin of error is 2.7%
  • 10% or 90% the margin of error is 1.9%
  • 5% or 95% the margin of error is 1.4%
  • 1% or 99% the margin of error is 0.6%

Low results for small parties is proportionally less accurate.

Putting that into a real poll

Using this in last week’s 3 News/Reid Research poll we get:

Poll Result Minimum
(at 95%)
Maximum
(at 95%)
National 44.5% 41.4% 47.6%
Labour 33.5% 30.6% 36.4%
Greens 12.4% 10.4% 14.4%
NZ First 5.7% 4.3% 7.1%
Maori 1.0% 0.4% 1.6%
Mana 0.3% 0.0% 0.6%
UnitedFuture 0.0% 0.0% 0.0%
Act 0.0% 0.0% 0.0%
Conservative 2.1% 1.2% 3.0%
Using opposite extremes of these ranges:
  • the difference between National (41.4) and Labour (36.4)  is 5%
  • the difference between National (47.6) and Labour (30.6)  is 17%

Either are a less likely result than middle-ish (11%) but are possible within a confidence of 95%.

Patrick Gower lead the news with this poll result stating “Winston Peters is up and over the 5 percent mark. He gets into Parliament on that, and it simply changes everything”.

Apart from the fact that a poll taken in January doesn’t tell us what people may think in the election this coming October (probably) NZ First is still not certain of making the 5% threshold based on this one poll, which ranges from 4.3% to 7.1%.

And the Roy Morgan poll a few days later had NZ First at 4.5% (3.2%-5.8% at 95% confidence).

Sub samples

The margin of error also increases when talking about sub samples because you asre working with a small sample size.

If a poll is of 1,000 people of which 45% (+/-3.1%) support National, then if quoting a preference of National supporters their sample size is 450 so the margin of error is 4.6% on a result of 50%, or 2.8% on a result of 10%.

* Margin of error versus margin of sampling error

Margin of sampling error is an actual quantity we can measure.  There is no such thing as a measurable margin of error for a poll.

Poll results are subject to lots of sources of errors ranging from how well the questions were designed and asked to how well the interview was conducted to how well the sample design was implemented.  Good pollsters and researchers do everything in their power to minimize these other possible sources of errors, but they are non-measurable in any case, and one can never know the precise amount of error associated with any poll finding.

Ref: http://www.aapor.org/Margin_of_Sampling_Error1.htm

Also ref: How to Calculate the Margin of Error for a Sample Proportion

Prompted by discussions on the margin of error at Pundit.

Any clarifications or corrections welcome.

Leave a comment

3 Comments

  1. I’d add that the sampling error assumes a random probability sample. Only a couple of pollsters in NZ actually use this approach.

    Reply
  2. Quota sampling. NZ public poll methods grid | Grumpollie
    http://grumpollie.wordpress.com/nz-public-poll-methods-grid/

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s