“Below the margin of error” an inexcusable error

Referring to “below the margin of error” is a common error made when talking about polls. It’s an inexcusable error for media like One News and 3 News. It’s impossible to be below the margin of error.

Gavin White from UMR slams margin of error ignorance in a blog post:

In passing I’d like to express my irritation with a comment made about this on TVNZ Breakfast, along the lines of “the Internet Party polled 0.4%, below the margin of error for this poll’. That is a stupid statement that displays the journalist’s lack of knowledge of how polls work.

I heard similar, repeated several times, on 3 News’ Firstline on Monday morning. This is a common mistake and shows a basic lack of understanding of what the margin of sample error is.

Gavin explains:

The margin of error given for surveys is almost always ‘for a 50% figure at the 95% confidence level’.

The important thing to note is that the margin of error shrinks as the percentages expressed in the survey move away from 50%.

If a survey of n=1000 New Zealanders has National on 50%, for example, then probability theory indicates that we can be 95% confident that their actual vote lies between 46.9% and 53.1%.

If the same survey has NZ First on 5%, then we can be 95% confident that their actual vote lies between 3.6% and 6.4%. Why? Because the margin of error for that 5% figure is only 1.4%.

By the same token with the Internet Party on 0.4%, we can be 95% confident that their actual vote lies between 0.0% and 0.8% – the fact that they are below the margin of error for the survey as a whole is about as relevant to the discussion as the fact that the British Conservative Party got 36% of the vote at their 2010 General Election.

This week’s 3 News poll report doesn’t explain it well, they only say:

The maximum sampling error for a simple random sample of 1000 eligible voters is +/- 3.1 percent at the 95 percent confidence level.

One News has a much better explanation on their poll report from Colmar Brunton:

SAMPLING ERROR: The maximum sampling error is approximately ±3.1%-points at the 95% confidence level. This is the sampling error for a result around 50%. Results higher and lower than 50% have a smaller sampling error. For example, results around 10% and 5% have sampling errors of approximately ±1.9%-points and ±1.4%-points, respectively, at the 95% confidence level.
These sampling errors assume a simple random sample of 1,000 eligible voters.

Reporting of polls by media needs improving.

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s