Severely flawed royal tour poll

Self selecting online polls are unreliable indicators of public opinion no matter how well they are done, but a Dominion Post poll on the royal tour leaves a glaring gap in it’s options.

             Are you hoping to see the royals? 

o  I’m lining up to see them every chance I get!

o  I’d be happy to get just a glimpse of them.

o  It would be nice to see them, but it’s OK if I don’t.

o  No! I’m a republican and don’t care for royalty.     

The poll is attached to an editorial praising the royals and the tour - Editorial: Give royals a jolly good time – so people interested in the tour are far more likely to click on the editorial link and see the poll than those who don’t care.

The questions are unbalanced, with three options for people who might like to see the royals and only one for those who don’t want to see them.

Worse, the “don’t care” option is tied to being a republican. It’s quite likely there are far more people who don’t consider themselves republicans and don’t care than republicans who don’t care.

This poll is severely stacked against showing a representative sample of opinion.

UPDATE: I linked this to Twitter and got a response from someone from the Dom-Post:

Hi Pete – we hear you. We will amend the poll to make the ‘not interested’ option unlinked to republicans. Thanks

That’s a bit better but changing a poll question part way through the polling is not going to help accuracy, although it’s been changed early in the polling period.

And it doesn’t address the other flaws. Self selecting polls associated with news or opinion articles are for entertainment, they can’t be considered representations of public opinion.

New question in poll:

Are you hoping to see the royals?

 o  I’m lining up to see them every chance I get!

 o  I’d be happy to get just a glimpse of them.

 o  It would be nice to see them, but it’s OK if I don’t.

 o  I don’t care for royalty.


Cannabis law posts and polls

There’s been two cannabis law discussions today on blogs so I have looked for polls on the issue. The most indicative polls from UMR from 2011 and 2013 suggest a slight trend towards a softening of laws with nearly two thirds thinking there should be some change.

Whale Oil: SIMPLE REALLY, BUT ONE IS LEGAL AND ONE IS NOT which has links to US poll data. Cameron Slater concludes:

It is high time (snigger) that our politicians acted responsibly and moved toward legalisation of cannabis. There is no better way to reduce the harm of synthetic cannabis than to allow organic cannabis onto the market freely and without sanction save for similar legislative processes similar to alcohol and tobacco.

The Standard had a guest post What do we do about synthetic cannabis? but the discussion that followed also covered cannabis and the question came up about public opinion on law reform. I searched and found information on several relevant polls.

A Curia poll run for Family First

Media Release 10 September 2013

Only one in three NZ’s believe that marijuana should be decriminalised, according to an independent poll of NZ’ers.

In the poll of 1,000 NZ’ers by Curia Market Research, respondents were asked whether they agreed with the statement “If an adult wishes to use a drug such as marijuana, they should be able to do so legally.”

Only 33% of respondents agreed, with 60% disagreeing and 7% being unsure or refusing to say.

This will have been a properly scientific poll the headline the claim doesn’t match the odd poll question.

“Use a drug such as marijuana” opens up possibilities of drugs other than cannabis so people are not saying no “to dope”, they are saying no to ‘drugs such as dope which is significantly different.

It’s quite possible people would have decided based on synthetic cannabis and other concoctions as well as natural cannabis.

It’s quite possible that a question referring to natural marijuana only would have more agreeing and fewer disagreeing than this poll result.

I also found a more useful reference to two other polls, both done by UMR.

Two marijuana specific polls from UMR which show a slight move to more softening of marijuana laws:

…the results of a SAYit survey of n=1000 New Zealanders conducted in August 2011, and showed that at that time:

14% wanted marijuana fully legalised
45% wanted it decriminalised, so anyone caught using it would get fined but would not get a criminal record
38% believed that it should remain illegal and anyone caught using it should get a criminal record.
3% were unsure.

​We re-asked this question in July 2013 (again with a survey of n=1000 New Zealanders), which showed a small change in attitudes.

17% now want marijuana fully legalised
46% now want it decriminalised
35% believe that it should remain illegal.
2% are now unsure.

​The proportion favouring a softening in the laws is therefore up from 59% to 63%.

This suggests a possible trend towards at least softening cannabis laws with nearly two thirds in favour of some change.

Three polls

Reid Research and Colmar Brunton polls were published on Sunday and Roy Morgan’s fortnightly poll was published yesterday. They covered similar but different polling periods.

  • National – 45.9, 47, 43 – 45.3
  • Labour – 31.2, 31, 32 – 31.4
  • Greens – 11.2, 11, 13 – 11.7
  • NZ First – 4.9, 7, 5.5 – 5.8
  • Conservative – 1.9, 2.3, 2.5 – 2.2
  • Maori Party – 1.5, 0.9, 1.5 – 1.3
  • ACT Party – 1.1, 0.3, 0.5 – 0.6
  • Mana Party – 1.1, 0, 0.5 – 0.5
  • Internet Party – 0.4, 0, 0.5 – 0.3
  • UnitedFuture – 0.1, 0.1, 0.5 – 0.2

Reid Research (18-26 March), Colmar Brunton (22-26 March), Roy Morgan (17-30 March) – Average

Colmar Brunton round to whole numbers above 5
Roy Morgan round to 0.5

There’s good discussions on polls and polling on this post at Public Address: Hard News: Gower Speaks

There’s talk of ignoring low polling parties. I don’t think sub 1% parties should be discounted. It makes a difference whether they are wobbling from 0-0.5 or wavering from 0.5 to 1. Small parties can move late in an election and it could make quite a difference whether Mana, ACT or UF look possible for picking up a list seat off the party vote.

Gavin White of UMR says in his latest post What chance do the Internet Party and the Conservatives really have?

“Here’s another angle on this – again assuming that they (Internet Party) don’t win an electorate, has a party ever increased its vote from below 1% at the beginning of election year to over 5% by election day? In our first poll of 2002, United Future polled 0.3%, and they were below 1% in every poll that year until the election campaign, where they polled 6.7% in the actual election.”

I don’t think UnitedFuture have a great chance of repeating anything like that but one or more of the currently low polling parties could make significant moves.

National down to 43 in Roy Morgan poll

Significant changes in the latest Roy Morgan poll are National down to 43 and New Zealand First up over 5% again.

  • National 43% (down 2.5%)
  • Maori Party 1.5% (down 0.5%)
  • ACT NZ (0.5%, unchanged)
  • United Future 0.5% (unchanged)
  • Labour Party 32% (up 0.5%)
  • Greens 13% (down 1%)
  • New Zealand First 5.5% (up 2%)
  • Mana Party 0.5% (up 0.5%)
  • Conservative Party of NZ 2.5% (up 1%)
  • Internet Party (0.5%, up 0.5%)
  • Others 0.5% (down 0.5%)
  • Polling period 17-30 March

This the lowest National have been since last August-October when the bottom out at 41.

Labour are up slightly and Greens down a bit, cancelling each other out.

The Conservative Party rises ate National’s expense

NZ First have been bobbling around the threshold, this time up and over, and the Conservatives are up again, both probably at National’s expense.

The Internet Party had appeared in January/early February at 0.5, then dropped off for a month and now comes back in, presumably helped by their party being launched towards the end of the polling period.

Roy Morgan New Zealand Voting & Government Confidence Tables

Margin of Error

The margin of error to be allowed for in any estimate depends mainly on the number of interviews on which it is based. The following table gives indications of the likely range within which estimates would be 95% likely to fall, expressed as the number of percentage points above or below the actual estimate. The figures are approximate and for general guidance only, and assume a simple random sample.

Sample Size 1,000
Percentage Estimate

40%-60% ±3.2
25% or 75% ±12.7
10% or 90% ±1.9
5% or 95% ±1.4

Polling 101 and kudos for Colmar Brunton

Opinion polls have been poorly reported and poll details have been hard to find. The latest Colmar Brunton poll is a major improvement.

Most polls have been dominated by associated media organisation news reports that have often had very narrow headlines that ignore how polls work as they try to create headlines. This still seems to be the case with the One News and 3 News polls reported on Sunday and Monday, but a least if all the relevant poll information is made available it’s possible to evaluate much wider implications.

Roy Morgan has been good at presenting their polls factually and with very good explanatory poll information – see their last New Zealand political poll which includes a link to extensive poll data.

New Zealand based polling companies are all linked to media organisations, and they have been big on making up headlines and poor at providing details.

Sunday’s 3 News poll coverage headlined Poll: National up, despite Oravida saga and gave all all party results but over dramatised without justification.

And New Zealand First is on 4.9 percent. It is so close, but leader Winston Peters would not make it back. If he got that little bit extra to 5 percent, it would change everything.

It wouldn’t necessarily “change everything”, 0.1% is statistically irrelevant. At 4.9% (with 95% confidence with a sample size of 1000) the margin of error is +/- 1.4% giving a range of 3.5-6.3.  That’s an approximately 50-50 chance of making the threshold – and research shows that polls tend to under-measure NZ First support.

3 News linked to READ MORE: Full 3 News Reid Research results but this is very disappointing. It did have some detail and some standard poll information but it is inadequate.

It included a “projected number of seats” that means little this far out from the election. Worse, it excluded NZ First from the seat count despite a strong statistical confidence of them making Parliament but it includes the Conservative’s with 2 seats despite there being no statistical or factual basis for them winning an electorate seat.

One News coverage of their poll on the same day further emphasises the nonsense of making definitive predictions based on a single poll. Their headline was NZ First the big winner in poll.

Winston Peters is likely to be the man other party leaders are lining up to make deals with after the election, according to the latest ONE News Colmar Brunton political poll.

New Zealand First is the poll’s big winner, its support more than doubling to 7% this month.

Apart from this significant difference One News gave very superficial poll results, with percentages rounded to the nearest whole number and small parties that didn’t make 1% were ignored as if they had not featured in the poll at all.

And One News had no link to poll details or explanations. This is very poor.

In contrast One News’ polling company Colmar Brunton has provided extensive poll details – if you look for it.

On their Current ONE News Colmar Brunton Poll website page they have a poll summary only (again excluding sub 1% parties) but they provide detailed polling information, including:

Note: The interview introduction was changed in this poll to remove any reference to politics, and the weighting specifications were updated. This may impact comparability with the previous poll.

Please click here to download the ONE News Colmar Brunton Poll report, or read more at the TVNZ politics page.

The poll report gives extensive poll details with polling history, charts and wording of questions plus relevant commentary. They also include detailed methodology.

Methodology summary

CLIENT: Television New Zealand.

RELEASED: Sunday 30 March 2014.

POLL CONDUCTED: Interviewing took place from Saturday 22 – Wednesday 26 March 2014.

MEDIAN FIELDWORK DAY: Monday (50% of sample size target typically reached on this day).

TARGET POPULATION: Eligible New Zealand voters.

SAMPLE POPULATION: Eligible New Zealand voters who live in New Zealand households that have a landline telephone.

SAMPLE SELECTION: Nationwide random digit dialling of landline telephones using stratified probability sampling to ensure the sample includes the correct proportion of people in urban and rural areas. Interviewers ask to speak to the person in each household aged 18 years or over with the next birthday. When required, multiple calls are made to reach that person. Voting eligibility is determined at the first question.

SAMPLE SIZE: n = 1,003 eligible voters.

SAMPLING ERROR: The maximum sampling error is approximately ±3.1%-points at the 95% confidence level. This is the sampling error for a result around 50%. Results higher and lower than 50% have a smaller sampling error. For example, results around 10% and 5% have sampling errors of approximately ±1.9%-points and ±1.4%-points, respectively, at the 95% confidence level.
These sampling errors assume a simple random sample of 1,000 eligible voters.

INTERVIEW METHOD: Conducted by CATI (Computer Assisted Telephone Interviewing).

WEIGHTING: The data have been weighted to align with Statistics New Zealand population counts for age, gender, household size and ethnic identification.

REPORTED FIGURES: Reported bases are unweighted. For Party Support, percentages have been rounded up or down to whole numbers, except those less than 5%, which are reported to 1 decimal place. For all other figures percentages have been rounded up or down to whole numbers except those less than 1%, which are reported to 1 decimal place.

METHODOLOGY NOTES: The party vote question has been asked unprompted since February 1997.

The interview introduction was changed in this poll to remove any reference to politics, and the weighting specifications were updated. This may impact comparability with the previous poll.

The data does not take into account the effects of non-voting and therefore cannot be used to predict the outcome of an election. Undecided voters, non-voters and those who refused to answer are excluded from the data on party support. The results are therefore only indicative of trends in party support, and it would be misleading to report otherwise.

This poll was conducted in accordance with the New Zealand Political Polling Code. Publication or reproduction of the results must be acknowledged as the “ONE News Colmar Brunton Poll”.

It’s worth repeating this:

The data does not take into account the effects of non-voting and therefore cannot be used to predict the outcome of an election. Undecided voters, non-voters and those who refused to answer are excluded from the data on party support.

The results are therefore only indicative of trends in party support, and it would be misleading to report otherwise.

One News and all journalists reporting on polls should learn this thoroughly.

Kudos to Colmar Brunton for providing all this information. If One News and other media organisations and polling companies did likewise, and if political journalists did Polling 101, political polls would be less abused and better used.

Polls – One News Colmar and 3 News/Reid Research

Two new polls out tonight.

One News/Colmar 3 News/Reid Research
National 47% (-4) 45.9% (+1.4)
Labour 31% (-3) 31.2% (-2.3)
Greens 11% (+3) 11.2% (-1.2)
NZ First 7% (+3.9) 4.9% (-0.8)
Maori Party 0.9% (-0.2) 1.5% (+0.5)
Conservative Party 2.3% (+1.0) 1.9% (-0.2)
Mana Party  0% (nc) 1.1% (+0.8)
ACT Party  0.3% (-0.1) 1.1% (+1.1)
UnitedFuture  0.1% (-0.2) 0.1% (+0.1)
Internet Party 0.4% (+0.4)


ONE News Colmar Brunton Poll 22-26 March 2014, margin of error 3.1% (at 50%)
Don’t know 13%, Refused 5% out of 1003

This 3 News Reid Research poll was carried out among a random sample of 1000 eligible New Zealand voters from March 18 to March 26, 2014. The maximum sampling error for a simple random sample of 1000 eligible voters is +/- 3.1 percent at the 95 percent confidence level.


Internet Party polling

Stuff have reported on poll support for the Internet Party in Media man takes on role with Dotcom:, quoting party media advisor Jim Tucker:

At the moment the party was polling at 2.6 per cent, he said.

David Farrar has queried this in Gamekeeper turned poacher:

So here’s my question to any of those journalists who were trained by Jim Tucker. Considering that the Internet Party hasn’t registered above 0.1% in any published poll, should a Jim Tucker trained journalist just report such an assertion without challenge, knowing that gullible members of the public may read it and assume it to be true?

Or would a Jim Tucker trained journalist ask the person making the claim to substantiate it?

As one commenter said, it’s up to Tucker to promote the party he’s now working for. It’s up to the media reporting on this to provide details of the polling – perhaps Helen Harvey should look at the just published Research Association Code of Practice which suggests:

Polling Best Practice Guidelines for Media

  1. If possible, get a copy of the full poll report and do not rely on a media release.
  2. The story should include the name of the company which conducted the poll, and the client the poll was done for, and the dates it was done.
  3. The story should include , or make available, the sample size, sampling method, population sampled, if the sample is weighted, the maximum margin of error and the level of undecided voters.
  4. If you think any questions may have impacted the answers to the principal voting behaviour question, mention this in the story.

I asked Internet Party Chief Executive Vikram Kumar about poll details.  He said:

The figure is from independent polling commissioned by and for the Internet Party.

It’s impossible to judge the poll from that. Kumar then said  they won’t give out polling details, and points out that other parties don’t either (they don’t, although the sometimes mention favourable results).

The Internet Party is happy for people to make up their own mind about its polling numbers via public, widely accepted results that are likely to show a rapidly increasing level of support in time

We’ll have to see over the next month whether public polls start to reflect similar support. Obviously publicity around the possible Mana co-operation and their launch this Thursday could make a difference.

One thing in their favour is that the media will give them a chance to boost their profile, although their main target is younger demographics via social media. Tucker in the Stuff article:

“The target demographic is 18 to 35, people who are disaffected with politics, people who have never voted, who are looking for something different. That’s a good reason not to plant ourselves right or left or whatever. There’s a strong feeling about not ruling out anyone along the spectrum.”

It will be a challenge getting accurate poll results from that demographic, let alone party members and votes.

Roy Morgan – National = Labour+Greens

The latest Roy Morgan results:

  • National 45.5% (down 3.0)
  • Labour 31.5% (up 1.0)
  • Greens 14.0 (up 3.5)
  • NZ First 3.5% (down 1.0)
  • Maori Party 2.0% (up 0.5)
  • Mana Party 0% (down 0.5%)
  • ACT  0.5% (down 0.5)
  • UnitedFuture 0.5% (no change)
  • Conservative Party 1.5% (down 1.0)
  • Internet Party 0% (no change)
  • Other 1.0% (up 1.0)

National look to have taken a hit from the Collins/Oravida issue.

Labour have moved slightly but together with Greens they equal National.

The Greens seem to be able to recover from blips quickly. Labour is still struggling to make much impression although this is up on the Herald result earlier this week of 29.5%.

The polling period was Monday 3 – Sunday 16 March.

RoyMorgan 21 March 2014

Herald Digipoll – National up, Labour crash

The latest Herald Digipoll shows National rising 4 to 50.8% despite the Judith Collins issue happening during the polling period, and Labour is down 6 to 29.5%. This is not good for David Cunliffe, who also drops 5 in ‘preferred PM’ to 11.1, lower than David Shearer ever got.

  • National 50.8% (up 4 from Dec 2013)
  • Labour 29.5% (down 5.9)
  • Greens 13.1% (up 2.3)
  • NZ First 3.6% (down 0.3)
  • Conservative 1.3% (no change)
  • Act 0.8% (up 0.8%)
  • Other 0.5% (up 0.1)
  • Maori 0.2% (down 1.1)
  • Mana 0.1% (up 0.1)
  • Undecided 11.4%

750 eligible voters were polled from Thursday March 6 to Sunday March 16. That was a period of major negative coverage of Cunliffe but only some of the Collins milk issue.

The margin of error is 3.6% (presumably at a confidence of 95%) – note that +/-3.6 only applies at a polling level of 50%, see Poll ‘margin of error’ explained.

Preferred PM:

  • John Key 66.5% (up 4.6)
  • David Cunliffe 11.1% (down 5.4)
  • Winston Peters 6.5 (down 0.8)
  • Russel Norman 4.5 (up 1.1)
  • Helen Clark 3.3 (up 0.1)
  • David Shearer 1.1 (up 0.3)
  • Shane Jones 1.1 (up 1.1)
  • Jacinda Ardern 0.8 (up 0.2)
  • Metiria Turei 0.6 (down 0.3)
  • Grant Robertson 0.5 (down 0.2)
  • Tariana Turia 0.4 (unchanged)
  • Annette King 0.2% (down 0.5)

Key is far more supported than National (16%).

Cunliffe is far less supported than Labour. Cunliffe+Clark+Shearer+Jones+Ardern+Robertson+King=18.1%

Russel Norman rates significantly higher than Metiria Turei – in the draft Green list released yesterday Turei was ranked number 1 for Greens so that would presumably put her at the top of the list for a position in a Labour/Green coalition. I’d expect Greens to be pushing for a deputy PM spot, especially on these poll results.

Comprehensive poll results including regional and gender breakdowns at NZ Herald – National, Greens up, Labour at new low.

On polls and cellphones

When polls are favourable people applaud. When they are not favourable they tend to look for reasons. It’s often other reasons than a poorly performing party, like ‘unfair’ media coverage. And the lack of polling of cellphones often comes into the discussions.

Pollster ‘Andrew’ blogs:

Calling cells is not, and will never be, the magic bullet for opinion polling.

There are many aspects to getting accuracy in polling.

Rob Salmond (Labour adviser) at Polity has posted Endangered: Polls without cell phones and looks at trends away from landline phone use and to cellphone use in the US.

The US is a few years ahead of New Zealand on mobile adoption and decoupling form landlines, but I think within 5 years we will see these kinds of proportions in New Zealand. This will make current pollsters’ policies of refusing to call cell phones hugely problematic – they will cut out almost half the population. No amount to weighting can reliably undo a sampling frame that unbalanced.

This has been reposted at The Standard and has more comments there.

Andrew also comments on this at Grumpollie: Rob Salmond’s post on cell phone polling and says “I agree with Rob Salmond that within five years polling methodologies will likely change” but goes on to make some important points, including correcting a common misconception.

The company that I work for has no policy on “…refusing to call cell phones.” In fact, they do randomly dial cell phones for telephone surveys. They will also call them for the poll if a non-qualifying person in the household gives them a cell number to call.

Roy Morgan also states that it polls cellphones.

This latest New Zealand Roy Morgan Poll on voting intention was conducted by telephone – both landline and mobile telephone…

Colmar Brunton only polls landlines:

Nationwide random digit dialling of landline telephones using stratified random probability sampling to ensure the sample includes the correct proportion of people in urban and rural areas.

Reid Research, Digipoll and IPSOS don’t state (as far as I could see) whether they poll cellphones.

Andrew points to what he things is a far bigger issue than whether cellphones are polled or not:

At present my view is that, in New Zealand, non-response is a far far bigger source of error than non-coverage. If non-coverage of cell only households is such a big issue, how come most polls seem to over-state support for the Green party? And why don’t they under-state support for the Labour Party?

In New Zealand, does calling cell phones decrease non-response or increase it? Don’t underestimate the importance of this.

And he updates his post:

UPDATE: I’ve read, here and there, some comments that polls use a) published landline listings, or b) an outdated list of number banks for RDD sampling. I can categorically state that ‘a’ is absolute rubbish. None of the main media-client public polls use published listings. At the company I work for ‘b’ is also rubbish. It’s quite possible to uncover new number ranges.

For those interested, RDD works by randomly generating numbers within number banks, then connection testing them, and then re-sampling the connected numbers.

I’m sure all polling companies do what they can to be as accurate as they can. And people look for reasons other than the failings of their own parties for unfavourable poll results.


Get every new post delivered to your Inbox.

Join 184 other followers