Published:

11 comments

SHAKESPEARE Stephan headshot

Stephan Shakespeare is a founder and global CEO of YouGov.

We’ve now had four years of YouGov daily voting intention polling, which has undermined the false excitement and pointless speculation generated in the era of monthly polls by small margin-of-error variations falsely interpreted as significant.

We can see clearly now that voting intentions are very stable. Even when one sees a bigger flutter than just margin-of-error, it reverts quickly to a slow-moving trendline. Does anyone seriously believe two per cent of people changed their voting intention because of Junker? And then switched back? No: it’s just a few frothy “don’t knows” (who are probably also “won’t votes:) responding with whatever was top-of-mind.

Back during the 2010 UK election, I predicted on this site that the leaders’ debates would cause plenty of this froth, but wouldn’t really change votes. Remember the excitement over Clegg’s performance, and how it was a game-changer? The Liberal Democrat rise in the polls quickly disappeared.

Then during the 2012 US election, we ran a project with Microsoft polling a huge sample – about 25,000 people – before the first Obama-Romney debate, and recontacting the same people afterwards. Although it was generally agreed that Romney had “won” the debate, almost no-one changed their mind about who they would vote for. That study decisively demonstrated that debate-driven polling flutters are of no significance (see here for the academic paper “The Mythical swing Voters”).

That doesn’t mean nothing changes, of course. There are two kinds of real change that you can see from continuous tracking.

There is slow change, in which attitudes really do move, but in tiny increments over a long period. I would include in this category the inevitable disenchantment with a new government and the expected gradual return to it as an election comes closer. This recovery is usually predicted because the resentment people often feel towards whoever is in power tends to diminish a little as a switch to the alternative becomes real. It doesn’t always happen of course: when people have decided they really do want change, we can see the opposite. But right now, you can’t detect any deep longing for that.

Then there is dramatic change, which we saw when the Liberal Democrats reversed their policy on tuition fees. It had been their landmark policy – so dropping it felt, for their strongest supporters, like a real betrayal. It needs that kind of clear and fundamental event to make a real difference in the polls. Even then, I expect we will see some return of their disgruntled votes.

All very dull, I realise – but I invoke Twyman’s Law (nothing to do with my colleague Joe Twyman) which states: “if a polling result is surprising, it’s probably wrong”. We had an important example recently, when we saw a series of exciting polls suggesting that Scotland’s referendum was becoming a close-run thing. YouGov and most other reliable polls showed no such change. So what was happening? Peter Kellner spotted evidence that was briefly on Survation’s website (and was hurriedly taken down again) which clearly showed the fundamental flaw in the method, creating repeated false results (we believe). For a superb analysis, see here.

Survation responded with this curious comment: “the practice of commissioning your own research solely for the purpose of criticising another polling company’s methodology seems, in our view, to be unprecedented in the industry and not something that we would wish to repeat”.

Firstly, of course, we didn’t commission polling – we conduct our own polling continuously, and looked for a reason why one set of pollsters were telling a different story to the others. But, gosh, is Survation saying we shouldn’t seek to understand these differences? That we shouldn’t find out about these important errors? If those polls that showed the Yes vote rising significantly were based on systematic flaws in the method, it is of the first importance that we examine them using evidence. All pollsters make mistakes and get things wrong, there’s no shame in that, but we have to be committed to critique and correction. These polls changed the political news at a critical time in this important referendum campaign.

Continuous polling may be dull but it’s valuable, because it provides plenty of reliable information that shows you what’s really happening. It clearly demonstrates that the events and initiatives which excite the political class have little effect on real voting intention. Over the next ten months, we’ll see strenuous efforts from all parties trying to move the dial, and equally strenuous arguments about whether it’s working, but unless there’s something unusually dramatic, all that will happen is the marginal extension of the trend-lines.

11 comments for: Stephan Shakespeare: Beware the coming waves of froth over polling

Leave a Reply

You must be logged in to post a comment.