Thoughts on the Scottish Referendum Part 1: Polling
A Note About My Personal Views
Everyone has biases, and it is best that I get them out of the way up-front. I have mostly steered clear of commenting on or even following Scottish politics particularly closely. Despite having spent two years there at the University of St. Andrews, I always felt an ambiguous relationship Scottish Nationalism, seeing in it the populist tendency that in the United States has promoted the more self-destructive policies of the Tea Party, paired to a leftist rather than rightist platform. Just as the Tea Party in states like Kansas or the South defines their opponents as the culturally alien professional class in the coastal cities, along with their local collaborators, the SNP almost by definition seemed to view success as un-Scottish. Moving to London to join a bank or law firm was un-Scottish; Edinburgh, the one city in Scotland that has been economically prosperous without oil is viewed with suspicion, struggling and impoverished Glasgow as the heart of national identity. For all the talk about the prospects for prosperity that an independent Scotland would possess, the SNP government seemed to determined to base that independence on hostility to exactly the sort of people and businesses they would need to attract to make that success possible.
That said, in the Kingdom of the Blind the one-eyed man is king. And if Alex Salmond may not have had an idea for economics, he certainly possessed a good one for politics. By turning devolution into a systemic blackmail scheme against Westminster he succeeded in extracting concessions above and beyond what any other region of the UK has managed to achieve, from both Labour and Tory governments. This then enabled him to buy the support of voters, selling himself even to those who opposed outright independence as the man best equipped to make autonomy as prosperous as possible.
When he declared for independence most observers, especially in the London media but also in Scotland itself, were quick to declare the act a blunder, a description they have applied to most things Alex Salmond has done since becoming First Minister. After-all he was forcing the issue, and was sure not only to be defeated, but to the split the SNP by alienating the Unionist votes he had "borrowed" in order to win his majority in 2011. History does not always repeat, but with Salmond the record player seems to have consistent skip, as not only have those observers suddenly woken up to the prospect that Salmond might actually win, with YouGov's most recent poll showing Yes leading NO by 2%, but the government in Westminster is preparing another package of bribes to try and prevent a Scottish exit. It is doubtful whether these bribes will change the outcome, but regardless, what they will do is solidify Salmond's power, demonstrating conclusively that even in the event of an Independence defeat that he was correct to push for the vote in the first place.
The Polling Picture
But how likely is that victory? Part of the problem is no one particularly knows. British polling is problematic at the best of times, as anyone who observed Fivethirtyeight.com's efforts to extend their successful model to the 2010 General Election, an effort I played a role in, can testify. The Cleggmentum that dominated polling and media coverage of the campaign failed to materialize in practice. Was the media wrong? Only to the extent they focused on the polling.
Why then was the polling off? There are several reasons why UK polling is generally less reliable than its American equivalent. For one thing, "partisan weighting" the effort to ensure that your sample is politically and not just demographically representative of the electorate is an obsession for British pollsters, and has been ever since John Major's surprise victory in the 1992 elections prompted a search for the "shy Tory" voter. The result were a variety of mathematical methods, based more on speculation than any sort of social science, that pollsters embraced in an effort to artificially raise the number of Conservative voters in their polls. There is a certain irony in the fact that pollsters chose to attempt this in the lead-up to the 1997 and 2001 General Elections, when the effect was least likely to manifest itself due to tactical voting and the unpopularity of the Major government, though this cancelled out by the fact that the Labour landslide of 1997 was so massive that the media missed the fact that most pollsters were off until the final days.
Since then however pollsters seek to "weight" their polls by asking who respondents voted for in the last election and who they intend to in the next. While superficially useful, this method has the problem that Nate Silver among others brought up in the American 2012 Presidential campaign; when you deliberately weight the poll based on a specific party breakdown you are in effect making a guess about the make-up of the electorate, and that guess by and large presumes the outcome. This was fundamentally the problem with the "unskewing" movement in 2012; they attacked mainstream pollsters for using an electorate in their polls that looked like 2008, arguing that minority turnout that year could not be repeated. This in and of itself was an assumption about what the electorate would look like, one that the advocates of "unskewing" on the US right insisted should be the basis for polls. Despite all of their insistence, and the adoption of many of their claims by elements of the media, in the end the raw samples proved more accurate as the minority percentage of the electorate was actually greater in 2012 than in 2008.
Most partisan weighting in UK polls has been based on asking voters to recall who they voted for in the last general election. While this is good at measuring swings between parties among voters, it is far less effective at measuring actual voter sentiment since it by and large carries with it the assumption that the electorate as a whole will be largely static between elections. In the UK this has to a large extent been the case for General Election contests, but it has been far less true of lower turnout contests such as the local and European elections, one reason why UKIP and the SNP outperformed the polls during them.
These misses however were by and large irrelevant for the same reason that UKIP and the SNP tended to over-perform their polling numbers. Because the elections were perceived as meaningless, only the voters most interested in sending a message were inclined to turn out, which in practice meant the most alienated. Labour and Liberal Democratic voters, even in Scotland, were likely to view who ruled in Holyrood as a less important question in 2011 than who ruled in Westminster, whereas SNP voters had the inverse set of priorities, explaining not just that party's overpeformence in 2011, but its disappointing Westminster results in 2010.
The challenge for pollsters approaching the Scottish referendum is not so much how to poll a low-turnout contest, which they have done in the past, but whether it will be low-turnout at all. Will the electorate for the Scottish Referendum look more like the 2011 Scottish elections, or like the 2010 General Election? Or as may seem more plausible, somewhere in between with SNP turnout resembling 2011, and Labour, Tory, and SNP turnout looking a bit more like 2010?
YouGov's "Yes" Lead
Pollsters have taken a variety of different approaches to this question, with YouGov weighting on voter recall of the 2011 Scottish Elections as well as for Age & Gender, Social Grade, Newspaper readership, and country of birth. As samples are unlikely to be purely representative, they are then multiplied by whatever number is necessary to attain the proper balance. As such, if a sample of twelve voters contains 10 men and two women, the women's votes are multiplied by three, while the men's are multiplied by .6.
This system, whatever its merits, still relies on a number of assumptions, first and foremost that it has to presume the makeup of the electorate. To cite the example utilized above, in order to figure out what to do with a sample that includes men and twelve women, YouGov has to assume that the eventual electorate will have an equal number of men and women in it. At the end of the day this is a guess. It is perhaps a well-informed and justified guess, but it remains a guess, and one of the downsides of hard-weighting is that genuine changes in the enthusiasm or voting intentions of demographic groups can be written off as sampling error and "corrected" for, while in turn pollsters can as easily assume that a shocking result is the consequence of their own error or random chance, rather than movement. On the other-hand, efforts to correct for past error can also produce results that are equally easy to conflate with genuine movement.
These factors need to be borne in mind when comparing YouGov's recent polls showing a Yes lead of two points with an earlier sample of mid-August showing a No lead of 13, one sees that the major shift has been among voters who claim that they intend to support Labour in a potential future Scottish election. In mid-August these voters favored the NO side 78-15, a margin that fell to 73-23 in September. Yet as anyone capable of doing math can figure out, there is no way that an 8 point shift among Labour supporters can explain a swing of that magnitude unless Labour supporters made up more than 100% of the sample, an impossibility even if Tower Hamlets Mayor Luftur Rahman were running their postal ballot effort. Something else had to change as well.
In the case of YouGov, it was the weighting of the sample rather than change in sample composition. Taking a look at the Holyrood question, if we are to assume that substantial movement in terms of sentiments occurred, the shift in polling should be in response to a surge in support for the SNP among actual respondents - in plain English, YouGov should have found more SNP supporters in their September results than in their August results. Looking at their unweighted results however, this does not seem to have occurred. For their August 4-5th poll which found a 37-36 Labour-SNP split, YouGov's raw sample included 356 Labour supporters and 346 SNP supporters, a near tie. In their September 2nd-4th poll they also found a near tie, 335 Labour supporters to 338 SNP supporters, yet these results were weighted to a 40-33 SNP lead. If the 2011-recall question is used as a control and its results are fixed, it is unclear how that result can be justified by the same weighting. X+5 and X+5 cannot ever equal the different numbers, its basic mathematics. Nor can the same equation produce the following results:
Raw Respondents Weighted
Aug 4-5 Sept 2-4 Aug 4-5 Sept 2-4
Labour 356 335 342 304
SNP 346 338 340 373
Con 144 154 146 147
LD 45 39 39 34
As we can see therefore, the change that occurred between August and September was driven not by an overall change within YouGov's sample, which remained fairly constant, but in the multipliers applied to the raw results. As YouGov's weighting is listed at the end of their polling, this indicates that this shift resulted in a much more substantially Pro-SNP sample.
We see this in the answers to the Holyrood voting intention question. In August, Yougov's sample expressed a split of 37-37-15-4 between the SNP, Labour, Conservatives, and Liberal Democrats. This past week, they found that these intentions had shifted to a 40-33-16-4 split. With SNP supporters going over 90% for Yes and Labour supporters in the mid-70s for NO, such a shift is a pretty direct transfer of support from the NO to the YES side.
YouGov Samples
August 4-5
August 12-15
August 21-Sept 2
September 2-5
SNP
36%
37%
39%
40%
Labour
37%
37%
34%
33%
Conservative
16%
15%
15%
16%
Liberal Democrat
4%
4%
5%
4%
Other
7%
7%
7%
7%
Yes Aug
No Aug
Unsure Aug
Yes Sept
No Sept
Unsure Sept
SNP
87%
4%
9
92
5
3
Labour
15
78
7
23
71
6
Conservative
5
94
1
7
93
1
Liberal Democrat
22
75
3
13
83
5
If we re-weight the September numbers regarding Holyrood intentions back to the August sample, we turn a 47-45 Yes lead into a 49-43 No lead, which without undecideds ends up as about 53-47 NO, in line with most other polls. That result may very well hold a hint as to why YouGov's results shifted so far. YouGov's August numbers showed No drawing substantially more support than any other pollster. With the benefit of hindsight we can see that this was largely due to an unusually Unionist Labour sample, and that with another round of polling, YougGov's results would have fallen into line with everyone else's at 53-47. Instead, with the sampling, it swerves all the way to 51-49 YES.
We can measure the impact by reweighting YouGov's September breakdowns by their sampling for August 12th-15th and 4th-5th. We find that the same breakdown that is giving a 47-45 lead for yes would yield leads of 5.5% and 7.8% respectively. In effect a large portion of the closing has been generated by YouGov reweighting its polls every two weeks to artificially inflate the Yes vote. The same polling responses that yielded a 51-49 Yes margin without undecideds would have yielded 55-45 for No a month ago.
Weighted
Published Results
Yes
No
Yes
No
September Weighting
47
45
47
45
Mid August Weighting
48.5
42.9
51
38
Early August Weighting
49.9
41.2
55
35
Did Scottish voters change their minds and move to the SNP because of the success of the "Yes" campaign? Or did the "Yes" campaign gain because the SNP lead in the sample went from less than a point to seven? There is no way to tell, and even if we could, nothing but speculation will give us an idea of which of those universes is a more plausible representation of who will actually vote on September 18th? What we can say is that the August Labour sample was most likely an outlier, with the result that the poll was too friendlier to NO. To that can be added that if we give YouGov's long-standing methodology more credit than YouGov itself evidently provided it with, we end up with a 53-47 result, which is in line with most other polls. This leads to a conclusion on my part that the election is likely close, but that NO is still in all likelihood ahead by 4-6 points.
What we also get is reinforcement of the lesson that there is wisdom in going with the "totality of polling" a standard which served Nate Silver so well during the 2012 American elections. We then accept s that YouGov's August numbers were an outlier in favor of the No, and that they over-corrected by changing their sample when the Labour numbers would have fixed the results in any case, resulting in an outlier in favor of Yes in September.
In conclusion, YouGov's Yes lead is the result of changes in sample composition rather than a clear shift, though a substantial shift in the preferences of Labour supporters was detected. The only way to examine that is to ask ourselves the qualitative rather than the quantitative question of whether anything has happened to produce such a shift. I will look at that in my next post.
