Can sentiment analysis help predict the 2017 general election?

Share on facebook
Share on twitter
Share on linkedin

Ten days ago Theresa May called a general election. Since Tuesday 18th psephologists – a.k.a pollsters – have been churning out predictions: these show Conservative leads of between 11% and 25%, with conventional wisdom telling us to watch for a Tory landslide. A Conservative majority is on the cards in Wales, and if we take opinion polls at face value, Labour face total wipe-out in Scotland.

Theresa May’s lead on who would make the most capable Prime Minister is now 61% to Corbyn’s 23%, the highest Ipsos MORI have recorded since they began asking the question in 1979 (Thatcher hit 48% against Michael Foot, Blair 52% against William Hague).

But could the pollsters get it wrong again?

Since the start of the general election campaign, both Theresa May and the Labour Party have questioned the wisdom-of-polls; May points to historic errors, whilst Corbyn has flagged that he was given odds of 200-to-1 when he first stood for the Labour leadership. The Conservatives want to avoid voter complacency, whilst Labour seek to energise their membership.

Though they have revised their methodology since the last election, faith in pollsters’ practices is hard to come by. This time around, questions are being raised, over whether they have over-adjusted for their false Conservative trail last general election.

It is becoming more and more difficult for pollsters to identify representative samples; without representative samples, they cannot be sure the views of the people they quiz reflect the views of the voting public. After 2015’s debacle (which saw the Conservatives clinch an unexpected majority) and a number of wrong calls (albeit within the margin of error) in the days approaching the EU Referendum, many are wondering whether an alternative approach is needed.

Figure 1: General Election Opinion Polling (Source: Mark Pack’s PollBase)

Is there a better way?

As the unreliability of traditional polling becomes more apparent and less trusted, there is a burgeoning need for new ways of assessing public opinion. Big data and machine learning offer this opportunity. For the first time in history, we are able to observe what huge populations are saying about the issues of the day, live, and train bots to analyse it through social media data. Though these technologies are still embryonic, we can start to use them to develop a picture of changing public perceptions over the next six weeks. As these tools are refined, using data from historic elections, they could provide key insight into the effectiveness of political campaigns (potentially on a local or candidate-by-candidate basis). They can also potentially be used to observe the damage done by a scandal or the benefit reaped from a policy announcement.

We at GK, alongside sister agency onefourzero, are using used our sentiment analysis tools to track changing perceptions of political parties and their leaders in the run-up to the election. So far, these tell a largely familiar tale. Despite his social media savvy fan base, chatter concerning Jeremy Corbyn remains broadly negative.
The party leaders and their national net sentiment scores are as follows:

These figures demonstrate that Theresa May is more well-liked than Jeremy Corbyn, in line with polls. Digging deeper we find something more revealing. When discussing leader competence, Jeremy Corbyn scores a dismal -78%, compared to Theresa May’s more palatable -32%: a net difference of 46% in favour of May. This indicates that the ‘safe pair of hands’ narrative being crafted by the Conservative Party is working.

The Conservatives’ strong popularity is being driven by the collapse of UKIP and the absence of the expected Lib Dem resurgence. Both Paul Nuttall and Tim Farron have been involved in a string of controversial stories, coinciding with rises in negative sentiment over the past week.

The lack of Lib Dem support could be the result of a ‘shy Lib Dem’ phenomenon, harking back to the ‘shy Tories’ of 2015. Young people may be reticent to admit a Lib Dem affiliation, following their tuition fee blunder, but prepared to vote for the party to support their pro-remain stance. Sentiment analysis of conversations referencing the Liberal Democrats, however, gives a very different view to opinion polling: nationwide, they are the least unpopular party (hovering around -1%), despite the relative unpopularity of their leader.

Overall, our sentiment analysis is not as damning of Labour’s chances as traditional polling, but it still doesn’t paint a pretty picture for the party. Theresa May has a higher net sentiment than Jeremy Corbyn and outside of London, the Conservative Party has a higher net sentiment than the Labour Party (-18% and -23%, respectively). Whether you look at traditional polls or our own sentiment analysis – which shows Theresa May pitted against a trio of unpopular leaders – it is hard to believe the Prime Minister will not increase her majority. Whether the opinion polls get it right, or social listening tools are more accurate, we won’t know until June 8th. What’s certain is that more data is never a bad thing, and it’s about time polls caught up with the digital age.

To find out more about our digital capabilities please contact