Guest post by Kevin Cunningham, TCD (posted by David Farrell, February 22, 2011)
This article explains some of the problems of common polling interpretation and applies the leading solution to the general election campaign of 2011. It also offers a more accurate interpretation of trends and generates a current poll of polls as at Monday February 21st.
Why a new technique
Analyzing polls can be particularly fruitless in Ireland as each opinion poll typically has a confidence interval of approximately 3%. For example, some polls are giving Fianna Fail 12%, others 16%. IT is difficult to tell to what extent is the 12% incorrect, or within the realms of possibility.
One of the main outcomes of this is that we cannot rely on a single poll to determine what is going on now, or, with any degree of reliability what the trends, if any, have been. A typical approach to this problem is to aggregate all polls (i.e. a poll-of-polls). However, in the context of any underlying trend in the sample of polls, this method will yield skewed results because it attributes an equal weighting to the oldest polls as it does to the most recent polls. It also offers nothing in terms of trends.
A second approach is to combine recent polls only. There are difficulties and problems with this also. First, the more significant the trends, the more polls that need to be discarded and therefore we reduce the number of polls to be used. Furthermore, interpretation of which set of polls to use is inherently unclear due to the aforementioned variance of the polls themselves.
In this election psephologists are faced with the typical dilemma as there now appears to have been a trend in support from Fine Gael and away from Labour.
A standard approach to interpreting polling trends involves using loess regression. This generates a trend based on the entire dataset. It has huge benefits as it negates the influences of exceptional data and other sources of random variation. However, loess regression models have a big drawback. Computations using loess will only run smoothly when polls are plentiful, and in the case of Ireland, we simply do not have enough polls to deploy such a technique.
The solution to this problem involves employing technique known as Kalman Filtering and Backwards Smoothing (KF). It is a technique employed in a wide range of applications from trajectory estimation for the Apollo program to FM Radios.
In relation to polling data, it applies a similar smoothing algorithm to the loess smoother in that it accounts for all data points (in this case, polls). However, KF also assigns weights to poll results, by taking into account such factors as sample size and the amount of time elapsed between polls. It further accounts for the relative spuriousness of different polls by conducting autoregression on the data at the outset (see Green, Gerber and DeBoef, 1999 for further details).
In essence, the technique yields estimates from polling data at each point in time based on a variety of factors. It minimizes error from the two sources of uncertainty in polling: the true level of support for a party, which may change on a day-to-day basis by some unknown amount; and the error in polling.
I applied the KF and smoothing algorithm to the polling data thus far and ran the results again through a loess smoother. It produces the story of the election campaign as depicted in Figure 1. Each point is a polling result for a party at that date and the trend-line is the trend as determined by the technique.
Figure 1: Kalman Filtering and Smoothing Election Campaign 2011, data points relate to the actual polling data, the smoothed line is a loess curve of the KF estimates.
What we can say about the election is therefore that:
• Support for Fianna Fail, the Independents, Sinn Fein and the Green Party has remained constant.
• Support for Labour has fallen dramatically and the trend-line suggests this may continue to fall in the short-term future.
• Support for Fine Gael has increased, but appears to have stabilised at its current elevated level. It does not appear that they will continue to increase their support to levels where a single-party government is achievable.
• The Green party also appears to be dwindling of late.
• It appears that commentary in relation to changes in Fianna Fáil support so far has been spurious. Although, another poll like the 12% Millward Browne recently put them on might change this interpretation, that particular poll now looks open to scrutiny.
• Variance in relation to the Independent candidates is wider than other parties, this may be due to varying interpretations of independents and other candidates. Variance of the Labour party is also relatively wide, suggesting that they might be particularly difficult to predict on election day.
So, as according to the KF technique, upon the release of The Irish Times poll of February 21st, a poll of polls would give:
Fianna Fáil 15.81%
Fine Gael 36.62
Sinn Féin 11.27
Ind. /Other 14.44
7 thoughts on “Too may polls, they’re all insignificant, but what’s going on? (Or Fine Gael level off, Labour point south and the Greens dwindle)”
So essentially until the votes are counted no one knows what the results will be!
Also, I don’t think any other country has the same issue that Ireland does with a dominant party for 80s years of the scale of FF and the inexplicable refusal of your average voter to vote in a national election on national issues and then to be surprised when their is such an abject failure of natinal politics.
Re: the first point. There are ways around the identified problems, they just involve using fairly heavy statistics like the method I describe here.
Thank you for the first statistically literate analysis of polling trends I have seen. I am so fed up of the Indo (and others who should know better) assigning significance to minor changes within the margin of error of individual polls.
Of course the individual polls within your dataset also use different methodologies – landline, mobile, face to face, different sample sizes, structures and “adjustments” – not to mention different question orders and wordings.
However your methodology is the best I have seen for filtering out spurious random and other biases and giving us a reliable trend indicator.
What it doesn’t measure is the last minute trends as the undecided get off the fence, differential turnouts, transfer patterns and local candidate brand values (independent of their party affiliation).
So there is still plenty for us all to ponder and speculate about. But at least we have a better defined ball-park within which to speculate.
If it wasn’t for opinion surveys politicians could not use one of their favourite cliches: the only poll that counts is the one on polling day.
Frank Schnittiger says “Thank you for the first statistically literate analysis of polling trends I have seen.” – I second this proposal!
Many people are getting “so fed up of the Indo (and others who should know better) assigning significance to minor changes within the margin of error of individual polls.” RTÉ’s Brian Dowling picked up the hype-it-up virus today on the News at One Programme – when all the dust settles it seems probable that Fine Gael and Labour will have the numbers to form a coalition government.
Thanks to Kevin Cunningham for a good professional approach to polls
It’s hard to see how you can say Labour is heading south.