Karnataka Polls – Opinion Polls the world over

These 2 posts from Offstumped on CNN-IBN’s opinion poll for the upcoming Karnataka Assembly elections have generated quite a bit of debate.

So how does the world conduct opinion poll, some Offstumped Research to put this into perspective.

First Gallup which has this excellent summary extract from a Book on how it conducts polls.

The current U.S. adult population in the continental United States is 187 million. The typical sample size for a Gallup poll, which is designed to represent this general population, is 1,000 national adults. The actual number of people that need to be interviewed for a given sample is to some degree less important than the soundness of the fundamental equal probability of selection principle. In other words — although this is something many people find hard to believe if respondents are not selected randomly, we could have a poll with a million people and still be significantly less likely to represent the views of all Americans than a much smaller sample of just 1,000 people — that sample is selected randomly.

So if we go by Gallup which has the most experience, the litmus test for CNN-IBN is how good its selection technique in ensuring equal probability.

Next the Pew Center which also has a good track record in Polls in the United States. Its methodology has an interesting comment on the bias introduced by non-responses.

Non-response in telephone interview surveys produces some known biases in survey-derived estimates because participation tends to vary for different subgroups of the population, and these subgroups are likely to vary also on questions of substantive interest. In order to compensate for these known biases, the sample data are weighted in analysis.

The demographic weighting parameters are derived from a special analysis of the most recently available Census Bureau?s Current Population Survey (March 2005). This analysis produces population parameters for the demographic characteristics of households with adults 18 years of age or older, which are then compared with the sample characteristics to construct sample weights. The analysis only includes households in the continental United States that contain a telephone. The weights are derived using an iterative technique that simultaneously balances the distributions of all weighting parameters.

This reinforces Offstumped’s argument about the 5124 respondents of the 9000 in CNN-IBN’s sample likely having introduced a bias in its findings.

Another international pollster John Zogby has this to say on biases in sampling

If I draw 1000 out at random, chances are I will get the same numbers of each 95 times out of 100 within a margin of error of +/-3%. However, there are other sources of survey error – e.g. how questions are phrased, etc…..

we apply weighting for party identification to ensure that there is no built-in Democratic bias in our sampling

Next up is Scott Rasmussen who was the only one to project both Bush and Kerry’s vote total within half a percentage point of the actual outcome during the 2004 U.S. Presidential election with this comment on the polling during the 2004 election

As always, it’s useful to use common sense when reviewing poll data. If a poll suggests that 10 or 20 percent of Americans are changing their mind on a regular basis, it should be viewed with caution. Most of the time, you will find that the partisan mix of the polling sample is changing more than the actual perceptions of voters.

Once again a question of what was the partisan mix in the CNN-IBN sample of 5124.

Finally the mother of them all, Real Clear Politics which averages all the polls tracking the current U.S. Presidential elections?for the 2008 race between Barak Obama, Hillary Clinton and John McCain has an interesting chart on the sample sizes and the spread across polls.

Offstumped Bottomline: The world over there is a lot more rigor in demographic profiling to account for biases and there is a lot more polling to accurately prognosticate. CNN-IBN has much explaining it do on how its sampling is free of bias. But before that does it even know what partisan and demographic biases exist beyond the obvious on religion and caste.

Filed under: Karnataka Polls 2008