A primer on polls

[quaoar]

With all the attention here on polls, I thought some would like to read this on how to assess polls:

http://www.ncpp.org/qajsa.htm#11

[depakid]

1. EVERYONE needs to read this.

It's very straightforward- and although it presents a little too rosy of a picture, it's good food for thought.

Some excerpts:

You must know who paid for the survey... Polls are not conducted for the good of the world. They are conducted for a reason – either to gain helpful information or to advance a particular cause.
It may be the news organization wants to develop a good story. It may be the politician wants to be re-elected. It may be that the corporation is trying to push sales of its new product. Or a special-interest group may be trying to prove that its views are the views of the entire country.

Most scientific samples use special techniques to be economically feasible. For example, some sampling methods for telephone interviewing do not just pick randomly generated telephone numbers. Only telephone exchanges that are known to contain working residential numbers are selected – to reduce the number of wasted calls. But even a random sample cannot be purely random in practice since some people don't have phones, refuse to answer, or aren't home. (the key question here is whther the non-respondents differ as a group from the respondents- historically, the answer to that question is yes, which has produced embarrassing results like "Dewyey Defeats Truman").

You should always ask if the poll results have been "weighted." This process is usually used to account for unequal probabilities of selection and to adjust slightly the demographics in the sample. You should be aware that a poll could be manipulated unduly by weighting the numbers to produce a desired result. While some weighting may be appropriate, other weighting is not. Weighting a scientific poll is only appropriate to reflect unequal probabilities or to adjust to independent values that are mostly constant.

[quaoar]

2. And this is significant as well:

Interviews with a scientific sample of 1,000 adults can accurately reflect the opinions of nearly 200 million American adults. That means interviews attempted with all 200 million adults – if such were possible – would give approximately the same results as a well-conducted survey based on 1,000 interviews.

What happens if another carefully done poll of 1,000 adults gives slightly different results from the first survey? Neither of the polls is "wrong." This range of possible results is called the error due to sampling, often called the margin of error.

This is not an "error" in the sense of making a mistake. Rather, it is a measure of the possible range of approximation in the results because a sample was used.

Pollsters express the degree of the certainty of results based on a sample as a "confidence level." This means a sample is likely to be within so many points of the results one would have gotten if an interview were attempted with the entire target population. They usually say this with 95% confidence.

Thus, for example, a "3 percentage point margin of error" in a national poll means that if the attempt were made to interview every adult in the nation with the same questions in the same way at about the same time as the poll was taken, the poll's answers would fall within plus or minus 3 percentage points of the complete count’s results 95% of the time.

This does not address the issue of whether people cooperate with the survey, or if the questions are understood, or if any other methodological issue exists. The sampling error is only the portion of the potential error in a survey introduced by using a sample rather than interviewing the entire population. Sampling error tells us nothing about the refusals or those consistently unavailable for interview; it also tells us nothing about the biasing effects of a particular question wording or the bias a particular interviewer may inject into the interview situation.

[depakid]

3. Statistical concepts are important- but they need context

Most people in America haven't had a basic statistics class- so a lot of this kind of information is too abstract to grasp. Completely understandable. I've seen high priced consultants pull the wool over very intelligent people's eyes using what is sometimes profoundly lame methodology. Slick graphics look great, but if you ask one or two of the right questions, the conclusions all come tumbling down.

Producing quality results is hard work and it's expensive. Campaigns spend a ton of money on their internals- they actually want to know what's going on, but sometimes even they're deceived by their own pre-conceptions. "News" organizations, on the other hand, do things on the cheap, and have a completely different agenda.

My thoughts are that anyone who truly enjoys political analysis would find the social science behind polling to be absolutely fascinating- especially when you look at all the stories involved.

[DemocratSinceBirth]

4. I Did Post Grad Work In Government

worked with SPSS....


I still don't understand how the Gallup poll had such violent swings in Election 00 but miraculously was able to come "close" on Election Day...


And I don't find the "intervening events" explanation plausible. Certainly not plausible enough to account for a fourteen point shift for Gore in the final week of the campaign...

The only logical conclusion is Gallup is "under weighting" Democratic and " over weighting" Republican voters....

[depakid]

5. Gallop definitely has an agenda

swings like the one you describe- or the ones in the AJC poll/Diebold results in Georgia in 2002 obviously aren't due to chance error or intervening events. In Gallop's case, they have been known to play fast and loose with weighting.

This little piece was posted yesteday:

"Values Seen as Most Important Characteristic of Presidential Candidates."

http://gallup.com/content/default.aspx?ci=12544

Check out how they did the weighting!

btw: SPSS is a fine program- it's been around a long, long time... although, as I'm sure you know, if you put garbage in you get garbage out! One of my old prof's has some hilarious examples that students turned in....

[DemocratSinceBirth]

6. I Did Post Grad Work In The Mid 80's

so I have been out of the loop a long time....


I follow politics now as an informed observer not a participant, academic or practioner...


I just don't understand how Gallup is so out of sync with Zogby and Newsweek even controlling for the one and two day differences in poll taking...


I do not believe Gore closed a fourteen point deficit in seven days in Election 2000.... The more likely explanation is that Gallup manipulated their polls in the waning days of 2000 so as not to look like charlatans when the actual results came in...


How many Dems?.... How many Reps?.... How many Indys? are in the Gallup poll........

[depakid]

7. Sounds like your "bullshit detector" is intact

the basic concepts haven't changed since the 80's... although the integrity of the media reporting has changed a lot... which means that people really do need to look at poll results with a jaundiced eye. I haven't looked closely at that set of Gallop polls- but your analysis seems plausible enough to me.

Frankly, I wish that basic stats was a required high school course. People need and actually use these skills more than ever in their daily lives- whether they know it or not- and not just in politics.

[DemocratSinceBirth]

9. Pollsters Give Statiscticians A Bad Name

but polling like DNA is based on inferential statistics...


I think there are only about 30,000 dna samples in the entire data base but scientists can infer from that odds in the billions of a match...


I want to know how Gore was losing 52% -39% on October 27 and went on to win 48.5% to 48.0% on Election Day.... Even Shrub's twenty year old DUI can't account for that kind of swing...

I guess when ABC and CBS release their post convention polls we will get a better idea of what's happening...

[tedoll78]

8. What many here should take solace..

The Gallup poll seems to be an anomaly, completely inconsistent with the results from most other polls. Newsweek, Rasmussen, Zogby.. no poll is perfect, but the overall trend is definitely in Kerry's favor. What's more, the state-by-state polling looks very promising. We have Kerry up in Minnesota and Michigan, and even Novak is admitting that Kerry's pulling-away in Pennsylvania. Recent polls even have Kerry leading in Nevada, West Virginia, and New Hampshire. And if the Colorado referendum on electoral vote splitting passes, it means 4 more unexpected electoral votes. We have no real reason to be fretting about the horserace right now.

A side-note..

What worries me more is this "terror alert" b.s. I'll bet that there will be more alerts at these times:
- the death of the 1000th G.I. in Iraq
- Kerry putting the smackdown on Bush in a debate
- the home video release of 'Fahrenheit 9/11'
- a major Bush campaign gaffe

We've seen it happen like clockwork. After the Edwards announcement, and now here after the convention. Will the Democrats have the guts to call Bush on this, or even just have surrogates float the idea as a trial balloon for the American people to consider?