GALLUP's Credibility Gap

[liberalpragmatist]

http://www.opinionjournal.com/diary/?id=65000474

Gallup's Credibility Gap
What's with those wild swings in the polls?

Thursday, October 26, 2000 12:01 a.m. EDT

"The Gallup Poll just has no credibility with me anymore," the network producer sighs. "Even if the survey methods are valid, how can you tell viewers that a swing of 15 points in four days is realistic?"

They might be even more skeptical of Gallup's tracking poll if they knew that the swings are due in large part to the company's nightly sample having large fluctuations in how many Democrats vs. Republicans are interviewed. One three-night sample, on Oct. 4, had 37% Democrats and 30% Republicans. Perhaps not surprisingly, Al Gore had an 11-point lead. Three nights later, on Oct. 7, a completely new sample consisted of 39% Republicans and only 31% Democrats. Lo and behold, George W. Bush suddenly had an eight-point lead. Typical exit polls of actual voters, taken as they leave the polls, show the parties evenly matched or with just a point or two advantage for Democrats.

Gallup and CNN, a major sponsor of its poll, insist they have a plausible explanation for this. Gallup told the Washington Post that in 1996 fully 25% of the people they contacted changed their party identification at least twice. That seems highly implausible, and James Campbell, a polling expert at the State University of New York at Buffalo, says it's "crazy" and misleading to base nightly poll results on a hypothetical electorate that switches party identification so readily.

<snip>

It seems likely that the Gallup pendulum swings are in part the consequence of tracking polls that sample only 400 voters a night and then pare that number down to 240 or so likely voters when results are published. Not weighting a sample properly for party identification doesn't seem to make sense. Pushing undecided voters too hard can make people give unconsidered answers. And Gallup seems strangely convinced that its voter samples can have big mood swings from one night to the next. Finally, there's the problem that, as we're always told, the chances of getting a truly odd sample are at least 5% on any given night. That means in 60 days of nightly tracking, Gallup can expect a rogue voter pool about three nights. That could skew any three-day average of nightly tracking polls significantly.

UPDATE: A number of posters are pointing out that this article specifically refers to the 3-day tracking polls that Gallup hasn't yet started doing. Whatever - the point of this is to show that Gallup tends to have big swings. So what are people's theories on this? Is it just their LV models that are wild and unpredictable? Why is this so? Feedback would be appreciated.

[w4rma]

1. After Gallup got bought out, their accuracy went downhill, imho (nt)

[tritsofme]

2. This is talking about their tracking poll

toward the end of the 2000 election.

tracking polls that sample only 400 voters a night and then pare that number down to 240 or so likely voters when results are published

This is not true of their samples in polls being released around now.

[liberalpragmatist]

4. That's a good point

Still, I do think the fact that they use a Likely Voter model is troubling and I posted this just to show that Gallup has had wild swings in that past election and just this past year, for instance, when they had Kerry way up when it was probably much closer.

My point is that Gallup is unreliable, at least until Election Day.

[tritsofme]

6. They provide RV numbers with every poll,

and that has been much tighter throughout the race.

The logic of their LV model makes sense, but it has been putting out lopsided results this year.

[unfrigginreal]

12. Clue us in on the logic they're using for LV

from what I've seen, that's where Gallup fails. The same was true of the 2000 election.

[tritsofme]

14. This is what Gallup says about their Likely Voter model

Gallup tries to avoid the lack-of-interest problem by specifying that 55% of its sample (which corresponds with a 55% projected turnout in the election) will be considered likely voters. Gallup asks each respondent seven LV screening questions, and gives each person an LV score of 0 to 7. The top 55% are classified as likely voters. This classification works even if interest is low early in the year, because low-interest respondents are considered likely voters as long as they are in the top 55% of the sample on the overall LV scale.

[unfrigginreal]

18. That's all well and good but it doesn't tell us anything about how...

they're weighted. Here's a theory put out about the 2000 race that suggests that Gallup was underrepresenting minority voters.

http://chris-bowers.mydd.com/story/2004/5/19/225039/641

"Likely Voter" Bias: Why Gallup was Wrong in 2000
by Chris Bowers

On November 6th, 2000, the final Gallup/CNN/USA Today tracking poll showed Al Gore at 45% and George Bush at 47%. This poll was similar to others conducted by different services in the final days of the election. During the same time period, NBC/Wall Street Journal showed Gore at 44% and Bush at 47%; ABC/Washington Post showed Gore at 45% with Bush at 48%. Tarrance showed Gore 41%, Bush 46%; Christian Science Monitor showed Gore 46% and Bush 48%. Among all final polls, only CBS (45-44) and Zogby (47-46) showed Gore ahead.

I am not a subscriber to Gallup, and thus I do not have access to old poll internals. However, if you are willing to take my word for it, in early October of 2000 I spent a couple hours after class one day pouring over the internals of the latest Gallup tracking poll in an attempt to better understand the state of the campaign. One discovery, which has stayed with me to this day, led me to conclude that they were underestimating Gore's support by around 2-3%, and overestimating Bush's support by the same margin. The discovery was that they had incorporated race into their model, and had done so in a way that appeared to more heavily weight white opinion than minority opinion.

In 2000, Gallup's national tracking model identified 87.5% of "likely voters" as "white" and 12.5% as "non-white." However, according to exit polls, only 81% of voters were "white" while 20% of voters were "non-white." If Gallup had incorporated an accurate model of minority turnout into their polls in 2000, their final tracking poll results would have been amazingly accurate. If only 12.5% of voters had in fact been "non-white" in 2000, then Bush would have won 50.13-46.13. Conversely, had Gallup projected that 20% of voters would have been "non-white," then their final poll results would have shown 47% for Gore and 46% for Bush. Perhaps not coincidentally, Zogby's Election Day numbers were 47% for Gore and 46% for Bush.

Although I lack access to internals from the time, I submit that Gallup and almost every other polling agency was wrong about Gore and Bush in 2000 because their polling models over emphasized white opinion. Further, it is entirely possible that the reason Zogby and CBS were more accurate than other polling services was because they incorporated an accurate model of minority turnout into their data. Considering demographic changes, it is quite likely that minority turnout will constitute an even larger percentage of the national vote in 2004 than it did in 2000. If, for example, the "non-white" vote will make up 22% of the national electorate in 2004, than even a model that assumes 20% minority turnout will be inherently inaccurate. Such a model would add nearly 1% to Bush and subtract nearly 1% from Kerry.

[tritsofme]

19. Gallup doesn't weight their polls at all

They use random sampling for all their polls.

And then they use the formula I posted above for LV.

[unfrigginreal]

20. Then, I disagree with you that their LV logic makes sense.

If their polls aren't weighted in such a highly partisan atmosphere they're useless.

[sonicx]

7. actually, i think their polls then and now have about 1000 total polled

and 700 or so likely voters.

[tritsofme]

8. That was my point. nt

[sonicx]

9. i think that was what the article said but...

they were just cutting it up by days (400 per day).

[tritsofme]

10. IIRC Gallup was releasing daily tracking polls the last week of

the 2000 campaign.

I'm not sure though.

[sonicx]

11. oh, i didn't know that.

i think i got it confused. :crazy:

[annabanana]

3. flip coins..... more accurate than Gallup..................n/t

[kentuck]

5. They will be much more accurate about 3 days before the election...

These present polls are all for show and manipulation of public opinion.

[Iceburg]

13. Not likely ... look at Oct 27, 2000 Bush 52% Gore 39% (Gallup)

http://www.cnn.com/2000/ALLPOLITICS/stories/10/27/cnntime.poll/index.html
October 27, 2000
Web posted at: 8:36 p.m. EDT (0036 GMT)

...
The poll of 2,060 adult Americans, including 1,076 likely voters, has a margin of error of plus or minus three percentage points and is thus in essential agreement with a CNN/USA Today/Gallup tracking poll also released Friday. That poll gives Bush a 52 percent-39 percent edge over Gore.

...
ABC News and The Washington Post both have daily tracking polls today putting the race at 48 percent for Bush and 45 percent for Gore. The latest Reuters/MSNBC/Zogby tracking poll has the contest at 45 percent for Gore and 43 percent for Bush.

================
Zogby was the only pollster to come close in 2000!

[tritsofme]

15. Gallup was very close in their final 2000 poll

It showed Bush 48% Gore 46% Nader 4%

Actual results were approx. Gore 48% Bush 48% Nader 3%

[sonicx]

16. yup, but i think

gallup should sit it out and poll only in the last 2 or 3 days. ;)

[tritsofme]

17. That's not how you make the bucks.

;)