Interviews with some Actual Exit Pollsters - they messed up

[Mistwell]

Gee...and exit pollster who appeared to be with Move-On.org...an exit pollster who admits they were not sampling randomly...an exit pollster who admits someone shoved in their own unsolicited survey to the batch...and exit pollster who says white males were avoiding her until she got better at her approach at the end of the day...

It's a VERY small sample, and I am not currently claiming this is representative (though I feel 100% positive that some people in this thread will ignore what I just said and claim I am trying to make a hasty generalization). But it's enough to want to interview a lot more of these exit pollsters. If this does turn out to be representative of this exit poll, you can take those exit polls and burn them, because firewood is about what they are worth. :eyes:

http://www.mysterypollster.com/main/2005/01/professor_m.html

January 28, 2005

Professor M

Following up on a lesson learned from the last post, that a story can sometimes make a point more powerfully than a lot of arcane data, I decided to share an excerpts from a series of emails I received about the experiences of four NEP interviewers from the state of Minnesota. The information comes from the Minnesota college professor - let's call him Professor M -- who helped recruit these college student interviewers for NEP. I have shared much of the substance of this story in previous posts, but in light of the findings of the Edison-Mitofsky report, I thought it would be useful to share his verbatim comments.

In early November, intrigued by the controversy surrounding the exit polls, Professor M decided to interview his four students about their experiences as interviewers. As he points out, a sample size of 4 is truly "anecdotal" -- it is by no means representative of the experiences of the 1400 odd interviewers who worked for NEP on Election Day. However, it is remarkable how many of the problems he notes help explain patterns in the data on "within precinct error" in the Edison-Mitofsky report.

The following are excerpts from our email dialogue:

The information that I got from my students is quite intriguing, but of course it cannot in any way be considered a representative sample. Also, the students I spoke with kept no independent notes of response rates or other details while serving as interviewers; their impressions of who responded and who didn't were entirely from memory.

The geographic distribution of the four interviewers was as follows: one outer exurbia, one inner ring suburb, one "exclusive" upscale suburb and one precinct in an ethnically and economically diverse Minneapolis neighborhood.

The badge did display the logos of the networks prominently.

However, this could not be easily seen from a distance, and at least one of my students was hampered by the fact that a contingent of folks from MoveOn.Org was stationed right next to her at the 100-ft line. This made her appear from a distance as being connected with them, and being forced to stand 100 ft from the polls, people were able to easily turn aside to avoid her and MoveOn. Also, it gets dark early up here, so the badge was not visible from a distance after 4pm or so.

The two students in suburban areas commented that they had the hardest trouble getting participation in the early morning -- probably due to lines and people needing to get to work.

I believe all of the students reported receiving requests from voters who wanted to participate in the survey despite the fact that they were not the nth person to emerge. Nearly all of the students reported some inclusions that were somewhat less than random. Most commonly this occurred when a couple emerged together and the person the poll worker approached refused but the partner offered to participate. None of the students saw any "difference" in which one of the two participated as long as one of them did.

As I understood what the students told me (who did not see themselves as doing anything wrong, by the way) they would not have coded a refusal at all in that situation.

One student reported at least one instance of a person simply taking a survey from her supplies (which were out in the open at her table) filling it out, and dropping it into the survey box. By the time she realized what had happened (she was busy trying to buttonhole legitimate respondents), there was no way to determine for certain which of the surveys in the box had been incorrectly included.

A few additional observations from -- she noted that she had more refusals among white males, although she was not sure if that was related to her own appearance (she is African-American). Also, she observed (and this makes sense, when you think about it) that her response rate improved over the course of the day as she became better at honing her "sales pitch." Still, despite the fact that she had perhaps the most advantageous placement of any of my four students (she was indoors at the only entrance/exit and had full cooperation from the staff on-site), she still recalls a fairly low response rate -- 40-50% perhaps.



To clarify one point: Each interviewer was given an "interviewing rate" which ranged from 1 to 10 nationally. Here is the way the Edison-Mitofsky training materials (passed along by Professor M) describe what was supposed to happen:

We set an interviewing rate based on how many voters we expect at your polling place. If your interviewing rate is 3, you will interview every 3rd voter that passes you. If it is 5, you will interview every 5th voter that passes you, etc. We set an interviewing rate to make sure you end up with the correct number of completed interviews over the course of the day, and to ensure that every voter has an equal chance of being interviewed.

If the targeted voter declines to participate or if you miss the voter and do not get a chance to ask him or her to participate, you should mark them as a "Refusal" or "Miss" on your Refusals and Misses Tally Sheet and start counting voters again (for a more thorough explanation of refusals and misses, refer to page 9). For example, if your interviewing rate is 3 and the 3rd "person refuses to participate, you do not interview the 4th person. Instead, start counting again to three with the next person.

The point: If interviewers allowed "inclusions that were somewhat less than random" but did not tally refusals appropriately, then the "completion rates" now getting so much scrutiny in the Edison-Mitofsky report are not only inaccurate, but the inaccuracies will probably occur most, on average, in the same precincts shwoing the biggest "within precinct error" (WPE).

The bigger point: Consider that all of the above comes from just four interviewers. Imagine how much we might learn if we could talk to hundreds. Apparently, that is exactly what Edison Mitofsky says they will soon do (or are already doing) with the interviewers in Ohio and Pennsylvania (p. 13):

We are in the process of an in-depth evaluation of the exit poll process in Ohio and Pennsylvania...We will follow up with in-depth interviews with the exit poll interviewers in the precincts in which we saw the largest errors in an attempt to determine if there were any factors that we have missed thus far in our investigation of Within Precinct Error.

I think I can speak for others in the survey research profession when I say we hope they ultimately share more of what they learn. It will help us all do better work.

[mhr]

1. Then What About This New Analysis?

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203x309093

[intensitymedia]

3. MItofsky swamp gas refuted in 5 clean pages

The original poster's anecdotal data is typical of the effort to obscure the power of statistical analysis, and cloud peoples' minds with trivial nonsense.

The study cited above already refutes problems with exit polls.

btw the respondents mentioned in the original post were in the state of Minnesota. Is that where they did the exit polls? Doesn't matter, it has no bearing on the validity of the argument, but it's a striking omission - leaving out this single crucial fact shows how sloppy and illogical the argument is.

peace - but never give up the struggle

che el garbantho

[Igel]

12. The new analysis provides wriggle room.

Saying X is not conclusively proved does not mean "not X" is conclusively proved.

Mitofsky claims WPE accounts for the error. Freeman shows that WPE may not account for the error, and complains that sufficient proof hasn't been given by Mitofsky.

Mitofsky claims his poll doesn't admit proving fraud. Freeman says that since Mitofsky's claims aren't iron-clad, therefore his poll must therefore prove fraud.

Freeman doesn't prove Mitofsky's analysis is false. And, even if he did, that wouldn't prove Freeman's conclusion is true.

[mhr]

15. Ahh, But You Forget The Probabilities - They Are Astronomical

eom

[Mistwell]

17. Astronomical only if the MOE is accurate

Which it is not. We are pretty much certain at this point that the MOE for the early polls was off, perhaps by a lot.

Once that basic assumption is in doubt, all the probability analysis is dead in the water.

[merh]

2. Yeah, and I believe everything posted on the internet

especially when posted by "mystery" people after the fact (and with time to create excuses and rationalizations).

Maybe like Moyers told the Ohio 4, this is not evidence, just hearsay, not even attempts to be legal affidavits or "sworn testimony".

:shrug: Internet is full of mysteries and lies.

[Helga Scow Stern]

4. Yes, suddenly produced as evidence and fact,

when they ignore much more valid data sitting in front of them, time after time. Mystery Pollster's credibility is sinking with me. He just doesn't want to look at the facts...except in a very biased way. His whole argument is always based on "Anything but fraud." I am beginning to wonder if he is on the administration's payroll.

[RaulVB]

11. This is a "smoke screen"

Irrelevant.

[dzika]

41. Check the source of this...

Isn't this guy paid by the media? Doesn't he have a history of being a media/pollster yes-man? In fact, hasn't he already stated that he is mostly in agreement with the explanation in Mitofsky's report?

Alan Combs says he's a registered Democrat too.

[RaulVB]

42. Blumenthal's agenda is clear, indeed (n/t)

[RaulVB]

13. Yep, I don't believe anything posted on the internet (n/t)

[BrklynLiberal]

5. All of this does not change the FACT that tens of thousands, and more

probably, millions of voters were intimidated, suppressed, turned away, frustrated, left out, had their registrations torn up or thrown away, or had their votes stolen or their right to vote denied in any number of ways.
Even if the exit polls were totally wrong, that does not in any way affect the truth of the voter fraud that took place all over this country during the past 4 years, and perhaps even longer.

[New Earth]

6. NOT VOTER FRAUD

except for the FEW cases where a voter committed the fraud.

It is ELECTION fraud. The election was fraudulent, not the voters.

please read these
http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=306090&mesg_id=306090

http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=306119&mesg_id=306119

[BrklynLiberal]

21. ELECTION fraud ELECTION fraud ELECTION fraud. sorry.

:D I will learn that!!!

[Mistwell]

18. Indeed

And that has always been my point - focus on the evidence of actual fraud, and stop focusing on the theoretical, flawed exit poll analysis that niether proves nor disproves ANYTHING.

[kster]

7. You may be

right thats why we need a full blown investigation .

[Wilms]

8. Thank God for Dirty Tricks.

They've proven, once and for all, that exit polls should be ignored.

[BrklynLiberal]

9. Opposing view?

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203x309093

[RaulVB]

10. This is old and ridiculous

Blumenthal's agenda and your own are just beyond the pale.

I could give 5.000 anecdotes that would reresent the opposite view.

Simply ridiculous and quite tiring, but keep it up!

I'll be here.

[Mistwell]

19. Old? It's from yesterday, and the interviews are all new

If by old you're speaking in nanoseconds, then you're right. But, for this issue, it's brand new.

[RaulVB]

20. This is old, a very old tale

Blumenthal will be in trouble if he pushes this "construction".

He HAS NOTHING, and I'm glad to see him trying this "childish" argumentation.

[tandem5]

14. God be praised!

Mitofsky's contorted and funky theories have been confirmed by anecdotal evidence! ...And just in the nick of time no less! Damn all people under 30 - They're stupid, stupid people!

[skids]

16. It would be nice if he had bothered to correlate...

(apologies for mincing Dem and Rep terms vs voting behavior, but
it is so much easier to write that way.)

...which of the accounts comes from which of the poll workers. Like, was the person jamming a survey into the box in a highly Republican precinct or a highly Democratic precinct?

Keep in mind, folks, that even if the "Democratic non-response" were a valid explanation, and it isn't, that it is just an aggregate effect. Not all Republicans were evasive and not all Democrats were enthusiastic. There would have been enthusiastic Republicans, and evasive Democrats as well, regardless of how the groups would have averaged.

So, while this guy says he's all up for more research, his anecdote seems specifically contrived to prove one and only one point: that there was room for selection bias based on the respondent's willingless to participate. If he wants me to take his anecdote seriously, sorry, same thing as Mitofski: raw data please. How were
those errors distributed between your exurbs, inner-city, etc?

The one point he does make that *is* interesting is that the accuracy of the completion rate statistics themselves is in question. What effect would that have? If there was any noticable actual error introduced by the identified problems in the selection process, and that error was not randomly distributed, it would have the effect of giving better completion rates to precincts with more error than they deserved. Since they are correlated anyway, this would only serve to magnify the quantity of the observed effect, and wouldn't really affect the conclusions drawn that suggest the "nonresponse" theory is bogus. And note the "if". There's no way to tell who these errors favored, and how much they contributed to the overall error.

[Karenca]

22. Methinks thou doth protest too much.

:hi:


:dem: :dem: :dem: :dem: :dem: :dem:
KERRY!
KERRY!!
KERRY!!!
KERRY!!!!!

[IndyPriest]

23. Opposite anecdote posted here weeks ago

I remember reading the story in this forum of an exit pollster who happened to wander into the same bar as a couple of our DUers after the election. He looked sad and frustrated. They asked him why. He related that he had just come from a small gathering of other exit pollsters who could not believe the outcome of the elction. WHy? Because they were absolutely certain they had gotten the exit poll right, absolutely nailed it. Or so they had thought. Anyone else remember seeing that story?

Well, so now we're going to have battling anecdotes, is that it? SHould I dig back into the archives to find this guy, find his fellow poll takers, do some interviews with them and compare notes?

[Goldeneye]

24. Did that really happen?

[IndyPriest]

25. Yes. Absolutely. n/t

[Goldeneye]

26. go search, go search!!!

I'd love to see that.

[Mistwell]

28. Sigh

I tell people I am not making a generalization from this, and just asking that we look further into it and get more interviews. I predict people will ignore what I just said and claim I AM making a hasty generalization. And there you go and prove I am write...totally ignoring what I wrote, and claiming I said exactly the opposite of what I wrote.


Sigh...

[tandem5]

32. I think the parenthetical caveat was kind of nullified...

by the whole firewood statement. Inflammatory and flammable... inflammabletory, if you will.

[Mistwell]

38. All I ask is that we investigate the exit pollsters

...and get a lot more interviews from them to find out if these incidents are representative or not.

Is it really so inflammatory for me to want to look deeper into how exit polls are conducted, and see if we can improve them?

[tandem5]

40. absolutely, but...

Mitofsky must publicly release all the pertinent information and raw data so that the investigation can be done independently and by multiple organizations.

[anaxarchos]

35. Some of my best friends are black... really! n/t

[sunnystarr]

27. Assuming that the premise is true ....

How would that differ from any other exit poll ever conducted??? I don't see anything that would be unique to the '04 election. If we accept this as true, then all exit polls are wrong and pollsters are receiving money for 'tried and false' polls and should be refunding monies for every poll they've conducted.

[Mistwell]

I agree with you 100%

Given that ALL prior early exit polls, the ones that were not adjusted by the actual vote, WERE actually incorrect, then you are totally correct.

I honestly and sincerely do not know why people pay for these things. They have never, NOT EVEN ONCE, been anywhere near the margin of error when compared to the actual result, prior to "weighting" by the actual vote itself.

My best guess is that, thought they don't work, they still give you more information than no information at all. But really, given their history of ALWAYS being wrong, I personally wouldn't be shelling out a dime for them.

You can read about it here:

http://www.tcf.org/publications/pow/nov17_2004.pdf

(Warning, this IS a PDF)

"The exit polls have always drawn samples that are off the real world results and have always had to be corrected (weighted) to
eliminate bias, reflect new turnout patterns, and, in the end, just flat-out conform to the election results."

Duke University attempted to analyze why the exit polls are always off, and here is their study (again, warning, it's a PDF):

http://www.duke.edu/~mms16/non_response2000.pdf

In the end, I agree with you "all exit polls are wrong and pollsters are receiving money for 'tried and false' polls and should be refunding monies for every poll they've conducted."

[fasttense]

29. Anecdotal evidence is not proof.

Freeman and numerous other Phd's state the Mitofsky's claims about over reporting democrats provides no proof of that argument. Though Mitofsky says they over sampled Democrats, he provides no statistical information to prove it. Just like your post here.

In fact, if you actually look at the numbers Mitofsky provides in his report you will see that more people in Repuke dominate precincts completed the poll than did in Democratic dominant precincts.

Sorry but your claim does not hold water.

[Mistwell]

30. Please read it

I wrote: "It's a VERY small sample, and I am not currently claiming this is representative (though I feel 100% positive that some people in this thread will ignore what I just said and claim I am trying to make a hasty generalization). But it's enough to want to interview a lot more of these exit pollsters. "

you wrote: "Anecdotal evidence is not proof...etc..."

Thanks for, once again, proving me right. Every time I've had faith that people will actually read what is written before responding, I am shown to be incorrect in having that faith. So I gave up, and now tell you up front that not reading what is written is apparently the norm for some around here, and I was correct to be certain that some people (like yourself) will try to claim that I am trying to make a hasty generalization.

By the way, if you're interested, both of Freeman's reports have been fairly well refuted by Mystery Pollster, in detail.

[Carolab]

31. Look at the locations of the pollsters here in Minnesota

"The geographic distribution of the four interviewers was as follows: one outer exurbia, one inner ring suburb, one "exclusive" upscale suburb and one precinct in an ethnically and economically diverse Minneapolis neighborhood."

The exurban locations and "upscale" suburban areas are heavily Republican here. The inner ring suburb and the ethnically and economically diverse Minneapolis neighborhood would represent more of a "mix" of Democratic and Republican voters.

This in no way explains Mitofsky's higher Kerry voter participation versus Bush voter participation.

[Mistwell]

33. It does

If you read what the interviewers said...that one appeared to be from MoveOn.org, that one seemed to be scaring off white males, etc...that would tend to explain higher Kerry voter participation versus Bush voter participation.

It's not PROOF of anything. However, it's enough for me to want to support a full investigation that would include interviews from a LOT more exit pollsters. I have no idea if these experiences were representative of all the exit pollsters, or anomolies. But, I think we have enough to want to know more.

[Carolab]

36. That one that "appeared to be from Move On"...

where were they stationed? In a location where there were LOTS of Republican white male voters who would have been scared off?

FYI, I was a MoveOn GOTV volunteer and I manned the check-in station at my precinct. We were no where near the polling place and there were plenty of other locations away from our table where a pollster could have stood that would not "scare white male voters away". In fact, in Minnesota on election day, a bunch of Republican operatives were busy making a stink on public radio that MoveOn volunteers were standing "too close to the polls" or handing out Kerry-Edwards literature, and this was UNTRUE. We even had a 100-foot string that we stretched out from the door of the polling place to our stand on the boulevard to prove that we were 100 feet away. And, rest assured, we were NOT handing out any "literature". In fact, this was CLEARLY against the rules. MoveOn tables were set up ONLY to provide for "check-in" of voters who had said they were going to vote and who agreed to check in with us if/when they did.

[Mistwell]

37. Here is what it said

Exit pollsters are ALSO 100% away. Exit polls must also obey the 100 ft. rule, as they are conducted by a private company, not a publically approved poll. So, in at least this particular case, the exit pollster was stationed right next to MoveOn.Org. I can certainly see that, for that particular polling station, Republicans would almost certainly be scared off from taking the exit poll.

"...a contingent of folks from MoveOn.Org was stationed right next to her at the 100-ft line. This made her appear from a distance as being connected with them, and being forced to stand 100 ft from the polls, people were able to easily turn aside to avoid her and MoveOn".

The "scaring off white male voters" was an unrelated incident to the one detailed above. That person "...had perhaps the most advantageous placement of any of my four students (she was indoors at the only entrance/exit and had full cooperation from the staff on-site)." That claim at least implied racism on some level from the white male voters, and had nothing to do with MoveOn.org, but just the appearance of the exit pollster (african-american) and her approach (which apparently got better the longer she had practice approaching voters and refined her sales pitch).

It is not at all hard for me to believe that the attitudes of some Republicans (perhaps subconsciously for some) would cause them to not want to speak with a black female exit pollster.

[Carolab]

39. Again, where were they placed?

It is not at all hard for me to believe that the attitudes of some Republicans (perhaps subconsciously for some) would cause them to not want to speak with a black female exit pollster.

But were there a LOT of white male Republicans at this location? And were they all turned off? And she herself said that the situation improved as she improved her "pitch", thereby negating the "racial argument".

[Carolab]

43. Also read this new report from uscountvotes.org re: exit poll response

http://uscountvotes.org/ucvAnalysis/US/USCountVotes_Re_Mitofsky-Edison.pdf

[Mistwell]

44. Those are fine questions

...that I wish we had answers for. But, we can only get answers if we ask for more interviews. That's a pretty hard thing to get if everyone is going around saying the exit polls are done just fine, no need to look into it, nothing going on here, move along...

I don't think it negates the racial claim. I think someone who is mildly racist can get past it, if approached better. If left to the mildly racist voter, I can see them choosing to move past the pollster. However, if this voter is approached in a manner that is more disarming, then I can see more of these voters getting past it and taking the interview. It doesn't negate the racism, it just gets better at overcoming it.

[SaveAmerica]

34. People do make mistakes, to say that all the pollsters were perfect

would be stupid. At the same time so many pollsters would have to have made the exact same mistakes in only battleground states for this to have any meaning against the accuracy of early exit polls.

[necso]

45. Let me describe the sampling problem as I see it.

There is a huge container of balls. These balls are of different colors, each color representing some voting "group", whose further decomposition provides no more (significantly) useful information about the voting behavior of the group. Each ball also has a mark on it to indicate its "vote" (Kerry/*/other).

We do not know the relative proportions of the balls (or even the number of colors), nor do we know the relative proportions of the various marks for any color ball. -- And we know that the balls are not uniformly distributed throughout the container.

Short of counting all the balls, what formula does one use for coming up with a sample that adequately represents the whole? -- Or short of this, how does one take some (easy to take perhaps) sample then weight it to adequately represent the whole?

Of course, the way around this difficulty is to throw out this model or its basic assumptions -- but try to solve the problem without doing so.

...And I realize that this response is rather skewed to your original post. But is is useful to look at the larger problem before even looking at the sampling methodology used -- much less the various difficulties or failures in its actual practise.

Besides, it is to be expected that the originator of the poll is not going to fault basic methodology, but rather the details. But then, what can this person be expected to say: "I really don't have a clue, but please keep sending me your money."?