Why do Clinton and Trump do better in crappy-method polls and worse in established-method polls?

[Attorney in Texas]

The fact that Sanders has an 8.6% aggregate lead in New Hampshire at Real Clear Politics, but only a 4.3% aggregate lead in New Hampshire according to Pollster is due, in part, to the stricter methodology requirements imposed over at Real Clear Politics.

Nate Silver over at 538 discussed this phenomenon in connection with Trump's poll numbers (how he does better in poor methodology polls and does worse in polls that employ more accepted methods), and the same is true for Clinton's polling.

If you look at a graph of all the polls (the best, the good-but-flawed, and the crap), Sanders has an aggregate 4.3% lead in New Hampshire:



If you focus on likely voters (drop the polls of all adults and polls of all registered voters), Sanders' aggregate lead grows to 4.9%:



If you drop the robo-call polls, which employ the most doubtful of methodologies, Sanders' aggregate lead grows again to 7.2%:



If you drop the pollsters who have not conducted at least 4 polls in New Hampshire (drop the one-off and infrequent pollsters), Sanders' aggregate lead grows to 8%:



If you narrow down to the live phone (landline and cell) polling of the most frequent pollster, Sanders' lead peaks at 10.6%:



And this 10.6% live phone polling lead for Sanders is bigger than the internet-polling 10.1% lead in the CBS News polls:



Contrast this with a graph that includes only robo-call polls, which give Clinton the lead in New Hampshire (and make the aggregation of polls look like a closer race than the more reliable polls suggest):



The same trend holds true in Iowa.

If you look at an aggregation of all the polls (good, bad, and ugly), Clinton appears to have a 15% aggregate lead:



Again, if you drop the robo-polls and the registered voter polls and focus on the most frequent pollsters who have done the most polling in Iowa this cycle, Clinton's aggregate lead is cut in half to 7.9%:



Conversely, if you focus only on the robo-call polls in Iowa (which are affecting the aggregate pollster graphs), you would think Clinton has a 21.2% lead:



You have to ask -- why do Clinton and Trump do so much better in bad-methodology polls and so much worse in polls employing a reliable methodology?

[JaneyVee]

1. All polls matter.

Except for online polls.

[Attorney in Texas]

14. The robo-call polls are the worst

[Proserpina]

2. Please stop! I took the final Thursday, and I'm done with statistics for life!

my poor head!

(I admire the effort you put into the post, but basically you are calling the primary for Bernie, right? Carry on, then! )

[Attorney in Texas]

11. Feel free to ignore all polling (especially all national polling) until mid-February. Does that help

your head?

Iowa polling is notoriously inaccurate because Iowa is a caucus state and too much of the caucus results are dependent on enthusiasm which is almost impossible to factor into the polling.

New Hampshire polling is notoriously inaccurate because it is so often influenced by the results in Iowa that you cannot really predict any results in the primary based on polling conducted without the benefit of the post-Iowa effect calculated into the result.

As unreliable as Iowa and New Hampshire is, polling in the other states is even less reliable.

As unreliable as the polling is in post-Iowa and post-New Hampshire states is, it is still many times more predictive than national polling which has almost zero predictive value.

Polling is crack for political junkies. Just say NO.

Better?

[Proserpina]

13. Much, thanks!

[virtualobserver]

3. This all adds up to another painful shock for Team Hillary on caucus night in Iowa.

Even the best polls may not pick up the underlying intensity for Bernie.

[Attorney in Texas]

7. Agreed. The impact from the Iowa caucus has traditionally been measured in relation to expectations.

Many candidates have come out of Iowa with strong momentum even when they did not win the caucus but merely beat expectations.

If Clinton's supporters want to raise expectations for her supporters so a narrow Sanders victory is read as a huge upset and a narrow Sanders loss is seen as huge underperformance far short of expectations for the Clinton campaign, I'm OK with that.

I guess we might see a narrow Sanders win in Iowa, a bigger win for Sanders in New Hampshire, and then the rest of the race will have to be reassessed in light of those results, but the voting is more than a month away and so all predictions are little more than guesses.

[JoePhilly]

4. Are we to believe that you gathered all this data ...

... and yet lack the ability to draw specific conclusions on your own, and then spell them out?

I call Bullshit.

You gathered the data, selecting these charts, excluding others.

But you want others to explain it to you.

Take a stats class and come back.

[Attorney in Texas]

5. I gathered the data because I was curious. I asked the question because I don't know the answer.

There is a discussion (in the context of Trump polling which shows the same phenomenon) at 538, and they also admitted they don't know why there is a consistent pattern of very different results from polls using the established methodology and robo-polls using less established methodologies, but they threw out some possibilities.

For example, the robo-polls may be oversampling non-voters, but the live-interview polls may be understating the support for a candidate with high negatives because people may be more inclined to vote for such a candidate than to admit supporting that candidate in a live-phone interview.

PS - I've taken a stats class. I did pretty good.

[Chan790]

6. Don't mind the chattering mass of Clintonites.

They get really bent out of shape whenever someone points out that Hillary is basically not the indomitable front-runner they want to claim she is. She's going to get her ass handed to her in IA and NH and when she under-performs her polling but still wins in SC, her candidacy is going to be over before Super Tuesday even happens. The eulogies for Clintonism will be heard on all the Sunday shows and the headlines will read "Why the Clinton campaign failed? Twice considered inevitable, twice a participation award and an early exit." It'll take her another two months and several primaries again but she's likely done before she's barely out of the gate.

Then, I'm going to gloat because the thing I want to see most in the entire world is the look on her face when Hillary knows her career in politics and government is over forever; that's she's though, that there is no third act or place of prominence on the stage for her and she just doesn't matter.

[Attorney in Texas]

8. I just can't see us winning a general election with a candidate whose dishonesty/untrustworthy

rating by every potential demographic other middle-aged and older Democrats is underwater.

I prefer Sanders ideologically, but I fear a Clinton candidacy on electability concerns for the general election.

[Attorney in Texas]

15. Please, by all means, raise the expectations for Clinton in Iowa. But remember that the importance

of expectations cannot be overstated.

Remember that Hart in 1984 won New Hampshire based on momentum from a better-than-expected loss in Iowa, Harkin in 1992 collapsed in New Hampshire after a worse-than-expectations win in Iowa, and Buchanan in 1996 won New Hampshire based on momentum from a better-than-expected loss in Iowa.

Often, beating expectations is more important than winning so please help Sanders as much as you can by raising expectations for Clinton in Iowa.

[JoePhilly]

9. You mean you did "pretty well".

[Attorney in Texas]

12. I did even better in English.

[Attorney in Texas]

18. By the way, the period comes within the quotation marks and not after the closing quotation mark.

[Betty Karlson]

16. Thank you so much for this break-down.

These lessons in statistic analysis are very valuable. Please keep up the good work.

[Attorney in Texas]

17. I just hope everyone working on the ground for the Sanders campaign understands that their sacrifice

and their hard work is paying off no matter what the Crown Her Now! squad says.

[Attorney in Texas]

19. Today's national robo-call horse race poll by Emerson College is the perfect example of junk polling

where Clinton polls more than TWICE as well as she polls in live phone polling.