HomeLatest ThreadsGreatest ThreadsForums & GroupsMy SubscriptionsMy Posts
DU Home » Latest Threads » Forums & Groups » Main » General Discussion (Forum) » Understanding margins of ...

Sat Oct 31, 2020, 10:42 AM

Understanding margins of error

I was listening to Lawrence O'Donnell earlier this week discussing some polls, and talking about what the polls said when you take into account their margins of error.

What he said was roughly correct, in a non-mathy sort of way, but also off enough to be a bit misleading.

First of all, margins of errors don't say anything about how good a polling model is, so it's not an expression of faith in the model. What the margin of error tells you is, if the model is good, how much would random variation in the particular people who get sampled typically throw off how well the model reflects reality.

Also, if a poll says, say, Biden is likely to get 52% of the vote, with a margin of error of ±3%, that does NOT mean Biden is just as likely to get 49% as he is to get 52% -- it's a "bell shaped curve" -- and values near the middle are favored over values near the edges of the given range.



Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 49-55 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range.

Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%.

What O'Donnell got wrong, however, is speaking about this situation as if, "Hey, this essentially is a tie", since the margins of error overlap. Nope! This would still be a poll that looks much better for Biden than it does for Trump.

Without getting into the exact math, for Trump to actually be ahead in a poll like that requires that Trump pushes well into the more unlikely upper end of his range at the same time Biden happens to fall into the more unlikely lower end of his range. The sampling errors aren't very likely, however, to line up in just that way very often.

20 replies, 736 views

Reply to this thread

Back to top Alert abuse

Always highlight: 10 newest replies | Replies posted after I mark a forum
Replies to this discussion thread
Arrow 20 replies Author Time Post
Reply Understanding margins of error (Original post)
Silent3 Oct 31 OP
Wounded Bear Oct 31 #1
PCIntern Oct 31 #2
DonaldsRump Oct 31 #3
jimfields33 Oct 31 #6
BComplex Oct 31 #4
Spazito Oct 31 #5
beachbumbob Oct 31 #7
Silent3 Oct 31 #9
beachbumbob Oct 31 #11
DrToast Oct 31 #8
Fiendish Thingy Oct 31 #10
Silent3 Oct 31 #12
Fiendish Thingy Oct 31 #16
Silent3 Oct 31 #18
Fiendish Thingy Oct 31 #19
Silent3 Oct 31 #20
grantcart Oct 31 #13
grantcart Oct 31 #14
SharonClark Oct 31 #15
treestar Oct 31 #17

Response to Silent3 (Original post)

Sat Oct 31, 2020, 10:44 AM

1. K & R...for visibility...nt

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 10:47 AM

2. You explained much better than I could

Which is why I never attempted a post like yours. K and r.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 10:47 AM

3. Am so glad there are folks like you on DU

Statistics was not something I excelled at school...and I run away from it today.

K&R!

Reply to this post

Back to top Alert abuse Link here Permalink


Response to DonaldsRump (Reply #3)

Sat Oct 31, 2020, 11:06 AM

6. You are not alone

I got a D in statistics and the school only accepted C- and above. Took it again next semester with same professor and got a C-. I’m convinced he passed me just to get rid of me. I think they let a few lost causes get a participation grade. Lol.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 10:57 AM

4. Thanks Silent3! That really helps me to understand the polls.

Good visuals, too.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 10:58 AM

5. Thank you very much, very informative! n/t

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 11:07 AM

7. to me the only use of polls is to indicate the trends and why I was skeptical in 2016 as everything

 

showed Hillary was dropping the moment Comey kneecapped her campaign. Nothing of the like is indicated in 2020, the exact opposite

Reply to this post

Back to top Alert abuse Link here Permalink


Response to beachbumbob (Reply #7)

Sat Oct 31, 2020, 11:16 AM

9. The trends were a factor, and so were modeling errors

With both candidates having such high negatives (Clinton very unfairly so), the likely voter models were probably shot to hell.

Even so, I think 538 still had Trump at about a 30% chance of winning, so leaning toward Clinton, but far from a sure thing. I was very worried at the time, because so close to a 1-in-3 chance of Trump winning was terrifying to me.

And as it turned out, not just from the way the election went, but how Trump has behaved as President, I was right to be terrified.

This time around my only significant concerns are blatant electoral fuckery, with Republicans trying to steal the election with voter suppression and court battles to invalidate votes.

Take those concerns away, and I'd be personally be 99% confident of a Biden victory.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Reply #9)

Sat Oct 31, 2020, 11:25 AM

11. the 2016 polls were more correct than not and definitely NOT Wrong

 

in any way. The Comey hit was all it took to stop any Hillary momentum and in fact caused a reversal. In 2020 every element of this election is different from 2020

- Biden does not have high unfavorables
- is favorables are great
- he is NOT a flawed candidate
- there is NO fracture in the democratic base
- there is NO bernie bro movement, russian influence in social media has been a failure
- we have 4 years of trump's failures
- we democrats are on the hunt and 2018 is the outcome as will 2020
- we have trump lying and deflecting with every sentence he speaks and only his rabid base listens


this no 2016 and the outcome will reflect it

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 11:11 AM

8. The term "statistical tie" needs to die a horrible death and never be heard from again

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 11:24 AM

10. You discounted the most important part:

Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 49-55 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range.

Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%.


The scenario above is indeed an example of a “statistical tie”, and shouldn’t be discounted, especially when compared to a poll that isn’t a statistical tie.

For example, a poll from an A rated pollster that gives Biden a 9pt. Margin, but has a 5% MOE is a statistical tie (double the MOE to get the range of possible results between two candidates w/95% confidence). Contrast that with an A rated poll that has Biden with an 8pt. Margin, but only a 3% MOE- the latter represents a stronger, more stable representation of the actual outcome.

You said O’Donnell was wrong to point out the statistical tie - he was not, it was good journalism IMO.
Of course, following the trend/average will give the most accurate results overall, but one should not overlook the significance of the MOE.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Fiendish Thingy (Reply #10)

Sat Oct 31, 2020, 11:40 AM

12. As some else pointed out, the term "statistical tie" is, in and of itself, a problem.

Whoever leads in a poll (given the poll is based on a good model) has a higher chance of winning. Period. No matter how narrow the lead, nor how wide the MoE, the odds never convert to 50/50 when you take the MoE into account.

If there were a well-understood meaning for "statistical tie", like "this poll only gives the leading candidate a 60% chance of winning", that would different. But there is no such well-understood meaning, so the terminology, especially in the hands of people who aren't good at communicating statistics, can be very misleading.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Reply #12)

Sat Oct 31, 2020, 01:07 PM

16. Would you agree that a lead outside the MOE is better than one inside the MOE?

Even if the lead outside the MOE is smaller than the one inside the MOE?

In Clinton’s 2016 Rust Belt polls, her lead was within MOE, and most, but not all, of Biden’s Rust Belt leads are outside the MOE (and thankfully much larger than Clinton’s, and most significantly, above 50%). Most of Biden’s other swing state/Sun Belt polls are within the MOE, so I expect those races will be closer than MI, WI and MN.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Fiendish Thingy (Reply #16)

Sat Oct 31, 2020, 01:21 PM

18. Certainly it's better to be outside the MoE

And it's even better still to be five times outside the MoE.

That doesn't make the phrase "statistical tie", for something inside the MoE, any less misleading, however.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Reply #18)

Sat Oct 31, 2020, 01:44 PM

19. Nevertheless, there is a significant difference between leads outside/inside the MOE Nt

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Fiendish Thingy (Reply #19)

Sat Oct 31, 2020, 04:24 PM

20. Significant in what way?

The wider the spread between two candidates, the more certain the apparent leader will win. But there's no special, magical change that happens when you transition from inside the MoE to outside the MoE. It's a smooth continuum.

I don't know how to do the exact calculation off the top of my head, but if the MoE is ±3@95% for both polling values for two candidates, and their individual scores are 3% apart, the apparent leading candidate has something like an 80-90% chance of winning, because of the way the two bell curves interact.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Fiendish Thingy (Reply #10)

Sat Oct 31, 2020, 11:42 AM

13. O'Donnell and all the others who conflate MOE and statistical tie are wrong, the OP is correct


Further O'Donnell, who I love, is not a journalist and doesn't exercise journalism. He is an opinion commentator, it is as different as reading the opinion page and the rest of a newspaper.

Using a poll of poll like 538 which also weighs pollster performance a candidate with a 52 - 47 lead and a 5% MOE isn't in a statistical tie.

There is as much chance for the outcome to be 56-44 for the leading candidate as it is for there to be a 50-50 tie.

In addition, if there have been 10 polls showing the lead candidate in the lead then there is significantly less chance that the MOE will move 100% to the lesser candidate.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 11:48 AM

14. Excellent post


Morning Joe is another frequent "statistical tie" offender.


I would be very interested if you were interested in expanding the OP for a second edition on how a 538 poll of polls also narrows the MOE and how a race where one candidate has led in 10 straight polls would be even less likely to result in a tie when a 5% margin is within a 5% MOE.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 12:04 PM

15. Thank you! I'm bookmarking this post for future reference.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Silent3 (Original post)

Sat Oct 31, 2020, 01:09 PM

17. all pundits need it to be close

a dead heat. They want you to watch on election night.

Reply to this post

Back to top Alert abuse Link here Permalink

Reply to this thread