Sat Oct 31, 2020, 10:42 AM
Silent3 (10,384 posts)
Understanding margins of error
I was listening to Lawrence O'Donnell earlier this week discussing some polls, and talking about what the polls said when you take into account their margins of error.
What he said was roughly correct, in a nonmathy sort of way, but also off enough to be a bit misleading. First of all, margins of errors don't say anything about how good a polling model is, so it's not an expression of faith in the model. What the margin of error tells you is, if the model is good, how much would random variation in the particular people who get sampled typically throw off how well the model reflects reality. Also, if a poll says, say, Biden is likely to get 52% of the vote, with a margin of error of ±3%, that does NOT mean Biden is just as likely to get 49% as he is to get 52%  it's a "bell shaped curve"  and values near the middle are favored over values near the edges of the given range. Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 4955 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range. Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%. What O'Donnell got wrong, however, is speaking about this situation as if, "Hey, this essentially is a tie", since the margins of error overlap. Nope! This would still be a poll that looks much better for Biden than it does for Trump. Without getting into the exact math, for Trump to actually be ahead in a poll like that requires that Trump pushes well into the more unlikely upper end of his range at the same time Biden happens to fall into the more unlikely lower end of his range. The sampling errors aren't very likely, however, to line up in just that way very often.

20 replies, 736 views
20 replies  Author  Time  Post 
Understanding margins of error (Original post) 
Silent3  Oct 31  OP 
Wounded Bear  Oct 31  #1  
PCIntern  Oct 31  #2  
DonaldsRump  Oct 31  #3  
jimfields33  Oct 31  #6  
BComplex  Oct 31  #4  
Spazito  Oct 31  #5  
beachbumbob  Oct 31  #7  
Silent3  Oct 31  #9  
beachbumbob  Oct 31  #11  
DrToast  Oct 31  #8  
Fiendish Thingy  Oct 31  #10  
Silent3  Oct 31  #12  
Fiendish Thingy  Oct 31  #16  
Silent3  Oct 31  #18  
Fiendish Thingy  Oct 31  #19  
Silent3  Oct 31  #20  
grantcart  Oct 31  #13  
grantcart  Oct 31  #14  
SharonClark  Oct 31  #15  
treestar  Oct 31  #17 
Response to Silent3 (Original post)
Sat Oct 31, 2020, 10:44 AM
Wounded Bear (42,175 posts)
1. K & R...for visibility...nt
Response to Silent3 (Original post)
Sat Oct 31, 2020, 10:47 AM
PCIntern (20,991 posts)
2. You explained much better than I could
Which is why I never attempted a post like yours. K and r.

Response to Silent3 (Original post)
Sat Oct 31, 2020, 10:47 AM
DonaldsRump (4,913 posts)
3. Am so glad there are folks like you on DU
Statistics was not something I excelled at school...and I run away from it today.
K&R! 
Response to DonaldsRump (Reply #3)
Sat Oct 31, 2020, 11:06 AM
jimfields33 (4,409 posts)
6. You are not alone
I got a D in statistics and the school only accepted C and above. Took it again next semester with same professor and got a C. I’m convinced he passed me just to get rid of me. I think they let a few lost causes get a participation grade. Lol.

Response to Silent3 (Original post)
Sat Oct 31, 2020, 10:57 AM
BComplex (4,959 posts)
4. Thanks Silent3! That really helps me to understand the polls.
Good visuals, too.

Response to Silent3 (Original post)
Sat Oct 31, 2020, 10:58 AM
Spazito (36,137 posts)
5. Thank you very much, very informative! n/t
Response to Silent3 (Original post)
Sat Oct 31, 2020, 11:07 AM
beachbumbob (9,263 posts)
7. to me the only use of polls is to indicate the trends and why I was skeptical in 2016 as everything
showed Hillary was dropping the moment Comey kneecapped her campaign. Nothing of the like is indicated in 2020, the exact opposite

Response to beachbumbob (Reply #7)
Sat Oct 31, 2020, 11:16 AM
Silent3 (10,384 posts)
9. The trends were a factor, and so were modeling errors
With both candidates having such high negatives (Clinton very unfairly so), the likely voter models were probably shot to hell.
Even so, I think 538 still had Trump at about a 30% chance of winning, so leaning toward Clinton, but far from a sure thing. I was very worried at the time, because so close to a 1in3 chance of Trump winning was terrifying to me. And as it turned out, not just from the way the election went, but how Trump has behaved as President, I was right to be terrified. This time around my only significant concerns are blatant electoral fuckery, with Republicans trying to steal the election with voter suppression and court battles to invalidate votes. Take those concerns away, and I'd be personally be 99% confident of a Biden victory. 
Response to Silent3 (Reply #9)
Sat Oct 31, 2020, 11:25 AM
beachbumbob (9,263 posts)
11. the 2016 polls were more correct than not and definitely NOT Wrong
in any way. The Comey hit was all it took to stop any Hillary momentum and in fact caused a reversal. In 2020 every element of this election is different from 2020
 Biden does not have high unfavorables  is favorables are great  he is NOT a flawed candidate  there is NO fracture in the democratic base  there is NO bernie bro movement, russian influence in social media has been a failure  we have 4 years of trump's failures  we democrats are on the hunt and 2018 is the outcome as will 2020  we have trump lying and deflecting with every sentence he speaks and only his rabid base listens this no 2016 and the outcome will reflect it 
Response to Silent3 (Original post)
Sat Oct 31, 2020, 11:11 AM
DrToast (6,334 posts)
8. The term "statistical tie" needs to die a horrible death and never be heard from again
Response to Silent3 (Original post)
Sat Oct 31, 2020, 11:24 AM
Fiendish Thingy (6,110 posts)
10. You discounted the most important part:
Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 4955 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range.
Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%. The scenario above is indeed an example of a “statistical tie”, and shouldn’t be discounted, especially when compared to a poll that isn’t a statistical tie. For example, a poll from an A rated pollster that gives Biden a 9pt. Margin, but has a 5% MOE is a statistical tie (double the MOE to get the range of possible results between two candidates w/95% confidence). Contrast that with an A rated poll that has Biden with an 8pt. Margin, but only a 3% MOE the latter represents a stronger, more stable representation of the actual outcome. You said O’Donnell was wrong to point out the statistical tie  he was not, it was good journalism IMO. Of course, following the trend/average will give the most accurate results overall, but one should not overlook the significance of the MOE. 
Response to Fiendish Thingy (Reply #10)
Sat Oct 31, 2020, 11:40 AM
Silent3 (10,384 posts)
12. As some else pointed out, the term "statistical tie" is, in and of itself, a problem.
Whoever leads in a poll (given the poll is based on a good model) has a higher chance of winning. Period. No matter how narrow the lead, nor how wide the MoE, the odds never convert to 50/50 when you take the MoE into account.
If there were a wellunderstood meaning for "statistical tie", like "this poll only gives the leading candidate a 60% chance of winning", that would different. But there is no such wellunderstood meaning, so the terminology, especially in the hands of people who aren't good at communicating statistics, can be very misleading. 
Response to Silent3 (Reply #12)
Sat Oct 31, 2020, 01:07 PM
Fiendish Thingy (6,110 posts)
16. Would you agree that a lead outside the MOE is better than one inside the MOE?
Even if the lead outside the MOE is smaller than the one inside the MOE?
In Clinton’s 2016 Rust Belt polls, her lead was within MOE, and most, but not all, of Biden’s Rust Belt leads are outside the MOE (and thankfully much larger than Clinton’s, and most significantly, above 50%). Most of Biden’s other swing state/Sun Belt polls are within the MOE, so I expect those races will be closer than MI, WI and MN. 
Response to Fiendish Thingy (Reply #16)
Sat Oct 31, 2020, 01:21 PM
Silent3 (10,384 posts)
18. Certainly it's better to be outside the MoE
And it's even better still to be five times outside the MoE.
That doesn't make the phrase "statistical tie", for something inside the MoE, any less misleading, however. 
Response to Silent3 (Reply #18)
Sat Oct 31, 2020, 01:44 PM
Fiendish Thingy (6,110 posts)
19. Nevertheless, there is a significant difference between leads outside/inside the MOE Nt
Response to Fiendish Thingy (Reply #19)
Sat Oct 31, 2020, 04:24 PM
Silent3 (10,384 posts)
20. Significant in what way?
The wider the spread between two candidates, the more certain the apparent leader will win. But there's no special, magical change that happens when you transition from inside the MoE to outside the MoE. It's a smooth continuum.
I don't know how to do the exact calculation off the top of my head, but if the MoE is ±3@95% for both polling values for two candidates, and their individual scores are 3% apart, the apparent leading candidate has something like an 8090% chance of winning, because of the way the two bell curves interact. 
Response to Fiendish Thingy (Reply #10)
Sat Oct 31, 2020, 11:42 AM
grantcart (51,063 posts)
13. O'Donnell and all the others who conflate MOE and statistical tie are wrong, the OP is correct
Further O'Donnell, who I love, is not a journalist and doesn't exercise journalism. He is an opinion commentator, it is as different as reading the opinion page and the rest of a newspaper. Using a poll of poll like 538 which also weighs pollster performance a candidate with a 52  47 lead and a 5% MOE isn't in a statistical tie. There is as much chance for the outcome to be 5644 for the leading candidate as it is for there to be a 5050 tie. In addition, if there have been 10 polls showing the lead candidate in the lead then there is significantly less chance that the MOE will move 100% to the lesser candidate. 
Response to Silent3 (Original post)
Sat Oct 31, 2020, 11:48 AM
grantcart (51,063 posts)
14. Excellent post
Morning Joe is another frequent "statistical tie" offender. I would be very interested if you were interested in expanding the OP for a second edition on how a 538 poll of polls also narrows the MOE and how a race where one candidate has led in 10 straight polls would be even less likely to result in a tie when a 5% margin is within a 5% MOE. 
Response to Silent3 (Original post)
Sat Oct 31, 2020, 12:04 PM
SharonClark (7,257 posts)
15. Thank you! I'm bookmarking this post for future reference.
Response to Silent3 (Original post)
Sat Oct 31, 2020, 01:09 PM
treestar (78,123 posts)
17. all pundits need it to be close
a dead heat. They want you to watch on election night.
