NYT admits partisan non-response is source for Trump's lead in poll

[Fiendish Thingy]

Link to tweet

So they admit a Dem unwillingness to respond to their poll questions results in Oversampling Republicans with no scientific adjustment to correct for their error.

Proof, once again, that the NYT polls should be ignored.

[CaptainTruth]

1. If they know it they should attempt to quantify it & statistically adjust for it.

The fact that they're not implies they're either lazy or they don't understand math & statistics. Or maybe they have an agenda.

[Ms. Toad]

9. They did.

Even though more Republicans responded to this Times/Siena poll, we weight the results by party registration, so the poll still shows the “correct” number of Democrats and Republicans to reflect the country as a whole. This kind of statistical adjustment reduces the chance that sampling errors yield outliers. But it doesn’t necessarily prevent outliers. It would not be enough if it turns out that Mr. Trump’s supporters were simply likelier to respond to the survey, regardless of party. In other words, weighting by party wouldn’t be enough if the higher response among Trump supporters also yielded too many independents who favor Mr. Trump and too many Democrats who favor him.

https://www.nytimes.com/2024/06/26/upshot/biden-trump-poll-outlier.html

[Fiendish Thingy]

28. An unscientific methodology at best

The honest, but less profitable response would be to say “we couldn’t get enough Dems to complete the survey, so our results are unreliable and we aren’t going to publish them”.

Either that, or they could exclude the excess Republican responses, resulting in a smaller sample size and higher MOE.

[Ms. Toad]

34. I don't think you understood what you read.

It is standard to adjust polls to account for over/under responses by mathematically reducing the responses by the over-responding party (as they did in this case). All polls are going to have a disproportionate response - that's why one of the standard questions is party affiliation or, if none, which party do you favor more. The purpose of that question is to be able to "exclude the excess Republican responses" by mathematically adjusting number of republican votes in the poll.

But what this standard statistical adjustment cannot take into account is a motivation skew within the Democrats and Independents: If Democrats and Independents who intend to vote for Biden chose not to respond - and those who intend to vote for Trump chose to respond, it would look like Trump has more support because of the disproportionate Trump responses within the Democrats and Independents. No adjustment of the Republican responders would fix that skew.

[Fiendish Thingy]

36. They didn't simply reduce the over responses

To do so would have shrunk the overall sample size and significantly increased the MOE.

What they did instead is use an unscientific “reweighting” tool to essentially “clone” Democratic responses, theoretically diluting the Republican oversample and balancing out the overall sample.

Oversampling/reweighting is a cost saving measure that impacts the reliability of the poll. It affects many, but not all pollsters, but the practice has been exceptionally egregious by the NYT this year.

This kind of statistical adjustment reduces the chance that sampling errors yield outliers. But it doesn’t necessarily prevent outliers.

Every NYT poll this year has been an outlier compared to the trends of other polls.

Every. Single. Poll.

[Ms. Toad]

38. You are still completely ignoring the concern the potential error the NYT identified:

The motivation of Trump voters to respond - including those Trump voters within the Democratic party and independent voters.

And, as for whether you describe it as cloning Democratic responses, or mathematically reducing the Republican responses, it is a standard, statistically valid, and accepted process.

And even had the responses from Democrats and Republicans been proportional to their representation in the population (so no mathematical adjustment was needed), it still does not address the skew which may have been introduced by the motivational difference between Trump voters and Biden voters to respond to polls - regardless of party affiliation.

[Fiendish Thingy]

41. It is not standard or statistically valid

Reweighting is an opaque, often proprietary tool that varies from pollster to pollster and is not scientific, but instead based on the best guesses to obtain the most valid sample.

If you don’t believe me, go to the Siena website and copy and paste the formula they use to reweight the sample; I’ll wait here.

(hint: you won’t find the formula; the most you will find is some non-transparent reference to some parameters/guidelines they use)

“partisan non-response” doesn’t seem to be as big an issue for other pollsters this year.

[Ms. Toad]

42. Transparency would be helpful -

But the lack of transparency does not inherently make reweighting non-standard or not statistically valid.

[Fiendish Thingy]

43. It does indeed impact reliability.

Reweighting was uncommon in political polling until the past few cycles when it became more difficult, time consuming and expensive to gather a truly random, representative sample.

So, pollsters invented their own reweighting tools, tools that are not standardized across the industry or vetted by independent researchers. They are largely opaque, proprietary instruments implemented out of desperation to attempt to construct a sample that they can say with a straight face is representative.

If the Siena reweighting tool was reliable in constructing representative samples, then why has every one of their national polls this year been an outlier?

Every. Single. Poll.

[Ms. Toad]

44. And the lack of transparency does that?

Hmm. Last I checked, applying an algorithm will produce the same results whether I know what the algorithm is or not.

Lack of transparency does not impact whether the reweighting algorithm is reliable. What it impacts is the ability for you to verify whether it is reliable. Those are two different things. As I said - lack of transparency itself does not inherently impact reliability.

[Fiendish Thingy]

46. Lack of transparency means the reliability of the methodology cannot be independently verified

Basically, the pollster is saying “trust me, would I lie to you?”

[Ms. Toad]

47. I never claimed otherwise.

You were the one who insisted that lack of transparency impacts reliability. As I said - it only impacts the ability to verify the reliability.

[CaptainTruth]

51. Weighting results CAN be scientific, when done properly.

As I noted in another post, to be most accurate (which yes, might not be all that accurate, there's always some uncertainty, hence MOE), the weighting would need to reflect the electorate, that is, the folks who actually vote, & how they're expected to vote. That means developing a detailed model of the electorate, as detailed & accurate as you can, & that's another project with it's own MOE.

Simply using party registration doesn't get you there, I guarantee party registration will not accurately reflect the composition (& most importantly, the votes of) the electorate.

[Fiendish Thingy]

52. AFAIK, there is no standardized methodology for reweighting

And no independent verified research base which to guide reweighting.

It’s essentially just a “guesstimate” than can vary from polling house to polling house, and even individual poll workers.

[CaptainTruth]

54. There's an entire field of mathematics dedicated to exactly things like this.

And yes there is a standardized scientific methodology for it.

Now, do pollsters always use that methodology? No, they frequently don't, & that's the problem.

The problem is not that there's no standardized methodology, there is. The problem is that pollsters don't use it. Math is hard. Flashy click bait headlines are easy.

[Fiendish Thingy]

55. Any links to info on reweighting methodology? Nt

[CaptainTruth]

45. Ok, so I'm an idiot. Not the first time, won't be the last.

[CaptainTruth]

50. But in the most accurate way? "we weight the results by party registration" is not the best way.

Party registration does not necessarily reflect the electorate, especially when there are more registered Independents than Democrats or Republicans. The best way to do it is to model the expected electorate as accurately as possible, using all the data available, & then weight the poll data according to the expected electorate, not just party registration.

When votes are cast I guarantee that "party registration" will not equal "the electorate." Hence, inaccuracy in the poll.

I'm one of those geeks who took way too much math in college, even convinced the dean to let me take more advanced courses in things like stochastic processes & random events (statistics out the wazoo) when I didn't have the prerequisites because I loved the math, it fascinated me. Of course in my free time I was reading books on subatomic particle physics & cosmology (again, statistics out the wazoo) & they gave me the "prereqs" I needed. So, I get frustrated when I see pollsters who don't seem to grasp the basics of data sets & statistics.

[Ms. Toad]

53. Who knows.

I was simply addressing the contention that they didn't adjust at all for oversampling Republicans (so of course Trump would poll higher).

How good their questions were (on the front end - to provide enough information to adjust their data to the expected electorate) and how good the algorithm (on the back end to actually make the adjustment) is a different question.

If you ask the right questions to be able to sort out who is likely to show up at the polls, the adjustment to account for imbalanced sampling is just a bunch of math (and I've got two math degrees - the second in applied math). They did a bunch of math in this poll, but whether they did the right math is part crystal ball, part asking the right questions to be able to sort who is actually likely to vote. One of the polling entities I respond to frequently asks a lot of unrelated questions (some political, some not overtly). Early in the poll they usually ask if I voted in the last election - and they usually ask who I voted for in a race that someone who didn't actually vote probably wouldn't even recognize. That gives them some ability to sort out likely voters based on semi-verified past behavior. Later in the poll - enough later that someone trying to fool the pollster wouldn't necessarily remember how they answered the first questions - they ask about likelihood of voting in the next election (and who I might vote for). And at the end they ask demographic information - including party information (and which way you lean if you indicate you don't belong to a party). With that information, and a bit of a crystal ball, there is a reasonable chance they can simulate the predicted electorate (balanced for age, gender, race, political affiliation - and historical voting pattern). I didn't participate in the poll in question, so I don't know what kind of questions they asked, or what (if anything) they adjusted for beyond party affiliation.

[ananda]

2. I'm telling you, polls are just worthless these days.

Too much money along with corporate and billionaire
greed involved.

[Andy823]

3. I agree

And trump did admit "buying" polls in his first run, I am sure he is doing the same thing now

[AZSkiffyGeek]

29. The poll he bought was for "best New York businessman" or some nonsense

It predated his run.

[SWBTATTReg]

4. And they (NYT) has lost all creditability now. But, I think they were already well on their way of losing it long

before.

[DENVERPOPS]

7. They have been losing creditability

for a few decades now............It has happened to much of the Media with the "He Said, She Said", bullshit and publishing of Alternate Facts.
People are dropping their subscriptions to NYTimes like wildfire these last few months due to some of their recent atrocious and outrageous Right Wing articles..........

[markodochartaigh]

14. At least they have good recipes.

This is what we want from "The US newspaper of record", correct?

/s

[LiberalFighter]

5. Polls have been oversampling Republicans for several elections.

Especially since Covid.

[maxrandb]

6. Yes, Democrats actually lead fulfilling lives outside of politics

But we will take a short break from living our best lives, to beat the ever-loving piss out of the MAGAts at the ballot box.

We then will return to being productive citizens in a democratic civil society, and the MAGAts can to back to swilling warm beer and arguing about whose sister is better in bed, or whether Dennis Rodman should be Secretary of State.

[peppertree]

8. True. Fascists and Fundamentalists have always had that advantage: they tend to be more committed

Quite literally, in some cases.

[modrepub]

10. Given How Much The M$M Is Expected to Get In Political Ads

I'd expect them to want a close race. Collectively we spend billions on election advertising. The closer the race appears, the more advertising will be done.

And as a side, TFG has an ability to dominate news cycles (and media clicks). So the M$M is going to keep posting because people keep watching.

[Xipe Totec]

11. I never respond to polls - don't know who's behind them

For all I know it's Republicans trying to identify targets for voter disenfranchisement.

[Attilatheblond]

19. Yep. Interrupt their script & INSIST on knowing who commissioned/paid for the poll

Either they answer (LOL) or they won't say, don't know. Hang up and block the number.

RWers are probably eager to have someone take down their answers. Most are naive and lonely.

Thinking people are unlikely to actually listen to pollsters, or even answer calls from unknown numbers.

The polls are not gonna give any real info on voters' choices.

[ITAL]

37. I mean the callers likely never know

I did phone surveys for awhile in college (rarely political). We called about stuff like home computer usage and chewing gum varieties...all sorts of random stuff. I'm sure our managers knew who we were trying to get the information for, but we almost never did. We could sometimes figure it out based on questions (if we were doing a computer survey and the questions were mostly about Dell Computers, it was safe to assume that's who commissioned the survey, but they specifically wouldn't tell us so we couldn't answer callers questions).

[Attilatheblond]

39. I get that, but I still ask and will not listen to anything if they don't have an answer

9 times out of 10, they don't understand the question. I say adios and block the number. No one is under obligation to answer questions from strangers, and frankly, the nation would be better off if we all hung up on such calls. Too much market research in this nation. Too much manipulation marketing of stuff we really can do without.

[ITAL]

40. I understand

I tend to be sympathetic to them given I had that job for awhile myself. I usually do surveys when called. I know they don't share information on me, because I'd be called a lot more often given the overall low response rates - LOL. The last time I was called about my opinion was probably over a year ago on telecom providers.

[ananda]

24. Me neither. Besides, I'm on the Democratic list anyway.

They all know I'm gonna vote for them, and/or donate.

[BumRushDaShow]

12. Not just them but ALL of them

(as noted earlier in the thread) - they have been promoting the "opposite" view that somehow, Republicans aren't participating in polls or were reticent, so they continue to over-sample GOP "voters", extracting as much as they can from the slime pits, and then introduce massive amounts of error (that they refuse to acknowledge) by "correcting" for supposed Democrats (and others) in the population.

But it has been all a lie and election after election (whether primary, special, or general) they end up finding that "Democrats over-performed".

We just have to make sure to GOTV.

[calimary]

16. Yep!

Nothing matters as much as YOUR VOTE.

[paleotn]

13. And that's why the only poll that matters is the one on the first Tuesday of November.

The first Tuesday after the first Monday to be precise.

[calimary]

17. VOTE. That's IT. Just VOTE!!!

NOTHING ELSE matters.

[paleotn]

20. exactly

[Escurumbele]

15. I have been saying that for a long time, Democrats do not like to answer polls, and that is why I believe trump

will loose with bigger numbers than he did on 2020, and I hope that is the last we hear from him besides a status here and there about his time in jail. I can see the headlines "Ivanka refuses to visit his dad in jail, only Eric has shown a couple of times after two years, but no one knows where Don Jr has gone to, some people think he is hunting somewhere in Africa, or simply moved there."

That would be so nice...

And the last headline..."None of the trump 'children' showed up for his funeral which took place at hole #2 of the New Jersey Golf Club."

[agingdem]

21. I agree..

I don't answer my phone unless the caller is on my contact list...I block all numbers that are "scam likely"..and I'm betting many of those are polls...

[markodochartaigh]

18. I think

that in the case of an authoritarian Strong Leader it should be realized that the support for the Strong Leader is going to remain the same no matter what he does. Although Trump's Republican support briefly dropped after his attempted coup, it soon regressed to its previous level. His support dropped a bit after his felony convictions, we'll see how that compares to his support this fall. On the other hand, we are fortunate that Trump is such an unprecedentedly corrupt buffoon. His general support can't go up much. I think that polls are usually used to illustrate these points.

[lpbk2713]

22. Sounds reasonable to me.

I don't ever do any phone polls. There's no way of knowing if the caller is actually who they say they are.

[TheBlackAdder]

23. I refuse to answer any polls because I got hit with Republican Push Polls trying to change my view on topics.

They come off as neutral, but have that RW bent that they slowly veer their questioning.

[twodogsbarking]

25. Have they printed their story of how Trump won the debate yet ?

[Takket]

26. this has been a known phenomenon for years now........

so this cannot simply be written off as a "surprise" that catchers pollsters off guard.

They know this is happening which means one of two things..........

1. They are intentionally ignoring it because they know presenting data that shows drumpf close/winning is good for inspiring anger/fear/frustration in their readers, which drives click/sales.

2. They really are just that BAD at their jobs.

I think it is #1.

[emulatorloo]

27. FINALLY they confess. Thanks for posting, bookmarking.

[Think. Again.]

30. The nyt is the rightwing AM radio of print.

Of course they're going to lie and cheat to advance a rightwing agenda, that's what being rightwing IS.

[DoBW]

31. I think most blue voters instinctively knew it all was BS

We're just not the morons they think we are

[dchill]

32. Adjust for it? Whyever would they do that?

They already got the results they were after.

[dchill]

33. Dupe.

[czarjak]

35. The Donald will not win the popular vote.

Period. Not saying that he won't be president again. We need to change that part.

[Algernon Moncrieff]

48. I think the big polling organizations are still traumatized by 2016

After Hillary Clinton lost, there were accusations of polling bias. But since then, we've tended to see overcompensating. The "Red Wave" of 2022 didn't play out. The 20-30 seat majority projected in the house turned out to be far less and led to the speakership chaos. Youngkin was projected to get one house for sure and possibly nothing Virginia -- he got neither.

[UpInArms]

49. Or, Dems are tired of their sill push polls

With questions that push you to answers that do not reflect anything near your response.