As the political pundits parse last night’s Republican debate – a topic I will tackle later – I want to return to a story that attracted quite a bit of media play earlier this week. Three days ago New Hampshire television station WMUR in conjunction with CNN released a poll that showed Hillary Clinton and Bernie Sanders in a “statistical tie” in New Hampshire. Not surprisingly, the poll generated quite a bit of media coverage, with The New York Times headline for its story on the poll proclaiming that Clinton and Sanders were in a “dead heat.” Other news outlets, citing the same poll, made similar claims. In fact, the survey, which was in the field during the last week of July, showed 42% percent of likely Democratic primary voters saying they will vote for Clinton, with 36% saying they are backing Sanders. How can the New Hampshire race be a “tied” when the poll shows Clinton with a 6% lead? The answer is that because the two candidates’ numbers fall within the poll’s sampling margin of error (a measure of how confident pollsters are in their results), one can’t discount the possibility that Sanders is actually tied, or perhaps even ahead, of Clinton. Remember, surveys are simply estimates of the sentiments of an underlying population – in this instance, likely Democratic primary voters in New Hampshire – and one’s confidence in the results depends in part on how many people are surveyed and what confidence level we are willing to accept in evaluating the results. In this case, the WMUR poll’s margin of error at the 95% confidence level for the Democratic nominating race is +/-5.9%. In describing the race as a “statistical tie”, then, the WMUR pollsters are acknowledging the possibility that despite Clinton’s 6% lead, Sanders’ actual support might be at the upper end of the margin of error, and Clinton’s at the lower end. (Of course, it’s possible their support lies outside the margin of error, but this is even more unlikely.) Hence, WMUR’s decision to label the race a “statistical tie.”
At the risk of nitpicking, however, I would argue that a “statistical tie” is not the equivalent of a “dead heat”, The Times’ headline notwithstanding. To understand why, one should also ask: what is the probability that a purely random sample of 274 likely Democratic voters (the size of the WMUR poll on the Democratic side) would show Clinton ahead by 6% if in fact there is no difference in polling support between Clinton and Sanders in the underlying population – that is, that they really are tied? It turns out that it is not very likely – in fact, a simple test of the difference in survey sample results suggests there is a less than 10% probability that the race is actually tied, given the survey findings showing Clinton ahead by 6% (and making certain other assumptions about how the WMUR poll was conducted.) So, it is true that we can’t be sure that Clinton is ahead, at least not using the conventional 95% uncertainty level. But it is much more likely, given these poll’s parameters, that she is leading Sanders than that they are in a true dead heat. My quibble with most of the media stories reporting the WMUR poll is that they don’t make the difference between a “statistical tie” and an actual tie very clear.
“Fine,” you respond. “At least I can take comfort in knowing that Bernie is closing the gap with Hillary.” And, in fact, the first line of The Times story notes that “Senator Bernie Sanders of Vermont continues to tighten the race with Hillary Rodham Clinton in New Hampshire, according to a poll released on Tuesday.” As evidence, the author notes that a previous WMUR/CNN poll of likely Democratic voters that was in the field from June 18 to 24 found Clinton leading Sanders by 43%-35% (with a margin of error of +/- 5.2%). Based on these two polls, then, it appears that Sanders has gained 2% on Clinton – evidence that, according to the Times, “Mr. Sanders continues to gain momentum after months of negative publicity about Mrs. Clinton’s use of a private email server as secretary of state.”
Well, maybe. Again, it is useful to put this claim in proper context too. If you parse the polling numbers, the one-month change from a 43%-35% Clinton lead in early July to 42%-36% margin this week comes out to approximately a handful of respondents expressing a preference for Sanders rather than Clinton this month compared to last month. Now, this might reflect an actual change in the sentiments of the underlying population. Or, it might just be the result of picking up a couple more Bernie supporters in the random sampling process, even though there’s been no actual change in voter preferences. The bottom line is that we should be cautious about extrapolating that Sanders is gaining on Clinton based solely on a 2% change in the WMUR polling results across a one-month period.
This doesn’t mean Bernie hasn’t gained ground on Hillary in New Hampshire. As this Pollster.com poll aggregation shows, if we widen our time horizon it’s clear that Bernie has closed the polling gap, particularly when Elizabeth Warren’s name was dropped from the survey options.
But there hasn’t been a lot of recent polls in New Hampshire. A NBC poll in late July had Clinton up by a larger margin, at 47%-34%, while a recently-released Gravis poll has it 43%-39% in Clinton’s favor. (I haven’t looked closely at the internals of either poll.) Right now the aggregate Pollster polling has Clinton up 43.3%-38.8%. RealClearPolitics, which uses a slightly different aggregating algorithm, has Clinton with a more substantial lead over Bernie, at 44.8%-31.6%.
The bottom line is that rather than a “dead heat”, Clinton is probably leading Bernie in New Hampshire, and that it is not even clear, despite an abundance of recent negative news coverage for Clinton and Bernie’s well-attended campaign events, that he has gained all that much ground over the last month. Alas, for a media with a vested interest in seeing a competitive race for the Democratic nomination, that narrative is probably a lot less interesting, even if it is likely to be a bit more accurate.
I would not put too much trust in political polls. After all, if they were that accurate, Scotland would be independent, and the British Labour Party would be running the UK.
New NH poll just came out with Sanders ahead by 7 pts…care to try again Tex?
No, they’re not in a dead heat; Bernie’s winning.
https://www.bostonherald.com/news_opinion/us_politics/2015/08/bernie_sanders_surges_ahead_of_hillary_clinton_in_nh_44_37
I think it’s hilarious that you published this piece hours before The Boston Herald posted the results of their latest New Hampshire poll.
The Boston Herald reports, that there is a new Franklin Pierce University/Boston Herald poll of 442 New Hampshire voters, showing Sanders at 44 percent, and Clinton at 37 percent. Makes the paragraphs and paragraphs the writer spends on talking about how Clinton’s 6 percent lead means the two candidates are not in a “tie” seem kind of ridicilous.
It seems like more and more people learn about Sanders’ policy proposals, and like them, thus support his campaign. Back in the beginning of May, Sanders’ New Hampshire poll average was about 13 percent. So to some, it might seem impressive how so much momentum built in just a few months, and Sanders is now at 44 percent leading Clinton in New Hampshire, or at least being in a “tie” with her. Over time, as more and more people get to know Sanders, and the momentum might continue to build, then more and more might also realize that he indeed could win…
Here is a link to the Boston Herald story:
Poll: Bernie Sanders surges ahead of Hillary Clinton in N.H., 44-37
http://www.bostonherald.com/news_opinion/us_politics/2015/08/poll_bernie_sanders_surges_ahead_of_hillary_clinton_in_nh_44_37
Thanks info for the link – I’m up with a post discussing the results right now. See: http://wp.me/plZpC-40X
Jesse – As I note in my next post (http://wp.me/plZpC-40X) I could reprint the previous post to make the same point about this poll!
“Tex”,
Yep. At the risk of repeating the previous post, I’ve made a new one, but the argument is pretty much the same: beware relying on a single poll! See: http://wp.me/plZpC-40X
Matthew Dickinson wrote: “At the risk of repeating the previous post, I’ve made a new one, but the argument is pretty much the same: beware relying on a single poll!”
No, this was not pretty much the same argument. The argument of the blog entry above on August 11 was “Bernie and Hillary are not in dead heat in NH”. Well, turns out, they are not only in a statistical tie, so at least in dead heat, Bernie was even already leading in the FPU/Boston Herald Poll.
So instead of admitting “Ok. I was wrong. I wrote paragraphs and paragraphs that Bernie and Hillary are not in dead heat in NH, but they indeed are, with Bernie even leading in the latest poll”, the blogger claims that the blog entry on August 11 was about “Don’t trust a single poll”. Well, the August 11 blog entry clearly wasn’t. It was about something else, and it turned out to be wrong.
Instead of admitting to have been wrong (what’s so hard about that, at least everybody is wrong sometimes), a new blog entry was written on August 12, supposedly emphasizing “Don’t trust a single poll” and “Pollster.com aggregate polls continue to have Hillary holding a slim lead over Bernie in the Granite state”. Well guess what? The Boston Herald reports that there is a new Public Policy Polling survey of 370 “usual” Democratic primary voters in New Hampshire, conducted August 21 to 24, showing 42 percent support for Senator Bernie Sanders and 35 percent support for former Secretary of State Hillary Clinton.
So once again, the blogger using his unserious line of arguments to show Sanders is not in dead heat or leading in NH, this time turning to “Don’t trust a single poll”, ended up in a dead end, as now there are two polls showing Bernie solidly leads Hillary in NH by 7 percent points.
And for “Note, however, that [the] Pollster.com aggregate polls continue to have Hillary holding a slim lead over Bernie in the Granite state” – will the blogger now write that “Note, however, that Pollster.com aggregate poll has Bernie with a solid 3.8% lead over Hillary in the Grante state”? At least some might think that it’s the blogger’s pattern to not to do this, still find some other twist. Now the blogger might possibly revert to writing something like “Don’t trust two polls”, and that that is what he said all along, and that until at least three polls say that Bernie is leading, Hillary is leading and that most importantly, Bernie and Hillary are not in dead heat in New Hampshire. No way. 🙂
The future is not written, still to many it might look like the campaign of Bernie Sanders currently has a lot of momentum, and anything is possible. At the same time, even when Sanders is leading in polls, also in the future, one has to expect denial of any Sanders lead (like in “or does he?”) and some twisting by pointing to certain demographic groups, as in “Sanders can’t win, because while Sanders leads among younger voters, he’s still trailing among seniors”, even when he’s leading in total. And even in case Sanders was not only leading generally, but also in each and every subgroup of voters, then one might still expect to hear “Don’t trust that poll” and “He can’t win”. The general public will just have to learn to deal with it and ignore it where it seems unsubstantiated, and go with the actual numbers and facts.
In the current Pollster.com aggregate for the 2016 New Hampshire Democratic Presidential Primary, Cinton is at 39.8%, while Sanders is at 43.6%. Now wait for the spin how though earlier the blogger wrote one should have “however noted” the aggregate, now all of a sudden it might not meaning anything… 🙂
Info,
Thanks for the thoughtful (and passionate!) response. In the initial post that you refer to, I made two points. The first is that based on the NH poll the New York Times article discussed, it might be that Sanders and Clinton were in a “statistical tie” given the poll’s margin of error (although I dislike the term statistical tie), but – again based on that one poll! – they almost surely were not in a “dead heat” (actual tie.) Instead, it was more probably that Clinton was ahead. My second point is that one should probably never rely on a single poll to assess the state of the race particularly if there are multiple ones available.
Subsequently a second NH poll came out, and again many news outlets made the same mistake, calling the NH race a “dead heat” when in fact – despite the margin of error, it was more likely – based on this one poll – that Sanders had a slight lead. As I wrote, “Once again, as the graph suggests, we can’t discount the possibility, given the margin of error, that Clinton and Bernie are tied, or that Clinton might even be slightly ahead. At the same time, however, based on this one poll, the odds are greater that Bernie is now ahead of Clinton in New Hampshire. But of course, one shouldn’t rely on one poll – hence my reference to the aggregate polling results.”
Longtime readers (of which I hope you become one!) will note that I am often mistaken and quite willing to acknowledge as much, particularly when called out by discerning (and passionate!) readers. Thus you will undoubtedly find my response here obstreperous, but in rereading both posts, I stand by my initial points! That it, in neither poll was it most probable that Clinton and Sanders were actually tied, the Times headline notwithstanding, and that we shouldn’t rely on one poll to get an accurate assessment of what’s going on in New Hampshire (or in any race.)
At the end of your comment, you point out that the aggregate polls are now showing Bernie with a slight lead in NH, so I feel a deep satisfaction that you seem to be accepting my point to rely on the aggregate polling over any single poll! Good for you! I trust you will continue to do so even if they show Clinton back in the lead! 🙂 Of course, and I know you understand this, the fact that today the aggregate polls show Sanders ahead says no more about who was ahead when I wrote those posts than today’s temperature tells us what the temperature was three weeks ago!
I appreciate your deep passionate support for Bernie. But, if you continue to interpret the validity of any of my analyses (and my motives!) by whether my post is favorable or unfavorable to your candidate, you are deeply misreading what my purpose is in writing this blog. As the title suggests, this is a nonpartisan site. If you want to immerse yourself in an echo chamber where every comment is favorable to your chosen candidate, no matter what the reality, then this is not the site for you. If, however, you want to read a non-partisan analysis based on an objective assessment of the data, then keep on reading. But – and here is my third point – don’t shoot the messenger! Keep those critical comments coming….
Quod erat demonstrandum
Thank you! It is always gratifying when a reader comes around….this is an educational blog, after all (or so I like to think!)
No. In this comment section there was no reader coming around, there was a blogger though who demonstrated exactly that kind of pattern that some readers might have been afraid he would, brushing aside the facts. As by at least to some readers it might be obvious that the blogger’s posts again and again proved inaccurate. Repeatedly, there have been statements, basically saying that whatever presidential candidate Bernie Sanders’ campaign achieved, it should be dismissed. The blogger’s first post basically said: “36% for Sanders and 42% for Clinton in New Hampshire doesn’t mean Sanders and Clinton are in dead heat”. Of course that seemed incredibly off when directly afterwards a poll came out showing Sanders leading 7% in New Hampshire. Just ignoring the flaws of the previous statements, now the 7% lead needed to be attacked along the lines of: “Bernie is leading by 7 percent in one New Hampshire poll – but that doesn’t mean anything, because it’s just one single poll”. Of course with that argument, the blogger was in a dead end afterwards as the 2nd poll came out, also showing Bernie leading with 7%, confirming the first poll. It turned out, one could indeed have trusted the first poll showing the 7% lead, though the blogger said one should not. The blogger’s posts were wrong, still understandably the blogger does not want to say that. That’s why in the long comment above, the blogger again had to dismiss the importance of the 2nd poll showing that Bernie is leading in New Hampshire, while at the same time (as before) getting across, that the importance of any polls or even hints showing Bernie Sanders somehow not leading in whatever sub group or location that might be, has to be noted. If he would not have dismissed the 2nd poll as well, he would have had to recognize, that not only his long blog post doubting Bernie’s lead is in New Hampshire after his first poll lead (“Bernie’s leading in New Hampshire – or does he?”) was off, but the initial “they are not even close, not in dead heat” as well.
Now a third poll came out showing the lead of Bernie Sanders in New Hampshire. And now Bernie is not only leading by 7 percent anymore, but by 11 percent. “The NBC News/Marist Poll in New Hampshire was conducted Aug. 26-Sept. 2 of 966 registered voters (+/- 3.2%), 413 potential GOP primary voters (+/- 4.8) and 356 potential Democratic primary voters (+/- 5.2).” Bernie Sanders now is at 49%, Hillary Clinton at 38%. So Clinton down from 42% (-4%), Sanders up from 36% (+13%) since the blogger’s August 7 post. Still just a few days ago, the blogger talked about the possibility of a future poll to “show Clinton back in the lead”. Turns out, quite the opposite, Clinton moved even further away from being back in the lead in New Hampshire, and Sanders continues to increase his lead. But meanwhile at least some readers know the pattern, and anticipate more blog posts about how this new poll – as long as it shows Sanders leading – doesn’t matter…
Info – I can appreciate your passion for Bernie’s candidacy. Just to be certain that I understand your logic, are you saying only one of the NH polls you are discussing is accurate, and the others are not, or that the polls are showing different results because the underlying sentiment in NH has been changing during the time the polls were in the field?
And to be doubly certain I understand your logic, I take it that you would think a headline stating “Bernie and Clinton in dead heat in NH” based on a poll showing Bernie ahead by 7% would be incorrect?