The Pundits vs. Political Science: Debating the Impact of the First Debate

As Election 2012 heads down to the wire, it is fascinating to see how the pundits view this race compared to where the political scientists do (at least as I interpret them).  They often seem to be looking at two different events.  For pundits, the race is a roller-coaster affair in which candidates gain and lose momentum based on a series of often unpredictable events.  Thus, on today’s Meet the Press, Joe Scarborough openly speculated about the impact of Hurricane Sandy in blunting Romney’s “momentum.” In Scarborough’s words, “The question is, whether the hurricane stopped the momentum for 48 hours and whether that in the end will stop him from overtaking the president.”

The only problem with this analysis is that it appears that Mitt’s support had stabilized at between 47-48% in the national polls back on October 9, and there hasn’t been very much movement beyond random statistical fluctuations since then.  In other words, there wasn’t much momentum for Mitt to lose, Sandy or no Sandy.

David Gregory, the host, then asked his panelists to identify the moments of the campaign that stood out to them.   Four of the five participants cited the first debate as the turning point in this campaign.  Scarborough, taking what might be perceived as a shot at political scientists, argued, “And, third, just a remarkable first debate that if Mitt Romney wins will be a debate that political scientists will be looking at fifty years from now.  It really could be the big turning point.”  The usually reliable Tom Brokaw concurred: “I– I think the first debate.  I think that history will long record that if he survives this– that debate was something unprecedented.  I have never seen anything like that in my lifetime, when a man who had to convince the country, he was a strong leader, disappeared from that stage.”

At this risk of repeating myself (see here and here), there is not much evidence that the first debate had nearly the impact the pundits ascribe to it.  Previously I published Anna Esten’s research indicating that Mitt may have earned a net gain of about 2.7% when comparing the average of the polls in the week prior to the first debate compared to the average of those polls in the week after.  This is not an inconsequential gain, mind you, but it’s worth remembering what that change signifies.  As I’ve argued before, that net gain did not come from converting Obama supporters to Romney supporters – instead, it came from shifting the proportion of people who made it through the various polling voter screens. In short, the first debate was a focusing event that likely accelerated the polling toward the dead-heat equilibrium that the political science forecast models, looked at in the aggregate, have been suggesting this race would become all along.

Moreover as Peter Kellner explains in this post, there is some evidence, based on looking at a panel study that interviewed the same set of people both pre- and post-debate,  that even a 2.7% projected net gain for Mitt may overstate the impact of the first debate.  Kellner writes, “The key point is that this was a true panel study. We questioned the same people twice. This allowed us to investigate what change, if any, took place at the level of individual voters, NOT by comparing results from different samples.  Any change in the numbers in such panel studies reflects real changes by real voters. And our overall sample was much larger than normal. We polled almost 33,000 electors in September, and reinterviewed more than 25,000 of them after the first debate.

The message from this study was clear. The Romney bounce was tiny.  Overall, YouGov found just a one-point narrowing of Obama’s lead.”

Why did YouGov find only a 1% change, while Anna calculated Mitt gained a more sizeable advantage? The answer gets to an issue I’ve talked about before – most pollsters do not “weight” their polls to maintain a fixed proportion of partisans.  That is, they let the poll results determine how many “Democrats” and “Republicans” there are in any particular sample.  So, after the first debate, many pollsters likely picked up a shift in party identification indicating more Republican self-identifiers, which in turn suggested a net polling gain for Romney. The debate had the same impact on the YouGov panel study – more Republicans in the panel were likely to answer the follow up survey after the first debate than were Democrats.  However, in contrast, to most pollsters, YouGov adjusted their post-debate sample to keep the partisan distribution consistent with the pre-debate sample.  When they did that, they saw only minimal gains by Mitt.

Kellner’s conclusion?  “What we can therefore be fairly sure of is that the first TV debate made little or no difference to the (high) degree of loyalty Democrats and Republicans display towards the two candidates. It is NOT the case that many voters switched from Obama to Romney. The question, rather, is whether the first debate caused the number of Democratic-ID Americans to fall, and Republican-ID Americans to rise.”

The answer to just how big an impact that first debate had, then, turns in part on whether you think “partisanship” changes only slowly, or that it can in fact change rapidly in response to a single event, such as a debate.  Note that this is not one of those nerd fights that have little consequence in the real world.  Instead, it gets to the heart of the polling discrepancies we have found between some state-level polls and the national tracking polls by some firms, like Rasmussen, that do adjust their partisan composition to bring it in line with what they think the “true” party division currently is.  Decisions regarding whether and how to weight by party can determine whether a poll favors one candidate or the other.  Republicans have consistently argued that many pollsters are including too many Democrats in their samples. Democrats counter that some firms, like Gallup, are underweighting some groups that are likely to vote Democratic.  We will know who is right on Tuesday.

Meanwhile, in the long run,  I suspect Scarborough is right. Fifty years from now political scientists will be arguing that the first Romney-Obama debate didn’t have nearly the impact the media pundits said it did – and we will be ignored then as well!


  1. Professor,

    What can be made of all of the media hype over “perfect storm” situations for election day? MSNBC just had a large article about 4 nightmare scenarios. There seems this election cycle to be a lot of focus on how ballots are counted, what do you make of that? Also, what are the chances or rather, what are your thoughts about the nation waking up on Wednesday and not having a victor? Do you seen a 2000 type situation playing out this year?

    If the vote will be as close as political scientists believe, how will this impact true election results. Does it mean we will be seeing less of Wolf jumping up and down saying we have a prediction to make?

  2. Multiple linear equations be damned ! I’m on my knees praying that I can serve
    Him regardless of the outcome of this election.
    Plan B – Move to a less corrupt country. Dick

  3. Chris,

    Remember, it is the media’s job to hype the unexpected, improbable and controversial, as opposed to covering the usual, highly probable and thus less newsworthy events. So, while it is certainly possible that in a close elections like I am forecasting for tomorrow there is a higher probability of some type of snafu,or tied Electoral College, or a repeat of Florida in 2000, I think it more likely that we’ll be pretty sure that one candidate beat the other guy. That won’t stop the extremists on either side from crying foul if their guy loses however!

  4. Professor,

    I very briefly wanted to clarify for your readers “partisanship” as described in your post, as I think a misunderstanding of the term leads to some of the distrust folks have in the polls. There are two types of partisanship as measured by pollsters: Party ID and Party Registration. You correctly add that the majority of serious public pollsters do not weight by the former. Party ID is an attitudinal variable…the example I like to use is that weighting by Party ID is akin to weighting by the results from a previous election. A common criticism goes “Party ID in Poll X is D +9, but in 2008 it was D+5!” At the same time, I highly doubt someone would say “Obama is receiving 46% in this poll, but in 2008 he received 49%!” but in truth it’s the same thing (even leaving aside for the time being that the former statement is likely a comparison of Party ID to Party Reg, so it’s not even apples to apples in the first place).

    Party registration, on the other hand can be included if the sample is drawn from a list that has been compiled based on the voter file. Most public polls are RDD polls, which makes this impossible, but it is why many internal polls do weight by “partisanship” (in this case Party Registration) which is far less of an attitudinal variable.

    The issues come when people try and conflate the two measures, which are in fact very different, and the result is a lot of the back-and-forth we’ve seen over many of the public polls over the last couple of months.

    Hope this helps for those that didn’t know the difference.

    Peter Baumann

  5. Peter – Thanks for the clarification reminder. It’s a point I’ve made before in this blog, but it can’t be repeated enough. Note, however, that even among those who view party as an attitudinal trait – there is some disagreement regarding whether sample should lets that trait be determined by a particular sample, or whether it make sense to adjust it to fit a presumed partisan distribution.

  6. Just released NYU study of President Obama and Mitt Romney’s body language shows on a word by word basis what each candidate emphasizes.

    be sure to click on the more info tab and read the information just so you have seen it.

Leave a Reply

Your email address will not be published. Required fields are marked *