The Pundits vs. Political Science: Debating the Impact of the First Debate

As Election 2012 heads down to the wire, it is fascinating to see how the pundits view this race compared to where the political scientists do (at least as I interpret them).  They often seem to be looking at two different events.  For pundits, the race is a roller-coaster affair in which candidates gain and lose momentum based on a series of often unpredictable events.  Thus, on today’s Meet the Press, Joe Scarborough openly speculated about the impact of Hurricane Sandy in blunting Romney’s “momentum.” In Scarborough’s words, “The question is, whether the hurricane stopped the momentum for 48 hours and whether that in the end will stop him from overtaking the president.”

The only problem with this analysis is that it appears that Mitt’s support had stabilized at between 47-48% in the national polls back on October 9, and there hasn’t been very much movement beyond random statistical fluctuations since then.  In other words, there wasn’t much momentum for Mitt to lose, Sandy or no Sandy.

David Gregory, the host, then asked his panelists to identify the moments of the campaign that stood out to them.   Four of the five participants cited the first debate as the turning point in this campaign.  Scarborough, taking what might be perceived as a shot at political scientists, argued, “And, third, just a remarkable first debate that if Mitt Romney wins will be a debate that political scientists will be looking at fifty years from now.  It really could be the big turning point.”  The usually reliable Tom Brokaw concurred: “I– I think the first debate.  I think that history will long record that if he survives this– that debate was something unprecedented.  I have never seen anything like that in my lifetime, when a man who had to convince the country, he was a strong leader, disappeared from that stage.”

At this risk of repeating myself (see here and here), there is not much evidence that the first debate had nearly the impact the pundits ascribe to it.  Previously I published Anna Esten’s research indicating that Mitt may have earned a net gain of about 2.7% when comparing the average of the polls in the week prior to the first debate compared to the average of those polls in the week after.  This is not an inconsequential gain, mind you, but it’s worth remembering what that change signifies.  As I’ve argued before, that net gain did not come from converting Obama supporters to Romney supporters – instead, it came from shifting the proportion of people who made it through the various polling voter screens. In short, the first debate was a focusing event that likely accelerated the polling toward the dead-heat equilibrium that the political science forecast models, looked at in the aggregate, have been suggesting this race would become all along.

Moreover as Peter Kellner explains in this post, there is some evidence, based on looking at a panel study that interviewed the same set of people both pre- and post-debate,  that even a 2.7% projected net gain for Mitt may overstate the impact of the first debate.  Kellner writes, “The key point is that this was a true panel study. We questioned the same people twice. This allowed us to investigate what change, if any, took place at the level of individual voters, NOT by comparing results from different samples.  Any change in the numbers in such panel studies reflects real changes by real voters. And our overall sample was much larger than normal. We polled almost 33,000 electors in September, and reinterviewed more than 25,000 of them after the first debate.

The message from this study was clear. The Romney bounce was tiny.  Overall, YouGov found just a one-point narrowing of Obama’s lead.”

Why did YouGov find only a 1% change, while Anna calculated Mitt gained a more sizeable advantage? The answer gets to an issue I’ve talked about before – most pollsters do not “weight” their polls to maintain a fixed proportion of partisans.  That is, they let the poll results determine how many “Democrats” and “Republicans” there are in any particular sample.  So, after the first debate, many pollsters likely picked up a shift in party identification indicating more Republican self-identifiers, which in turn suggested a net polling gain for Romney. The debate had the same impact on the YouGov panel study – more Republicans in the panel were likely to answer the follow up survey after the first debate than were Democrats.  However, in contrast, to most pollsters, YouGov adjusted their post-debate sample to keep the partisan distribution consistent with the pre-debate sample.  When they did that, they saw only minimal gains by Mitt.

Kellner’s conclusion?  “What we can therefore be fairly sure of is that the first TV debate made little or no difference to the (high) degree of loyalty Democrats and Republicans display towards the two candidates. It is NOT the case that many voters switched from Obama to Romney. The question, rather, is whether the first debate caused the number of Democratic-ID Americans to fall, and Republican-ID Americans to rise.”

The answer to just how big an impact that first debate had, then, turns in part on whether you think “partisanship” changes only slowly, or that it can in fact change rapidly in response to a single event, such as a debate.  Note that this is not one of those nerd fights that have little consequence in the real world.  Instead, it gets to the heart of the polling discrepancies we have found between some state-level polls and the national tracking polls by some firms, like Rasmussen, that do adjust their partisan composition to bring it in line with what they think the “true” party division currently is.  Decisions regarding whether and how to weight by party can determine whether a poll favors one candidate or the other.  Republicans have consistently argued that many pollsters are including too many Democrats in their samples. Democrats counter that some firms, like Gallup, are underweighting some groups that are likely to vote Democratic.  We will know who is right on Tuesday.

Meanwhile, in the long run,  I suspect Scarborough is right. Fifty years from now political scientists will be arguing that the first Romney-Obama debate didn’t have nearly the impact the media pundits said it did – and we will be ignored then as well!