How reliable is the generic ballot survey question in helping forecast the size of the Democratic losses that will occur in November? As we move closer to November 2, a growing proportion of my blog posts will undoubtedly focus on the midterm elections. Many of these posts will likely mention the generic ballot question that is asked by a number of polling firms. As most of you know, this question typically take some version of the following form: “Looking ahead to the Congressional elections in November, which party do you plan to vote for if the election were being held today?” The survey results, as many of you have heard me say, are actually a useful indicator of the likely outcome of the November midterm election.
But how useful? In a recent blog post, Nate Silver at fivethirtyeight.com raised important questions about the utility of the generic ballot question results. Silver writes: “It might be the case that the generic ballot is fairly stable, but that doesn’t necessarily mean it’s all that useful an indicator. In addition to the fact that the consensus of polls (however careful we are about calibrating it) might be off in one or the other direction, there’s also the fact that the thing which the generic ballot is ostensibly trying to predict — the national House popular vote — is relatively irrelevant to the disposition of the chamber, or the number of seats that each party earns.”
Political scientist Alan Abramowitz, (via Brendan Nyhan) takes issue with Silver’s comments, calling portions of them “pretty silly.” I wouldn’t characterize Silver’s comments as silly, but I certainly disagree with portions of his post, particularly his claim that the national House popular vote is “relatively irrelevant to the …number of seats each party earns.” Although it is not a perfectly precise indicator of how many seats a particular party will win (or lose) come November, it does provide a useful approximation. Similarly, the generic ballot question does help us predict, within a margin of error, what the likely popular vote will be come November. To be sure, in some midterms it has proved more useful than others. But it is not irrelevant.
The bottom line, then, is that the generic ballot results, properly understood, helps us predict the November midterm results. To see why, it helps to understand how political scientists factor the results of this question into their midterm forecast model.
When a generic ballot survey comes back with results showing, as the most recent Gallup survey poll does, that Republicans are favored over Democrats by 7%, 50%-43%, that doesn’t necessarily mean Republicans will win 7% more congressional seats.
Indeed, as Silver correctly points out, the generic ballot result can’t even tell us how many overall votes each party will get in the midterm, never mind how they will do in each of the 435 districts. But political scientists understand this. As it turns out, they don’t necessarily have to perform a district-by-district level analysis to get a decent handle on the total number of seats each party will win. Instead political scientists can use the generic ballot results as one factor in statistical models that also take into account the existing political context as well as “structural” attributes associated with a midterm election. So, for example, Abramowitz’s midterm forecast model uses only four variables to predict the midterm outcome, as measured by seats lost/gained: the results of the generic ballot question, presidential popularity (as measured by the Gallup poll), the number of seats held by Republicans going into the midterm election, and a simple indicator variable that signals this is a midterm election, rather than a presidential one. (This last variable is important because the president’s party typically loses seats in the midterm election.)
How do we know that the Abramowitz model (or others like it) is reliable? Because it is constructed using previous midterm election results. In effect, by looking at previous midterms, Abramowitz can build an overall “generic” statistical model that says, on average, how important each of these four variables is in determining the midterm election results. Then, by plugging the current values for each variable into the existing model, he comes up with a forecast. Political scientists have constructed different forecast models, but they typically all include some mix of variables that measure voters’ partisan sentiment, structural attributes associated with the midterm, and some indicator of the environment (political and otherwise) in which the election is being held.
Now, these models aren’t completely foolproof. They are predicated on the assumption that the factors influencing the current midterm will behave pretty much as they have in the past. Say something unexpected happens – terrorists attack the World Trade Center, for example. That may unexpectedly distort the relative importance of some variables, thus throwing the forecasts slightly off. Moreover, even without unexpected events, there is always some uncertainty involved with these forecasts – some unexplained variance for which the model cannot account.
But it would be wrong to suggest, as Silver does, that the generic ballot question is “irrelevant.” In fact, it is a highly relevant and useful predictor of the midterm outcome – as long as it is evaluated within an overall understanding of the factors that drive midterm elections. And right now, it favors the Republicans by 7% – not a good sign for Democrats at all.
I will have much more to say about the generic ballot in the next several posts. Before doing so, however, there’s another prediction that I need to discuss: Elena Kagan’s Senate confirmation vote to the U.S. Supreme Court. Long-time readers will recall that in an earlier post I set the over/under for the Kagan no votes at 35. In a sign that my predictive powers may be slipping (remember that I nailed Sotomayor’s exact confirmation vote) Kagan was confirmed with 37 votes in opposition, so I was two votes off. (If you must know, it was Nelson and Brown). Will Loubier, on the other hand, hit the final vote margin squarely on the head, thus winning a Presidential Power “It’s the Fundamentals, Stupid!” t-shirt. Here’s Will, looking justly proud of his prognosticative abilities:
For the envious among you, note that I’ll be giving away another t-shirt in the “predict the midterm outcome” sweepstakes. Stay tuned!