Brace yourselves. With the two main party nominees established, we are now entering the poll-driven media phase of presidential electoral politics. For the next several months, media coverage will be dominated by analyses of trial heat polls pitting the two candidates head-to-head. Journalists will use this “hard news” to paint a picture of the campaign on an almost daily basis – who is up, who is not, which campaign frame is working, how the latest gaffe has hurt (or has not hurt) a particular candidate. Pundits specializing in election forecasting, like the New York Times’ Nate Silver, meanwhile, will use these polls to create weekly forecasts that purport to tell us the probability that Obama or Romney will emerge victorious in November.
Be forewarned: because polls are so volatile this early in the campaign, it may appear that, consistent with journalists’ coverage, candidates’ fortunes are changing on a weekly, if not daily basis, in response to gaffes, debates, candidate messaging or other highly idiosyncratic factors. And that, in turn, will affect the win probabilities estimated by pundits like Silver. Every week, Silver and others will issue a report suggesting that Obama’s chances of winning has dipped by .6%, or increased by that margin, of some such figured based in part on the latest polling data.
Pay no attention to these probability assessments. In contrast to what Silver and others may suggest, Obama’s and Romney’s chances of winning are not fluctuating on an almost daily or weekly basis. Instead, if the past is any predictor, by Labor Day, or the traditional start of the general election campaign, their odds of winning will be relatively fixed, barring a major campaign disaster or significant exogenous shock to the political system.
This is not to say, however, that polls will remain stable after Labor Day. Instead, you are likely to see some fluctuations in trial heat polls throughout the fall months, although they will eventually converge so that by the eve of Election Day, the polls will provide an accurate indication of the election results. At that point, of course, the forecast models based on polls, like Silver’s, will also prove accurate. Prior to that, however you ought not to put much stock into what the polls are telling us, nor in any forecast model that incorporates them.
Indeed, many (but not all) political science forecast models eschew the use of public opinion polls altogether. The reason is because they don’t provide any additional information to help us understand why voters decide as they do. As Doug Hibbs, whose “Bread and Peace” model is one of the more accurate predictors of presidential elections, writes, “Attitudinal-opinion poll variables are themselves affected by objective fundamentals, and consequently they supply no insight into the root causes of voting behavior, even though they may provide good predictions of election results.” In other words, at some point polls will prove useful for telling us who will win the election, but they don’t tell us why. And that is really what matters to political scientists, if not to pundits like Silver.
The why, of course, as longtime readers will know by heart, is rooted in the election fundamentals that determine how most people vote. Those fundamentals include the state of the economy, whether the nation is at war, how long a particular party has been in power, the relative position of the two candidates on the ideological spectrum, and the underlying partisan preferences of voters going into the election. Most of these factors are in place by Labor Day, and by constructing measures for them, political scientists can produce a reasonably reliable forecast of who will win the popular vote come November. More sophisticated analyses will also make an Electoral College projection, although this is subject to a bit more uncertainty.
But if these fundamentals are in place, why do the polls vary so much? Gary King and Andrew Gelman addressed this in an article they published a couple of decades ago, but whose findings, I think, still hold today. Simply put, it is because voters are responding to the pollsters’ questions without having fully considered the candidates in terms of these fundamentals. And this is why, despite my claim that elections are driven by fundamentals that are largely in place by Labor Day, campaigns still matter. However, they don’t matter in the ways that journalists would have us believe: voters aren’t changing their minds in reaction to the latest gaffe, or debate results, or campaign ad. Instead, campaigns matter because they inform voters about the fundamentals in ways that allow them to judge which candidate, based on his ideology and issue stance, better addresses the voter’s interests. Early in the campaign, however, most potential voters simply aren’t informed regarding either candidate positions or the fundamentals more generally, so they respond to surveys on the basis of incomplete information that is often colored by media coverage. But eventually, as they begin to focus on the race itself, this media “noise” becomes much less important, and polls will increasingly reflect voters’ “true” preferences, based on the fundamentals. And that is why Silver’s model, eventually, will prove accurate, even though it probably isn’t telling us much about the two candidates’ relative chances today, or during the next several months.
As political scientists, then, we simply need to measure those fundamentals, and then assume that as voters become “enlightened”, they will vote in ways that we expect them to vote. And, more often than not, we are right – at least within a specified margin of error! Now, if a candidate goes “off message” – see Al Gore in 2000 – and doesn’t play to the fundamentals, then our forecast models can go significantly wrong. And if an election is very close – and this may well be the case in 2012 – our models will lack the precision necessary to project a winner. But you should view this as strength – unlike some pundits who breathlessly inform us that Obama’s Electoral College vote has dropped by .02% – political scientists are sensitive to, and try to specify, the uncertainty with which they present their forecasts. It is no use pretending our models are more accurate than they are. Sometimes an election is too close to call, based on the fundamentals alone.
The bottom line is that despite what the media says, polls – and the forecast models such as Silver’s that incorporate them right now – aren’t worth much more than entertainment value, and they won’t be worth more than that for several months to come. As we near Election Day, of course, it will be a different matter. By then, however, you won’t need a fancy forecast model that incorporates a dozen variables in some “top secret” formula to predict the winner. Nor, for that matter, do you need any political science theory. Instead, as Sam Wang has shown, a simple state-based polling model is all you need to predict the presidential Electoral College vote. (Wang’s model was, to my knowledge, the most parsimonious and accurate one out there for 2008 among those based primarily on polling data.) Of course, this won’t tell you why a candidate won. For that, you listen to political scientists, not pundits. (In Wang’s defense, he’s not pretending to do anything more than predict the winner based on polling data alone.)
So, pay no heed the next time a pundit tells you that, based on the latest polls, Obama’s win probability has dropped by .5%. It may fuel the water cooler conversation – but it won’t tell us anything about who is going to win in 2012, and why.
Matthew,
I think this post would be more informative if you had described our forecasting model accurately. It looks at economic figures, and the past voting patterns in each state, along with polls. In fact, it discounts polls quite heavily this time of year.
http://fivethirtyeight.blogs.nytimes.com/2012/06/07/election-forecast-obama-begins-with-tenuous-advantage/
Likewise, looking at how well fundamentals-based models (those which do not look at all at polls) have actually performed at prediction, as opposed to at curve-fitting experiments that “post-dict” the election, is instructive. They’ve performed very badly and are beaten by naive methods like just guessing the election will be split 50:50.
http://fivethirtyeight.blogs.nytimes.com/2012/03/26/models-based-on-fundamentals-have-failed-at-predicting-presidential-elections/
You might also want to read Phil Tetlock’s book if you haven’t.
http://www.amazon.com/Expert-Political-Judgment-Good-Know/dp/0691123020
We believe that our model is informed by the political science, and by best practices in the forecasting literature. Of course, it still contains assumptions and approximations, as all models do, and some of them surely will be wrong. So we advise our readers to read broadly, if they might mostly avoid certain types of horse-race journalism that are biased by the need to create news when there isn’t any.
My view, nevertheless, is that it is good to translate one’s assumptions and approximations into testable predictions. Following Karl Popper, that it what puts the “science” in political science and separates it from punditry.
Nate,
Thanks for the reply. I confess that I don’t get to read your column nearly as much as I’d like to since the Times put in the paywall, so it’s possible I’ve misconstrued your forecast model.
But I don’t think so, at least based on the link you provided. Simply put, a model that discounts polling very early, but then increasingly relies on polls as election day draws nigh is likely to end up accurately predicting the outcome of the race in the end, but it is not very theoretically useful, and it doesn’t really tell us much about the state of the race today. To see why not, ask yourself this – why not rely exclusively on polls now? The answer, of course, is because they aren’t very useful right now – a point I made in my post. And why should they be, when fully 30% of the voters have paid no attention, at all, to the race so far. To the extent that the polls are predictive, it is because survey respondents are reacting to something – current issues, the candidates, the state of the economy – that causes them to favor one candidate over the other. Indeed, your model incorporates some measures of these “fundamentals” – how a state voted previously, and an economic index that is a composite of various economic measures. (I can’t evaluate your index without knowing how you weight the various measures, but I don’t doubt it is probably as good [albeit more complicated than probably necessary] as any of the measures political scientists use. So why not jettison polls entirely and focus on developing an explanatory framework? Indeed, you have a start toward developing a theory – you could explain to your readers that the previous state-level vote is useful because most people have a standing predisposition to vote a particular way (a short-hand way of capturing this is to measure the distribution of partisan I.D.), and that economic factors are a good way of gauging whether voters will be happy with the job the incumbent or incumbent’s party is doing or not.
But rather than stake your ground on some explicit voting theory, you instead gradually throw what little implicit theory you have away for the sake of getting the final results right. By the end, your forecast relies almost exclusively on polling data. And as Sam Wang has shown, it is incredibly easy to make an accurate state-by-state prediction based on public opinion polls and nothing else. So why bother with the fundamentals at all, particularly if, as you suggest, political scientist models haven’t fared much better than a coin toss in predicting presidential outcomes?
The reason, I think, is that you try to have it both ways – you want to suggest there is some theoretical underpinning to what is essentially an a-theoretic forecast model, but in the end you do what is necessary to get the forecast right. Look, if I wrote a national column focusing on election forecasting I’d be damn sure I got the final results right, even if it meant jettisoning any pretense that I am actually interested in what drove those results – which is essentially what you do when you make a final forecast that basically says “I believe people will vote this way tomorrow because that’s what the most recent polls say they will do.” And there’s nothing wrong with that. But let’s not pretend there’s more to that forecast model than that, which is what you are implicitly suggesting when you say Obama’s election chances are changing on a weekly basis. In fact, it’s almost certain that his actual reelection chances are not changing much at all – it’s your model’s results, under the partial influence of volatile polls – that is changing. The actual factors that drive the final results are not, at least not to the degree you suggest.
Put another way, to say that “if the election was held today, my model indicates Obama would have a 60% chance of winning” is intellectually misleading, because if the election was held today, you would have jettisoned most of the elements of your model on which today’s forecast is made in favor of relying on public opinion polls.
I hope this post doesn’t sound unduly harsh. I have a great fondness for your ability to present new information in graphically interesting ways and, as I noted above, I wish I could read your stuff on a regular basis. But I just want to be clear to my readers why I believe that, although I’m sure come Nov. 2 you will present a very accurate presidential election forecast, it is not based on any clearly articulated forecast model that allows us to say how the candidates are doing today. But I’m certainly open to and eager for correction.
Hi Professor,
I was wondering how this line of thought would apply to other races–like senatorial, congressional or even gubernatorial or AG races. The work I’m doing this summer is requiring me to look at some of these races and determine which ones are going to be “close.” There are other ways to determine whether a race will be close, I guess, but it would seem that polling data is the most “concrete” way. Right now these races are in the early stages–as you know, many of the primaries aren’t over yet.
So basically my question is, are polls at all a reliable indicator for possible “close” races at this point? If I’m looking for races that are polling within 5% or 10% right now, is that the best strategy for finding the potentially close races? Is there another (better) way to go about it.
Thanks in advance.
All the best,
Adam
I worry that there’s some misunderstanding happening here. If, when you talk about “Obama’s election chances,” you mean the probability that Obama will be elected in November, then we have to bear in mind that all probabilities, including this one, depend in a vital way on the information we have. These numbers are by their very nature _subject_ to what we know, and if what we know changes, it follows that these numbers must change as well.
For instance, suppose someone flipped a coin, and asked you what the probability was that it came up heads. If you could not see the coin, you’d likely reply that the probability was 50%. If you saw the coin land, and saw that it came up tails, you might answer 0%. Either way, you’re correct! The correct answer depends in a critical way on what you know about the situation. In short, probability is _defined_ relative to the information we have, and consequently, new information actually changes the probability.
Now let’s examine the first scenario (you don’t see the coin land) more carefully. You may know some more subtle information. You might realize that the coin is quite thick, perhaps, and that the ground on which it will land is quite soft. That makes you suspect that there’s a very tiny chance — 0.00004% — that the coin will land on its edge. So you might account for that and revise your probability downward 49.99998%. This is totally reasonable. The point here is that changes in your knowledge give rise to changes in the probability. In a real world scenario, there might be a variety of other things you know too; you probably can’t take them all into consideration, so you’re forced to make an estimate. But the more information you do take into account in your computation, the closer you will be to the actual probability.
My understanding of what Nate Silver does is this. He takes all the information he can reasonably use, and he computes two probability estimates. One, the forecast, includes all of this information, polls, fundamentals, and how much each factor should be weighed, given that the election is some time away. The other probability estimate is the “now-cast,” which uses only polling information; it’s supposed to be an estimate of the outcome in the parallel universe in which the vote, for some reason, took place today.
This all seems totally compatible with both mathematics and political science.
Matthew,
You have many strong opinions about my work. It is a little frustrating that some of these do not seem grounded in a careful effort to review it.
Nevertheless, it is clear that my views are quite different from yours in some important ways.
For instance, I find the whole distinction between theory/explanation and forecasting/prediction to be extremely problematic.
As a discipline, political science has done fairly poorly at prediction (see Tetlock for more, or the poor out-of-sample forecasting performance of the “fundamentals based” presidential forecasting models). Political science is hardly alone in this regard. Nevertheless, this ought to inspire questions about whether the theory has as much grounding in reality as we might like.
Instead, the theory may be a reflection of our prejudices — toward overconfidence, toward perceiving random data as signal, and toward oversimplifying complex processes that we do not understand very well.
Tetlock’s book also suggests that certain of the value preferences that you’ve asserted (e.g. toward parsimony, toward “staking your ground” rather than seeking consensus, toward disliking models that update with new information) are correlated with worse forecasting performance. These are satisfying beliefs, but they are not necessarily good heuristics for producing better science.
Nevertheless, my view is that some of the theory on voting is robust enough to have survived some empirical scrutiny. For instance, the propositions that (i) poor economic performance hurts the incumbent president on balance (ii) voters weigh relatively recent information more heavily than distant information, and (iii) that voting is highly partisan, are reasonably robust and are reflected in the design of the model.
Essentially, the model uses these propositions as Bayesian priors. It starts out “believing” in them, but concedes that the theory is probably wrong if, by the time we’ve gotten to Election Day, the polls and the theory are out of line.
Nate, I’d be interested in seeing your actual model (i.e. the math). Is this available anywhere? Like Matt I’d be curious about the weighting in the economic index. Also, are your parameter estimates (or Bayesian priors) based on analysis of previous elections, and if so, which ones?
-Bert
Adam,
Without knowing which elections you are focusing on, I can’t be certain what is most useful to you in predicting whether those elections will be “close” or not. While I wouldn’t dismiss polling data entirely, even this far in advance, I’d pay more attention to other factors if they are available. The most critical one is how the the previous election played out. How did the incumbent (if there is one) do? Are there any factors in play this time around that would suggest the need to alter the degree to which you think the fundamentals will influence outcomes this time around? The key point is that elections do not take place in a historical vacuum; factors that influenced an outcome in the previous cycle are likely to come into play in this one as well.
C. Barwick – I don’t take issue with the process you have laid out, provided some additional information is provided. To begin, let us start with the baseline model. In your example using a coin toss, I understand why the baseline begins with a simple 50/50 proposition. However, I have no idea how Nate constructed his initial baseline. He indicates that it is partially based on economic factors, but I can’t see how he established that baseline – from what previous elections did he construct his index? How are the variables weighted?
You then suggest that if the coin fell on soft soil, you might adjust your prediction to take into account the increased probability of a coin landing on its edge. That’s fine – you’ve adjusted your priors base on a change to your explanatory model. Nate does not do this. He adjusts his priors based on changes in polls – but there’s no theoretical basis to explain why these polls don’t comport with the starting model. It’s an entirely atheoretical adjustment. That’s perfectly acceptable if all you want to do is to predict the final result. But it’s not very theoretically satisfying – we still don’t know why voters aren’t behaving the way they did in previous elections.
Does this help clarify my objections to Nate’s use of polling data?
Thanks for the comment.
Nate,
You make a number of interesting points that deserve a more complete response than is possible in a comments section. I’ll try to respond more fully in a separate post.
It does … But I guess I’m confused. Shouldn’t a new poll or movement of the Dow Jones count as a small amount of new information? If it does, then to me it doesn’t seem problematic that it will have a small impact on an estimate of a probability, and that, over time, these estimates will actually converge to an excellent approximation of the actual vote. In your post, I understood you to be suggesting that this state of affairs is necessarily absurd.
As a mathematician, I would certainly love to know more details about Nate Silver’s computational methods, but I don’t regard the fact that there are small daily movements alone as evidence that these methods are flawed. Rather, it seems to me quite natural to expect such tiny variations.
I take your point that polling data do not explain what causes voters to behave as they do. But in order to develop a good theory about voters’ behavior, don’t we first have to sort out just what it is they are doing? As I understand it, Nate Silver’s computations provide nothing more than an estimate of the probability that voters will behave in a particular way, based on all the data we can collect now.
Am I still missing the point?
Clark,
In theory, there is nothing wrong with using new information to update one’s forecast, assuming the new information is meaningful in the context of what we are trying to measure. It’s hard to judge whether that is the case with Nate’s model, since we don’t have access to the forecast model’s parameters. As for sorting out exactly what voters are doing – well, that is precisely my point. You need a theory for that. If, in the end, all you are doing is basing your forecast on what people tell you they are going to do, there’s no theory to explain why they acted as they did. At some point you have to have a theory – I don’t know what Nate’s theory is, and – based on his last comment – my sense is he’s content not to have one, in the end. There’s nothing wrong with that. But it’s not what drives most political scientists – we are interested in explanation more than prediction.
That makes sense and is very helpful. I was considering some of those factors, but it is clear that I wasn’t giving enough weight to them.
Thanks for the advice.
Matthew,
First, let me say that I enjoy your blog a lot. It’s one of the very few I take seriously. So I’m not trying to be annoying or argumentative; I’m trying to understand your objections more carefully from first principles, precisely because I take your views seriously.
I say “from first principles,” because I’m a layperson in political science. In other branches of science (with which I’m more familiar), data collection and data analysis precedes data interpretation. So if we perform an experiment, we try not to have our expectations for the phenomenon we’re studying come to bear on our measurements. Nor do we want to build our prejudices into the mathematical analysis of that data. Then we’ll just be confirming our prejudices, which is dull. We go to great lengths to avoid this.
For example, Kepler developed his laws of planetary motion by studying the data from Brahe’s remarkably accurate observations. If Brahe had decided early on that all planets must be moving in perfect circles and fudged his numbers accordingly, Kepler would have developed bogus principles.
Similarly, I want to argue that if a goal of political science is to develop “laws of voter motion,” then the starting point should be remarkably accurate data.
Now when I speak about “voter motion” or “what voters are doing,” I just mean which way they are voting. I think you don’t need any theory to find that out; you just need an election! Theory would then hopefully explain subtler information: how voters made the choices they did, what values weighed on their minds, how they consumed news, etc., etc. This is all information that cannot — and should not — be packed into a single vote tally; rather, it should explain something nontrivial about it. But before you can explain that vote tally, you have to have that vote tally.
Now the problem is, elections are uncommon, so election data don’t come quickly enough. So we might try to cheat and get hold of some imperfect election data now. We might start with polls, but since the data they provide are very noisy, we might try to eliminate that noise by averaging polls using Bayesian statistics, and we might add whatever other information we might be able to scoop up to estimate probabilities. This gives you some bit of data against which to test theories that might purport to explain how people vote.
Now, how should the information used in this estimate be modeled? I’d argue that we want the best possible estimates of the real probability, given everything we know. How do we work out which model is best? I suggest only two criteria: (1) it’s rooted in sound mathematics, and (2) it gives excellent estimates for data from earlier elections.
I’d argue that theory specifically must not enter in analyzing the accuracy of the model, because you’re not yet trying to explain your data; you’re still just trying to collect and analyze it! (I mean “analyze” just in the sense of statistical analysis, not explanation.) Inserting the theory whose hypotheses you wanted to test against this data at this point would ultimately mean begging the question.
My view of Nate Silver’s work is that it is sophisticated data collection and analysis. If, as you say, he hasn’t got a theory in mind when he does this work, then so much the better. This provides us all with data — tied to no particular worldview — that can serve as testing ground for different theories that hope to explain why voters vote as they do.