No, Political Scientists Are NOT Lousy Forecasters – In Fact, They Are Pretty Good

First Nate Silver. Now Jacqueline Stevens!  Stevens is the Northwestern political scientist whose op ed piece in Sunday’s New York Times  sparked more than a little debate regarding the role of government efforts through National Science Foundation grants to fund political science research, and the quality of that research more generally.  Stevens leaves no doubt that she considers most NSF funding to have been wasted; as she writes,  “It’s an open secret in my discipline: in terms of accurate political predictions (the field’s benchmark for what counts as science), my colleagues have failed spectacularly and wasted colossal amounts of time and money.”

Not surprisingly, that rather sweeping statement generated a lot of pushback, much of it from political scientists (see here and here and here, for starters.) As the critics point out, her overly-broad generalization and cherry-picked examples misreads much of what political science tries to do, and how well it has done it, and in the process manages to distort Phil Tetlock’s argument in his recent book regarding when and why political “experts” often miss the mark in their predictions.  (Hint: Stevens’ op ed piece show evidence of falling prey to some of the cognitive biases Tetlock cites in his book.)

Rather than belabor these points, I want to focus on another claim she makes as a way of explaining more generally why her criticism misses its mark by such a wide margin.  In pointing out forecasting flaws in the discipline, Stevens’ writes, “Political prognosticators fare just as poorly on domestic politics. In a peer-reviewed journal, the political scientist Morris P. Fiorina wrote that ‘we seem to have settled into a persistent pattern of divided government’ — of Republican presidents and Democratic Congresses. Professor Fiorina’s ideas, which synced nicely with the conventional wisdom at the time, appeared in an article in 1992 — just before the Democrat Bill Clinton’s presidential victory and the Republican 1994 takeover of the House.”

The problem with Stevens’ assertion is that it completely mischaracterizes Fiorina’s published research on divided government.   (Full disclosure: Fiorina sat on my dissertation committee [something he rues to this day, no doubt], and was later a colleague of mine [again, not necessarily by his choice!]).  In fact, Fiorina’s research laid out a model that sought to explain the persistence of divided government in the post-World War II era despite on-going claims by pundits (and some political scientists) that it lead to policy gridlock and political stalemate.  And yet Americans more often than not voted in a way that maintained divided government.  Why did they so frequently act against their own presumed self-interest, and in favor of gridlock and stalemate? For Fiorina, the answer was rooted in part by the (not always fully articulated) desire among a subset of Americans to achieve a modicum of ideological moderation by “balancing” the parties in the face of increasingly partisan polarization.  In short, as the parties become more extreme, voters are more likely to split their tickets. Now, he does not claim this desire for “balance” is the only factor contributing to divided government.  Other structural factors, such as the increasing professionalization of state legislatures, which serve as a crucial breeding ground for individuals who eventually run for Congress, certainly influenced the long period of Democratic control of Congress, since that professionalization disproportionally advantaged Democrats.  That meant voters who sought to balance the two parties at the national level usually focused on voting in a Republican president. He acknowledges as well that his balancing model is but one of several possible ways that voters’ choices might produce divided government.

The crucial point, however, is that contrary to Stevens’ assertion, the transition to a Democrat president and Republican-controlled Congress in 1994 after the Republican midterm wave is perfectly consistent with his explanatory model! Indeed, in looking at state governments, Fiorina explicitly lays out a model that explains when states might choose a Democratic governor and a Republican-controlled legislature.  That model is applicable to the national level.  In her op ed piece, Stevens would have us believe that Fiorina’s model predicted that under divided government Democrats would control Congress and Republicans the presidency indefinitely.  But that is not what he predicted – he laid out a more general model of divided control which encompassed different political permutations that included varying patterns of Republican and Democrat presidents and Congresses.  The key to evaluating his model, then, is whether the change in the composition of divided government from 1990 to 1994 is consistent with his premises.  That’s the proper test of his forecast – one Stevens ignores.

More generally, Stevens’ mischaracterizes the nature of political science forecasting.  Thus, Fiorina’s forecast model is not designed to say that in 1994, because we had a Democratic president, Republicans would reclaim control of the House and the Senate.  (In fact, I know of very few political scientists who saw this happening!)  Instead, as with most forecast models, Fiorina constructed a probabilistic explanatory framework premised on some clearly articulated assumptions.  That is, he suggested that when certain factors were in place – extremely polarized parties, a difference in the perceived strength of the presidency and Congress – many voters would act in a particular way that would often lead to divided government.  The results in 1994 are consistent with his argument.

And this gets back to my debate with Nate Silver, and to the importance of theory more generally.  Anyone, looking back on national elections dating to the Truman presidency, can predict that the odds are good that we are going to see divided government again in 2012.  But without a theory, we don’t know why, which renders the prediction essentially useless.  What if, in fact, Republicans sweep to power and control all three branches come November?   That might not mean our model of divided government is wrong – in fact, it might be fully consistent with that model.  It all depends on the underlying theory – did we actually have the conditions that the theory says must be in place for divided government to occur?  For political scientists, then, it matters why a prediction is right or wrong.

And this leads to my second point: when outcomes do not comport with theory, we learn something that, hopefully, allows us to make the theory better.   (Of course, this doesn’t mean a post hoc readjustment of the theory to fit the latest data point.)  Forecasting is a way to test whether our understanding of an event is correct.  In making that forecast, however, political scientists try as well to signal how specific the likelihood of an outcome is.   We might not be able to state that the Soviet Union will collapse in 1992 anymore than the medical doctor can tell you if and when you will get cancer.  But perhaps we can make some general statements about the conditions under which authoritarian regimes are more likely to collapse, and with what probability, just as the doctor can recommend some dietary and environmental factors that medical science suggests are likely to decrease your chances of getting cancer.

The reality is that because of political science, we know a lot more about politics than would otherwise be the case.  In some areas – such as forecasting presidential elections – we actually know quite a bit (more than Nate Silver’s criticisms about election forecasting indicate, I would argue.)  Moreover, the best political scientists – because they rely on clearly articulated theories and have a method for testing that theory – are less prone to making the types of errors that Tetlock associates with “experts”.  This is not to say we don’t make mistakes.  But even when our forecasts prove wrong, learning can still occur.

For all these reasons, I disagree with Stevens’ assessment of the state of the discipline, at least that portion of the discipline with which I am most familiar.  Given the limits of what can be expected in a probabilistic world, political scientists are, in fact, pretty good forecasters, and they are getting better.