For me, its satisfying to have responsible poll aggregation. End of story.
It serves a useful purpose for sorting most if not all of the poll cherry-picking that happens almost hourly on every internet site, every newspaper, and every tv news show. I get calls every day from some breathless friend telling me this or that about some new poll.

In the face of ever-increasing high quality and accurate poll aggregation I find it amazing that there continues to be so many badly written cherry-picked poll stories, and that the state of the horse race itself comes to dominant the campaign to the extant that it does.

Ultimately the debate over who is going to win simply re-litigates the underlying policy debate, but at one level of abstraction removed from actual policy. If reliable poll aggregation can be used to debunk and quiesce irresponsible chatter about the horse race maybe the pols will re-focus on policy differences.

I think part of the problem is that responsible poll aggregators have been transparent in their partisan affiliations — Nate, Sam Wang, HuffPo etc. Because of this, “mainstrain media” types have gravitated toward Real Clear Politics, which somehow appears more neutral …. and Nate, the popular guy from , gulp, the NY Times. Naturally the Right will beat up on Nate. Let’s not go there right now.

To summarize my points.
- Accurate poll aggregation is, in and of itself very valuable, particularly in helping us sort through the blitz of polling data presented every day.
- I personally don’t require election prediction.
- Poll cherry picking is still the dominant way in which polls are reported in the main stream media.
- Were poll aggregation to be legitimatized and used in mainstream poll reports, it might put pressure on candidates to move poll numbers through policy rather than spinning the horse race or spawning biased pollsters.
- Poll aggregators have work to do before they are seen as credible sources by mainstream media types for sorting out incoming polling data.

I would like to see poll aggregation become credible and move into mainstream reporting.

Paul Collacchi

I want to address what I believe to be an inadvertent mixing-up of concepts that has popped up in this discussion. Basically, I think the term “prediction” is used too loosely. I’ll make an analogy to weather forecasting.

Weather forecasting begins from immediate measures such as temperature, wind speed, and other current conditions. Adding likely future changes, whether on a short time scale (prevailing wind) or a long time scale (climate modeling) can add useful information to inform the future.

When you’re watching a hurricane, what you want is the person with current conditions and short-term trends. However, to learn about the next hurricane, different expertise comes in.

(1) What a poll analyst like Nate Silver or myself starts from is a snapshot of current polling conditions, sometimes with uncertainties added into the mix. In the analogy, we are weathermen.

(2) Political scientists like Ray Fair and successors provide a prediction of a future event. In the analogy, he is the climatologist.

These approaches often overlap, though it is not acknowledged explicitly. Combining them produces a third category (3), in which current conditions (the snapshot) and likely future changes can both inform a true prediction for a current race.

For example, Silver makes efforts to draw upon (2) a bit, and calls his hybrid calculation a prediction. Calling it a prediction satisfies his readers’ desire for one. He also provides very interesting ongoing commentary. All of this is legitimate, but I don’t think it’s appropriate to be too negative about approach (2). The weatherman should not castigate the climatologist.

To my own taste, such a hybrid approach does not add all that much information to the Presidential race. It tends to conceal what (1) and (2) above can each tell us separately — and approach (2) is open to debate, as seen in this thread. Thus at the Princeton Election Consortium I have in the past provided (1), a pure snapshot of polls.

However, a hybrid approach (3) is very useful for assessing individual races where less data are available, such as House and Senate races. This is an important practical application. If done cleanly, one could imagine using a variety of variables, including past trends and campaign spending, to inform a true prediction.

Now, a longer response, which arose from some recent correspondence I had with Carl Bialik at the Wall Street Journal.

All the best,
Sam Wang

>>>>>>>>>>>>>>>>>>>>

Nearly all online commentary on polls is not really a forecast, but instead a snapshot of current polls. Sometimes people add uncertainty to the snapshot, either inadvertently (suboptimal statistical analysis) or on purpose (assuming future trends), then call the product a forecast. But let’s unpack that a bit.

Most readers here probably agree that a traditional forecast is what Ray Fair and others do with the bread-and-peace model. (Aside: Pardon me if I fail to cite later efforts, which are interesting but don’t change the essential thrust of my argument). For the 2012 Obama v. Romney race, models like that suggest a very close race. This is not much fun, but they could be useful for evaluating where a current snapshot of polls might go in the future. Bread-and-peace models could be used to add random drift in national opinion (in the form of a single parameter that drives polls in one direction, together) that generally nudges things toward the bread-and-peace prediction. In some sense, this is basically a Bayesian prediction.

It is possible to embed such drift into a model in a more hidden manner, but that makes it hard to see the current snapshot. Since predictive models have only moderate power, I would recommend that they be added on separately from the polling snapshot.

>>>>>>>

I want to bring up a separate topic I am thinking about. In fact, I’d be interested in having a collaborator to work on it. Readers here are welcome to contact me at sswang@princeton.edu. The core question is: how can one use polling data to generate an optimal snapshot of conditions today?

I am thinking about a common analytical error in producing a poll snapshot. Here are some observations:

(1) Nearly all pollsters are professional enough in their methods to provide statistically informative information. By this I mean that their results can usually be understood as being a sample of true current opinion, plus/minus some consistent, ongoing bias due to their individual stratification and sampling methods. This view is well supported by multiple sources, including Charles Franklin at the University of Wisconsin, as well as David Shor from the Stochastic Democracy blog, who visited me a few years ago.

For example, Rasmussen and Gallup are both useful sources of information, despite the fact that they often appear to say different things. It’s like having a bunch of clocks, some of which are set fast and some of which are set slow, but all of them run at the correct rate.

(2) Therefore finding that correction offset first, then putting polls together in a meta-analysis, is an approach that approximates the true uncertainty.

Early approaches to a correction were not quite right. In 2008, Silver’s Pollster-Introduced Error turned the offset into a multiplicative factor. However, this leaves the original offset error in place, which might explain a tendency toward underconfidence in his expressed probabilities. Even in 2010, knowledgeable professional statisticians calculated an average before smoothing, which again baked the offsets into the final estimate. Both of these approaches tended to overestimate the true uncertainty. They were also susceptible to having too many polls from one source such as Rasmussen. See some interesting related commentary by Andrew Gelman here: http://andrewgelman.com/2010/11/some_thoughts_o_8/

Based on past Presidential races, it’s possible to have a day-by-day barometer that is effectively accurate to 0.1-0.2% swings in public opinion. That is the equivalent of my error on Election Eve 2008 of 1 electoral vote: http://election.princeton.edu/2008/11/11/post-election-evaluation-part-2/. I note that I did not calculate pollster-specific offsets, which made my apparent uncertainty rather high. This worked because pollsters are, when averaged, not very biased.

(3) A high-quality estimator would help identify truly game-shifting events (i.e. adding Sarah Palin to the GOP ticket), as distinct from individual events that mostly don’t have immediate effects on the Presidential race.

However, economic and competitive forces work against a commercial poll aggregator taking on such an approach. The resulting snapshot would be very stable (see my 2004 and 2008 graphs at http://election.princeton.edu/history-of-the-2004-race/), and would therefore be less interesting to watch. This is not to the advantage of aggregators who depend on pageviews: RealClearPolitics, FiveThirtyEight, TalkingPointsMemo, and so on.

In my opinion, where commercial and amateur aggregators add a lot of value is by analyzing individual House and Senate races. Potentially, they can do very well at filling in missing information.

All the best,

Sam Wang
Princeton Election Consortium, http://election.princeton.edu
Associate Professor, Molecular Biology and Neuroscience
Princeton University

Again, much of the post-election reporting gave at least some credit for Walker’s victory to his sizable money advantage. In this case, of course, a widely held distaste among Wisconsin voters for the recall process itself was also reported to be one of the key factors working to Walker’s advantage. No shortage of variables in the mix, I guess! But, while I take your point that campaign financing doesn’t deserve a place at the forecasting fundamentals table, I’m not so certain that the Walker/Barrett race is the best example to prove that point.

As Dwight suggests, my view is that the reaction to Citizens United has been out of proportion, at least inasmuch as that reaction presumes that money is buying election outcomes. If Citizens United has done anything, it may have been to increase the level of competition by providing a source of funds for underfunded challengers to tap. Indeed, in the 2012 presidential primaries, Gingrich and Santorum might not have had the money to continue as long as they did, had it not been for Super PACs. In the case of the Wisconsin recall election, Republican Governor Walker outspent Democrat Tom Barrett 89% to 11% in terms of total money spent WITHOUT including outside spending in the equation. When you add in the outside spending on both sides, Walker and Barrett were actually more even: Walker and his allies spent 72% of the total, while Barrett and his allies spent 28%.

So in the admittedly few cases we have so far, the outside spending from Super PACs and others seems to have made elections more competitive. But this effect of money on election outcomes is probably only because money helps underfunded candidates make their names and messages to the public. The public still has to accept or reject these messages. This is why, in presidential general election campaigns in which both sides are well funded, forecasting models do not include campaign finance figures — nor would they improve by doing so.