Another nice teaching moment, this one courtesy of Jay Cost at the Horserace blog and Mark Blumenthal at Pollster.com. Gallup released their latest generic ballot results under the headlines “Democrats Jump Into Six Point Lead on the Generic Ballot”.
As the following table shows, this jump comes after many months of polling data in which Democrats either trailed or were tied (within the poll’s margin of errors) with Republicans in the generic question.
As I’ve blogged about before, the generic ballot question is a useful predictor of the number of House seats won by each party in midterm elections, so people pay attention to these results. (Note, however, that Gallup is not yet using their likely voter screen – when they do, history suggests Republicans will gain about 5% over the results of their polling of registered voters.)
Not surprisingly, progressive bloggers like Tom Schaller at Fivethirtyeight.com and Andrew Sullivan – while issuing the usual caveats (it’s only one data point, it could be an outlier, etc.) – jumped on these latest results as a possible sign that Democrats were regaining support among voters. Sullivan opined: “It is unwise to discount the intelligence of the American people – a trait more endemic among liberals than conservatives. The latest Gallup generic poll is striking – because it suggests that voters in the end may vote on substance not spin and ideology.”
But what substantive issue explains the reversal in voter sentiment– was it the financial legislation that just passed Congress? Voter backlash to the Republican handling (see Joe Barton) of the Gulf oil spill? Belated recognition that the health care legislation is a good thing? Growing disgust with Republicans as the “party of no”?
How about none of the above? Yes, the six-point Democrat advantage lies outside the poll’s margin of error. But as Blumenthal reminds us, Gallup’s results are based on probability sampling – remember, each Gallup poll is followed by some version of this methodological blurb: “one can say with 95% confidence that the maximum margin of sampling error is ±3 percentage points”.
What does this mean? Essentially, if Gallup repeatedly sampled the population of registered voters, approximately 95% of the time the sample results would fall within 3 percentage points (plus or minus) of the actual proportion of Democrat and Republicans supporters in the generic ballot question. Put another way, 5% of the time the Gallup poll results might fall more than 3 percentage points from the actual proportion of Republican and Democrat supporters. That means we can expect, very rarely, a result that lies outside the margin of error even if there’s been no actual opinion change.
Now, we can’t know what that actual number of Republican and Democrat supporters is – Gallup can only estimate it by taking a random sample of all registered voters. Let’s assume, however, that the actual support for the Democrat on the generic ballot question is pretty close to the average Democratic support among registered voters, as polled by Gallup for the last 20 weeks. That average number is 45.6%. In that 20-week period, Blumenthal finds only one poll that lies more than 3% (plus or minus) from this average – and that is the most recent poll showing 49% support for Democrats in the generic ballot. Note that this most recent result lies is just barely (.4%) outside the margin of error.
Bottom line? I can understand why Sullivan equates wisdom with support for liberal policies, and why his world view might color his interpretation of the polling results. But, as always, we need to separate our personal preferences from our analysis of the facts. It’s possible Democrats are picking up support among voters as we move closer to the midterm. It’s also possible, however, that this is a perfectly predictable statistical fluctuation associated with polling based on random sampling, and that there’s been no real change in voter sentiment at all. We won’t know for certain until several more polls are in. But I would be cautious about discarding 20 weeks of results on the basis of a single poll.
In the meantime, skepticism rather than certitude should be your watchword.
Addendum: I’ve tweaked the wording of my original post in order to better describe the meaning of the phrase “margin of error” in the context of probability sampling. The substance of the post – that we can’t be sure this most recent result indicates a real shift in voter sentiment – hasn’t changed.