Monthly Archives: September 2008

Interpreting the polls: Likely versus registered voter surveys

I will have only brief opportunities to post during the next several days since I am traveling to give an election talk, but I wanted to respond to several of your comments and follow up on some points I made in my last posting regarding how to interpret surveys.  Those of you who are following the RCP (RealClearPolitics) site see that the Republican post-convention bounce seems to have peaked, although McCain retains a slight lead – within the margin of error – on most polls. However, a new tracking poll – the DiageoHotline tracker (Diageo, the sponsor, is in the drinks business) – has Obama up 4% over McCain.   Even factoring in the margin of error (about 3% on their poll), this makes Diageo an outlier from the other recent polls which have McCain in the lead.  How can this be?

The answer, I suspect, is that Diageo is sampling registered voters.  Most other major pollsters are winnowing their sample to focus on likely voters.  Which is more accurate?  This is still a matter of debate, but most pollsters believe that as you get closer to the election, the likely voter model is a bit more accurate. So why is McCain doing better in the likely voter models?  As I explained in my last post, historically Republicans tend to turnout better than their registered numbers would indicate, so pollsters adjust their likely voter models to oversample from Republicans.  Since Diageo doesn’t present crosstabs, I can’t tell how many Republicans they have in their sample, but my guess is that it is less than what pollsters using a likely voter model are including.  More generally, if you average the RCP national polls using likely voters versus those using registered voters, you’ll see a slight advantage for Obama in the registered voter models, while McCain does better in the likely voter surveys,

This is a reminder of two points I’ve made earlier: First, the RCP rolling average of the polls lump polls using different methodology together, so use with caution.  Second, if these pollsters using likely voter models are in fact underestimating Democratic turnout, or overestimating Republican turnout, they could be underestimating the actual level of support for Obama.  What I will try to do as the campaign continues is to look at individual surveys to see whether there are obvious differences in the results that can be traced to their sampling techniques.  This might provide clues as to whether some pollsters are missing hidden Obama support or not.

In a related issue, several of you have suggested that because so many people rely solely on cell phones, telephone surveys that use only landlines may be inaccurate. In 2004 about 7% of households had cell phone-only coverage.  That total may be as high as 15% today (I need to double check these figures).  And statistics show that younger and more affluent people are more likely to use cell phones. So doesn’t that suggest that pollsters calling landlines are likely under sampling from Obama voters?  Not necessarily.  In 2004 several studies were done to estimate whether the failure to sample cell phones was distorting survey results.  It turns out it didn’t.  The reason is because pollsters were very careful to weight their sample by age.  This meant they captured enough young people in their sample – even without calling cell phones – to accurately predict the impact of the youth vote in the 2004 election.  As long as they do so in this election campaign – as long as they accurately sample the demographic most likely to use cell phones – the failure to actually survey cell phone users should not bias the survey results.  Note also that some firms – such as Gallup – are now including cell phones in their survey.  Again, the lesson is to pay attention to the sampling techniques – not all polls are alike, although RCP treats them all alike.

Finally, we have one more indication of the Palin effect on the race. Obama adviser David Axelrod claimed in the Washington Post today that the post-convention Republican bounce, as reflected in national polls, is really only picking up movement toward McCain in red states.  In fact, this is only partly true – while Axelrod is right that the red states have become “redder”, McCain also gained about 4% in the battleground states as well since picking Palin.  That’s why in many of these online electoral vote calculators McCain has now pulled even or is ahead of Obama – because the Palin pick is swinging some independents and women into the McCain column.  But will that bounce stick in light of “troopergate” and increased scrutiny on Palin?   More on that in a bit.

Are the polls systematically underestimating Obama’s support?

Those of you who followed these posts during the nomination campaign will remember my constant refrain that not all surveys are alike and my determination to make you look at their sampling techniques in order to evaluate their accuracy. We should have the same concerns when looking at polling numbers in the general election. For much of the summer, most polling organizations simply polled registered voters.  This is appropriate for understanding the potential outcome of a presidential race.  But in trying to forecast the likely outcome, many pollsters believe it makes little sense to rely on a random sample of all eligible voters, or even registered voters, since we know that only 6 or so out of every 10 such voters (or even less) will actually cast a presidential ballot (and yes, I’m not one of them). As a result, at this stage of the race, most of the national polling organizations switch their polling and begin surveying likely voters. They want to randomly sample from that subset of voters who are actually going to vote, as opposed to simply being eligible to vote.  As Garrett Saito suggested in an email to me, however, that raises an important question: how do they determine the likely voters?  Garrett, and others, have suggested that it is possible these polling organizations might be underestimating Obama’s support.  Why might this be?

Some of you might recall that earlier this year in June Gallup ran two polls – one of likely voters, and one of registered voters. Obama led McCain among registered voters, 47% to 44%, but McCain led among likely voters, 49-44%.   And, in fact, historically (at least as far back as we have polling data), Republicans have been more likely to be included in likely voter models despite usually lagging behind Democrats in terms of registered voters.  In the actual presidential elections, the Republican candidate has often won despite the fact that there are usually more registered Democrats which appears to validate this sampling approach.  So Gallup and other organizations typically oversample in their likely voter model from Republicans, relative to their numbers among registered voters.  This is usually appropriate, but the key question is whether that dynamic still holds true in this election; is the traditional means by which pollsters identify likely voters still valid, or are there reasons to believe that the prevailing methodology needs to be reassessed?  Some observers  suggests that because Obama is attracting such strong support among younger people and others who have never voted before, the likely voter samples might be overestimating McCain supporters, and underestimating Obama voters.  In a close election, the argument goes, these newly registered voters might swing the election to Obama, but the polls will miss this.  As evidence, they point to party registration figures in a number of states which indicate that enthusiasm for the Democratic ticket is much higher than in past elections.

Is there any evidence to support this argument?

To answer that question, we need to understand how polling organizations determine likely voters.  Unfortunately, each polling organization has their own method for determining likely voters and not all of them reveal how they do so. (If you are interested in looking at this topic in more detail, here’s a link to a discussion from 2004 by polling expert Mark Blumenthal which is the one I rely on most frequently.  Keep in mind, however, that some organizations may have changed their methodology since then):

http://www.mysterypollster.com/main/2004/11/likely_voters_v.html

In examining how polling organizations determine likely voters and whether they are underestimating Obama’s support, I look at the following issues.

1.      Does the polling organization use previous voting as one of their indicators of a likely voter?  This could potentially lead to underestimating Obama’s support if most of these newly registered voters are likely to vote Democrat. (In 2004 survey organizations using previous voting as part of their likely voter screen included ABC Washington Post, AP-IPSOS, ARG, CBS/New York Times, Democracy Corps, FOX/Opinion Dynamics, Gallup, Harris, LA Times, Newsweek, Pew, Quinnipiac, Rasmussen and Time.)

2.      Does the polling organization weight its sample of likely voters by party – that is, does it assume a priori that a certain percentage of voters will be Republican, Democratic, Independent, etc., and adjust its survey results accordingly? If so, and this weighting underestimates the percentage of Democrats who are likely to vote, then again the poll could underestimate Obama’s support, assuming most Democrats vote for Obama.

3.      Does the polling organization attempt to calibrate their likely voters sample according to expected turnout?  For example, Gallup historically chooses an expected cutoff figure for turnout – say 60% – and uses that to adjust its likely voter model. Again, if turnout is much higher due to an influx of Obama supporters, that could skew the likely voter model.  In 2004 seven survey organizations – ABC/Washington Post, Gallup, LA Times, Newsweek, Pew, Quinnipiac and Time – used this method.

So, what does this mean for polling results in this election cycle?  Is there reason to believe the polls are systematically underestimating Obama’s support?  There is no single answer to this question, in part because each polling outfit uses slightly different methods for determining a likely voter. Certainly there is the potential for bias, and the polling organizations are well aware of this.

One way to counter potential bias is not to rely on any single methodology for determining likely voters. In previous posts I have cautioned about relying on the RCP average of the polls because it is rolling average that includes polls from different time periods in its estimate, and so can be slow to react to trends. On the flip side, however, by averaging polls using different methodologies, it is less likely to be biased toward any one method for determining the likely voter and hence is less likely to be biased against Obama.  So the RCP poll may be less biased in terms of likely voter models.

More generally, keep the following points in mind when considering the issue of bias in the polls. Most importantly, we know, based on past elections, that already 80% of voters have likely already determined for whom they will vote in the 2008 election. From this perspective, the ups and downs in tracking polls can be understood as partly statistical noise reflecting errors in estimation, and partly a measure of which candidate, in voters’ minds, is getting the better of the argument or media coverage, at the time.  These variations aren’t necessarily capturing changes in how people are planning to vote.  The exception is that subset of voters – particularly independents – who tend to make up their mind very late in the election cycle. Historically, likely voter models becomes more accurate as the campaign winds down, for the simple reasons that as more people become interested in the election, it is easier to choose a likely voting sample.  Second, every four years we hear about how younger voters will make a difference in the election, and every four years they don’t.  In 2004, voting among 18-29 year olds was up, but the increase was not as much as among older voters, and Bush benefited by the overall increase in turnout more than did Kerry, and expanded his support from 2000.  So, until voting patterns change, it is not unreasonable for pollsters to rely on models that have worked well in previous elections. Third, while it is true that enrollments are up disproportionately among Democrats in many states, we have to see if the Sarah Palin factor begins to counteract this – it may be that likely voter models are underestimating her impact as well.  In particular, if Palin’s support is drawn disproportionately from the bitter, religious-leaning, gun-toting blue collar workers who supported Clinton, then likely voter models based on party could be skewed against the Republicans.

The bottom line is that this is a precedent breaking election.  This means that at least some likely voter models will be off this time around. However, until we have conclusive evidence that voting patterns are systematically different, it is difficult to say which models are wrong or why.  At best, then, we need to read these polls results with caution and avoid relying on any single result.  Pay attention to the fine print describing how they determine a likely voter. Taken collectively, however, these polls remain the best source of data we have regarding voter opinion during the campaign.  And – at this point – I don’t see any evidence that they are systematically underestimating Obama’s support.  But we can’t be sure of this.  What we can do, however, is compare polls and understand why they differ – it almost always has to do with how they construct their samples.

Having spent all this time discussing polling data, I am now going to explain why political scientists don’t need any of it to predict who will win the 2008 presidential election.  In fact, their predictions are already in!

A final thought: I have received some excellent comments from many of you in my email inbox.  In particular, many of you have taken issue with some of my observations, or provided your own election analysis.  Everyone would benefit from these comments, and so I urge you to post them on my blog (you need not attach your name).  We all can benefit from a broader discussion of the issues that I am raising. So join in!

Explaining the McCain/Palin surge

Since Sept. 2, when Obama led McCain in the RCP average of polls by almost 7%, 49.2 to 42.8 (his biggest margin since late June) the polling dynamics of this race have undergone a sharp turnabout.  The last RCP average of polls showed McCain up by 2.3%, 47.4 to 45.1.  So in a span of 9 days Obama lost 4 points, McCain gained almost 5 for a net turnaround of almost 9% in RCP’s poll of the polls. Of the 11 most recent national surveys conducted, McCain leads in 7, they are tied in 2, and Obama holds 1% leads in the remaining two.  In the various Electoral College scenarios available at various websites (I’ll look at these in a later post) McCain has drawn even, or in some cases ahead, of Obama.   Even the “futures markets” are now placing most of their money on McCain winning the election. All in all, it has been a good stretch for the McCain/Palin ticket since the Republican convention.

I have avoided commenting on this shift until today in order to give time for the Republican convention bounce to peak and to see whether it will recede, but now seems like an appropriate time to answer two questions: what accounts for the McCain/Palin surge, and will it last?

First, the dynamics behind the surge are pretty clear, and can be captured in two words:  Sarah Palin.  (I’m tempted here to remind readers of my earlier posts on this topic but that would be quite unseemly.  However you are free to go back and read them!)  There are two underlying factors driving the Palin dynamics – her impact on Republicans, women and independent voters, and the ineptness with which Democratic strategists, abetted by Obama’s off the cuff  lipstick remark, have responded to her selection.  Let’s look first at the survey numbers.

To begin, Palin has very favorable polling numbers – Rasmussen reports as of Sept. 10 that Palin was viewed favorably by 56% of voters, including 41% with a Very Favorable opinion.  (Compare this to Biden’s numbers: 53% favorable, with 25% Very Favorable).  Her selection, as I discussed earlier, has energized the Republican base, particularly evangelicals.  According to one survey of evangelicals (unfortunately this polling outfit doesn’t provide sampling data or crosstabs, so use with caution), McCain held 73 percent of the evangelical vote before the Republican convention. Just one week after announcing Palin as his running mate, that percentage jumped to 80 percent and the number of undecided evangelicals dropped by half.  As I discussed earlier, support among evangelicals was not affected by the news that Palin’s unwed daughter was pregnant.

Perhaps more importantly, however, the Palin pick has shown signs of moving independents and women into McCain’s column. Among unaffiliated voters, 54% have a favorable opinion of Palin (49% say the same about Biden). What must be particularly worrisome to Obama supporters is that among independents, a slight plurality now views the McCain/Palin ticket as more likely to bring change than the Obama/Biden ticket. (Diego/Hotline poll from Sept. 5-7 -for actual data go to: http://diageohotlinepoll.com/documents/diageohotlinepoll/Diageo%20Hotline%20Tracker%20release%20-%2009%2008%2008%20data.pdf). )  More worrisome still, independents are now evenly divided between McCain and Obama on who is more likely to bring effective health care reform.  McCain also now leads Obama among independents on dealing with the economy (37-33), and the energy (40-32).  This is a big turnaround from before the conventions, when independent thought Obama was the agent of change and better able to handle all these issues.  In short, among independents, it is the McCain/Palin ticket, and not Obama/Biden, that is now viewed as the reform/change vote.  By choosing Palin, McCain has repositioned himself among the crucial independents, and made it harder for Obama to link him to the Bush/Cheney administration.  Conversely, by choosing Biden, Obama undercut his own image as a reformer.

Now let’s look at women.  Remember, this was probably the single most important demographic to contribute to Bush’s victory in 2004 – although Kerry won the overall women’s vote, Bush gained roughly 3-5% among all groups of women (black, white, latino) from his support among these groups in 2000.  That pushed him from 48 to 51% of the popular vote.  So, where do women stand now, after Obama (apparently) brushed Clinton aside and McCain responded by putting Palin on the ticket?  (All data are from the ABC News/Washington Post survey released Sept. 8 – go to:

http://abcnews.go.com/images/PollingUnit/1070a1AftertheConventions.pdf)

Most noticeably, ABC reports (but does not show the crosstabs, unfortunately!) a 20% shift among white women toward McCain in the period between before the conventions, when they supported Obama 50-42, to after the convention, where they now support McCain 53-41. This same poll has McCain up among independents – who number about 18% of those surveyed – 50-43%, and it also shows McCain closing the gap or pulling ahead on the various policies discussed above (“better able to handle”): the economy, security, and energy.  (However, in this poll Obama stills lead McCain among all voters by 51-39 on most likely to bring change to Washington.)  So, those disaffected Clinton voters are, right now, deserting Obama in huge numbers. According to the Rasmussen tracking poll, Obama led McCain among women voters by 14% at the height of the Democratic convention bounce – since that it has been reduced to 7% as of Sept. 12.  If this trend continues, McCain will win the election. Women are the most important voting bloc in this country and given his clear support among men, McCain needs only to draw within a couple of percentage points to Obama among women and the election is his. In a clear effort to achieve this goal, Palin – no fool – took time to praise Hillary Clinton in her ABC interview, and went so far as to say Obama probably regrets not putting Clinton on the ticket.  (I wonder…)

So, what are we to make of this shift in survey dynamics? I have indicated that in past elections, the vice presidential selection rarely influences the final vote decision, and I persist in believing that the Palin effect will recede somewhat, a function of increased media scrutiny of her candidacy (notice the flap over her cautious response to Charlie Gibson’s question about the “Bush Doctrine”), possible fallout from Troopergate, and the inevitable shift in voter attention to more fundamental issues.   Still, this has been an election that has broke with previous precedents, and it is worth noting that 68% of voters (Rasmussen poll) say that the presidential candidates’ running mates are more important this year than in previous election cycles.  It’s hard to tell how much of that is simply the halo effect of the Palin choice.  But it may be that the choice of a vice president, for the first time, will have a demonstrable and enduring impact on the election results.  If so, it will reinforce my claim that Obama simply missed the call on Clinton VP pick, thus opening the window for McCain to choose Palin.  If Obama loses this campaign – and the underlying fundamentals, as I’ll discuss in a post to come, favor him – the most direct cause will be his failure to choose Clinton as his running mate. Candidates have very few opportunities to control their own destiny in a presidential campaign; election outcomes, for the most part, are determined by factors outside their immediate influence. One of the few exceptions is their choice of the vice presidential candidate.  McCain seized the opening provided by Obama, and is now reaping the benefits. As one of my audience members put it last night, in considering the vice presidency, Obama was playing checkers and McCain chess.

But the outcome of this election game – whether chess or checkers – is not in my view likely to be determined by the vice presidential choices (although I am less confident of this than I once was). And, in fact, there are underlying reasons to believe that the electoral situation in many ways is still quite favorable to Obama, and that survey data may be understating his support. I’ll look at that more closely in my next post.

A brief digression

Responding to the steady stream of election-related events often prevents me from addressing some other issues about which I have promised to write.  So let me deal quickly with one housekeeping matter in this post before writing more extensively in a separate post about the problems of forecasting presidential elections. Several of you expressed curiosity – and skepticism – about my recent comment that I do not vote in presidential elections.  As one of you wrote, “I was raised at a time and come from a generation where voting is viewed as a civic duty and the price of democracy.  I am befuddled by your declaration.” Another suggested that, “it is possible to make these posts fairly non-partisan even when you do have a strong opinion and do choose to vote.” Given these reactions, which I am certain reflect the sentiment of many of you, my statement that I don’t vote in presidential elections deserves brief elaboration.  First, let me make clear that I do vote in all local elections in which my wife is running for office. (That’s actually quite a few since she serves multiple positions in our local government!)  Even in these elections, however, I refuse to admit for whom I vote.  Second, as I succinctly stated in that previous post, I urge everyone else to vote in the upcoming election.  I trust I need not spell out why voting is important to the health of a democracy.

So why don’t I vote?  I certainly don’t claim to be without bias when it comes to presidential politics – I have certain preferences and values, just like everyone else.  And I agree that it is possible to analyze elections in an objective, non-partisan fashion, and still exercise one’s right to vote.  But I believe it is harder to do so when one is openly (or even secretly) committed to a candidate.  Voting requires one to invest time to learn about the candidates’ personalities and issue stances.  Once one decides for whom to vote, that commitment becomes an emotional investment – a sunk cost – in the outcome of the race. You want your candidate to win because she’s the better person for the office. That’s why people vote, after all – to pick the best person for the office.  My experience is that this emotional investment alters how people view presidential elections – my candidate is wise and virtuous, my candidate’s opponent is sleazy and underhanded.  This is an exaggeration, of course, but I hope you see my point.  In this election cycle, I have found that supporters of all the candidates often reveal this emotional investment when discussing election particulars, although it seems particularly strong among Obama supporters.  (This may reflect the fact that Obama has strong support in my immediate community particularly among professors and students, so inevitably I see more reaction from Obama supporters than from other candidates’ backers.  It may also reflect the fact that he has aroused such passion among his backers.)  This passion is a good thing for prospective voters, but not, in my judgment, for analysts (or the media, for that matter – but that’s another topic.)  Were I to decide to support a particular candidate, I think I would be susceptible to the same tendency.  By detaching myself from the process, then, I hope to avoid letting any emotional investment color my analysis.  Am I completely objective?  Hardly. But I think I am more so for choosing from the outset not to acquire any rooting interest in the outcome.

There is a second, related reason I do not vote – I have a public (admittedly very minor) presence.  In addition to teaching courses on presidential (and American) politics, I give many public presentations on the election, as well as writing this blog.  I have had more than one student come to me complaining about what they see as a failure of professors to present material in an evenhanded way because their political bias colors their teaching.  I take my teaching and presentations seriously (as do my colleagues).  It is important that I not signal to my audience in any way that I have a personal stake in the issues about which I speak.  By not voting, this is literally true. Some of my colleagues take a different but certainly equally defensible approach – they admit up front what their political views are, but also make a determined effort not to let their personal views influence their teaching.   I certainly respect that approach, but I lack their inner strength of character.

For these reasons, I have adopted the Walter Cronkite strategy and simply do not vote.  I don’t expect that everyone will agree that this is a wise decision. Nor do I expect my explanation to quell your criticisms. But now at least you understand my rationale.

Now on to the election forecast. I will begin my next post with a cautionary tale….

McCain won the post-convention “bounce.” Does that mean he will win the election?

Several of you have asked whether the candidate who wins the “bounce war” – who gets the biggest bounce in approval coming out of their respective convention – is more likely to win the general election.  In response, I went back and looked at the bounce winner dating back to 1960.  In 7 of 11 elections, the winner of the bounce went on to win the general election (I did not include Gore’s victory in the 2000 “bounce” war, since he won the popular vote, but lost the Electoral College vote).   So, since McCain won the “bounce” this year, does this indicate that he is more likely to win the general election?   I discussed this question with my colleague, Professor Amy Yuen, and she ran some simple statistical tests to try to find an answer.  Without going too deeply into the statistics, what we want to do is compare our sample’s outcome – 7 out of 11 bounce winners go on to win the general election – with what we think is the “real” underlying relationship between the bounce and the election outcome.  We might think that the winner of the bounce should always win the election.  In this case, we ask whether our sample outcome is, statistically speaking, different from the “real” relationship in which the bounce winner actually goes on to win the general election every time.

Another way to think about it is to assume that there is no relationship between winning the bounce and winning the general election.  That is, a candidate’s odds of winning the election after winning the bounce are really no better than 50/50.  So here the comparison is between winning 7 of 11 times versus winning 5.5 out of 11.

In short, we are asking the same question, but testing it in two different ways. So, what does our statistical testing tell us?  Based on past history, can we infer that McCain is more likely to win the general election because his post-convention bounce was bigger than Obama’s? Amy was kind enough to run the statistical tests (all tests using the STATA 10 statistical program) and found that, in fact, one cannot be confident that there is any difference (within a commonly accepted level of statistical significance) between simply flipping a coin and the bounce winner going on to win 7 out of 11 elections.  On the other hand, we can reject the idea that winning 7 of 11 times is, statistically speaking, equivalent to the bounce winner going on to win every election.  In short, both results indicate that we can say that McCain’s winning the bounce does not tell us that he will win the general election; statistically, we find no relationship between the two events.  (By the way, if you are interested in learning more about this test and about political science research methods more generally, I recommend taking my colleague Bert Johnson’s course Frontiers in Political Science Research this fall. He takes you through a variety of research issues, many of which we will touch on in the coming weeks in this blog.  I hope to have both Amy and Bert guest blog on issues touching on their specialties as time permits.)

Having said this, there is tantalizing evidence based on the post-convention survey data, that something is happening among likely voters in this race as a consequence of the two conventions. I’ll address this issue in my next blog.