The pollsters: Who's good, who's not, and how bouncy they all are

Polls, polls, and more polls. They can make your head spin. We've had about 200 independent samples of the national presidential race alone since the beginning of the year.

Which means we have enough data to play with and draw some lessons about the polls and pollsters in 2012. In brief: for a heavily polled contest like the national presidential numbers, you can, if you like, preserve your sanity by ignoring individual polls, and follow a polling average instead, like this. This is advice which is not new in the least but nice to have confirmed again by the data. Sequential polls by the same pollster can and often do have fairly large changes in the Obama-Romney margin that are consistent with random error alone. Polling averages smooth out the random error and retain the trends, although they take a little longer to respond to current events. Below, you will see just how jumpy the polls can be for no particular reason, and how different Rasmussen and Gallup are from the rest.

Onward to the data. First, we need a standard to judge against, and a way to judge that standard. I decided to use a local regression of all polls from pollsters who sampled registered voters, called cell phones, and released polls at (reasonably) regular intervals over the entire period. This resulted in a total of 48 polls from Pew, NBC/WSJ, Fox, ABC/WaPo, CBS/NYT, Reuters/Ipsos, and CNN. The local regression is the green line below. How close is the green line to reality? We can't tell, but I can tell you that the average of the final 2008 polls for the pollsters listed above was Obama +7.6; the actual result was Obama +7.3. I find that somewhat reassuring.

The blue line is the regression for all 208 polls (I probably missed some). Notice how they both have similar trends, even though they diverge a bit after January, so we're not missing any major dynamics with the green line. The Republican primary sent the polling margin soaring in Obama's favor as the country got to know Romney; then it gradually came back down as the primary resolved and Republicans came home, and it's been stable since the end of the primary. The grey line, however, is a more sensitive regression, and it shows a dip in early May that is quite likely real (it's not just driven by one pollster), following Obama's interview in support of marriage equality. The green line misses this entirely, so keep that in mind.

So here's how the most prolific individual pollsters stack up compared to the green line. What we want to see for a good pollster is for the data points to be evenly distributed above (erring in Obama's favor) and below (erring in Romney's favor) the green line, with no particular trend over time. Click on the graph to enlarge it.

So who can you trust when it comes to polls? Nobody, especially not Rasmussen. Pick almost any pollster in the chart above, and you would get seasick watching it bounce up and down, even while the average is relatively stable. Conclusion: Don't worry about large changes in individual polls. Fluctuations in the margin of polls from any one pollster of more than 5 points are fairly common, and, in isolation, should be cause for neither concern nor jubilation. Also, there will sometimes be trends in a series of any one poll's data during periods of stability in the average of all polls; again, just watch the polling average.

More details below.