Five students in the Hamilton College Public Policy department posed a question at the outset of their thesis presentation on May 2: are the columnists you read and talking heads you watch better than a coin flip?
Holly Donaldson, Russ Doubleday, Kate Tummarello, Scott Hefferman and Evan Klondar spent the next hour discussing the findings of their thesis, "Are Talking Heads Blowing Hot Air? An Analysis of the Accuracy of Forecasts in the Political Media." In their thesis, they found that Paul Krugman was the best in their sample, while Cal Thomas was the worst; liberals fared better than conservatives, and people with a law degree were poorer predictors than those without.
Before discussing the results, they first explained how they came to evaluate this question. Philip Tetlock did a study over a 20-year period that showed experts did not fare much better than simple extrapolation when it comes to making predictions. The Hamilton students sought to narrow the focus of the study and evaluate solely the most popular prognosticators: those people see in print and on TV on a weekly basis.
The group next discussed their method for analysis. The study was confined to the 2008 election season, starting at the beginning of September, 2007 and ending on December 31, 2008. They sought out the most widely syndicated print columnists and the most frequently-appearing talking heads on Sunday morning talk shows. They collected prognosticator's predictions and used a regression analysis to discover what contributed to prognosticator accuracy.
Prognosticators who used conditionals (“if X, then y”) were more likely to predict inaccurately. Similarly, lawyers were more likely to be incorrect than non-lawyers. Liberals were more likely to be right than their conservative counterparts. The finding about liberals being better predictors held true even when taking out predictions about the election.
The group also compiled their findings into a list of “The Good, the Bad, and the Ugly.” This was the group’s way of “naming names” about the best and worst predictors. Paul Krugman was the best, and was joined among the ranks of “the Good” by Ed Rendell, Nancy Pelosi, Kathleen Parker, and David Brooks among others. The six “Ugly” prognosticators included George Will, Joe Lieberman, and Carl Levin; Cal Thomas was the worst of the Ugly.
The presentation was live-streamed over the Internet and the presenters were able to take questions from people who used the hashtag #hcpundit on Twitter. Questions from Twitter and the audience asked specific questions about the methodology, and one Twitter question even came from Howard Wolfson, one of the prognosticators evaluated in the paper.
