The point of regression analysis is to make predictions based on past relationships. “My concern,” one of the authors of the paper said, “is that when reading economics journal articles you get the impression that the world is much more predictable than it is.”
What Soyer and Hogarth did was get 257 economists to read about a regression analysis that related independent variable X to dependent variable Y, then answer questions about the probabilities of various outcomes (example: if X is 1, what’s the probability of Y being greater than 0.936?).
When the results were presented in the way empirical results usually are presented in economics journals — as the average outcomes of the regression followed by a few error terms — the economists did a really bad job of answering the questions. They paid too much attention to the averages, and too little to the uncertainties inherent in them, thereby displaying too much confidence.
When the economists were shown the numerical results plus scatter graphs of the same data, they did slightly better. The economists who were shown only the graphs and none of the numerical results, meanwhile, actually got most of the answers right, or close to right.
The bigger point here, which Soyer and Hogarth have elaborated in other research, is that we tend to understand probabilistic information much better when it’s presented in visual form than if we’re just shown the numbers. (This was also a key argument of Sam Savage’s edifying and entertaining 2009 book The Flaw of Averages.) What’s so interesting is to learn that statistically literate experts are just as likely to glom onto the point estimate and discount the uncertainty as, say, innumerate journalists reporting the results of political polls.