And the Numbers to Prove It – When Statistics Lie

Among the empty platitudes that fill our politics, perhaps none is so pervasive as the idea that no matter what ails us, the solution is to be found in common sense. As a rule, voters would rather have their intuitions validated than have their viewpoints challenged.

That turns out to be a pretty basic human trait. A useful term to keep in mind when thinking about this is ‘motivated reasoning’ – or the tendency to look at information in a way that makes a particular conclusion much likelier than the alternative. If you want a simple way to think about this, just look at the way people watch sports.

Whether you think a pitch should have been called a strike or a ball can often depend more on whether you’re rooting for the batter or the pitcher than on the placement of the ball. Was a football game called fairly? That will depend on whether your team won or lost. That’s just the way it is.

Among the different ways in which people prejudice their thinking, special mention should be given to ‘confirmation bias’ – or the tendency to give special weight to evidence that confirms our positions while discounting evidence that points the other way. In politics, the most relevant example of this is often ‘asymmetric skepticism’. Basically, it’s the tendency to tear the other guy’s argument to shreds while being relatively complacent of arguments that reach conclusions you like.

Listen to the way politicos talk about economic research and you’ll see what I mean: any study that agrees with your position is groundbreaking, any study that disagrees with it is deeply flawed.

The best way to avoid this pitfall is to make sure that you don’t stop thinking about a problem just because you found a piece of evidence that you like – and the best way to learn the importance of that is to get an appreciation for just how easy it is to find really bad evidence in support of a position.

Let’s say, for example, that you wanted to know whether or not there’s a relationship between the number of lawyers in Wisconsin and the per capita consumption of cheese in the United States:

wisconsin and cheese

Hmm. That’s odd. But I guess we can imagine that both of those things would increase over time since they might both increase with population growth. Let’s try something more obscure, – is there a relationship between the number of points scored by the winning team in the Super Bowl and the number of occupants of heavy vehicles killed in a collision with a stationary object?

Super bowl and vehicle accidents

On its face, that graph looks like compelling evidence of something. But when we know that something is the relationship between the score of a game and motor vehicle accidents, we become a bit more wary. And rightfully so:, both graphs are taken from Tyler Vigen’s Spurious Correlations blog, which is designed to show how trivially easy it is to see patterns in data that don’t actually tell us anything informative about the real world.

But what if these graphs instead showed us the relationship between stimulus spending and unemployment, or between tax rates and economic growth? Would they be more informative then? Some people would probably think so.

The right answer, of course, is “no” – at least not on their own. On its own, every piece of data is circumstantial, and just because a pretty graph shows a statistical relationship doesn’t mean you should be prepared to accept the underlying claim.

The lesson is to always question the data, especially if it’s data that you think agrees with you. Remember that the world is under no obligation to be pithy.

Follow Pedro on Twitter @IamPedroA.

Want to help The Fog of Policy grow? Then take a minute and share this piece! Or let me know what you think in the comments section.

Have a question or suggestion for a new piece? Submit it through the Feedback form – and don’t forget to subscribe on the homepage to get posts and features automatically sent to your inbox.

One thought on “And the Numbers to Prove It – When Statistics Lie

  1. As someone with some understanding I can point to the following example of the same statistic being used for opposing reasons.

    Lets say that you have a drug that is 99.8% safe
    You make some changes to it and it becomes 99.9% safe.

    The following statistics are both true:

    1) The percentage of people who experience problems was cut in half.
    2) The percentage of people who had no problems only changed by .1%

    Don

Leave a Reply