This Sunday's N.Y. Times had an interesting review of Ian Ayres new book, Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart. In a nutshell, this book piles on to a theme developed by Moneyball and Freakanomics -- you can profit by data mining, especially when the conventional wisdom is wrong.
NY Times economics reporter, David Leonhardt, begins his review of Super Crunchers with an anecdote on Orley Ashenfelter's theory that the weather in Bordeaux is the best predictor of a wine's eventual price. Apparently, Ashenfelter's theory irritated the wine establishment, which heaped scorn upon him. In fact, Ashfelter is part of lore of the early days of the American Law & Economics Associations. Many years ago, Ashenfelter gave the ALEA key note address over dinner and entertained the crowd with his musings on weather and wine prices, noting (to the delight of his listeners) that his theories had already produced a tidy profit.
To my mind, the principles of Moneyball-Freakanomics-Super Crunchers are hard to dismiss. Indeed, over the last several years, I cannot recall a single frontal attack on any of these ideas. Instead, it is my observation that most people (and thus organizations and institutions) never take the time to think through their simple, stark implications--i.e., don't be complacent, look at the data, or eventually you are going to get smoked by the competition.
Here is the key passage in Leonhardt's review:
"For all its successes ... statistical analysis continues to face tremendous skepticism and even animosity. For one thing, Ayres notes, statistics threaten the 'informational monopoly' of experts in various fields. But even to many people without a vested interest, relying on cold, hard numbers rather than human instinct seems soulless.
The problem, as Aryes points out, is that there is overwhelming evidence that human beings are notoriously overconfident in their own judgments and intuitions. Indeed, ever since Moneyball, every where I turn I am astonished by the overconfidence of the well-placed and the well-credentialed (today's winner being Michael C. Dorf). This creates a lot of opportunities for the rest of us.