A very interesting and helpful post (and comments) by John Donohue (Yale) over at the Balkinization blog illustrates some all-too-common problems with empirical research. To illustrate some of these problems, John comments on a recent paper by Royce de Rohan Barondes (Missouri-Columbia), provocatively titled: Want Your Opinions Questioned or Reversed? Hire a Yale Clerk.
As John notes, "unfortunately, counterintuitive empirical results almost always turn out to be wrong if they are not based on an appropriate empirical methodology for the inquiry at hand. In my opinion, the methodology of the Barondes (paper) is flawed, and the conclusions drawn from this research are either incorrect or unfounded." (To be fair, Barondes responds to some of John's critiques in comments to John's post.)