Dan Kahan, Professor of Law and Psychology at Yale, has two posts (so far) up at the Cultural Cognition Project blog about the uses and abuses of multivariate regression in particular and empirical research more generally. He takes as his jumping-off point two analyses by the National Research Council -- one, from 2004, about whether concealed-carry laws increase or decrease gun violence and another, from 2012, about whether the death penalty increases murder rates (presumably through deterrence) or decreases them through "a cultural 'brutalization effect.'" In both of these analyses, the NRC panels conclude that the many studies relying multivariate regression to try to tease out the effects of the relevant laws are simply useless. The problem that the NRC finds in both sitautions is that the outcomes of these analyses depend on model specification and on the underlying assumptions of those models.
Kahan then goes on to discuss the ways that opinion leaders rely on such studies when they support their policy views and dismiss them -- often on the very basis that, as the NRC explains, they are inherently inconclusive -- when they do not. Likewise, ordinary people, perhaps relying on this "opportunitistic" use of scientific research, often believe that scientific consensus supports their own world-views or policy preferences even when it does not or even when scientific consensus does not exist.
Kahan hardly dismisses the usefulness of empirical research, however. In his second post, he explains:
But I do have an idea (a conviction, in fact) about the sensible way to make sense of empirical evidence. It's that it should be evaluated not as "proving" things but as supplying more or less reason to believe one thing or another. So when one is presented with empirical evidence, one shouldn't say either "yes, game over!" or "pfffff ... what about this that & the other thing..." but rather should supplement & adjust what one believes, and how confidently, after reflecting on the evidence for a long enough time to truly understand why it supports a particular infernece and how strongly.
These posts raise important and interesting questions for empirical scholars of law and policy, who are rarely, if ever, able to rely on controlled experimental conditions, and for the consumers of their work. See the previous post on this very blog for a related discussion.
Comments