I would like to thank Jason Czarnezki for inviting me to be this week's guest blogger on the Empirical Legal Studies web site. Readers will find also an extended comment I made in response to a comment by Bill Henderson on Richard Sander's research, but this is not the intended topic of my blogging (Though I may return to it at the end of the week.) Rather I want to use my space to simply to reflect on the Empirical Legal Studies movement and to suggest some cautions for those doing empirical research on legal topics.
I have been doing empirical work on legal issues as a law professor for almost 40 years, long before anyone spoke of empirical legal scholarship and legal empiricists. I have been making the case for the virtues of empirical scholarship on law for almost that long. Thus I am delighted to see the expansion of empirical legal studies (within and) beyond economics, and to know that there is a sufficiently large community of scholars who identify with the area as to justify a journal and a web forum, not to mention an AALS Meeting theme. At the same time I have concerns about the quality of some of the empirical work being done and how empirical work, even good scholarship, is used. To put it bluntly too much empirical scholarship is being deployed normatively, downplaying caveats that should be attached to findings, and some research, including work by outstanding scholars, seems strongly agenda driven (sometimes to the point of being financed by parties building records for litigation). I also see in some work a divorce of empirical analysis from theory and context which not only can diminish the utility of empirical studies in building more general understandings, but can also lead to poor research designs and misunderstandings of data. Moreover, even when research is of high quality and done with great care, its results can be hijacked by groups that oversimplify what was found in order to "sell" positions they hold and would hold even if the empirical work had come out differently.
Too often researchers encourage misuses of their results in conclusions that push the practical implications of their research, even when the more detailed analysis emphasizes proper cautions. While this occurs with empirical students of the law in liberal arts schools by political scientists, sociologists, economists and psychologists among others, the problem tends to be more severe in the empirical work of law professors, perhaps because most see their business not as building social or behavioral theory but as criticizing laws and legal institutions and recommending reform.
A study that illustrates how results of how even high quality empirical research on the law can be somewhat oversold and then hijacked by agenda-drive groups is one done by the sociologists/criminologists Richard Berk and Lawrence Sherman, which appeared in the American Sociological Review more than two decades ago. The study of the effects of mandatory arrest for spouse abuse on recidivism is one of the first field experiments in criminal law enforcement, and in many respects it is outstanding. The authors (principally Sherman, I would guess) implemented a field experiment in Minneapolis in which police randomly chose to arrest, separate or merely lecture men whose wives, girlfriends or neighbors had called on the police to come in response to spouse abuse, so long as that abuse had not escalated to the point of justifying a felony arrest. Random implementation was not perfect, but the authors (principally Berk I suspect) corrected statistically for these failures. Thus we have a top notch research team, implementing a powerful and creative research design and analyzing the results with great sophistication.
Two outcome measures were used in the study. One was subsequent spouse abuse arrests and the other was reports by the abused women on whether there had been subsequent spouse abuse. Although the measures were not perfectly in accord, taken together they indicated that mandatory arrest followed by at least a night in jail (but usually no formal punishment)was associated with lower recidivism than the other treatments. This was interpreted as suggesting that arrest deterred future spouse abuse. Evidence in the data that the mechanism might have had more to do with breaking up relationships than deterring violent behavior was not dwelt on, and with some appropriate caveats, though not strong enough for my taste, the authors suggested that mandatory arrest seemed likely to deter spouse abuse. Moreover, Professor Sherman made a special effort to publicize this research including, if I recall correctly, filming some police calls for spouse abuse and making the resulting footage available to television for stories on the occasion of the report of preliminary results.
The research was done at an opportune time. Conservatives were pushing toughness on crime and many liberals, influenced by initiatives within the women's movement, were pushing toughness on spouse abusers. This research, along with several other studies that were not as well designed, became ammunition of both liberals and conservatives in pushing for mandatory arrest for spouse abusers, and many police departments adopted such policies. No doubt, the political winds were such that the research would have been used the same way, even if Sherman had been less proactive in publicizing it and even if relevant caveats had been more strongly emphasized ,and the possible policy relevance of the study's findings further downplayed.
Writing an editorial in the Law and Society Review, I was both an early admirer of this research and an early critic of its potential for having unjustified effects on policy. Later NIJ, which had funded this study, did something as commendable as it was (unfortunately) unusual. They funded a series of studies replicating (with some variations) this research in different locations. The results indicated a far more complex picture and suggested substantial context dependence. The best of the replications was, in my view, one done by Professor Sherman in Milwaukee. Here, where the citizenry was far less homogeneous than the citizens of Minneapolis, Sherman found that in some cases arrest seemed to deter future spouse abuse while in other cases it seemed to enhance the prospect of later violence. Those who had a lot to lose from arrest (whites, the highly educated, those with good jobs) seemed to be deterred from future spousal violence while those with much less to lose became, if anything, more violent if their treatment was arrest. To oversimplify only somewhat, it appeared that an arrest policy that would protect middle class white women was likely to endanger lower class black women. The policy implications of this research, and of the replications taken together, are thus much less clear than those of the original Minneapolis study, but the results of the follow on research never received the same publicity and did not have the same quick effects.
There is little empirical scholarship in the law that implements designs as innovative and powerful as the Minneapolis spouse abuse experiment, and there are almost no law professors with the statistical acumen of a Richard Berk. The moral I draw from this tale, or more precisely would like to have my readers draw, is that empirical legal scholars should be humble about their findings; aware, in a more than boiler plate fashion, of the limitations and shortcomings of their studies, and hesitant to draw strong policy prescriptions from their work alone. For policy workers I offer a firm rule: no important policy should ever be changed based on the results of a single study, no matter how well the study has been done and how intuitively plausible its results.
Tomorrow I will comment on law as an empirical endeavor and speculate on why empirical legal scholarship seems to be taking off.
Rick Lempert
Dohonue also recently did something similar to what Rick just recounted: Donohue and Justin Wolfers tried to replicate the curiously recent proliferation of studies showing that the death penalty deters future crime by others. They found serious data, methodological, and ethical lapses in those "yes there's deterrence" studies, most of which trace to a blatant political agenda of "proving" deterrence -- the legal equivalent of Philip Morris or ExxonMobil researchers pushing questionable "studies" to reach their own desired conclusions.
Posted by: Scott Moss | 08 August 2006 at 04:38 PM
John Donahue had a line at the 2006 AALS Annual Meeting that really stuck with me. "One study does not settle an important academic dispute; a literature does." That sounds right. bh.
Posted by: William Henderson | 07 August 2006 at 12:32 PM