Empirical Legal Studies

ELS Past, Present and Future

In the 1930s, Yale Law Professor Underhill Moore, enlisting the young Lloyd Cutler as a research assistant (along with all of Cutler’s classmates – they had no choice), did a massive empirical study of how New Haven’s citizens responded to the presence of parking meters and sanctions for not feeding them. The resulting article, whose length was exceeded only by its tedium, may have doomed empirical legal studies for the next 20 years. Yet in many ways the project was well conceived. Parking behavior is a nice way of studying how ordinary citizens respond to mild sanctions. Parking behavior is easily observable, as are the costs of parking and the cost and likelihood of sanctions. Various independent variables, such as the value of a car (as an index of wealth) and the gender, race and approximate age of the driver may also be captured with considerable reliability, and enough cases may be generated to allow detailed analysis. But Moore lacked the technology to adequately analyze the data and to present his findings in ways that could be easily understood. This was more or less true of quantitative research on law, and most other areas, until the 1960s.

Outside of economics and psychology, the classic empirical social science offerings before that decade were mainly qualitative studies, books like William Foote Whyte’s Street Corner Society and Philip Selznick’s TVA and the Grass Roots, though a decade earlier, with the leadership of people like Paul Lazardsfeld at Columbia and the survey researcher Angus Campbell at Michigan and their students, modern quantitative social science was being born. By 1970, in all the social sciences except anthropology, quantitative research and formal modeling had replaced qualitative studies and informal theorizing as the hottest, and in some departments the only respectable, way to do social science. Yet except for a few celebrated works, like Kalven and Zeisel’s The American Jury and Wayne LaFave’s massive study of arrest processes, works that used numbers crudely if at all, legal scholarship was barely affected.  Underhill Moore’s ghost had not risen.

I think there were many reasons for this, including a law school culture that reserved its highest accolades for those most adept at analyzing legal doctrine, particularly the pronouncements of the Supreme Court; the view that those who had excelled academically at top law schools and clerked for the right judges needed no further training beyond, perhaps, a bit of practice experience before joining the nation’s leading law faculties, and, in many of the nation’s law schools, a two-and-a-half year, one grand article, path to tenure that did not provide the time needed for empirical data collection and analysis and did not recognize that good quantitative social science more often consists of the production of a series of fine bricks than the construction of one imposing wall.

Some Simple Cautions - Day 4

 Central to most empirical research is the control of plausible rival hypotheses. If we are trying to show empirically that the world is or operates in one way, we need to convince the reader that our results are not likely to be attributable to explanations that undercut or are inconsistent with the description or explanation we provide. The key concept here is plausible. We need not dispose of all possible explanations for our results. If we had to do this, we never could publish. But some law professors, many lawyers, and too often courts, are adept at ignoring the plausibility constraint when they want to reject an empirically-based argument. Instead they treat, and get away with treating, possible alternative explanations as if their mere possibility seriously undercuts reliable data and well done analysis.

 David Baldus and his coauthors, if I recall correctly, included more than 600 independent variables in a model aimed at investigating whether death sentences in Georgia were tainted by race effects. The goal, I assume, was to fend off this kind of uninformed (or bad faith) prosecutorial and judicial nit picking. From a scientific standpoint, the far more parsimonious model they included, with 32 variables as I recall, was a better way of examining the data. But their tactic worked – in a way. When the McCleskey case, in which the Baldus study was the key evidence, reached the Supreme Court, the conservative majority chose not to contend with the analysis, but instead held that in order to show the racial discrimination needed to reverse a death sentence a defendant had to show that in his particular case discrimination played a role in his sentence.  This meant the Court never had to confront  the Baldus finding of substantial race of victim discrimination when cases were of intermediate seriousness.  It had become irrelevant.

Theory, Context and Common Sense

Day – 3

 I would like to talk to today about three aids to assessing empirical work: theory, common sense and knowledge of context. They are equally valuable for those doing the work.

 In 1975 the economist Isaac Ehrlich published a time series study of the deterrent effects of capital punishment running from the late 1930s to the mid 1960s, using what were at the time advanced techniques of statistical modeling. His results, which indicated that one execution deterred 7 or 8 homicides, were attention grabbing to say the least, particularly since they were set against a background of numbers of earlier studies, using apparently simpler methods, almost all of which found no evidence that capital punishment deterred homicides, and some of which suggested a possible brutalization effect. Ehrlich’s findings gained credibility by the apparent statistical sophistication of his methods (with tongue only slightly in cheek, I would define statistically sophisticated methods as methods with lots of equations and numbers that many readers cannot or do not have the time to follow).

 Because the topic was so controversial, other social scientists reexamined Ehrlich’s data using equally sophisticated, and often similar, statistical approaches. These reanalyses revealed that Ehrlich’s model and his results were acutely sensitive to the years analyzed. If the last years of the time series – the 1960s – were removed from the analysis, evidence for a deterrent effect disappeared, and in some studies a brutalization effect seemed possible. The 1960s were, of course, a period when executions for murder dropped to almost zero and all sorts of crimes were rapidly increasing. This replication research pretty much destroyed the credibility of Ehrlich’s results in the academic community, though their influence on the popular debate, like the Cheshire cat’s grin, long lingered.

Day 2 _ What's in a Name?

     I might as well say it at the outset; I don't like the nomenclature adopted by the ELS movement.  I don't like "Empirical Legal Studies" as a name for a type (dare I say "school") of scholarship, and I positively bristle when people are called by their colleagues or describe themselves as "empiricists" or "legal empiricists."  One reason for this, I am sure, is that where I (in part) come from - Sociology - to call someone an empiricist - or, even worse, a "dust-bowl empiricist" was not flattering.   It implied a person who crunched numbers with no concern for theory and often without an inkling of what the numbers represented.  Relationships are often easy to spot in data, but understanding what they mean and, in particular, the mechanisms that underlie and produce them is unlikely to result from number crunching alone.   I wish I could say that legal empiricists have gotten beyond this, but too often I feel that the power of the number crunching that one now sees in law reviews is not matched by an equally sophisticated sense of how plausible theories are built and tested and how causal arguments are structured. 

    This is not to say that there is not considerable value in organizing complex data sets and laying our legally-relevant relationships.  There is, and people who do this in an unbiased methodologically sophisticated manner make a genuine contribution.  But if the goal is not simply to see what is out there, but to understand processes and institutions, too often empirical scholarship  in law (though not only in law) not only falls short of the mark but overstates what analysis reveals.

     But there is another more fundamental reason why I don't like the idea of defining some portion of the legal scholarly establishment - namely those doing quantitative data analysis - as empiricists.  It seems to me that this ignores or denies the way in which legal scholarship, as traditionally done, is the most empirical of professions.  The most traditional lawyers work with empirical data all the time - case texts.  Texts are out there as part of the real world, and from the point of view of empirical scholarship have some substantial virtues.  In particular they are transparent (even when opaque), equally accessible to all scholars and open to various methods of interpretation, some of which are better than others.  They also often matter.  Would that the data "legal empiricists" examined always had these virtues.

      There is also the question of qualitative data.   I am distant enough from the ELS movement that I do not know how its core advocates regard qualitative research, but taking down 5 volumes of the Journal of Empirical Legal Studies that happen to be close at hand I could not help but note that every article in every volume had a quantitative dimension.  Each had at least a graph, table, equation or regression and most analyzed and presented results using more than one of these analytic modality.  Yet qualitative research is as empirically-based as quantitative research and it can be as unbiased and as rigorous.  Moreover, it is often more revealing of relationships legal scholars seek to understand, not to mention more accessible and interesting.    Lawyers have done many quantitative studies I find useful and admire, but I would not elevate any of them above, for example, Bob Ellickson's study of Shasta county when it comes to developing and sharing an understanding of the real world or, in this case, illuminating the limitations of the Coase Theorem. 

   New tools allow for the more systematic analysis of qualitative data and for sharing qualitative data in ways never before possible.  A cutting edge in social science research, including such heavily quantitative specialties as survey research, is the integration of qualitative and quantitative methodologies.   Should the law school world fission into communities that exacerbate divisions between qualitative and quantitative researchers and between those interested in numbers and those analyzing doctrine, I think we will be falling behind the times in social science, and our understanding of the law and legal systems will be the worse for it.

     I am not saying this is happening or that the movement to create "empirical legal scholarship" is pushing us in this direction.  Indeed, the best people doing such scholarship are often the most interested in what non-quantitative research has to offer.  But I do wish the law school world would find a different term than "empiricist" to describe people who work more with quantitative than with qualitative or doctrinal data.  In the law school world, we are, almost all, empiricists.   

   Rick

Tomorrow:  Common Sense and Theory

Reflections on Empirical Legal Studies

    I would like to thank Jason Czarnezki for inviting me to be this week's guest blogger on the Empirical Legal Studies web site.  Readers will find also an extended comment I made in response to a comment by Bill Henderson on Richard Sander's research, but this is not the intended topic of my blogging (Though I may return to it at the end of the week.)  Rather I want to use my space to simply to reflect on the Empirical Legal Studies movement and to suggest some cautions for those doing empirical research on legal topics. 

       I have been doing empirical work on legal issues as a law professor for almost 40 years, long before anyone spoke of empirical legal scholarship and legal empiricists.  I have been making the case for the virtues of empirical scholarship on law for almost that long.  Thus I am delighted to see the expansion of empirical legal studies (within and) beyond economics, and to know that there is a sufficiently large community of scholars who identify with the area as to justify a journal and a web forum, not to mention an AALS Meeting theme.  At the same time I have concerns about the quality of some of the empirical work being done and how empirical work, even good scholarship, is used.  To put it bluntly too much empirical scholarship is being deployed normatively, downplaying caveats that should be attached to findings, and some research, including work by outstanding scholars, seems strongly agenda driven (sometimes to the point of being financed by parties building records for litigation).  I also see in some work a divorce of empirical analysis from theory and context which not only can diminish the utility of empirical studies in building more general understandings, but can also lead to poor research designs and misunderstandings of data.  Moreover, even when research is of high quality and done with great care, its results can be hijacked by groups that oversimplify what was found in order to  "sell" positions they hold and would hold even if the empirical work had come out differently.   

     Too often researchers encourage misuses of their results in conclusions that push the practical implications of their research, even when the more detailed analysis emphasizes proper cautions.  While this occurs with empirical students of the law in liberal arts schools by political scientists, sociologists, economists and psychologists among others, the problem tends to be more severe in the empirical work of law professors, perhaps because most see their business not as building social or behavioral theory but as criticizing laws and legal institutions and recommending reform.

Lempert Comment on Affirmative Action Post

A couple of weeks ago I posted a lengthy essay on the debate surrounding Richard Sander's affirmative action research.   Richard Lempert, our guest blogger for the coming week and one of the most prominent Sander critics, posted a detailed comment to my post, which I encourage all our regulars to read.   The comment is here.

Richard, we look forward to your posts.  Welcome.  bh.

Guest Blogger: Richard Lempert

This week's guest blogger is Richard Lempert of the Michigan Law School.  Professor Lempert is particularly concerned with the problem of applying social science research to legal issues. This is reflected in much of his work, particularly his work on juries, on capital punishment and on the use of statistical and social science evidence by courts, as well as in his service as an original panelist in the National Science Foundation's Law and Social Science Program and with the National Research Council's Committee on Law Enforcement and the Administration of Justice, which he chaired from 1987-89. He is the author (with Joseph Sanders) of An Invitation to Law and Social Science and co-editor (with Jacques Normand and Charles O'Brien) of Under the Influence? Drugs and the American Work Force. From 1982-85, he edited the Law & Society Review. He is a recipient of the Law & Society Association's Harry Kalven Jr. Prize prize for outstanding socio-legal scholarship and has held visiting fellowships at the Center for Advanced Study in the Behavioral Sciences and at the Russell Sage Foundation. In 1993 he was elected a fellow of the American Academy of Arts and Sciences. He currently serves as the secretary of Section K of the American Association for the Advancement of Science.

Video Game Violence in Court

On Monday, the United States District Court for the District of Minnesota struck down the Minnesota Restricted Video Games Act. See Entertainment Software Assoc. v. Hatch, No. 06-CV-2268 (D. Minn. 2006). The Act prohibited minors [below 17] from knowingly renting or purchasing a video game carrying an ESRB rating of AO (Adults Only) or M (Mature). As of this moment, only 23 titles carry an AO rating (including only one game of any note, Grand Theft Auto: San Andreas), but 989 titles carry an M rating (including my own current favorite, Splinter Cell: Chaos Theory). The penalty imposed on a minor for violating the Act was a $25 fine. I'll focus on the use of the empirical evidence in this case.

For the Minnesota law to have survived under Eighth Circuit precedent, the state needed to “come forward with empirical support for its belief that ‘violent’ video games cause psychological harm to minors.” Interactive Digital Software Assoc. v. St. Louis County, 329 F.3d 954, 959 (8th Cir. 2003). It needed “substantial supporting evidence of harm.” Id. (internal quotation marks omitted).

The state relied primarily on a meta-analysis by Professor Craig Anderson of Iowa State University, who summarized his findings as follows:

Variable Transformation and Correct Parameter Estimations

Over the last few weeks, Andy Morriss and I have been working with U.S. News rankings, focusing primarily on post-graduation outcomes such as employment upon graduation and bar passage.   Much of the raw data does not satisfy the assumptions of ordinary least squares regression (linear relationship between X and Y, constancy of variance, etc.).  Thus, to create a proper linear model, I have been experimenting with various tranformation methods.   

One of the lessons this experience drove home was the importance of examining your data BEFORE you rely upon the various OLS parameter estimates.   In our case, the estimates changed in important ways after transforming the relevant variables.   It also reduced the amount of heteroskedasticity (i.e., nonrandom distribution of errors) in our various specifications, which will save us some additional corrective steps.

Here are some graphics that I thought readers might find useful.  This graph [Click to enlarge, excerpted from Fox, Applied Regression Analysis, Linear Models, and Related Methods 70 (1997)] shows a monotone curvalinear relationship, which can plausibly be corrected by a power transformation of the dependent (Y) or independent (X) variable:

"End of Term Statistics and Analysis"

Time to trek down to the Law Library... The recent issue of Preview of United States Supreme Court Cases has an article titled End of Term Statistics and Analysis (of the 2005 Term) by Thomas Goldstein.  I can't find the online version of the article, though the dagger footnote says it's available on SCOTUSblog.  The Harvard Law Review also annually provides SCOTUS Stats, usually in the November issue (see, for example, the data on the 2004 Term).