At the AALS Annual Meeting today, the Section on Law and Social Sciences organized a panel entitled, “Designing Empirical Legal Inquiry.” The four panelists were Andrew Martin (Wash U), Lee Epstein (Northwestern), Tracey George (Vanderbilt), and Ahmed Taha (Wake Forest). The panel was designed for each of the participants to discuss different empirical methods to address the same question—How do judges make decisions?
Epstein kicked off the panel by discussing the Dark Side of Archival Research. While she normally reports about the rewards of archival treasure hunts, she wanted to make clear that while worthwhile, archival work requires money, includes uncomfortable workspace and lots of photocopies, and mandates the creation of detailed coding rules.
Martin discussed how cases filled words become lines of data filled with ones and zeros, often coding for votes. There a number of advantages to vote coding: easy to understand who wins/loses, data availability and reliability. Disadvantages include lack of available data on the lower courts, concerns of case selection bias, and possible validity concerns. In the future he expects case coding to be far richer with more focus on the lower courts, citation networks, and the actual text of the opinions.
Taha discussed his paper on How Panels Affect Judges: Evidence from U.S. District Courts. The data include the nearly 300 district court opinions on the constitutionality of the sentencing guidelines. What is interesting is that 14 district courts heard decisions in en banc. (Raise your hand if you knew district courts could do this!) Taha looked at whether panels created unanimity and why, combining his quantitative work with interviews of district court judges.
George presented “An Experimental Approach to the Study of the Judging” (on behalf of Chris Guthrie and Jeff Rachlinski who were unable to attend). Asking the question of whether judges can avoid bias, an example taken from (I think) Guthrie, Rachlinski & Wistrich, "Inside the Judicial Mind," 86 Cornell Lew Review 777 (2001), she focused on the strengths and limitations of between-group experimental methods that have a control and experimental group. The strengths: high internal validity, straight-forward statistics, can assess a particular independent variable because of random judge assignment. The limitations: external validity concerns, tells us about central tendencies, but not individuals, the need to go through IRB Human Subjects protocol, and the difficulty in recruiting the subject pool (especially judges).
The panel answered questions about selection bias in qualitative research, the difference between reliability versus validity, publishing empirical work in law reviews, and how can empirical results be conveyed in an accessible way to the reader.