Empirical Legal Studies

Influence of Mandatory and Discretionary Court Dockets on Case Outcomes

In an intriguing paper forthcoming in the Journal of Empirical Legal Studies (2012), Does the Judge Matter? Exploiting Random Assignment on a Court of Last Resort to Assess Judge and Case Selection Effects, Ted Eisenberg (Cornell), Talia Fisher (Tel Aviv), and Issachar Rosen-Zvi (Tel Aviv), exploit a database of criminal cases appealed to the Israel Supreme Court. In the course of seeking to assess individual judge effects on case outcomes, and controlling for the usual complexities that contribute to non-random case assignment to justices, the authors lever a factor that distinguishes the Supreme Courts in Israel and the U.S. Specifically, the Israel Supreme Court's docket includes an important cohort of mandatory cases. The paper notes how "studies limited to discretionary jurisdiction case outcomes can distort perceptions of judges’ preferences." The abstract follows.

"We study 1410 criminal law cases appealed to the Israel Supreme Court in 2006 and 2007 to assess influences on case outcomes. A methodological innovation is accounting for factors — case specialization, seniority, and workload — that modify random case assignment. To the extent one accounts for nonrandom assignment, one can infer that case outcome differences are judge effects. Individual justices cast 3986 votes and differed by as much as 15 percent in the probability of casting a vote favoring defendants. Female justices were about two to three percent more likely than male justices to vote for defendants but this effect is sensitive to including one justice. Defendant gender was associated with outcome, with female defendants about 17 percent more likely than male defendants to receive a favorable vote on appeal. Our data’s full samples of mandatory and discretionary jurisdiction cases allow us to show that studies limited to discretionary jurisdiction case outcomes can distort perceptions of judges’ preferences. Justices’ ordinal rank in rate of voting for defendants or the State was uncorrelated across mandatory and discretionary jurisdiction cases. For example, the justice who sat on the most criminal cases was the fourth (of 16 justices) most favorable to the State in mandatory jurisdiction cases but the twelfth most favorable in discretionary jurisdiction cases. This result casts doubt on studies of judges on discretionary jurisdiction courts, such as the U.S. Supreme Court, in which votes in the selection process are not observed."

The Journal of Law and Courts

As Chair of the Law and Courts section of the American Political Science Association, I am delighted to announce the inauguration of the Journal of Law and Courts, the official journal of the section.

The Journal is edited by David Klein, of the University of Virginia's Department of Politics, and published by the University of Chicago Press. The JLC is a double-blind peer-reviewed, single-submission journal, indexed by Lexis-Nexis, Westlaw, EBSCO, JSTOR, and others. While formally an APSA section journal, the JLC aims to be the premier outlet for the publication of work on law, courts, and things judicial from a wide range of perspectives.  More information about the journal and instructions for authors can be found here.

Submissions are being accepted now, via the journal's Editorial Manager page. We anticipate publication of the inaugural issue in spring 2013.

A Critical Appraisal of Empirically "Judging Judges"

A while back we noted, among other papers in the "judging judges" genre, one by Stephen Choi (NYU), Mitu Galati (Duke), Mirya Holman (Duke/UNC), and Eric Posner (Chicago) that sought to hold claims about Justice Sotomayor’s now-infamous assertion, uttered prior to her Senate confirmation hearing, that female judges might be “better” than male judges. As noted in 2009, the Judging Women paper "estimate three measures of judicial output: opinion production, outside state citations, and co-partisan disagreements." The paper found that "male and female judges perform at about the same level."

The enterprise of empirically "judging judges," however, recently generated its own push-back. In The Costs of Judging Judges by the Numbers, the authors, Marin Levy (Duke), Kate Stith (Yale), and José Cabranes (CA2), advance some plausibly dramatic claims about the efficacy of "judging judges," at least empirically. The paper's abstract follows.

"This essay discredits current empirical models that are designed to “judge” or rank appellate judges, and then assesses the harms of propagating such models. First, the essay builds on the discussion of empirical models by arguing that (1) the judicial virtues that the legal empiricists set out to measure have little bearing on what actually makes for a good judge; and (2) even if they did, the empiricists’ chosen variables have not measured those virtues accurately. The essay then concludes that by generating unreliable claims about the relative quality of judges, these studies mislead both decision-makers and the public, degrade discussions of judging, and could, if taken seriously, detrimentally alter the behavior of judges themselves."

Exploring Judicial Self-Interest and Judicial Decisions

In an Essay forthcoming in the University of Illinois Law Review (2011:5), Measuring Maximizing Judges: Empirical Legal Studies, Public Choice Theory, and Judicial Behavior, Joanna Shepherd (Emory) surveys and summarizes the existing empirical literature assessing a public choice model of judicial decisionmaking. An excerpted abstract follows.

"Although legal scholars have traditionally been reluctant to engage in a critical inquiry into the role of judicial self-interest on judicial behavior, recent empirical studies confirm many of the predictions of the model. As a result, the public choice model has gained broad acceptance across a range of disciplines, courts, and even the U.S. public."

Judges (and Justices), Judicial Opinions, and Legal Scholarship

Forging ahead on a trail previously blazed by Judge Harry Edwards, Chief Justice Roberts recently endorsed the "disconnect thesis" by noting the separation between legal academics and the practicing Bar. CJ Roberts made clear his view that legal scholarship simply doesn't help judges all that much.

While Tun-Jen Chiang's (George Mason) post on PrawfsBlawg details what was said (and what was not said and how to interpret all of this), of more interest to me is subjecting CJ Robert's premise to data.

And this is where a recent paper by Lee Petherbridge (Loyola-LA) and Dave Schwartz (Chicago-Kent) comes in. An Empirical Assessment of the Supreme Court’s Use of Legal Scholarship does just that and will interest those engaged in this debate. The abstract follows.

"Derogating legal scholarship has become something of a sport for leading figures in the federal judiciary. Perhaps the chief antagonist in recent years has been none other than the Chief Justice of the U.S. Supreme Court, John G. Roberts Jr. His most recent salvo includes the claim that because law review articles are not of interest to the bench, he has trouble remembering the last law review article he read. This claim, and others by the Chief Justice, may represent the end of an uneasy détente concerning the topic of the utility of legal scholarship to the bench and bar. At a minimum, Justice Roberts’s recent comments represent a vigorous invitation to a discussion, which this article accepts. To that discussion we contribute an empirical study that is based on an original and unprecedented body of data derived from every Supreme Court decision over the last sixty-one years. This article presents several surprising results and makes two major novel contributions. The first is evidence describing the amount and patterns of the Supreme Court’s use of legal scholarship over the last sixty-one years. The second, and perhaps most striking contribution of this article, is empirical evidence on the nature and quality of the Court’s use of scholarship. This article provides the first report, as far as we can determine, of evidence that the Supreme Court not only often uses legal scholarship, it also disproportionately uses scholarship when cases are either more important or more difficult to decide. It thus presents results strongly counterintuitive to claims that scholarship is useless or irrelevant to judges and practitioners. The article also discusses areas for future work."

Evidence of Political or Judge Effects?

Responding to my previous post, Ted Eisenberg notes:

"Brian Fitzpatrick’s study of class actions (forthcoming in 7:4 JELS) finds no evidence that the fee award was associated with the political affiliation of the judge who made the award.  This result is similar to the finding in Individual Justice or Collective Legal Mobilization? Employment Discrimination Litigation in the Post Civil Rights United States, by Laura Beth Nielsen, Robert L. Nelson, and Ryon Lancaster, recently published in 7 J. Empirical Legal Stud. 175 (2010), which also found no political effect in an excellent dataset of employment discrimination cases, which, unlike most studies (but like Fitzpatrick’s study), included the mass of settlements.  Since settlement is the modal outcome for civil litigation, including settled cases in studies of judge effects is important. The Nielsen et al. results echo the absence of judge effects in an earlier civil rights study, that included settlements, by Orley Ashenfelter, Stewart Schwab, and me.  See Politics and the Judiciary: The Influence of Judicial Background on Case Outcomes, 24 J. Legal Studies 257-81 (1995). 

This raises the following question: Is there a large-scale study of trial court level civil litigation, that includes settlements, that finds political or other judge effects?"

Presidents, Picking Judges, and Politics

Over at Concurring Opinions, Corey Yung (John Marshall) is on a blogging roll. His latest post summarizes results from his study addressing: "which President(s) appointed the most ideological judges?"

UPDATE: Part 2.

Measuring Judicial Ideology Conversation

Nice discussion (the first of a promised series of posts on the topic) over at Concurring Opinions prompted by Corey Yung (John Marshall) here.

UPDATE: Part 2.

UPDATE #2: Part 3.

UPDATE #3: Part 4.

Identifying and Measuring Ideology in Judging

In a recent article, The Context of Ideology: Law, Politics, and Empirical Legal Scholarship, 75 Missouri L. Rev. 79 (2010), I argue for a new approach to identifying the ideological nature of Supreme Court cases.  The abstract follows:

In their confirmation hearings, Chief Justice Roberts and Justice Sotomayor both articulated a vision of the neutral judge who decides cases without resort to personal perspectives or opinions, in short, without ideology. At the other extreme, the dominant model of judicial decisionmaking in political science has long been the attitudinal model, which posits that the Justices‟ votes can be explained primarily as expressions of their personal policy preferences, with little or no role for law, legal reasoning, or legal doctrine.

Many traditional legal scholars have criticized such scholarship for its insistence on the primacy of ideology in judicial decisionmaking, even as empirical legal scholarship has grown in significance and influence in the legal academy. Recently, however, empirical scholars and traditional legal academics have begun to engage in serious discussions with each other about how to think about and evaluate the balance between law and ideology and about how to harness the powerful tools of quantitative analysis to study such questions. In this Article, I offer several contributions to this discussion.

First, the Article evaluates current efforts by empirical scholars to identify the ideological character of cases. These efforts generally assume that the ideological character of a case can be determined by reference to a single liberal-conservative spectrum, and they generally presume that all or most cases present only a single issue. Through a recoding and quantitative analysis of a random sample of recent Rehnquist Court cases, as well as through a qualitative analysis of many of the cases, I establish concretely some of the limitations of these efforts. Specifically, I demonstrate that these approaches are indeterminate and oversimplified, and often prevent scholars from identifying cases in which the Justices face issues that pull them in different ideological directions. At the same time, however, I identify their strengths, particularly the strengths of approaches that leverage information derived from the Justices‟ actual voting patterns.

Finally, I propose a new approach for empirical scholars interested in studying the role of ideology in Supreme Court cases. Rather than starting with the assumption that ideology is the most important factor in Supreme Court decisionmaking, my approach focuses on just how important ideology was to the Justices in a particular case. Measuring ideological salience, I argue, would allow empirical scholars to study the empirical question of when and how ideology – and other factors – affects decisionmaking. Doing so would also open the door to a wide variety of important and interesting research questions. Most importantly, this approach would allow empirical scholars to engage with more traditional legal academics in the important normative debates about when and how ideology should play a role in the work of the Supreme Court.

Social Science in Law: Reflections on Then and Now

In their brief essay (forthcoming in Law and Human Behavior), John Monahan & Laurens Walker (both at UVa) assess the continuities and changes that have occurred in the application of social science research to law during the past 25 years. It is an appropriate moment to reflect on changes and their velocity over time. An excerpted abstract follows:

"When the first edition [of the casebook Social Science in Law (2010)] appeared, courts’ reliance on social science was often confused and always contested. Now, courts’ reliance on social science is so common as to be unremarkable. What has changed - sometimes radically - are the substantive legal questions on which social science has been brought to bear."