Empirical Legal Studies

Criminal Sentencing Variations and Judge "Intersections"

While efforts to empirically assess the independent influence of, e.g., judge gender and age on an array of judicial outcomes are not uncommon in the judicial decisionmaking literature, a recent paper, The Intersectionality of Age and Gender on the Bench: Are Younger Female Judges Harsher with Serious Crimes?, by Morris Hoffman (Dist. Ct. Judge, St. of Colorado) et al., explores how judge age and gender may interact in ways that account for some of the observed variations in criminal sentencing. Or, more specifically, the paper considers whether "trial judges in their thirties sentence the same thirty years later, when they will likely still be on the bench.”

The paper benefits from an unusually rich data set: "2,995 individual state criminal sentences imposed in Colorado, covering 183 different types of crimes and 285 different judges—180 male and 105 female—over a sixteen-year time span (2001–2016)." Because the data set allows the authors to observe "the same judges sentencing similar crimes as they grow older," a longitudinal view of "the interactive effects of judicial age, gender, and crime level in determining sentences in actual criminal cases” becomes possible.

Some of the paper's core findings might surprise some. "We uncovered a three-way interaction not previously reported in the empirical or experimental literature: For high harm crimes, younger female judges sentenced convicted defendants more harshly than their male and older female colleagues. We controlled for the independent effect of judicial experience, leading to the conclusion that age—and not just experience on the bench—is driving the results.”

The analyses exclude, however, plea bargains, crimes where sufficient sentencing discretion does not exist (e.g., first-degree murder), and crimes which impede efforts to code for the crime's "harm" (e.g., drug possession). Despite any analytical attractiveness, such data exclusions invite questions about the results' generalizability. After all, a decision to exclude plea bargains, for example, removes a potentially sizable slice of criminal sentencing activity from the analyses. Data limitations aside, the paper's core results will likely interest judicial decisionmaking scholars and criminologists. The paper's abstract follows.

“We analyzed sentencing data from sixteen years of criminal trials in the State of Colorado, consisting of almost 3,000 individual sentences, and discovered an interaction effect of harm, gender, and age not reported in any of the empirical or experimental literature. Young female judges punished high harm crimes substantially more than their male and older female colleagues. These results, if confirmed, could have significant strategic and tactical implications for practicing lawyers. They may also inform policies surrounding judicial selection, education, training, and retirement.”

Evidence of More Revolving Doors

"Revolving doors," evidently, are everywhere. While in a few sectors (e.g., financial regulators/financial services industries) research on revolving doors is well-developed, empirical research in other sectors is less so. The latter sectors include, e.g., an individual's path from a Supreme Court clerkship, to an appellate litigation group, and then back to the Supreme Court as an advocate. While "hunches" about this "revolving door" have persisted over time, data on former clerks' efficacy arguing in front of their former bosses remain scant. A recent paper, The Influence of Personalized Knowledge at the Supreme Court: How (Some) Former Law Clerks Have the Inside Track, exploits decisions in the Supreme Court Database between the 1979 and 2018 Terms and contributes to this data void. In their paper, political scientists Ryan Black (Mich. St.) and Ryan Owens (Wisc.) find evidence of an effect as the following figure implies.

The paper's abstract, summarizing core results, follows.

"When arguing at the U.S. Supreme Court, former High Court law clerks enjoy significant influence over their former justices. Our analysis of forty years of judicial votes reveals that an attorney who formerly clerked for a justice is 16 percent more likely to capture that justice’s vote than an otherwise identical attorney who never clerked. What is more, an attorney who formerly clerked for a justice is 14 to 16 percent more likely to capture that justice’s vote than an otherwise identical attorney who previously clerked for a different justice. Former clerk influence is substantial, targeted, and appears to come from clerks’ personalized information about their justices. These results answer an important empirical question about the role of attorneys while raising normative concerns over fairness in litigation."

"Younger" Federal Judges & Executive Power

While modeling judicial decisions--especially for federal judges--is not a terribly new endeavor, a recent paper in the Journal of Law and Economics (2020), Younger Federal District Court Judges Favor Presidential Power, caught my eye. In it, the authors, Tom Campbell (Chapman) and Nathaniel Wilcox (Chapman--economics), assess how federal judges' votes on challenges to assertions of executive power evolve over a judge's life cycle. And they do so with an eye toward federal district court judges' potential appetite for judicial "promotion."

Drawing on district court opinions spanning from 1960 to 2015, what the authors find is that “The probability that district judges favor the executive (statistically) significantly increases with the hazard for elevation (and so declines with age) for all specifications with that regressor. Moreover, the estimated average marginal effects show that the estimated size of the effect is highly robust to the particular specification: in about one of every five cases, 50-year-old judges appear to rule differently than 65-year-old judges.” The paper's abstract follows.

"From 1960 to 2015, opinions of US federal district court judges (trial judges) in cases involving challenges to executive branch authority show that these judges favor executive authority less as they age. We suggest that district judges know that elevation to the federal circuit court of appeals becomes increasingly improbable, and hence have less reason to cooperate with the executive, with advancing age. Political variables, seniority of judges, and other variables introduced as extra regressors do not reverse this main result, nor does it appear to be the product of cohort effects or selection off the district court. When there are contemporaneous vacancies on their circuit courts, district judges in the 11 state circuits (but not the District of Columbia Circuit) are also more likely to favor the executive."

Submitting Bail Reforms to Data: Competing Interpretations in Cook County, Illinois

Various bail reforms are quickly emerging across jurisdictions. Proponents argue, at bottom, that such reforms will not impose undue increased risk on public safety. Such a claim is, of course, an empirical one. And, as is far too common in the “legal world,” such claims are made without the benefit of careful empirical study.

Enter bail reform initiatives in Cook County, Ill (functionally, Chicago) that were implemented in 2017 and studied in 2019. What has ensued—and is the focus of a recent paper by Paul Cassell (Utah) and Richard Fowles (Utah—econ), Does Bail Reform Increase Crime? An Empirical Assessment of the Public Safety Implications of Bail Reform in Cook County, Illinois—is a dispute over how to best interpret the study’s (that is, the “Bail Reform Study,” supervised by the Cook County Circuit Court) findings.

While the Bail Reform Study’s authors landed on far more “optimistic”  conclusions, the Cassell and Fowles paper, by contrast, has “serious doubts about its upbeat conclusions.” Their paper drills down on two main points. First, “the [Bail Reform] Study fails to recognize that, given that more defendants are being released after the reforms, even a 'stable' rate of community safety will inexorably lead to more crimes. That stable rate of safety—and, inversely, the stable rate of failure or public safety danger—applies across a larger pool of released defendants, which necessarily means that the public suffers additional crimes. In other words, at least in Cook County, more bail reform apparently means more crimes." Second, “we find that, contrary to the Study’s suggestion of stability, the number of crimes committed by pretrial releasees appears to have significantly increased.” A slightly excerpted abstract follows.

“Recently bail reform issues have been in the news across the country, as concerns about fair treatment of defendants and possible public safety risks from expanding pretrial release have collided. These issues involve important empirical questions, including whether releasing more defendants before trial leads to additional crimes. An opportunity to investigate this public safety issue has developed in Chicago, our nation’s third largest city. There, the Office of the Chief Judge of the Cook County Courts adopted new bail reform measures in September 2017 and reviewed them empirically in May 2019. Cook County’s Bail Reform Study concluded that the new procedures had released many more defendants before trial without any concomitant increase in crime. This article disputes the Study’s conclusions. This article explains that, contrary to the Study’s assertions, the new changes to pretrial release procedures appear to have led to a substantial increase in crimes committed by pretrial releasees in Cook County. Properly measured and estimated, after more generous release procedures were put in place, the number of released defendants charged with committing new crimes increased by 45%. And, more concerning, the number of pretrial releasees charged with committing new violent crimes increased by an estimated 33%. In addition, as reported by the Chicago Tribune, the Study’s data appears to undercount the number of releasees charged with new violent crimes; and a substantial number of aggravated domestic violence prosecutions prosecutors dropped after the changes, presumably because batterers were able to more frequently obtain release and intimidate their victims into not pursuing charges. These public safety concerns call into question whether the bail “reform” measures implemented in Cook County were cost-beneficial.”

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

A New Legal Cloud Over Statistical Analyses of Judicial Decisions in France

In a move that will strike many court scholars (and, likely, others as well) as verging on the "unbelievable" (in the literal sense of the word), according to emerging news reports (e.g, here) the French Government has now "banned the publication of statistical information about judges’ decisions–with a five year prison sentence set as the maximum punishment for anyone who breaks the new law." The new french law (Art. 33 of France's Justice Reform Act) makes clear that "[t]he identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.” (translated; emphasis added)

According to news accounts, the new french law targets "legal tech companies focused on litigation prediction and analytics–from publicly revealing the pattern of judges’ behaviour in relation to court decisions." That said, the implications for empirical legal scholars seeking to model judicial decisions, certainly at the individual judge-level, in France are obvious.

While empirical legal scholars seeking to analyze U.S. judicial decisions face bothersome "hurdles" when it comes to linking judge names and their published decisions (e.g., a long-standing AO policy that strips judge names/IDs from most publicly-available AO data sets), I am unaware of anything in the U.S. that comes close to what the French Government seeks to accomplish with its new law.

Constitutional Judicial Decisions: A Comparative Frame

Empirical assessments of judicial decisions are nothing new and includes well-plowed terrain. While assessing the influences on a judge's decision remains an inherently difficult research task, the degree of difficulty increases, in general, for constitutional cases. Moreover, when one considers the constitutional context from a comparative perspective, the difficulties mount exponentially. In Quantitative Analysis of Judicial Voting, Benjamin Alarie (Toronto) and Andrew James Green (Toronto) summarize the relevant literature, identify important challenges, and make a plea for more comparative work in this area. An excerpted abstract (below) summarizes what this paper sets out to accomplish.

"Empirical research on voting in constitutional cases is so difficult because there are so many potential influences on judges and such tremendous variance across countries. Some countries have written constitutions and some do not. For those that do, the scope of the rights and freedoms varies widely. Some countries have special courts to deal with constitutional matters while others allow lower courts to hear initial applications with appeals going to a generalist high court. At an even more basic level, empirical study of constitutional voting, and on judicial decision-making more generally, is made more challenging by the fact that there is not even a clear, agreed upon theory of how judges make decisions.

Does one judge vote differently from another because they are different people, with particular political or world views and certain characteristics such their gender? Studying how a judge’s attitudes influence her voting is a long-standing line of inquiry in empirical legal studies and political science. However, not surprisingly, a judge may also care about the law itself. For example, a judge may vote differently in a case involving striking employees if there is a broadly written right to association as opposed to a specific right to strike. What if there is no written constitution at all but instead some unwritten constitutional principles? Part 3 examines some approaches to determining how such differences in constitutional law may influence judges. Part 4 turns to the institutional structure surrounding the judge such as the structure of the court, the nature of the political system, and even the norms of decision-making by the judges themselves. We examine the influence of these institutional differences before considering in Part 5 contextual factors which may shift constitutional decision-making such as war or national security concerns. Our aim is to discuss both how existing analysis has been undertaken and the limitations of such analysis. We end the paper by pointing to the need for more comparative work."

New PACER Resource From Free Law Project and GSU Researchers

With appreciation to Charlotte Alexander (Georgia State) for alerting me to this development and assisting with this post, information on a new PACER resource follows.

For researchers who study the federal courts, litigation patterns and outcomes, and judicial decisionmaking, the Free Law Project (FLP) operates the RECAP Archive, which seeks to reproduce the contents of the Public Access to Electronic Court Records (PACER) system in a single, comprehensive, searchable database (RECAP is PACER spelled backwards). The archive sets out to address major problems with PACER that make life difficult for researchers: a per-page download fee, a lack of fine-tuned search capabilities that can mine a document's text, and a clunky search interface.

The RECAP Archive draws its materials from two sources: 1) PACER users’ "donated" documents, and 2) courts’ PACER Written Opinions Report (WOR) that includes all documents that each court has designated as a judge's "written opinion."

This second stream of RECAP contributions is new. In cooperation with Professors Charlotte Alexander (Georgia State) and Javad Feizollahi (Georgia State) and with support from a U.S. Department of Labor grant, FLP just completed a comprehensive download of all material available via the WOR, and updates the download daily. All told, the RECAP Archive now contains about 3.4 million documents from 1.5 million federal district court and bankruptcy cases from 1960 to the present, including all judicial opinions available via the Written Opinions Report. The RECAP search form and more information about the Archive are here.

While this development is exciting for researchers, some important caveats are in order. First, any RECAP documents other than court orders appear in the Archive only because a PACER user donated copies of documents to the archive. Second, WOR documents are the product of each judge’s determinations of which of his/her opinions to include. Despite efforts to standardize judge decisions (e.g., here) variation across judges introduces some level of selection bias.

Caveats notwithstanding, the RECAP Archive warrants attention.

Empirically Judging Judges

Assessing judicial outcomes and judicial decisionmaking not surprisingly endure as attractive topics for researchers. Indeed, the experimental literature is festooned with judicial "vignettes" that experimentally isolate critical variables. Of course, to really learn about judicial decisionmaking the "gold standard" sample remains actual judges (though, even when submitting real judges to experiments external validity concerns persist). On this front (and in this sub-field), my Cornell colleague, Jeff Rachlinski, remains a leading scholar. In a recent paper, Judging the Judiciary by the Numbers: Empirical Research on Judges, Rachlinski (Cornell) & Andrew Wistrich (CA Central Dist. Ct.) report results, consistent with past their efforts, that document how, just like most humans, judges succumb to various "mental shortcuts" that can lead them to mistakes. The paper's abstract follows.

“Do judges make decisions that are truly impartial? A wide range of experimental and field studies reveal that several extra-legal factors influence judicial decision making. Demographic characteristics of judges and litigants affect judges’ decisions. Judges also rely heavily on intuitive reasoning in deciding cases, making them vulnerable to the use of mental shortcuts that can lead to mistakes. Furthermore, judges sometimes rely on facts outside the record and rule more favorably towards litigants who are more sympathetic or with whom they share demographic characteristics. On the whole, judges are excellent decision makers, and sometimes resist common errors of judgment that influence ordinary adults. The weight of the evidence, however, suggests that judges are vulnerable to systematic deviations from the ideal of judicial impartiality.”

Bringing Data to Theories of USSC Review

In Why Does the Supreme Court Uphold so Many Laws?, the authors, Ben Johnson (Princeton--Politics) and Keith Whittington (Princeton--Politics), empirically explore the Supreme Court's judicial review practices regarding its discretionary review authority. As the paper notes, "Scholars spend a lot of time considering the legitimacy and implications of the Supreme Court striking down federal laws by use of judicial review. Similarly, there is a large literature focusing on the Court’s power and obligation to manage the federal judiciary through its certiorari powers over its own docket and its ability to reverse lower courts. However, there is almost no work that examines the interplay of the Court’s judicial review powers and its managerial authority."

Exploiting the Judicial Review of Congress (JRC) database, this paper brings much-needed data to prevailing competing theories. The JRC database is helpful in that it "identifies cases decided by the U.S. Supreme Court that substantively review the constitutionality of provisions of federal statutes from the founding to the present. Significantly, the JRC database includes not only cases in which the Court found a federal statutory provision to be unconstitutional but also cases in which the Court upheld a statutory provision against constitutional challenge.”

What the paper finds surprises: "Our examination of these data reveal that the category of cases we call the uphold-affirm set---where the lower court upholds a statute and the Court affirms---is the plurality category among judicial review cases. This is surprising not only because this category is completely incompatible with current theories of judicial review and certiorari, but also it shows that the Court, which usually reverses lower courts, deviates from its general practice when it exercises its judicial review powers where it tends to affirm.” Given the findings, the authors explore why "the U.S. Supreme Court taken and affirmed so many decisions that upheld so many laws against constitutional challenge."

AI & Predicting USSC Case Outcomes

As legal scholars begin to digest AI's ramifications for law and legal analysis, a recent paper exploits the Supreme Court Database to illustrate AI's ability topredict case outcomes. In A general approach for predicting the behavior of the Supreme Court of the United States, Daniel Katz (Chicago-Kent) et al. explore almost two centuries of Supreme Court decisions and use an algorithm to successfully predict 70.2% of the case outcomes and 71.9% of individual justice-level votes. The abstract follows.

"Building on developments in machine learning and prior work in the science of judicial prediction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the justice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications."

SCOTUS Tenure Data

An interesting (and timely) data crunching post over at Empirical SCOTUS focuses on Justice tenure (here). As the post notes, with an eye towards assessing President Trump's potential to influence the Court's future through appointments: "If we look over time to all Justices confirmed to the Court or elevated from Associate to Chief Justice, we see that the average time between appointments has increased only minimally to around two years (the Justices are listed sequentially in the order in which they were nominated/elevated and the minimum of zero years is when two Justices were confirmed within the same year)." Notably, the "time between nominations" trend remains essentially flat despite the concurrent emergence two related trends. First, the average age at nomination has "declined about five years on average since the beginning of the 20^th century." Second, Justices last on the Court "for more years and until they are older."

Reserach Projects at HLS' A2J Lab

Under Jim Greiner's direction, work at HLS' Access to Justice Lab (previously described here) is underway. One recent courthouse project in Dane County, WI, involves an evaluation of that court's Public Safety Assessment ("PSA"). As this post notes, "Criminal process in this jurisdiction now includes additional, scientifically based information in a randomly selected subset of cases to inform pre-disposition release decisions. The judicial official–known here as a Commissioner–receives risk scores and a recommendation for release through the PSA and its static criminal history inputs to consider in reaching those decisions." (The A2J link also includes an informative video.)

Federal Judicial Center, ICPSR, Data Access, and Change

Back in the "old days," virtually immediate access to ICPSR's massive--and massively important--federal court data holdings was simple, easy, and, quite literally, involved only a few "clicks" of the mouse. A few years ago, "something happened," and that something resulted in the imposition of a formal ICPSR application requirement along with pre-IRB approvals and restricted use agreements. And all this for publicly available data. Gone were the days when researchers could simply "browse" datasets without such impediments. While ICPSR data remain "available," access now requires a comparatively dogged effort and these barriers to access have dampened overall use of ICPSR data.

But change is coming, and some of it has now arrived. One supplier of ICPSR data germane to legal scholars, the Federal Judicial Center, in conjunction with the U.S. Administrative Office, has effectively "side-stepped" ICPSR and made the FJC's Integrated Database (through 2016) directly available to users. FJC's Joe Cecil, who was instrumental in this data access project, sums it up nicely:

"The Federal Judicial Center’s Integrated Database, containing records of all federal court cases, is now available without the necessity of a restricted use agreement at www.fjc.gov/research/idb (or here). An interactive query function allows selection of subsets of civil cases based on federal circuit, district, nature of suit, disposition, origin, jurisdiction, class action designation, filing date, termination date, and docket number. Annual case filing and termination datasets through 2016 for civil, criminal defendant, appellate, and bankruptcy cases also can be downloaded in SAS format."

The FJC deserves well-earned applause from empirical legal scholars for its efforts.

SCOTUS' Treatment of 4th Amendment Claims

Despite extensive scholarly attention the Court's Fourth Amendment jurisprudence, "there has been scant empirical research about how modern Justices have voted in Fourth Amendment cases." To this end, a recent paper by Brent Newton (US Sent. Comm'n), The Supreme Court's Fourth Amendment Scorecard, provides a helpful descriptive contribution. It does so by analyzing the "173 cases from 1982 through 2015 in which the Supreme Court granted certiorari and gave plenary consideration to Fourth Amendment issues.” An excerpted abstract, below, summarizes some of the paper's main findings.

“The key findings include: (1) the overwhelming majority of Fourth Amendment litigants (FALs) before the Supreme Court were criminal defendants rather than civil rights plaintiffs… ; (2) Fourth Amendment claims succeeded in slightly less than one in four plenary-review cases, and the Court reversed the judgment of the lower courts in the vast majority of Fourth Amendment cases in which certiorari was granted; (3) FALs were much more likely to succeed in cases concerning substantive issues than remedial issues; (4) civil rights plaintiffs were slightly more likely to succeed than criminal defendants; (5) individual Justices generally voted in Fourth Amendment cases in a manner that reflected their overall ideological dispositions, yet Justices Blackmun, Scalia, and Breyer bucked this trend, resulting in more ideologically mixed voting blocs in Fourth Amendment cases than in several other divisive areas of the Court’s docket, such as abortion rights or voting rights….”