Empirical Legal Studies

Misdemeanor (Non) Prosecution

Hidden in plain sight is a significant amount of criminal misdemeanor activity in state courts. Indeed, "every year approximately 13 million Americans are charged with misdemeanor offenses, and misdemeanor cases make up over 80 percent of the cases processed by the U.S. criminal justice system." Scholars and policymakers increasingly raise concerns with the sheer volume of prosecutions for low-level offenses. As well, "some district attorneys across the country have begun to implement alternatives to misdemeanor prosecution, particularly for nonviolent defendants."

As scholarly attention continues to focus on criminal misdemeanors (see, e.g., here), it is inevitable that larger data sets and more sophisticated analyses emerge. One such example includes a recent paper, Misdemeanor Prosecution. In it, Agan (Rutgers--econ) et al., assess a practice in the Suffolk County (MA) District Attorney's Office that imposes upon prosecutors a "presumption of nonprosecution" for "nonviolent misdemeanor offenses." To do so, the paper exploits "new data [N=67,553] on the prosecution of nonviolent misdemeanor criminal complaints from the Suffolk County District Attorney’s Office (SCDAO) in Massachusetts between 2004 and 2018 to estimate the impact of nonprosecution of nonviolent misdemeanors on future criminal justice system contact." (emphasis added)

Despite some missing ADA-level data concerns, one main result is that: "increasing nonprosecution reduced the likelihood of subsequent criminal complaints within a one-year post-arraignment window. In addition, there does not appear to have been an increase in reported crime due to the policy change." The abstract follows.

"Communities across the United States are reconsidering the public safety benefits of prosecuting nonviolent misdemeanor offenses. So far there has been little empirical evidence to inform policy in this area. In this paper we report the first estimates of the causal effects of misdemeanor prosecution on defendants' subsequent criminal justice involvement. We leverage the as-if random assignment of nonviolent misdemeanor cases to Assistant District Attorneys (ADAs) who decide whether a case should move forward with prosecution in the Suffolk County District Attorney's Office in Massachusetts. These ADAs vary in the average leniency of their prosecution decisions. We find that, for the marginal defendant, nonprosecution of a nonviolent misdemeanor offense leads to large reductions in the likelihood of a new criminal complaint over the next two years. These local average treatment effects are largest for first-time defendants, suggesting that averting initial entry into the criminal justice system has the greatest benefits. We also present evidence that a recent policy change in Suffolk County imposing a presumption of nonprosecution for a set of nonviolent misdemeanor offenses had similar beneficial effects: the likelihood of future criminal justice involvement fell, with no apparent increase in local crime rates."

What Would Happen if Plea Bargaining Was Banned?

For an array of reasons, plea bargaining endures as a concern for many criminal law scholars. Proponents of plea bargaining "point to its efficiency−since trials are costly if both sides form reasonably similar beliefs about the outcome to arise at trial they can just agree to that expected sanction beforehand and save the costs." Opponents, by contrast, fear that excessive plea bargaining rates--combined with various structural differences that distinguish many defendants (especially indigent ones) and prosecutors' offices--may help coerce some factually innocent defendants into pleading guilty rather than defending their innocence at trail.

Amid growing calls to "regulate the plea bargaining process in an attempt to reduce its prevalence," one state--Alaska--acted. Specifically, Alaska Attorney General Michael Geraghty announced a new policy limiting plea bargaining in July 2013. The limitation prohibits prosecutors from "negotiating over the length of the jail/prison sentence for those accused of committing violent crimes."

In a recent paper, Alaska's Plea Bargaining Ban, Bryan McCannon (Univ. West Vir.--Economics) sets out to assess the efficacy of Alaska's experiment. The paper exploits a data set that includes annual (2007-2019) court-year data on how all of Alaska's Superior Courts disposed of their criminal and civil dockets (N=874). That Alaska's plea bargaining ban can affect a variety of margins raises important identification concerns. One thing the ban cannot affect, however, are pretrial plea negotiations in civil cases. Thus, the paper compares changes in the civil and criminal contexts.

The primary variable of interest is the rate at which cases settled pretrial and the author estimates a diff-in-diff specification. The critical take-away from the findings is that Alaska's plea bargaining ban for defendants charged with violent criminal crimes "does not have a meaningful effect" (emphasis added). One explanation for the absence of any detected affect involves the (often-underappreciated) importance of prosecutors' charge-bargaining policies and practices (which fall outside of the Alaska's ban on negotiating jail/prison sentences).

To be sure, important study limitations exist. Alaska may be an anomalous state. As well, as the study draws from an aggregated panel data set it cannot account for various case-specific factors. Finally, aggregating the case outcome data to the court level invites ignoring various heterogeneous treatment effects. Research design limitations aside, however, while this study will certainly not be the final word on this topic it is a helpful first step. The paper's abstract follows.

“Plea bargaining dominates the U.S. criminal justice system and has garnered calls to reduce its prevalence. In 2013 Alaska’s Attorney General acted banning a practice known as sentencing bargaining where the prosecutor negotiates with the defense over the length of the incarceration. I provide the first causal identification of this policy’s impact on the plea bargaining rate. I show that the policy, by leaving open the charging discretion (in effect allowing charge bargaining), was ineffective at changing plea bargaining’s prevalence across the state. Policymakers should look elsewhere for tools if they want to mitigate its rate.”

A New and Interesting Take on Measuring Case Complexity

Most judicial decisionmaking papers can be criticized for inadequately controlling for variation in "case complexity." Lingering disputes over what "complexity" even means in terms of litigated cases only increase the degree of difficulty in constructing such a control. And even if a consensus on what "case complexity" means existed, constructing a reliable, consistent, and stable measure introduces additional challenges.

Difficulties, conceptual and mechanical, aside, a recent paper, Measuring Supreme Court Case Complexity, makes a helpful contribution. In it, Greg Goelzhauser (Utah State--Poli Sci) et al., both critique an array of earlier efforts and advance their own Bayesian measurement model to construct a latent (Supreme Court) case complexity measure that exploits issue and provision counts from the cases' merits briefs. By measuring case complexity from pre-decision data (merits briefs) the paper argues that its ex poste approach "diminishes endogeneity and post-treatment bias concerns when studying merits stage outcomes." To develop its measure, the authors draw from issue and provision (hand-coded) counts extracted from merits briefs filed in the 1954 through 2017 Court Terms, incident to Supreme Court Rules 24(1)(a) and 24(1)(f).

While perhaps limited to Supreme Court cases, new approaches to a difficult challenge can make important methodological contributions. The paper's abstract follows.

"Case complexity is central to the study of judicial politics. The dominant measures of Supreme Court case complexity use information on legal issues and provisions observed post-decision. As a result, scholars using these measures to study merits stage outcomes such as bargaining, voting, separate opinion production, and opinion content introduce post-treatment bias and exacerbate endogeneity concerns. Furthermore, existing issue measures are not valid proxies for complexity. Leveraging information on issues and provisions extracted from merits briefs, we develop a new latent measure of Supreme Court case complexity. This measure maps with the prevailing understanding of the underlying concept while mitigating inferential threats that hamper empirical evaluations. Our brief-based measurement strategy is generalizable to other contexts where it is important to generate exogenous and pre-treatment indicators for use in explaining merits decisions."

Empirical Conjectures About the Effects of Raising the Jurisdictional Amount

As my Civ Pro colleagues periodically remind me, we're likely overdue for an increase to jurisdictional thresholds. Even if they're correct, however, what might be the expected (and unexpected) consequences of various increases to jurisdictional thresholds? A recent paper by Steven Gensler (Oklahoma) and Roger Michalski (Oklahoma), The Million Dollar Diversity Docket, puts these counter-factual questions to data. And what they find should interest more than procedeuralists.

The authors hand-coded data from five (non-randomly selected) federal districts (Minnesota, Georgia Northern, New Jersey, Texas Northern, and Utah). From each of these districts they selected a random draw of diversity jurisdiction cases based on Federal Judicial Center (“FJC”) data on cases completed in 2018, resulting in 2,896 usable cases. The paper's methodological move involves substituting the temporal component in survival analysis with a measure of jurisdictional amount. That is, instead of asking on which day a machine fails, we ask at which jurisdictional amount a case fails. To do so, the authors estimated modified KaplanMeier survival functions and Cox Proportional Hazard Regression models.

The paper's main takeaway is that jurisdictional amounts "are not neutral throttles. They do not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others." The abstract follows.

"What would happen if Congress raised the jurisdictional amount in the diversity jurisdiction statute? Given that it has been almost 25 years since the last increase, we are probably overdue for another one. But to what amount? And with what effect? What would happen if Congress raised the jurisdictional amount from the current $75,000 to $250,000 or, say, $1 million?

Using modified Kaplan-Meier survival functions and Cox Proportional Hazard Regression models to analyze a novel hand-coded data set of pleadings in 2900 cases, we show that the jurisdictional amount is not a neutral throttle. It does not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others. As such, setting the amount in controversy at any level entails normative choices about which cases to bless and curse with a federal forum.

Our findings implicate not only policy discussions about diversity jurisdiction in general, but also debates currently underway in Congress to utilize modifications of the jurisdictional amount requirement to channel COVID-19 related litigation into federal courts. We hope our empirical work will inform these debates and enable a new wave of scholarship on the basic functions and functioning of the federal diversity docket. We outline some beginnings of this type of work by exploring radically different avenues of thought that develop and re-imagine the diversity docket anew. They range from incremental inflation-adjustments, to carve-outs, experiments with lotteries, and the radical replacement of an amount in controversy minimum with a maximum."

State Supreme Court Elections Data Set

Judicial decisionmaking scholars comparing outcomes from "elected" and "non-elected" judges invariably turn to state court systems (if the analytic focus is domestic) for the necessary variation. Those turning to the array of state court systems--and the range of state supreme court election types--invariably confront data gathering challenges that materially differ from challenges associated with, e.g., Art. III judges. One of the leading state court scholars, Bert Kritzer (Minn.), continues to make state court election data gathering efforts a bit easier for the rest of us. Not only has Bert engaged in the often under-appreciated yet Herculean effort of original data set building, but he shares his data sets (and associated documentation, codebooks, etc.) with other interested scholars. Bert's most recent updating effort, curated on Harvard's Dataverse, includes a collection of state supreme court election results that now range from 1946 through 2020.

"The archive includes three core datasets plus state-level Excel spreadsheets for all states that employed elections at least part of the time since 1946 to select or retain members of the state supreme court (known New York state as the Court of Appeals) including for Texas and Oklahoma both the state supreme court and the Court of Criminal Appeals which are the final courts for criminal cases in those states."

Those interested in the data sets can click here.

ELS in the "Real World"

The growth (and import) of MDL proceedings in federal courts are, by now, beyond obvious. One practice that has emerged organically among some (many?) of the larger MDLs involves the use of "fact sheets." Fact sheets can involve a range of "party-negotiated questionnaires, directed at parties to the case (both plaintiff and defendant), [and] provide judges and attorneys with information about the scope of the proceeding." The rise of MDLs has also prompted calls for new rules for MDL proceedings. Perhaps not surprisingly, the Advisory Committee on Civil Rules (Advisory Committee) recently began considering "specific rules related to multidistrict litigation (MDL) proceedings," including rules relating to fact sheets.

As a recent paper by Margaret Williams (Fed. Judicial Center), An Empirical Evaluation of Proposed Civil Rules for Multidistrict Litigation, makes clear, "the question of whether or not rules for MDLs, specifically rules regarding fact sheets, are necessary would have to be answered by evaluating how fact sheets are currently used in MDL proceedings, if proposed rules would change that, and how." As is too often the case in legal reform, however, reform efforts go forward absent anything even resembling a necessary empirical footing. As Williams notes, "there is very little published empirical work examining the use of fact sheets in MDL proceedings." While the Williams paper makes an important initial, largely descriptive contribution to our knowledge base, the paper also makes clear that much more work remains on fact sheets' efficacy as a case management tool in the MDL context. An excerpted abstract follows.

"Using a sample of 116 mass tort proceedings (typically involving products liability) centralized through MDL between 2008 and 2018, we examine when fact sheets were ordered, what were the procedures for complying with the case management order, what information was collected, and what effect they have on the termination of the proceedings. The proceeding ranged between 3 and 40,533 actions and were open a minimum of 118 days and a maximum of 3,811 days. Actions terminated within the proceeding at least 98% of the time, but little information is available on how the actions terminated. Proceedings were centralized in 40 districts. We find that fact sheets are ordered more than half the time, and were most likely to be used in the largest proceedings. The information in fact sheets is used in several ways within the proceeding, including identifying cases for bellwether trials and winnowing cases, and the use of fact sheet processes leads to faster termination of the proceeding, all else being equal. Our sample of proceedings suggests judges use fact sheets to organize products liability proceedings when judges perceive they are merited, after considering the size of the proceeding or the nature of the litigation. The frequency with which judges already employ fact sheets and the variation in the use calls into question both the need for a rule and how to write one without tying the hands of transferee judges. Many issues regarding how fact sheets are used remain to be studied more in-depth."

Criminal Sentencing Variations and Judge "Intersections"

While efforts to empirically assess the independent influence of, e.g., judge gender and age on an array of judicial outcomes are not uncommon in the judicial decisionmaking literature, a recent paper, The Intersectionality of Age and Gender on the Bench: Are Younger Female Judges Harsher with Serious Crimes?, by Morris Hoffman (Dist. Ct. Judge, St. of Colorado) et al., explores how judge age and gender may interact in ways that account for some of the observed variations in criminal sentencing. Or, more specifically, the paper considers whether "trial judges in their thirties sentence the same thirty years later, when they will likely still be on the bench.”

The paper benefits from an unusually rich data set: "2,995 individual state criminal sentences imposed in Colorado, covering 183 different types of crimes and 285 different judges—180 male and 105 female—over a sixteen-year time span (2001–2016)." Because the data set allows the authors to observe "the same judges sentencing similar crimes as they grow older," a longitudinal view of "the interactive effects of judicial age, gender, and crime level in determining sentences in actual criminal cases” becomes possible.

Some of the paper's core findings might surprise some. "We uncovered a three-way interaction not previously reported in the empirical or experimental literature: For high harm crimes, younger female judges sentenced convicted defendants more harshly than their male and older female colleagues. We controlled for the independent effect of judicial experience, leading to the conclusion that age—and not just experience on the bench—is driving the results.”

The analyses exclude, however, plea bargains, crimes where sufficient sentencing discretion does not exist (e.g., first-degree murder), and crimes which impede efforts to code for the crime's "harm" (e.g., drug possession). Despite any analytical attractiveness, such data exclusions invite questions about the results' generalizability. After all, a decision to exclude plea bargains, for example, removes a potentially sizable slice of criminal sentencing activity from the analyses. Data limitations aside, the paper's core results will likely interest judicial decisionmaking scholars and criminologists. The paper's abstract follows.

“We analyzed sentencing data from sixteen years of criminal trials in the State of Colorado, consisting of almost 3,000 individual sentences, and discovered an interaction effect of harm, gender, and age not reported in any of the empirical or experimental literature. Young female judges punished high harm crimes substantially more than their male and older female colleagues. These results, if confirmed, could have significant strategic and tactical implications for practicing lawyers. They may also inform policies surrounding judicial selection, education, training, and retirement.”

Evidence of More Revolving Doors

"Revolving doors," evidently, are everywhere. While in a few sectors (e.g., financial regulators/financial services industries) research on revolving doors is well-developed, empirical research in other sectors is less so. The latter sectors include, e.g., an individual's path from a Supreme Court clerkship, to an appellate litigation group, and then back to the Supreme Court as an advocate. While "hunches" about this "revolving door" have persisted over time, data on former clerks' efficacy arguing in front of their former bosses remain scant. A recent paper, The Influence of Personalized Knowledge at the Supreme Court: How (Some) Former Law Clerks Have the Inside Track, exploits decisions in the Supreme Court Database between the 1979 and 2018 Terms and contributes to this data void. In their paper, political scientists Ryan Black (Mich. St.) and Ryan Owens (Wisc.) find evidence of an effect as the following figure implies.

The paper's abstract, summarizing core results, follows.

"When arguing at the U.S. Supreme Court, former High Court law clerks enjoy significant influence over their former justices. Our analysis of forty years of judicial votes reveals that an attorney who formerly clerked for a justice is 16 percent more likely to capture that justice’s vote than an otherwise identical attorney who never clerked. What is more, an attorney who formerly clerked for a justice is 14 to 16 percent more likely to capture that justice’s vote than an otherwise identical attorney who previously clerked for a different justice. Former clerk influence is substantial, targeted, and appears to come from clerks’ personalized information about their justices. These results answer an important empirical question about the role of attorneys while raising normative concerns over fairness in litigation."

"Younger" Federal Judges & Executive Power

While modeling judicial decisions--especially for federal judges--is not a terribly new endeavor, a recent paper in the Journal of Law and Economics (2020), Younger Federal District Court Judges Favor Presidential Power, caught my eye. In it, the authors, Tom Campbell (Chapman) and Nathaniel Wilcox (Chapman--economics), assess how federal judges' votes on challenges to assertions of executive power evolve over a judge's life cycle. And they do so with an eye toward federal district court judges' potential appetite for judicial "promotion."

Drawing on district court opinions spanning from 1960 to 2015, what the authors find is that “The probability that district judges favor the executive (statistically) significantly increases with the hazard for elevation (and so declines with age) for all specifications with that regressor. Moreover, the estimated average marginal effects show that the estimated size of the effect is highly robust to the particular specification: in about one of every five cases, 50-year-old judges appear to rule differently than 65-year-old judges.” The paper's abstract follows.

"From 1960 to 2015, opinions of US federal district court judges (trial judges) in cases involving challenges to executive branch authority show that these judges favor executive authority less as they age. We suggest that district judges know that elevation to the federal circuit court of appeals becomes increasingly improbable, and hence have less reason to cooperate with the executive, with advancing age. Political variables, seniority of judges, and other variables introduced as extra regressors do not reverse this main result, nor does it appear to be the product of cohort effects or selection off the district court. When there are contemporaneous vacancies on their circuit courts, district judges in the 11 state circuits (but not the District of Columbia Circuit) are also more likely to favor the executive."

Submitting Bail Reforms to Data: Competing Interpretations in Cook County, Illinois

Various bail reforms are quickly emerging across jurisdictions. Proponents argue, at bottom, that such reforms will not impose undue increased risk on public safety. Such a claim is, of course, an empirical one. And, as is far too common in the “legal world,” such claims are made without the benefit of careful empirical study.

Enter bail reform initiatives in Cook County, Ill (functionally, Chicago) that were implemented in 2017 and studied in 2019. What has ensued—and is the focus of a recent paper by Paul Cassell (Utah) and Richard Fowles (Utah—econ), Does Bail Reform Increase Crime? An Empirical Assessment of the Public Safety Implications of Bail Reform in Cook County, Illinois—is a dispute over how to best interpret the study’s (that is, the “Bail Reform Study,” supervised by the Cook County Circuit Court) findings.

While the Bail Reform Study’s authors landed on far more “optimistic”  conclusions, the Cassell and Fowles paper, by contrast, has “serious doubts about its upbeat conclusions.” Their paper drills down on two main points. First, “the [Bail Reform] Study fails to recognize that, given that more defendants are being released after the reforms, even a 'stable' rate of community safety will inexorably lead to more crimes. That stable rate of safety—and, inversely, the stable rate of failure or public safety danger—applies across a larger pool of released defendants, which necessarily means that the public suffers additional crimes. In other words, at least in Cook County, more bail reform apparently means more crimes." Second, “we find that, contrary to the Study’s suggestion of stability, the number of crimes committed by pretrial releasees appears to have significantly increased.” A slightly excerpted abstract follows.

“Recently bail reform issues have been in the news across the country, as concerns about fair treatment of defendants and possible public safety risks from expanding pretrial release have collided. These issues involve important empirical questions, including whether releasing more defendants before trial leads to additional crimes. An opportunity to investigate this public safety issue has developed in Chicago, our nation’s third largest city. There, the Office of the Chief Judge of the Cook County Courts adopted new bail reform measures in September 2017 and reviewed them empirically in May 2019. Cook County’s Bail Reform Study concluded that the new procedures had released many more defendants before trial without any concomitant increase in crime. This article disputes the Study’s conclusions. This article explains that, contrary to the Study’s assertions, the new changes to pretrial release procedures appear to have led to a substantial increase in crimes committed by pretrial releasees in Cook County. Properly measured and estimated, after more generous release procedures were put in place, the number of released defendants charged with committing new crimes increased by 45%. And, more concerning, the number of pretrial releasees charged with committing new violent crimes increased by an estimated 33%. In addition, as reported by the Chicago Tribune, the Study’s data appears to undercount the number of releasees charged with new violent crimes; and a substantial number of aggravated domestic violence prosecutions prosecutors dropped after the changes, presumably because batterers were able to more frequently obtain release and intimidate their victims into not pursuing charges. These public safety concerns call into question whether the bail “reform” measures implemented in Cook County were cost-beneficial.”

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

A New Legal Cloud Over Statistical Analyses of Judicial Decisions in France

In a move that will strike many court scholars (and, likely, others as well) as verging on the "unbelievable" (in the literal sense of the word), according to emerging news reports (e.g, here) the French Government has now "banned the publication of statistical information about judges’ decisions–with a five year prison sentence set as the maximum punishment for anyone who breaks the new law." The new french law (Art. 33 of France's Justice Reform Act) makes clear that "[t]he identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.” (translated; emphasis added)

According to news accounts, the new french law targets "legal tech companies focused on litigation prediction and analytics–from publicly revealing the pattern of judges’ behaviour in relation to court decisions." That said, the implications for empirical legal scholars seeking to model judicial decisions, certainly at the individual judge-level, in France are obvious.

While empirical legal scholars seeking to analyze U.S. judicial decisions face bothersome "hurdles" when it comes to linking judge names and their published decisions (e.g., a long-standing AO policy that strips judge names/IDs from most publicly-available AO data sets), I am unaware of anything in the U.S. that comes close to what the French Government seeks to accomplish with its new law.

Constitutional Judicial Decisions: A Comparative Frame

Empirical assessments of judicial decisions are nothing new and includes well-plowed terrain. While assessing the influences on a judge's decision remains an inherently difficult research task, the degree of difficulty increases, in general, for constitutional cases. Moreover, when one considers the constitutional context from a comparative perspective, the difficulties mount exponentially. In Quantitative Analysis of Judicial Voting, Benjamin Alarie (Toronto) and Andrew James Green (Toronto) summarize the relevant literature, identify important challenges, and make a plea for more comparative work in this area. An excerpted abstract (below) summarizes what this paper sets out to accomplish.

"Empirical research on voting in constitutional cases is so difficult because there are so many potential influences on judges and such tremendous variance across countries. Some countries have written constitutions and some do not. For those that do, the scope of the rights and freedoms varies widely. Some countries have special courts to deal with constitutional matters while others allow lower courts to hear initial applications with appeals going to a generalist high court. At an even more basic level, empirical study of constitutional voting, and on judicial decision-making more generally, is made more challenging by the fact that there is not even a clear, agreed upon theory of how judges make decisions.

Does one judge vote differently from another because they are different people, with particular political or world views and certain characteristics such their gender? Studying how a judge’s attitudes influence her voting is a long-standing line of inquiry in empirical legal studies and political science. However, not surprisingly, a judge may also care about the law itself. For example, a judge may vote differently in a case involving striking employees if there is a broadly written right to association as opposed to a specific right to strike. What if there is no written constitution at all but instead some unwritten constitutional principles? Part 3 examines some approaches to determining how such differences in constitutional law may influence judges. Part 4 turns to the institutional structure surrounding the judge such as the structure of the court, the nature of the political system, and even the norms of decision-making by the judges themselves. We examine the influence of these institutional differences before considering in Part 5 contextual factors which may shift constitutional decision-making such as war or national security concerns. Our aim is to discuss both how existing analysis has been undertaken and the limitations of such analysis. We end the paper by pointing to the need for more comparative work."

New PACER Resource From Free Law Project and GSU Researchers

With appreciation to Charlotte Alexander (Georgia State) for alerting me to this development and assisting with this post, information on a new PACER resource follows.

For researchers who study the federal courts, litigation patterns and outcomes, and judicial decisionmaking, the Free Law Project (FLP) operates the RECAP Archive, which seeks to reproduce the contents of the Public Access to Electronic Court Records (PACER) system in a single, comprehensive, searchable database (RECAP is PACER spelled backwards). The archive sets out to address major problems with PACER that make life difficult for researchers: a per-page download fee, a lack of fine-tuned search capabilities that can mine a document's text, and a clunky search interface.

The RECAP Archive draws its materials from two sources: 1) PACER users’ "donated" documents, and 2) courts’ PACER Written Opinions Report (WOR) that includes all documents that each court has designated as a judge's "written opinion."

This second stream of RECAP contributions is new. In cooperation with Professors Charlotte Alexander (Georgia State) and Javad Feizollahi (Georgia State) and with support from a U.S. Department of Labor grant, FLP just completed a comprehensive download of all material available via the WOR, and updates the download daily. All told, the RECAP Archive now contains about 3.4 million documents from 1.5 million federal district court and bankruptcy cases from 1960 to the present, including all judicial opinions available via the Written Opinions Report. The RECAP search form and more information about the Archive are here.

While this development is exciting for researchers, some important caveats are in order. First, any RECAP documents other than court orders appear in the Archive only because a PACER user donated copies of documents to the archive. Second, WOR documents are the product of each judge’s determinations of which of his/her opinions to include. Despite efforts to standardize judge decisions (e.g., here) variation across judges introduces some level of selection bias.

Caveats notwithstanding, the RECAP Archive warrants attention.

Empirically Judging Judges

Assessing judicial outcomes and judicial decisionmaking not surprisingly endure as attractive topics for researchers. Indeed, the experimental literature is festooned with judicial "vignettes" that experimentally isolate critical variables. Of course, to really learn about judicial decisionmaking the "gold standard" sample remains actual judges (though, even when submitting real judges to experiments external validity concerns persist). On this front (and in this sub-field), my Cornell colleague, Jeff Rachlinski, remains a leading scholar. In a recent paper, Judging the Judiciary by the Numbers: Empirical Research on Judges, Rachlinski (Cornell) & Andrew Wistrich (CA Central Dist. Ct.) report results, consistent with past their efforts, that document how, just like most humans, judges succumb to various "mental shortcuts" that can lead them to mistakes. The paper's abstract follows.

“Do judges make decisions that are truly impartial? A wide range of experimental and field studies reveal that several extra-legal factors influence judicial decision making. Demographic characteristics of judges and litigants affect judges’ decisions. Judges also rely heavily on intuitive reasoning in deciding cases, making them vulnerable to the use of mental shortcuts that can lead to mistakes. Furthermore, judges sometimes rely on facts outside the record and rule more favorably towards litigants who are more sympathetic or with whom they share demographic characteristics. On the whole, judges are excellent decision makers, and sometimes resist common errors of judgment that influence ordinary adults. The weight of the evidence, however, suggests that judges are vulnerable to systematic deviations from the ideal of judicial impartiality.”