Empirical Legal Studies

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

A New Legal Cloud Over Statistical Analyses of Judicial Decisions in France

In a move that will strike many court scholars (and, likely, others as well) as verging on the "unbelievable" (in the literal sense of the word), according to emerging news reports (e.g, here) the French Government has now "banned the publication of statistical information about judges’ decisions–with a five year prison sentence set as the maximum punishment for anyone who breaks the new law." The new french law (Art. 33 of France's Justice Reform Act) makes clear that "[t]he identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.” (translated; emphasis added)

According to news accounts, the new french law targets "legal tech companies focused on litigation prediction and analytics–from publicly revealing the pattern of judges’ behaviour in relation to court decisions." That said, the implications for empirical legal scholars seeking to model judicial decisions, certainly at the individual judge-level, in France are obvious.

While empirical legal scholars seeking to analyze U.S. judicial decisions face bothersome "hurdles" when it comes to linking judge names and their published decisions (e.g., a long-standing AO policy that strips judge names/IDs from most publicly-available AO data sets), I am unaware of anything in the U.S. that comes close to what the French Government seeks to accomplish with its new law.

Constitutional Judicial Decisions: A Comparative Frame

Empirical assessments of judicial decisions are nothing new and includes well-plowed terrain. While assessing the influences on a judge's decision remains an inherently difficult research task, the degree of difficulty increases, in general, for constitutional cases. Moreover, when one considers the constitutional context from a comparative perspective, the difficulties mount exponentially. In Quantitative Analysis of Judicial Voting, Benjamin Alarie (Toronto) and Andrew James Green (Toronto) summarize the relevant literature, identify important challenges, and make a plea for more comparative work in this area. An excerpted abstract (below) summarizes what this paper sets out to accomplish.

"Empirical research on voting in constitutional cases is so difficult because there are so many potential influences on judges and such tremendous variance across countries. Some countries have written constitutions and some do not. For those that do, the scope of the rights and freedoms varies widely. Some countries have special courts to deal with constitutional matters while others allow lower courts to hear initial applications with appeals going to a generalist high court. At an even more basic level, empirical study of constitutional voting, and on judicial decision-making more generally, is made more challenging by the fact that there is not even a clear, agreed upon theory of how judges make decisions.

Does one judge vote differently from another because they are different people, with particular political or world views and certain characteristics such their gender? Studying how a judge’s attitudes influence her voting is a long-standing line of inquiry in empirical legal studies and political science. However, not surprisingly, a judge may also care about the law itself. For example, a judge may vote differently in a case involving striking employees if there is a broadly written right to association as opposed to a specific right to strike. What if there is no written constitution at all but instead some unwritten constitutional principles? Part 3 examines some approaches to determining how such differences in constitutional law may influence judges. Part 4 turns to the institutional structure surrounding the judge such as the structure of the court, the nature of the political system, and even the norms of decision-making by the judges themselves. We examine the influence of these institutional differences before considering in Part 5 contextual factors which may shift constitutional decision-making such as war or national security concerns. Our aim is to discuss both how existing analysis has been undertaken and the limitations of such analysis. We end the paper by pointing to the need for more comparative work."

New PACER Resource From Free Law Project and GSU Researchers

With appreciation to Charlotte Alexander (Georgia State) for alerting me to this development and assisting with this post, information on a new PACER resource follows.

For researchers who study the federal courts, litigation patterns and outcomes, and judicial decisionmaking, the Free Law Project (FLP) operates the RECAP Archive, which seeks to reproduce the contents of the Public Access to Electronic Court Records (PACER) system in a single, comprehensive, searchable database (RECAP is PACER spelled backwards). The archive sets out to address major problems with PACER that make life difficult for researchers: a per-page download fee, a lack of fine-tuned search capabilities that can mine a document's text, and a clunky search interface.

The RECAP Archive draws its materials from two sources: 1) PACER users’ "donated" documents, and 2) courts’ PACER Written Opinions Report (WOR) that includes all documents that each court has designated as a judge's "written opinion."

This second stream of RECAP contributions is new. In cooperation with Professors Charlotte Alexander (Georgia State) and Javad Feizollahi (Georgia State) and with support from a U.S. Department of Labor grant, FLP just completed a comprehensive download of all material available via the WOR, and updates the download daily. All told, the RECAP Archive now contains about 3.4 million documents from 1.5 million federal district court and bankruptcy cases from 1960 to the present, including all judicial opinions available via the Written Opinions Report. The RECAP search form and more information about the Archive are here.

While this development is exciting for researchers, some important caveats are in order. First, any RECAP documents other than court orders appear in the Archive only because a PACER user donated copies of documents to the archive. Second, WOR documents are the product of each judge’s determinations of which of his/her opinions to include. Despite efforts to standardize judge decisions (e.g., here) variation across judges introduces some level of selection bias.

Caveats notwithstanding, the RECAP Archive warrants attention.

Empirically Judging Judges

Assessing judicial outcomes and judicial decisionmaking not surprisingly endure as attractive topics for researchers. Indeed, the experimental literature is festooned with judicial "vignettes" that experimentally isolate critical variables. Of course, to really learn about judicial decisionmaking the "gold standard" sample remains actual judges (though, even when submitting real judges to experiments external validity concerns persist). On this front (and in this sub-field), my Cornell colleague, Jeff Rachlinski, remains a leading scholar. In a recent paper, Judging the Judiciary by the Numbers: Empirical Research on Judges, Rachlinski (Cornell) & Andrew Wistrich (CA Central Dist. Ct.) report results, consistent with past their efforts, that document how, just like most humans, judges succumb to various "mental shortcuts" that can lead them to mistakes. The paper's abstract follows.

“Do judges make decisions that are truly impartial? A wide range of experimental and field studies reveal that several extra-legal factors influence judicial decision making. Demographic characteristics of judges and litigants affect judges’ decisions. Judges also rely heavily on intuitive reasoning in deciding cases, making them vulnerable to the use of mental shortcuts that can lead to mistakes. Furthermore, judges sometimes rely on facts outside the record and rule more favorably towards litigants who are more sympathetic or with whom they share demographic characteristics. On the whole, judges are excellent decision makers, and sometimes resist common errors of judgment that influence ordinary adults. The weight of the evidence, however, suggests that judges are vulnerable to systematic deviations from the ideal of judicial impartiality.”

Bringing Data to Theories of USSC Review

In Why Does the Supreme Court Uphold so Many Laws?, the authors, Ben Johnson (Princeton--Politics) and Keith Whittington (Princeton--Politics), empirically explore the Supreme Court's judicial review practices regarding its discretionary review authority. As the paper notes, "Scholars spend a lot of time considering the legitimacy and implications of the Supreme Court striking down federal laws by use of judicial review. Similarly, there is a large literature focusing on the Court’s power and obligation to manage the federal judiciary through its certiorari powers over its own docket and its ability to reverse lower courts. However, there is almost no work that examines the interplay of the Court’s judicial review powers and its managerial authority."

Exploiting the Judicial Review of Congress (JRC) database, this paper brings much-needed data to prevailing competing theories. The JRC database is helpful in that it "identifies cases decided by the U.S. Supreme Court that substantively review the constitutionality of provisions of federal statutes from the founding to the present. Significantly, the JRC database includes not only cases in which the Court found a federal statutory provision to be unconstitutional but also cases in which the Court upheld a statutory provision against constitutional challenge.”

What the paper finds surprises: "Our examination of these data reveal that the category of cases we call the uphold-affirm set---where the lower court upholds a statute and the Court affirms---is the plurality category among judicial review cases. This is surprising not only because this category is completely incompatible with current theories of judicial review and certiorari, but also it shows that the Court, which usually reverses lower courts, deviates from its general practice when it exercises its judicial review powers where it tends to affirm.” Given the findings, the authors explore why "the U.S. Supreme Court taken and affirmed so many decisions that upheld so many laws against constitutional challenge."

AI & Predicting USSC Case Outcomes

As legal scholars begin to digest AI's ramifications for law and legal analysis, a recent paper exploits the Supreme Court Database to illustrate AI's ability topredict case outcomes. In A general approach for predicting the behavior of the Supreme Court of the United States, Daniel Katz (Chicago-Kent) et al. explore almost two centuries of Supreme Court decisions and use an algorithm to successfully predict 70.2% of the case outcomes and 71.9% of individual justice-level votes. The abstract follows.

"Building on developments in machine learning and prior work in the science of judicial prediction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the justice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications."

SCOTUS Tenure Data

An interesting (and timely) data crunching post over at Empirical SCOTUS focuses on Justice tenure (here). As the post notes, with an eye towards assessing President Trump's potential to influence the Court's future through appointments: "If we look over time to all Justices confirmed to the Court or elevated from Associate to Chief Justice, we see that the average time between appointments has increased only minimally to around two years (the Justices are listed sequentially in the order in which they were nominated/elevated and the minimum of zero years is when two Justices were confirmed within the same year)." Notably, the "time between nominations" trend remains essentially flat despite the concurrent emergence two related trends. First, the average age at nomination has "declined about five years on average since the beginning of the 20^th century." Second, Justices last on the Court "for more years and until they are older."

Reserach Projects at HLS' A2J Lab

Under Jim Greiner's direction, work at HLS' Access to Justice Lab (previously described here) is underway. One recent courthouse project in Dane County, WI, involves an evaluation of that court's Public Safety Assessment ("PSA"). As this post notes, "Criminal process in this jurisdiction now includes additional, scientifically based information in a randomly selected subset of cases to inform pre-disposition release decisions. The judicial official–known here as a Commissioner–receives risk scores and a recommendation for release through the PSA and its static criminal history inputs to consider in reaching those decisions." (The A2J link also includes an informative video.)

Federal Judicial Center, ICPSR, Data Access, and Change

Back in the "old days," virtually immediate access to ICPSR's massive--and massively important--federal court data holdings was simple, easy, and, quite literally, involved only a few "clicks" of the mouse. A few years ago, "something happened," and that something resulted in the imposition of a formal ICPSR application requirement along with pre-IRB approvals and restricted use agreements. And all this for publicly available data. Gone were the days when researchers could simply "browse" datasets without such impediments. While ICPSR data remain "available," access now requires a comparatively dogged effort and these barriers to access have dampened overall use of ICPSR data.

But change is coming, and some of it has now arrived. One supplier of ICPSR data germane to legal scholars, the Federal Judicial Center, in conjunction with the U.S. Administrative Office, has effectively "side-stepped" ICPSR and made the FJC's Integrated Database (through 2016) directly available to users. FJC's Joe Cecil, who was instrumental in this data access project, sums it up nicely:

"The Federal Judicial Center’s Integrated Database, containing records of all federal court cases, is now available without the necessity of a restricted use agreement at www.fjc.gov/research/idb (or here). An interactive query function allows selection of subsets of civil cases based on federal circuit, district, nature of suit, disposition, origin, jurisdiction, class action designation, filing date, termination date, and docket number. Annual case filing and termination datasets through 2016 for civil, criminal defendant, appellate, and bankruptcy cases also can be downloaded in SAS format."

The FJC deserves well-earned applause from empirical legal scholars for its efforts.

SCOTUS' Treatment of 4th Amendment Claims

Despite extensive scholarly attention the Court's Fourth Amendment jurisprudence, "there has been scant empirical research about how modern Justices have voted in Fourth Amendment cases." To this end, a recent paper by Brent Newton (US Sent. Comm'n), The Supreme Court's Fourth Amendment Scorecard, provides a helpful descriptive contribution. It does so by analyzing the "173 cases from 1982 through 2015 in which the Supreme Court granted certiorari and gave plenary consideration to Fourth Amendment issues.” An excerpted abstract, below, summarizes some of the paper's main findings.

“The key findings include: (1) the overwhelming majority of Fourth Amendment litigants (FALs) before the Supreme Court were criminal defendants rather than civil rights plaintiffs… ; (2) Fourth Amendment claims succeeded in slightly less than one in four plenary-review cases, and the Court reversed the judgment of the lower courts in the vast majority of Fourth Amendment cases in which certiorari was granted; (3) FALs were much more likely to succeed in cases concerning substantive issues than remedial issues; (4) civil rights plaintiffs were slightly more likely to succeed than criminal defendants; (5) individual Justices generally voted in Fourth Amendment cases in a manner that reflected their overall ideological dispositions, yet Justices Blackmun, Scalia, and Breyer bucked this trend, resulting in more ideologically mixed voting blocs in Fourth Amendment cases than in several other divisive areas of the Court’s docket, such as abortion rights or voting rights….”

POTUS Success With SCOTUS

In The Decline of Supreme Court Deference to the President, Lee Epstein (Wash U.) and Eric Posner (Chicago), submit the prevailing wisdom that presidents enjoy unusually favorable success rates at Supreme Court to data. Obviously, both in terms of case selection as well as benefits from having the Solicitor General's office argue the president's cases, plausible reasons support the conventional wisdom. What Epstein and Posner find, however, is that the conventional wisdom may warrant an updating.

The paper exploits "an original dataset consisting of cases of concern to each of the modern-day presidents (from Franklin D. Roosevelt forward). The modern U.S. Supreme Court Database served as our foundation. This version covers all orally argued cases (including per curiam opinions) decided between the 1946 and 2015 terms. To include the Franklin Roosevelt administration, we appended cases from 1932- 1945 terms using the legacy Supreme Court Database. In total, our dataset covers 13 presidents and 84 Court terms. For all terms, we exclude per curiam decisions.”

The paper's main findings include that the "historical dominance of the president in the Supreme Court reached its apex in the Reagan administration, which won nearly 80% of the cases, and has declined steadily since then. In the Obama administration, the presidency suffered its worst win rate, barely 50%." The authors' explanations for this trend include a "growing self-assertion of the Court and the development of a specialized private Supreme Court bar. We find no evidence for two other possible explanations—that the trend is due to greater executive overreaching than in the past, or ideological disagreements between the Court and the presidency.”

SCOTUS' (Mis-)Use of Empirical Evidence

While I am certainly among those inclined to applaud the Court's greater acceptance of empirical evidence, the Justices (and/or their clerks) simply have to know what they're doing. And in the empirical realm, sometimes they clearly do not.

The most recent example of this is highlighted by a piece (here) in today's NYT by Adam Liptak. If Liptak's reporting is accurate, Justice Kennedy’s majority opinion in Smith v. Doe (2003) includes a factual assertion about "the rate of recidivism [80%] of untreated offenders." Evidently, empirical support for the Court's factual assertion relies, at bottom, on a "1986 article in Psychology Today." One consequence, as Liptak observes, is that "the basis for much of American jurisprudence and legislation about sex offenders was rooted in an offhand and unsupported statement in a mass-market magazine, not a peer-reviewed journal."

SCOTUS Cert Analytics

Insofar as the U.S. Supreme Court grants merits review to approximately 80 or so cases per Term, attention to those cases selected for review (as well as those not selected) warrant attention. And the good folks at Empirical Scotus are beginning to do just that. Indeed, a recent post "looks at whether and how the timing of certiorari (cert) petition filings affects the likelihood of a grant.  It does so looking at data based on petitions the Justices granted from the 2015 and 2016 Terms."

What they find is that the sweet-spot for most petition grants is found between 78 and 131 days with a sizable number of grants also taking place between 131 and 184 days.

As the data in this quick-and-dirty cross-section of SCOTUS petition grants relate to the Court's 2015 and 1016 Terms, those with some time and the inclination might consider building a slightly larger dataset so to exploit something of a natural experiment and compare cert granting trends for the standard nine-member Court and trends for the current eight-member Court. While presumably the traditional "Rule of Four" remains constant, the game theoretics likely change slightly for the Justices when Court membership temporarily moves from nine to eight.

President-Elect Trump's SCOTUS Nominee Pool, "Replacing" Scalia, and Measuring "Originalism"

In Searching for Justice Scalia: Measuring the "Scalia-Ness" of the Next Potential Member of the U.S. Supreme Court, Jeremy Kidd (Mercer) et al. took then-candidate Trump's promise to "appoint judges [and Justices] very much in the Justice Scalia mold," and assessed this promise against most (15/21) of those on President-Elect Trump's public "short-lists" of potential SCOTUS nominees. More specifically, the paper sets out to distill "Scalia-ness" into three vectors, develop a "Scalia Index Score," and apply the "Scalia Index Score" to 15 of Trump's possible SCOTUS nominees. While the paper, as the authors correctly note, is "not without its limitations," it nonetheless provides a glimpse into what a Trump Court might look like, at least as implied by Trump's own public statements. Obviously, the paper's analytic frame is not limited to the SCOTUS. The abstract follows.

"The President-elect promised on the campaign trail to replace Justice Scalia with a judge like Justice Scalia (and Justice Thomas). But what does it mean to be like Justice Scalia? It surely means more than just being "conservative." This study proposes three empirical measures of what made Justice Scalia Justice Scalia. First, how often does a judge promote or practice originalism? Second, how often do they cite to Justice Scalia's non-judicial writings, writings that were not about the substance of the law but about how to think about interpreting the law. And third, how often does a judge write separately, something Justice Scalia did 25.9% of the time when he was not writing the majority opinion over his last 20 years on the court. The study then applies those measures to potential nominees, and provides a metric for determining just how Scalia-like they are: the Scalia Index Score. While not without its limitations, this metric provides an objective way to evaluate how much a potential nominee is like the famous jurist they may replace."