Empirical Legal Studies

Constitutional Judicial Decisions: A Comparative Frame

Empirical assessments of judicial decisions are nothing new and includes well-plowed terrain. While assessing the influences on a judge's decision remains an inherently difficult research task, the degree of difficulty increases, in general, for constitutional cases. Moreover, when one considers the constitutional context from a comparative perspective, the difficulties mount exponentially. In Quantitative Analysis of Judicial Voting, Benjamin Alarie (Toronto) and Andrew James Green (Toronto) summarize the relevant literature, identify important challenges, and make a plea for more comparative work in this area. An excerpted abstract (below) summarizes what this paper sets out to accomplish.

"Empirical research on voting in constitutional cases is so difficult because there are so many potential influences on judges and such tremendous variance across countries. Some countries have written constitutions and some do not. For those that do, the scope of the rights and freedoms varies widely. Some countries have special courts to deal with constitutional matters while others allow lower courts to hear initial applications with appeals going to a generalist high court. At an even more basic level, empirical study of constitutional voting, and on judicial decision-making more generally, is made more challenging by the fact that there is not even a clear, agreed upon theory of how judges make decisions.

Does one judge vote differently from another because they are different people, with particular political or world views and certain characteristics such their gender? Studying how a judge’s attitudes influence her voting is a long-standing line of inquiry in empirical legal studies and political science. However, not surprisingly, a judge may also care about the law itself. For example, a judge may vote differently in a case involving striking employees if there is a broadly written right to association as opposed to a specific right to strike. What if there is no written constitution at all but instead some unwritten constitutional principles? Part 3 examines some approaches to determining how such differences in constitutional law may influence judges. Part 4 turns to the institutional structure surrounding the judge such as the structure of the court, the nature of the political system, and even the norms of decision-making by the judges themselves. We examine the influence of these institutional differences before considering in Part 5 contextual factors which may shift constitutional decision-making such as war or national security concerns. Our aim is to discuss both how existing analysis has been undertaken and the limitations of such analysis. We end the paper by pointing to the need for more comparative work."

New PACER Resource From Free Law Project and GSU Researchers

With appreciation to Charlotte Alexander (Georgia State) for alerting me to this development and assisting with this post, information on a new PACER resource follows.

For researchers who study the federal courts, litigation patterns and outcomes, and judicial decisionmaking, the Free Law Project (FLP) operates the RECAP Archive, which seeks to reproduce the contents of the Public Access to Electronic Court Records (PACER) system in a single, comprehensive, searchable database (RECAP is PACER spelled backwards). The archive sets out to address major problems with PACER that make life difficult for researchers: a per-page download fee, a lack of fine-tuned search capabilities that can mine a document's text, and a clunky search interface.

The RECAP Archive draws its materials from two sources: 1) PACER users’ "donated" documents, and 2) courts’ PACER Written Opinions Report (WOR) that includes all documents that each court has designated as a judge's "written opinion."

This second stream of RECAP contributions is new. In cooperation with Professors Charlotte Alexander (Georgia State) and Javad Feizollahi (Georgia State) and with support from a U.S. Department of Labor grant, FLP just completed a comprehensive download of all material available via the WOR, and updates the download daily. All told, the RECAP Archive now contains about 3.4 million documents from 1.5 million federal district court and bankruptcy cases from 1960 to the present, including all judicial opinions available via the Written Opinions Report. The RECAP search form and more information about the Archive are here.

While this development is exciting for researchers, some important caveats are in order. First, any RECAP documents other than court orders appear in the Archive only because a PACER user donated copies of documents to the archive. Second, WOR documents are the product of each judge’s determinations of which of his/her opinions to include. Despite efforts to standardize judge decisions (e.g., here) variation across judges introduces some level of selection bias.

Caveats notwithstanding, the RECAP Archive warrants attention.

Empirically Judging Judges

Assessing judicial outcomes and judicial decisionmaking not surprisingly endure as attractive topics for researchers. Indeed, the experimental literature is festooned with judicial "vignettes" that experimentally isolate critical variables. Of course, to really learn about judicial decisionmaking the "gold standard" sample remains actual judges (though, even when submitting real judges to experiments external validity concerns persist). On this front (and in this sub-field), my Cornell colleague, Jeff Rachlinski, remains a leading scholar. In a recent paper, Judging the Judiciary by the Numbers: Empirical Research on Judges, Rachlinski (Cornell) & Andrew Wistrich (CA Central Dist. Ct.) report results, consistent with past their efforts, that document how, just like most humans, judges succumb to various "mental shortcuts" that can lead them to mistakes. The paper's abstract follows.

“Do judges make decisions that are truly impartial? A wide range of experimental and field studies reveal that several extra-legal factors influence judicial decision making. Demographic characteristics of judges and litigants affect judges’ decisions. Judges also rely heavily on intuitive reasoning in deciding cases, making them vulnerable to the use of mental shortcuts that can lead to mistakes. Furthermore, judges sometimes rely on facts outside the record and rule more favorably towards litigants who are more sympathetic or with whom they share demographic characteristics. On the whole, judges are excellent decision makers, and sometimes resist common errors of judgment that influence ordinary adults. The weight of the evidence, however, suggests that judges are vulnerable to systematic deviations from the ideal of judicial impartiality.”

Bringing Data to Theories of USSC Review

In Why Does the Supreme Court Uphold so Many Laws?, the authors, Ben Johnson (Princeton--Politics) and Keith Whittington (Princeton--Politics), empirically explore the Supreme Court's judicial review practices regarding its discretionary review authority. As the paper notes, "Scholars spend a lot of time considering the legitimacy and implications of the Supreme Court striking down federal laws by use of judicial review. Similarly, there is a large literature focusing on the Court’s power and obligation to manage the federal judiciary through its certiorari powers over its own docket and its ability to reverse lower courts. However, there is almost no work that examines the interplay of the Court’s judicial review powers and its managerial authority."

Exploiting the Judicial Review of Congress (JRC) database, this paper brings much-needed data to prevailing competing theories. The JRC database is helpful in that it "identifies cases decided by the U.S. Supreme Court that substantively review the constitutionality of provisions of federal statutes from the founding to the present. Significantly, the JRC database includes not only cases in which the Court found a federal statutory provision to be unconstitutional but also cases in which the Court upheld a statutory provision against constitutional challenge.”

What the paper finds surprises: "Our examination of these data reveal that the category of cases we call the uphold-affirm set---where the lower court upholds a statute and the Court affirms---is the plurality category among judicial review cases. This is surprising not only because this category is completely incompatible with current theories of judicial review and certiorari, but also it shows that the Court, which usually reverses lower courts, deviates from its general practice when it exercises its judicial review powers where it tends to affirm.” Given the findings, the authors explore why "the U.S. Supreme Court taken and affirmed so many decisions that upheld so many laws against constitutional challenge."

AI & Predicting USSC Case Outcomes

As legal scholars begin to digest AI's ramifications for law and legal analysis, a recent paper exploits the Supreme Court Database to illustrate AI's ability topredict case outcomes. In A general approach for predicting the behavior of the Supreme Court of the United States, Daniel Katz (Chicago-Kent) et al. explore almost two centuries of Supreme Court decisions and use an algorithm to successfully predict 70.2% of the case outcomes and 71.9% of individual justice-level votes. The abstract follows.

"Building on developments in machine learning and prior work in the science of judicial prediction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the justice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications."

SCOTUS Tenure Data

An interesting (and timely) data crunching post over at Empirical SCOTUS focuses on Justice tenure (here). As the post notes, with an eye towards assessing President Trump's potential to influence the Court's future through appointments: "If we look over time to all Justices confirmed to the Court or elevated from Associate to Chief Justice, we see that the average time between appointments has increased only minimally to around two years (the Justices are listed sequentially in the order in which they were nominated/elevated and the minimum of zero years is when two Justices were confirmed within the same year)." Notably, the "time between nominations" trend remains essentially flat despite the concurrent emergence two related trends. First, the average age at nomination has "declined about five years on average since the beginning of the 20^th century." Second, Justices last on the Court "for more years and until they are older."

Reserach Projects at HLS' A2J Lab

Under Jim Greiner's direction, work at HLS' Access to Justice Lab (previously described here) is underway. One recent courthouse project in Dane County, WI, involves an evaluation of that court's Public Safety Assessment ("PSA"). As this post notes, "Criminal process in this jurisdiction now includes additional, scientifically based information in a randomly selected subset of cases to inform pre-disposition release decisions. The judicial official–known here as a Commissioner–receives risk scores and a recommendation for release through the PSA and its static criminal history inputs to consider in reaching those decisions." (The A2J link also includes an informative video.)

Federal Judicial Center, ICPSR, Data Access, and Change

Back in the "old days," virtually immediate access to ICPSR's massive--and massively important--federal court data holdings was simple, easy, and, quite literally, involved only a few "clicks" of the mouse. A few years ago, "something happened," and that something resulted in the imposition of a formal ICPSR application requirement along with pre-IRB approvals and restricted use agreements. And all this for publicly available data. Gone were the days when researchers could simply "browse" datasets without such impediments. While ICPSR data remain "available," access now requires a comparatively dogged effort and these barriers to access have dampened overall use of ICPSR data.

But change is coming, and some of it has now arrived. One supplier of ICPSR data germane to legal scholars, the Federal Judicial Center, in conjunction with the U.S. Administrative Office, has effectively "side-stepped" ICPSR and made the FJC's Integrated Database (through 2016) directly available to users. FJC's Joe Cecil, who was instrumental in this data access project, sums it up nicely:

"The Federal Judicial Center’s Integrated Database, containing records of all federal court cases, is now available without the necessity of a restricted use agreement at www.fjc.gov/research/idb (or here). An interactive query function allows selection of subsets of civil cases based on federal circuit, district, nature of suit, disposition, origin, jurisdiction, class action designation, filing date, termination date, and docket number. Annual case filing and termination datasets through 2016 for civil, criminal defendant, appellate, and bankruptcy cases also can be downloaded in SAS format."

The FJC deserves well-earned applause from empirical legal scholars for its efforts.

SCOTUS' Treatment of 4th Amendment Claims

Despite extensive scholarly attention the Court's Fourth Amendment jurisprudence, "there has been scant empirical research about how modern Justices have voted in Fourth Amendment cases." To this end, a recent paper by Brent Newton (US Sent. Comm'n), The Supreme Court's Fourth Amendment Scorecard, provides a helpful descriptive contribution. It does so by analyzing the "173 cases from 1982 through 2015 in which the Supreme Court granted certiorari and gave plenary consideration to Fourth Amendment issues.” An excerpted abstract, below, summarizes some of the paper's main findings.

“The key findings include: (1) the overwhelming majority of Fourth Amendment litigants (FALs) before the Supreme Court were criminal defendants rather than civil rights plaintiffs… ; (2) Fourth Amendment claims succeeded in slightly less than one in four plenary-review cases, and the Court reversed the judgment of the lower courts in the vast majority of Fourth Amendment cases in which certiorari was granted; (3) FALs were much more likely to succeed in cases concerning substantive issues than remedial issues; (4) civil rights plaintiffs were slightly more likely to succeed than criminal defendants; (5) individual Justices generally voted in Fourth Amendment cases in a manner that reflected their overall ideological dispositions, yet Justices Blackmun, Scalia, and Breyer bucked this trend, resulting in more ideologically mixed voting blocs in Fourth Amendment cases than in several other divisive areas of the Court’s docket, such as abortion rights or voting rights….”

POTUS Success With SCOTUS

In The Decline of Supreme Court Deference to the President, Lee Epstein (Wash U.) and Eric Posner (Chicago), submit the prevailing wisdom that presidents enjoy unusually favorable success rates at Supreme Court to data. Obviously, both in terms of case selection as well as benefits from having the Solicitor General's office argue the president's cases, plausible reasons support the conventional wisdom. What Epstein and Posner find, however, is that the conventional wisdom may warrant an updating.

The paper exploits "an original dataset consisting of cases of concern to each of the modern-day presidents (from Franklin D. Roosevelt forward). The modern U.S. Supreme Court Database served as our foundation. This version covers all orally argued cases (including per curiam opinions) decided between the 1946 and 2015 terms. To include the Franklin Roosevelt administration, we appended cases from 1932- 1945 terms using the legacy Supreme Court Database. In total, our dataset covers 13 presidents and 84 Court terms. For all terms, we exclude per curiam decisions.”

The paper's main findings include that the "historical dominance of the president in the Supreme Court reached its apex in the Reagan administration, which won nearly 80% of the cases, and has declined steadily since then. In the Obama administration, the presidency suffered its worst win rate, barely 50%." The authors' explanations for this trend include a "growing self-assertion of the Court and the development of a specialized private Supreme Court bar. We find no evidence for two other possible explanations—that the trend is due to greater executive overreaching than in the past, or ideological disagreements between the Court and the presidency.”

SCOTUS' (Mis-)Use of Empirical Evidence

While I am certainly among those inclined to applaud the Court's greater acceptance of empirical evidence, the Justices (and/or their clerks) simply have to know what they're doing. And in the empirical realm, sometimes they clearly do not.

The most recent example of this is highlighted by a piece (here) in today's NYT by Adam Liptak. If Liptak's reporting is accurate, Justice Kennedy’s majority opinion in Smith v. Doe (2003) includes a factual assertion about "the rate of recidivism [80%] of untreated offenders." Evidently, empirical support for the Court's factual assertion relies, at bottom, on a "1986 article in Psychology Today." One consequence, as Liptak observes, is that "the basis for much of American jurisprudence and legislation about sex offenders was rooted in an offhand and unsupported statement in a mass-market magazine, not a peer-reviewed journal."

SCOTUS Cert Analytics

Insofar as the U.S. Supreme Court grants merits review to approximately 80 or so cases per Term, attention to those cases selected for review (as well as those not selected) warrant attention. And the good folks at Empirical Scotus are beginning to do just that. Indeed, a recent post "looks at whether and how the timing of certiorari (cert) petition filings affects the likelihood of a grant.  It does so looking at data based on petitions the Justices granted from the 2015 and 2016 Terms."

What they find is that the sweet-spot for most petition grants is found between 78 and 131 days with a sizable number of grants also taking place between 131 and 184 days.

As the data in this quick-and-dirty cross-section of SCOTUS petition grants relate to the Court's 2015 and 1016 Terms, those with some time and the inclination might consider building a slightly larger dataset so to exploit something of a natural experiment and compare cert granting trends for the standard nine-member Court and trends for the current eight-member Court. While presumably the traditional "Rule of Four" remains constant, the game theoretics likely change slightly for the Justices when Court membership temporarily moves from nine to eight.

President-Elect Trump's SCOTUS Nominee Pool, "Replacing" Scalia, and Measuring "Originalism"

In Searching for Justice Scalia: Measuring the "Scalia-Ness" of the Next Potential Member of the U.S. Supreme Court, Jeremy Kidd (Mercer) et al. took then-candidate Trump's promise to "appoint judges [and Justices] very much in the Justice Scalia mold," and assessed this promise against most (15/21) of those on President-Elect Trump's public "short-lists" of potential SCOTUS nominees. More specifically, the paper sets out to distill "Scalia-ness" into three vectors, develop a "Scalia Index Score," and apply the "Scalia Index Score" to 15 of Trump's possible SCOTUS nominees. While the paper, as the authors correctly note, is "not without its limitations," it nonetheless provides a glimpse into what a Trump Court might look like, at least as implied by Trump's own public statements. Obviously, the paper's analytic frame is not limited to the SCOTUS. The abstract follows.

"The President-elect promised on the campaign trail to replace Justice Scalia with a judge like Justice Scalia (and Justice Thomas). But what does it mean to be like Justice Scalia? It surely means more than just being "conservative." This study proposes three empirical measures of what made Justice Scalia Justice Scalia. First, how often does a judge promote or practice originalism? Second, how often do they cite to Justice Scalia's non-judicial writings, writings that were not about the substance of the law but about how to think about interpreting the law. And third, how often does a judge write separately, something Justice Scalia did 25.9% of the time when he was not writing the majority opinion over his last 20 years on the court. The study then applies those measures to potential nominees, and provides a metric for determining just how Scalia-like they are: the Scalia Index Score. While not without its limitations, this metric provides an objective way to evaluate how much a potential nominee is like the famous jurist they may replace."

The (Don) Songer Project--Data Access

In a fitting tribute to a leading appeals court scholar, the good folks at the Univ. of South Carolina have reorganized (and renamed) a helpful website in honor of their colleague, Don Songer. "The primary mission of the The Songer Project is to provide a comprehensive access point to the most recent and cutting-edge research on law and judicial politics. At this website, individuals interested in law and judicial politics can download electronic datasets of court cases, obtain smaller datasets or measures of judicially relevant phenomena, read various working papers on important topics, and link to other websites containing law and judicial politics information."

Well worth a visit, particularly for those seeking USCA datasets (click here).

District-Level Variation in Social Security Claims' Litigation

Social security claimants' experiences appealing claims can vary considerably across U.S. district courts. Drawing from a dataset of more than 20,000 appeals, Jonah B. Gelbach (Penn.) and David Marcus (Ariz.), in A Study of Social Security Disability Litigation in the Federal Courts, find, in part, that: "District judges in Brooklyn decide these cases pursuant to one set of procedural rules and have in recent years remanded about seventy percent to the agency. Magistrate judges in Little Rock handle this docket with a different set of rules and have in recent years remanded only twenty percent." An excerpted abstract follows.

"Our report makes four contributions. The first is a thorough introduction to the process by which a disability benefits claim proceeds from initial filing to a federal judge’s chambers. This description is intended to deepen understandings of where many of federal civil cases come from, and why they raise the same sorts of concerns repeatedly.

Second, the report provides some context for understanding why the federal courts remand claims to the agency at the rate that they do. We argue that the federal courts and the agency have different institutional goals, commitments, and resources. These differences would cause a sizable number of remands even if the agency adjudicated claims successfully and the federal courts applied the appropriate standard of review. Third, we undertake extensive statistical analysis to try to understand what factors explain the sharp variation in district-level remand rates. Circuit boundaries account for some, but not all, of this disparity. After excluding a number of other potential causes, we hypothesize that district courts remand claims to the agency at different rates in part because uneven adjudication within the agency produces pools of appeals of differing quality. Finally, the report analyzes contrasting procedural rules used by different districts to govern social security litigation. We argue that these differences are unnecessary and create needless inefficiencies. We conclude with a set of recommendations to improve social security litigation within the federal courts."