Empirical Legal Studies

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Do Patent Reforms Enhance Innovation?

Over the past two decades, academics and policy makers in the IP/patent space have focused on three specific developments: elimination of presumptive injunctive relief for victorious patent enforcers in eBay Inc., creation of the Patent Trial and Appeal Board (PTAB), and restriction of software’s eligibility for patent protection in the Alice Corp. decision. Despite general agreement on the importance of these three patent system changes, their individual and collective impact on innovation (construed in terms of firms’ patent filings and R&D investments) remains both contested and understudied.

A recent paper by Christian Helmers (Santa Clara—business) and Brian Love (Santa Clara), Patent Law Reform and Innovation: An Empirical Assessment of the Last 20 Years, jumps into this empirical void and seeks to assess these three reforms’ impact on innovation in the U.S. To do so, the paper constructs firm-level innovation measures and of exposure to each reform and submits them to various Dif-in-Dif specifications that compare relatively exposed and relatively unexposed firms. The paper finds that “Overall, our results suggest that these three reforms had a positive effect on innovation as measured by patent filings and R&D investment.” More detailed findings are described in the excerpted abstract.

“... We find: a positive association between eBay and R&D spending by firms that were relatively more exposed to patent litigation prior to the Court’s decision; a positive association between the introduction of PTAB proceedings and both R&D expenditures and patent filings by firms that innovate in tech classes where PTAB has been most active; and a positive association between Alice and both R&D spending by software firms and patenting by firms that held relatively more software patents prior to the Court’s opinion.”

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Federal Judicial Data: Greater Transparency or Opacity?

While not itself an empirical paper per se, The Necessary and Proper Stewardship of Judicial Data, by Zachary Clopton (Northwestern) and Aziz Huq (Chicago), engages in a discussion near-and-dear to most legal empiricists' heart. Specifically, their paper sets out to develop "the first comprehensive descriptive, doctrinal, and normative analysis of judicial data.”

As the authors note, judicial (especially Art. III courts) data are "valuable in governance and for those interested in understanding how they are being governed. In the absence of hurdles or walls, it can illuminate the operation of one branch of the national government. Constitutional lapses can be brought to light through retail or wholesale data." As such, data flowing from Art. III courts and institutions can promote various public goods by, in part, advancing a better understanding of, or improvements to, judicial institutions.

One critical anomaly is that, as compared to the legislative and Executive branches, judicial branch data remain "largely terra incognito for scholars and the general public alike. Instead, public attention rushes to high-profile cases of leaked Supreme Court opinions and private influence campaigns.” While why this is so remains unclear (to me, anyway), that it is so, however, is obvious to any empirical legal scholar who labors and writes in this space. And the stakes incident to the "transparency v. opacity" debate grow more acute as new scholarly tools emerge (e.g., natural language processessing, text-scrapping).

As the authors make clear, under current policy and practice judicial data "might be lost, dissipating into air, or kept under wraps, never yielding up their secrets." In contrast, a commitment to greater transparency invites characterizing judicial data as a "new public asset that can then be tapped to create public goods for the benefit of all."

School and Crime; Cause and Effect

Criminologists increasingly worry about a "school-to-prison pipeline" problem. Such worries are typically inflamed incident to public school districts' typical reaction (or, to some, over-reaction) to tragic episodes of student violence, especially mass-violence. An increasing number of empirical efforts seek to identify and better understand causal mechanisms. To this end, a recent paper by Todd Jones (Miss. St.-econ) and Ezra Karger (Fed. Res.-Chicago), School and Crime, exploits data from the National Crime Victimization Survey and explores "seasonality" patterns.

As it relates to "seasonality" patterns, the paper's main take-away is that juvenile criminal activity in general, and student-on-student crime in particular, largely maps onto the school calendar. More specifically, the paper documents a population-level outcome: across all children—those who are compliers for education-based policies and those who are not—attending school causes an average increase in the likelihood of student-on-student crime. While the overall patter is suggestive, the underlying mechanisms are less so. That is, whether a school's institutional setting itself (e.g., institutional-level policies and practices)--or, for example, that student-on-student crime is simply a function of student density levels--is the critical mechanism remains unclear. The paper's abstract follows.

“Criminal activity is seasonal, peaking in the summer and declining through the winter. We provide the first evidence that arrests of children and reported crimes involving children follow a different pattern: peaking during the school year and declining in the summer. We use a regression discontinuity design surrounding the exact start and end dates of the school year to show that this pattern is caused by school: children aged 10–17 are roughly 50% more likely to be involved in a reported crime during the beginning of the school year relative to the weeks before school begins. This sharp increase is driven by student-on-student crimes occurring in school and during school hours. We use the timing of these patterns and a seasonal adjustment to argue that school increases reported crime rates (and arrests) involving 10–17-year-old offenders by 47% (41%) annually relative to a counterfactual where crime rates follow typical seasonal patterns. School exacerbates preexisting sex-based and race-based inequality in reported crime and arrest rates, increasing both the Black-white and male-female gap in reported juvenile crime and arrest rates by more than 40%.”

Legal Precedent and Decay

While the idea of legal precedent implies some degree of durability, precedents are not immortal. Over time, they lose their capacity to influence action; eventually they cease to be of anything other than historical interest. Although precedential decay's existence is generally accepted, the determinants of precedential decay are comparatively under-studied.

Yun-chien Chang (Cornell) & Geoffrey Miller’s (NYU) recent paper, Decay of Precedent in State Supreme Courts, extends this literature by analyzing citation decay patterns in state supreme court opinions, drawing on three different (and large) data bases. Their paper examines both “backward-looking” precedential decay—the frequency with which opinions in a given year cite to opinions from prior years as well as “forward-looking” decay—the frequency with which specific precedents are cited in later opinions. One key finding includes the extraordinary persistence of exponential decay across all of the data. The paper’s abstract follows.

“This paper investigates the decay of precedents in three unique databases: citations in all published opinions by state supreme courts in 2019 (102,555 citations); citations in all published opinions by state supreme courts in 2003 (114,777 citations); and citations by all state and federal courts to state supreme court opinions issued in 2003 (259,627 citations). We find that the frequency of citation drops off by a roughly constant ratio with each passing year. The large scale of our yearly data and the time studied (1904–2019) suggests that this pattern must be the result of underlying forces not having to do with particular circumstances of time or place. We also find that cases with dissents tend to cite older cases and longer decisions tend to be cited more often and faster. Criminal law cases, as compared to constitutional law cases, tend to cite more recent precedents, while trust and estate cases tend to cite older precedents. We offer some conjectures about possible reasons for the observed differences in decay rates.”

How to Value Data

While a recent paper, Data Valuation and Law, is not itself an "empirical paper," it is certainly "ELS-adjacent" and, as such, I thought it might interest some. This is particularly so as both AI and "big data" become increasingly important.

Motivating Jordan Berry (USC) and D. Daniel Sokol's (USC) paper is the observation that "while data has become increasingly valuable and important, the law’s attempts to value data have lagged, remaining confused and underdeveloped." To improve matters in this regard, the paper advances a “real options” approach for framing data valuation not because the approach is perfect but, rather, because “it is the least bad alternative available.” The paper's abstract follows.

"Data has become an increasingly valuable asset. Numerous areas of law—including contracts, corporate law, IP, antitrust, tax, privacy, and bankruptcy—require parties and courts to determine the value of assets, including data. Unfortunately, data valuation has been hindered by a lack of clarity over what data is and why it is valuable. This lack of clarity also increases the chances of legal decisionmakers valuing data in inconsistent ways, which would create further confusion, inefficiencies, and opportunities for regulatory arbitrage.

This article proposes a unified framework for valuing data that will promote consistent valuations across fields of law. It begins by conceptualizing data as building blocks: It is of little value on its own. But when placed in skillful and creative hands, it can unlock choices for its holders—choices they would not otherwise have—that can generate tremendous profits. Thus, data constitutes what is known as a ‘real option.’ This article shows how using real options to value data can significantly improve upon existing data valuation practices.”

SCOTUS Nominations, By the Nimbers and Over Time

Charles Cameron (Princeton--politics ) & Jonathan Kastellec (Princeton--politics) recently published a book, Making the Supreme Court (OUP), that tracks 90 years of POTUS appointments to the Supreme Court. The Book's description (below) makes clear what the authors attempt to achieve with their book. The authors' execution benefits from data, including on:

• Party and presidential interest in the court, including the use of “litmus tests” for Supreme Court nominees;
• The type of selection processes employed by the president;
• The background characteristics of Supreme Court nominees, and those who made the president’s “short list”;
• Interest group participation in Supreme Court confirmation battles;
• Media coverage of Supreme Court nominees;
• Public opinion of nominees;
• Senate roll call voting on Supreme Court nominees; and
• Simulations on the future ideological trajectory of the Supreme Court.

But what particularly caught my eye was that the authors make available the data (and related coding) that they used in their analyses (click here).

“Appointments to the United States Supreme Court are now central events in American political life. Every vacancy unleashes a bitter struggle between Republicans and Democrats over nominees; and once the seat is filled, new justices typically vote in predictable ways. Making the Supreme Court examines 90 years of American political history to show how the growth of federal judicial power from the 1930s onward inspired presidents, the political parties, and interest groups to shape judicial policy through appointments. The result is a new politics aimed squarely at selecting and placing judicial ideologues on the Court, which has transformed the Court into an ideologically driven and polarized branch. Based on rich data and qualitative evidence, Making the Supreme Court provides a sharp lens on the social and political transformations that created a new American politics.”

Levering AI in the Service of Contract Interpretation

Contract interpretation canons and theories continue to evolve. The latest foray seeks to lever AI.

In a recent article, Generative Interpretation, the authors, David Hoffman (Penn) and Yonathan Arbel (Alabama), advance a new approach designed to assist in determining contracting parties’ meaning. The authors' impulse behind generative interpretation, is, according to them, "simple:" they apply large language models (LLMs) to contractual texts and extrinsic evidence to predict what the parties would have said at contracting about what they meant. The paper's abstract follows.

“We introduce generative interpretation, a new approach to estimating contractual meaning using large language models. As AI triumphalism is the order of the day, we proceed by way of grounded case studies, each illustrating the capabilities of these novel tools in distinct ways. Taking well-known contracts opinions, and sourcing the actual agreements that they adjudicated, we show that AI models can help factfinders ascertain ordinary meaning in context, quantify ambiguity, and fill gaps in parties’ agreements. We also illustrate how models can calculate the probative value of individual pieces of extrinsic evidence.

After offering best practices for the use of these models given their limitations, we consider their implications for judicial practice and contract theory. Using LLMs permits courts to estimate what the parties intended cheaply and accurately, and as such generative interpretation unsettles the current interpretative stalemate. Their use responds to efficiency-minded textualists and justice-oriented contextualists, who argue about whether parties will prefer cost and certainty or accuracy and fairness. Parties—and courts—would prefer a middle path, in which adjudicators strive to predict what the contract really meant, admitting just enough context to approximate reality while avoiding unguided and biased assimilation of evidence. As generative interpretation offers this possibility, we argue it can become the new workhorse of contractual interpretation.”

Lower Fedreal Courts and Religious Claims: An Update

The Supreme Court's ongoing readjustment of long-standing Free Exercise and Establishment clause tensions has renewed scholarly attention to courts' resolutions of religious claims. And in a recent paper, Trump's Lower-Court Judges and Religion: An Initial Appraisal, Stephen Choi (NYU) et al. contribute to a growing empirical literature by levering data on Trump-appointed Circuit Court judges in Free Exercise Clause cases and comparing Trump-appointed judge outcomes with those of Non-Trump Republican- and Democratic-appointed judges.

What the authors find includes that "Trump’s judicial appointments are more religious but no less qualified than other judges," and that "Trump judges are more likely to vote in favor of the plaintiff than not only Democratic judges, but also judges appointed by other Republican presidents."

Notably, however, while Trump-appointed judges vote “pro-religion” (that is, in favor of a religious plaintiff asserting a free exercise claim) more frequently than other judges, this tilt distributes unevenly across various religions. For example, and as the table below illustrates, the authors find “... different voting patterns for Islam. For cases involving Islam, Trump judges vote in favor of Muslim plaintiffs in only 19.2% of the cases compared with 34.5% for Non-Trump Republicans and 48.0% for Democrats.” The authors note that the overall pattern that emerges is “consistent with conventional wisdom: Democrats tend to protect minority religions, and Republicans tend to protect Christianity (and possibly Judaism).”

 

 

The paper's abstract follows.

“It is widely believed that President Donald Trump’s judicial appointments reflected a strategy of appeasing evangelical Christians and other religious groups that favored a more conservative, Christian judiciary, and that in pursuing this strategy Trump sacrificed quality. We explore evidence for this theory by examining the biographies and credentials of Trump’s lower court nominees and the voting records of his judicial appointments to the circuit courts in free exercise cases. We find that Trump’s appointments to the lower courts have stronger or more numerous religious affiliations, and are affiliated with the Federalist Society and the National Rifle Association at unusually high rates, but are no less well credentialed than other judges are. We also find that Trump’s appointments to the circuit court more frequently vote in favor of Christian plaintiffs, and less frequently vote in favor of Muslim plaintiffs, in free exercise cases than judges appointed by other Republican presidents and by Democratic presidents.”

Data on Local Prosecutor Campaign Contributions & Electoral Outcomes

Building on prior descriptive work (here) that focuses on local prosecutor elections in the U.S., in a recent paper, Donating to the District Attorney, Carissa Hessick (UNC) et al., broaden their focus to assess how district attorney campaign contributions map onto electoral success. To inform the authors' interest in "the local politics of mass incarceration," the paper exploits a unique data set that links contribution and election results data. Merging these two data sets allows one to know whether a given recipient [local prosecutor candidate] was an "incumbent or challenger, the recipient’s vote share in both the primary and general election, and whether the election was contested in the primary election, the general election, or either election."

Such data provide a "more nuanced picture of prosecutorial politics, allowing us to make important distinctions between offices and interest groups, suggest new accountability measures, and better evaluate how elections can lead to reform." What the authors find propels them to conclude that "accountability mechanisms beyond elections are necessary to ensure prosecutors do not exercise their significant discretionary power unchecked." The paper's excerpted abstract follows.

“… The study offers a more complete empirical account of prosecutor accountability by analyzing contributions to local candidates as well as their election results. It details the amount of money in local prosecutor elections, including from interest groups, and the relationship between candidate fundraising and success. The stark differences across the country underscore that the more than two thousand local prosecutors are not a monolith; some offices are best understood as political, with contested elections and significant amounts of campaigning, while most appear more bureaucratic...”

Judicial Decisionmaking: Moving Beyond Case Outcomes

While conceding that "judges matter" as a general matter, a recent paper, Do Judges Matter?, considers a slightly narrower question: what about judges matters when it comes to the more mundane aspects incident to the business of judging?" More specifically, the authors, Yun-chien Chang (Cornell) and Geoffrey Miller (NYU), explore potential relationships "between a judge’s personal characteristics and the day-to-day business of courts – the writing of opinions, the use of citations, the frequency of dissent, and like factors."

Exploiting a hand-collected data set of 4,591 substantive decisions issued by 48 state supreme courts in 2003 (excluding Alaska and Hawaii), the paper finds that while state supreme courts themselves vary in how they carry out their judicial business (e.g., citation distributions, frequency of dissents), personal characteristics of judges, by contrast, have only a "negligible impact." The abstract follows.

"An extensive literature examines whether characteristics of judges correlate with votes on cases. These studies generally consider the judges’ votes on the merits of cases. Examining a data set of 4,591 decisions issued by 48 state supreme courts in 2003, we consider whether judges’ personal features affect their opinion writing. We find virtually no significant differences along any of the dimensions under review. Judicial characteristics matter only indirectly — ideological differences increase dissensus in public-law cases, which increases opinion length and citation numbers. Our study suggests that when carrying out the ordinary business of courts, judges are almost all the same."

Data on the Fourth Amendment's "Good Faaith" Exception's Application

While the Fourth Amendment is designed to protect criminal defendants against prosecutors' use of unlawfully obtained evidence, an important exception exists for evidence obtained by law enforcement officers who rely in good faith on existing legal authority. Thus, if police rely on a statute that turns out to be unconstitutional or invalid warrants or precedent, the evidence obtained is nonetheless admissible under the good faith exception.

Empirical analyses on how the good faith exception works "on the ground," however, are scant. To this end, a recent paper, The Reality of the Good Faith Exception, by Matthew Tokson (Utah) and Michael Gentithes (Akron), helps to fill a void.

Their paper draws from available published (and unpublished) judicial decisions (N=5,531) that mention the Fourth Amendment or "motion to suppress" across three separate months (July 2015, 2018, and 2021). Among the paper's findings includes a discernible uptick in the exceptions salience over time (see Figure 1, below).

Figure 1: Good Faith Exception (GFE) Prevalence Over Time - Proportion of Fourth Amendment Suppression Cases Involving the Good Faith Exception

The authors are correctly mindful that selection effects certainly influence their descriptive findings. The authors argue that selection effects in this context likely lead, on balance, to an under-counting of suppression hearings to the extent that "defendants and their attorneys likely decline to file suppression motions because of the good faith exception." (emphasis added)

Overall, the paper argues that "the [good faith] exception, originally confined to narrow circumstances, is turning into a protection for nearly any investigatory activity with an arguable connection to an existing precedent or statute" and that "[c]ourts rely on the exception with remarkable frequency, and a large proportion of courts invoking the exception avoid any substantive constitutional ruling, stunting the development of Fourth Amendment doctrine."

Lay Understandings of Civil Settlements

When it comes to resolving formal civil disputes, popular imagination, partly fueled by the media and entertainment industry, is dominated by "courtroom drama" and jury trials. Empirical work over the decades, however, paints quite a different picture, and one dominated by the "disappearing trial." Thus, as Jennifer Robbennolt (Illinois) et al. make clear in Settlement Schemas: How Laypeople Understand Civil Settlement, "for all the attention paid to the real and imagined drama of the courtroom, these depictions overlook the reality of how many disputes are resolved: through any means but trial."

In the service of seeking a better understanding of how the public understands civil settlements, the authors surveyed a representative sample of more than 1,000 U.S. adults. Key findings include that while the public conveyed a widespread general understanding about civil settlements, evidence of misconceptions about the details, including the role of a jury and settlement scope, also emerged. The paper's abstract follows.

"What does the public think it means to ‘settle’ a civil case? Most legal disputes in the United States end in an agreement to settle, but little is known about what laypeople think about settlement. To fill this gap, we took a direct approach: we asked a nationally representative sample of U.S. adults – more than one thousand of them – basic questions about settlement. We found widespread understanding about the essential nature and frequency of settlement, but persistent, though not universal, misconceptions about the details, including the role of a jury and settlement scope. Because settlement is such a pervasive part of the U.S. legal system, the system’s legitimacy turns in part on how the public understands and views civil settlement. The survey reported here provides a foundational study of the understandings and framework – the schemas – that the public bring to settlement."

Sentencing as a Product of Plea Agreements

When it comes to explaining criminal sentences, conventional wisdom emphasizes either judges or, given that criminal convictions largely flow from negotiated plea agreements, prosecutors. A closer look, however, reveals far more nuance and complexity and a dynamic process involving multiple parties concurrently rather than a static event dominated by one actor.

In a recent "closer look," Jeffrey Bellin (William & Mary) & Jenia Turner (SMU) explore 77 plea bargains in the first half of 2019. Noting that comprehensive empirical work on judicial responses to negotiated plea deals is sparse, In Sentencing in an Era of Plea Bargains makes a few helpful contributions. First, the authors' deep dive into 77 pleas bargains, though descriptive, revels some initial, general contours of these agreements. Notably, while judges ultimately approved all of the 77 plea bargains in the paper's sample, judges departed from the negotiated sentencing recommendation in about one-third of the cases. Also notable is that in virtually all instances where the judge departed from the negotiated sentence, she did so in a direction that favored greater leniency. Second, in separate analyses the authors examine 31 plea agreements that elicited judicial rejections. Somewhat perplexingly, judges rejected most plea agreements (74%) because they felt the recommended sentence was too lenient.

Obviously, descriptive studies that lack statistical power are limited in important ways. At the same time, however, such studies can help by pointing future researchers in potential helpful directions. The paper's (excerpted) abstract follows.

“The literature offers inconsistent answers to a question that is foundational to criminal law: Who imposes sentences? Traditional narratives place sentencing responsibility in the hands of the judge. Yet, in a country where 95 percent of criminal convictions come from guilty pleas (not trials), modern American scholars center prosecutors – who control plea terms – as the decider of punishment. This Article highlights and seeks to resolve the tension between these conflicting narratives by charting the pathways by which sentences are determined in a system dominated by plea bargains.

After reviewing the empirical literature on sentence variation, state and federal plea-bargaining rules and doctrines, and conducting some empirical analysis of our own, we conclude that neither of the competing narratives is correct. Sentencing in the United States has become a dynamic process with substantial contributions from multiple actors, not a static event controlled by any single actor.”