Empirical Legal Studies

Do Patent Reforms Enhance Innovation?

Over the past two decades, academics and policy makers in the IP/patent space have focused on three specific developments: elimination of presumptive injunctive relief for victorious patent enforcers in eBay Inc., creation of the Patent Trial and Appeal Board (PTAB), and restriction of software’s eligibility for patent protection in the Alice Corp. decision. Despite general agreement on the importance of these three patent system changes, their individual and collective impact on innovation (construed in terms of firms’ patent filings and R&D investments) remains both contested and understudied.

A recent paper by Christian Helmers (Santa Clara—business) and Brian Love (Santa Clara), Patent Law Reform and Innovation: An Empirical Assessment of the Last 20 Years, jumps into this empirical void and seeks to assess these three reforms’ impact on innovation in the U.S. To do so, the paper constructs firm-level innovation measures and of exposure to each reform and submits them to various Dif-in-Dif specifications that compare relatively exposed and relatively unexposed firms. The paper finds that “Overall, our results suggest that these three reforms had a positive effect on innovation as measured by patent filings and R&D investment.” More detailed findings are described in the excerpted abstract.

“... We find: a positive association between eBay and R&D spending by firms that were relatively more exposed to patent litigation prior to the Court’s decision; a positive association between the introduction of PTAB proceedings and both R&D expenditures and patent filings by firms that innovate in tech classes where PTAB has been most active; and a positive association between Alice and both R&D spending by software firms and patenting by firms that held relatively more software patents prior to the Court’s opinion.”

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Side-By-Side vs. Overlayed Plots

At the risk of delving too deeply into Stata's graphics coding "weeds," two different ways to present the same graphs help illustrate graphing possibilities.

By convention, a "side-by-side" presentation of two distinct, though related, plots (with confidence intervals) in Stata looks like the following:

If the plots are snug enough in how they relate, however, additional coding can produce a single "overlayed" graph:

For those intrigued, the coding underneath these two possibilities is included here.

Federal Judicial Data: Greater Transparency or Opacity?

While not itself an empirical paper per se, The Necessary and Proper Stewardship of Judicial Data, by Zachary Clopton (Northwestern) and Aziz Huq (Chicago), engages in a discussion near-and-dear to most legal empiricists' heart. Specifically, their paper sets out to develop "the first comprehensive descriptive, doctrinal, and normative analysis of judicial data.”

As the authors note, judicial (especially Art. III courts) data are "valuable in governance and for those interested in understanding how they are being governed. In the absence of hurdles or walls, it can illuminate the operation of one branch of the national government. Constitutional lapses can be brought to light through retail or wholesale data." As such, data flowing from Art. III courts and institutions can promote various public goods by, in part, advancing a better understanding of, or improvements to, judicial institutions.

One critical anomaly is that, as compared to the legislative and Executive branches, judicial branch data remain "largely terra incognito for scholars and the general public alike. Instead, public attention rushes to high-profile cases of leaked Supreme Court opinions and private influence campaigns.” While why this is so remains unclear (to me, anyway), that it is so, however, is obvious to any empirical legal scholar who labors and writes in this space. And the stakes incident to the "transparency v. opacity" debate grow more acute as new scholarly tools emerge (e.g., natural language processessing, text-scrapping).

As the authors make clear, under current policy and practice judicial data "might be lost, dissipating into air, or kept under wraps, never yielding up their secrets." In contrast, a commitment to greater transparency invites characterizing judicial data as a "new public asset that can then be tapped to create public goods for the benefit of all."

School and Crime; Cause and Effect

Criminologists increasingly worry about a "school-to-prison pipeline" problem. Such worries are typically inflamed incident to public school districts' typical reaction (or, to some, over-reaction) to tragic episodes of student violence, especially mass-violence. An increasing number of empirical efforts seek to identify and better understand causal mechanisms. To this end, a recent paper by Todd Jones (Miss. St.-econ) and Ezra Karger (Fed. Res.-Chicago), School and Crime, exploits data from the National Crime Victimization Survey and explores "seasonality" patterns.

As it relates to "seasonality" patterns, the paper's main take-away is that juvenile criminal activity in general, and student-on-student crime in particular, largely maps onto the school calendar. More specifically, the paper documents a population-level outcome: across all children—those who are compliers for education-based policies and those who are not—attending school causes an average increase in the likelihood of student-on-student crime. While the overall patter is suggestive, the underlying mechanisms are less so. That is, whether a school's institutional setting itself (e.g., institutional-level policies and practices)--or, for example, that student-on-student crime is simply a function of student density levels--is the critical mechanism remains unclear. The paper's abstract follows.

“Criminal activity is seasonal, peaking in the summer and declining through the winter. We provide the first evidence that arrests of children and reported crimes involving children follow a different pattern: peaking during the school year and declining in the summer. We use a regression discontinuity design surrounding the exact start and end dates of the school year to show that this pattern is caused by school: children aged 10–17 are roughly 50% more likely to be involved in a reported crime during the beginning of the school year relative to the weeks before school begins. This sharp increase is driven by student-on-student crimes occurring in school and during school hours. We use the timing of these patterns and a seasonal adjustment to argue that school increases reported crime rates (and arrests) involving 10–17-year-old offenders by 47% (41%) annually relative to a counterfactual where crime rates follow typical seasonal patterns. School exacerbates preexisting sex-based and race-based inequality in reported crime and arrest rates, increasing both the Black-white and male-female gap in reported juvenile crime and arrest rates by more than 40%.”

New Stata Graphing "Toys"

While Stata's graphing capabilities are robust, coding remains complex in places. While the coding "load" has come down a bit over the years, the "bells & whistles" have remained relatively stagnant over the years. One recent change on the "bells & whistles" front involves an increased capacity to "colorize" standard graphs. While the example (below) illustrates how levering color in graphics can enhance (and/or clarify) what the data are telling you, I remain curious about how journals (and their publishers) will choose to deal with the increased production costs associated with color in published papers.

Any potential cost issues aside, a recent Stata newsletter (click here) explains and illustrates (with sample coding) how one can transform a standard twoway scatter plot from this:

To this:

CELS 2023 -- Schedule/Program

2023 Conference on Empirical Legal Studies

October 13-14, 2023

Schedule/Program

University of Chicago Law School

The program (overview version) for the 17th Annual Conference on Empirical Legal Studies (CELS 2023) is now available (click here).

Access to the detailed schedule, however, is password protected, as it will be the source for accessing the conference papers. It is available to all registered participants in the conference (click here).

Legal Precedent and Decay

While the idea of legal precedent implies some degree of durability, precedents are not immortal. Over time, they lose their capacity to influence action; eventually they cease to be of anything other than historical interest. Although precedential decay's existence is generally accepted, the determinants of precedential decay are comparatively under-studied.

Yun-chien Chang (Cornell) & Geoffrey Miller’s (NYU) recent paper, Decay of Precedent in State Supreme Courts, extends this literature by analyzing citation decay patterns in state supreme court opinions, drawing on three different (and large) data bases. Their paper examines both “backward-looking” precedential decay—the frequency with which opinions in a given year cite to opinions from prior years as well as “forward-looking” decay—the frequency with which specific precedents are cited in later opinions. One key finding includes the extraordinary persistence of exponential decay across all of the data. The paper’s abstract follows.

“This paper investigates the decay of precedents in three unique databases: citations in all published opinions by state supreme courts in 2019 (102,555 citations); citations in all published opinions by state supreme courts in 2003 (114,777 citations); and citations by all state and federal courts to state supreme court opinions issued in 2003 (259,627 citations). We find that the frequency of citation drops off by a roughly constant ratio with each passing year. The large scale of our yearly data and the time studied (1904–2019) suggests that this pattern must be the result of underlying forces not having to do with particular circumstances of time or place. We also find that cases with dissents tend to cite older cases and longer decisions tend to be cited more often and faster. Criminal law cases, as compared to constitutional law cases, tend to cite more recent precedents, while trust and estate cases tend to cite older precedents. We offer some conjectures about possible reasons for the observed differences in decay rates.”

Plotting Coefficients

As I'm seeing more examples of coefficient plots [coefplot command] in the literature (see, e.g., below), I thought the following links might interest some readers. For brief discussions and examples (including Stata coding) click here, here, here, and here.

How to Value Data

While a recent paper, Data Valuation and Law, is not itself an "empirical paper," it is certainly "ELS-adjacent" and, as such, I thought it might interest some. This is particularly so as both AI and "big data" become increasingly important.

Motivating Jordan Berry (USC) and D. Daniel Sokol's (USC) paper is the observation that "while data has become increasingly valuable and important, the law’s attempts to value data have lagged, remaining confused and underdeveloped." To improve matters in this regard, the paper advances a “real options” approach for framing data valuation not because the approach is perfect but, rather, because “it is the least bad alternative available.” The paper's abstract follows.

"Data has become an increasingly valuable asset. Numerous areas of law—including contracts, corporate law, IP, antitrust, tax, privacy, and bankruptcy—require parties and courts to determine the value of assets, including data. Unfortunately, data valuation has been hindered by a lack of clarity over what data is and why it is valuable. This lack of clarity also increases the chances of legal decisionmakers valuing data in inconsistent ways, which would create further confusion, inefficiencies, and opportunities for regulatory arbitrage.

This article proposes a unified framework for valuing data that will promote consistent valuations across fields of law. It begins by conceptualizing data as building blocks: It is of little value on its own. But when placed in skillful and creative hands, it can unlock choices for its holders—choices they would not otherwise have—that can generate tremendous profits. Thus, data constitutes what is known as a ‘real option.’ This article shows how using real options to value data can significantly improve upon existing data valuation practices.”

CELS 2023 -- Conference Registration

2023 Conference on Empirical Legal Studies

October 13-14, 2023

Conference Registration

University of Chicago Law School

The University of Chicago Law School and the Society for Empirical Legal Studies (SELS) are pleased to invite you to CELS 2023. The conference will take place October 13–14, 2023, at the University of Chicago’s flagship conference facility, the David Rubenstein Forum. CELS 2023 will feature more than 40 panels, a poster session, and a plenary session on the future of machine learning and AI in empirical legal studies.

As well, on October 12, 2023, UChicago Law and SELS will host a pre-conference workshop on empirical methods: “How to Use Machine Learning and AI in Empirical Legal Research.” This hands-on workshop will feature leading scholars in the field and will introduce participants to tools and techniques for ML- and AI-assisted empirical research.

CELS 2023 Registration: Please visit the CELS 2023 conference website (click here) for more information and to register.

Please direct any conference questions to the conference organizers at: [email protected].

Plotting Kernal Density Estimates as Colored Areas

By default, kernel density estimates are plotted as lines, meaning curves:

According to recent StataList discussions (see here and here), however, plotting kernal density estimates as "areas" has become the standard in some fields. While for publication purposes the coding requires color, if such a possibility exists the resultant information conveyed is quite helpful as the following example illustrates.

SCOTUS Nominations, By the Nimbers and Over Time

Charles Cameron (Princeton--politics ) & Jonathan Kastellec (Princeton--politics) recently published a book, Making the Supreme Court (OUP), that tracks 90 years of POTUS appointments to the Supreme Court. The Book's description (below) makes clear what the authors attempt to achieve with their book. The authors' execution benefits from data, including on:

• Party and presidential interest in the court, including the use of “litmus tests” for Supreme Court nominees;
• The type of selection processes employed by the president;
• The background characteristics of Supreme Court nominees, and those who made the president’s “short list”;
• Interest group participation in Supreme Court confirmation battles;
• Media coverage of Supreme Court nominees;
• Public opinion of nominees;
• Senate roll call voting on Supreme Court nominees; and
• Simulations on the future ideological trajectory of the Supreme Court.

But what particularly caught my eye was that the authors make available the data (and related coding) that they used in their analyses (click here).

“Appointments to the United States Supreme Court are now central events in American political life. Every vacancy unleashes a bitter struggle between Republicans and Democrats over nominees; and once the seat is filled, new justices typically vote in predictable ways. Making the Supreme Court examines 90 years of American political history to show how the growth of federal judicial power from the 1930s onward inspired presidents, the political parties, and interest groups to shape judicial policy through appointments. The result is a new politics aimed squarely at selecting and placing judicial ideologues on the Court, which has transformed the Court into an ideologically driven and polarized branch. Based on rich data and qualitative evidence, Making the Supreme Court provides a sharp lens on the social and political transformations that created a new American politics.”

Levering AI in the Service of Contract Interpretation

Contract interpretation canons and theories continue to evolve. The latest foray seeks to lever AI.

In a recent article, Generative Interpretation, the authors, David Hoffman (Penn) and Yonathan Arbel (Alabama), advance a new approach designed to assist in determining contracting parties’ meaning. The authors' impulse behind generative interpretation, is, according to them, "simple:" they apply large language models (LLMs) to contractual texts and extrinsic evidence to predict what the parties would have said at contracting about what they meant. The paper's abstract follows.

“We introduce generative interpretation, a new approach to estimating contractual meaning using large language models. As AI triumphalism is the order of the day, we proceed by way of grounded case studies, each illustrating the capabilities of these novel tools in distinct ways. Taking well-known contracts opinions, and sourcing the actual agreements that they adjudicated, we show that AI models can help factfinders ascertain ordinary meaning in context, quantify ambiguity, and fill gaps in parties’ agreements. We also illustrate how models can calculate the probative value of individual pieces of extrinsic evidence.

After offering best practices for the use of these models given their limitations, we consider their implications for judicial practice and contract theory. Using LLMs permits courts to estimate what the parties intended cheaply and accurately, and as such generative interpretation unsettles the current interpretative stalemate. Their use responds to efficiency-minded textualists and justice-oriented contextualists, who argue about whether parties will prefer cost and certainty or accuracy and fairness. Parties—and courts—would prefer a middle path, in which adjudicators strive to predict what the contract really meant, admitting just enough context to approximate reality while avoiding unguided and biased assimilation of evidence. As generative interpretation offers this possibility, we argue it can become the new workhorse of contractual interpretation.”

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Lower Fedreal Courts and Religious Claims: An Update

The Supreme Court's ongoing readjustment of long-standing Free Exercise and Establishment clause tensions has renewed scholarly attention to courts' resolutions of religious claims. And in a recent paper, Trump's Lower-Court Judges and Religion: An Initial Appraisal, Stephen Choi (NYU) et al. contribute to a growing empirical literature by levering data on Trump-appointed Circuit Court judges in Free Exercise Clause cases and comparing Trump-appointed judge outcomes with those of Non-Trump Republican- and Democratic-appointed judges.

What the authors find includes that "Trump’s judicial appointments are more religious but no less qualified than other judges," and that "Trump judges are more likely to vote in favor of the plaintiff than not only Democratic judges, but also judges appointed by other Republican presidents."

Notably, however, while Trump-appointed judges vote “pro-religion” (that is, in favor of a religious plaintiff asserting a free exercise claim) more frequently than other judges, this tilt distributes unevenly across various religions. For example, and as the table below illustrates, the authors find “... different voting patterns for Islam. For cases involving Islam, Trump judges vote in favor of Muslim plaintiffs in only 19.2% of the cases compared with 34.5% for Non-Trump Republicans and 48.0% for Democrats.” The authors note that the overall pattern that emerges is “consistent with conventional wisdom: Democrats tend to protect minority religions, and Republicans tend to protect Christianity (and possibly Judaism).”

 

 

The paper's abstract follows.

“It is widely believed that President Donald Trump’s judicial appointments reflected a strategy of appeasing evangelical Christians and other religious groups that favored a more conservative, Christian judiciary, and that in pursuing this strategy Trump sacrificed quality. We explore evidence for this theory by examining the biographies and credentials of Trump’s lower court nominees and the voting records of his judicial appointments to the circuit courts in free exercise cases. We find that Trump’s appointments to the lower courts have stronger or more numerous religious affiliations, and are affiliated with the Federalist Society and the National Rifle Association at unusually high rates, but are no less well credentialed than other judges are. We also find that Trump’s appointments to the circuit court more frequently vote in favor of Christian plaintiffs, and less frequently vote in favor of Muslim plaintiffs, in free exercise cases than judges appointed by other Republican presidents and by Democratic presidents.”

Fixed Effects vs. Random Effects

While debates persist, how fixed- and random-effect estimators differ and what they seek to accomplish, however, remain somewhat muddled in the various empirical literatures. A recent discussion on StataList, however, provides a rare moment of clarity (here) by nudging us to think about fixed- and-random effects as involving different tradeoffs (or preferences) between consistency and efficiency. As one commentator helpfully notes:

"The -fe- estimator is always consistent but inefficient if -re- is the way to go.
 The -re- estimator is inconsistent if -fe- is the way to go."

Skewed Data, Histograms & Axis "Breaks"

While skewed data distributions (e.g., a large proportion of zeros) invite an array of well-known methodological complications, they also complicated data presentation needs. For example, a standard histogram for a distribution that includes approx. 85% zeros looks something like the following:

Setting aside the methodological complications imposed by such skewed data, as it narrowly relates to descriptive data presentation needs one "dissenting" approach is to insert axis "breaks" (discussion here) that generates the following histogram:

Such a strategy may provide the optimal tradeoff between the visualization of "extreme" data and compression of the non-extreme data. This approach also has the virtue of keeping the underlying data in their raw rather than a transformed scale.

CELS 2023--Submission Deadline Extended

2023 Conference on Empirical Legal Studies

October 13-14, 2023

Call for Papers -- Submission Deadline Extension

University of Chicago Law School

The deadline for submitting papers to the 2023 Conference on Empirical Legal Studies (CELS) has been extended. The new deadline is Sunday, July 9, 2023 (11:59 PM Central Daylight Time).

Please note that this is the final deadline. The Conference website will not accept submissions after 11:59 PM (CDT) on Sunday, July 9, 2023.

To submit a paper, please visit the CELS 2023 website. We encourage preregistration of all experiments, and we encourage all authors to make their data, scripts, and code publicly available upon publication to the extent your methodology permits. We welcome replication studies and papers that develop and demonstrate empirical methods.

About: The University of Chicago Law School and the Society for Empirical Legal Studies (SELS) are pleased to host the 17th Annual Conference on Empirical Legal Studies (CELS) on October 13–14, 2023, in person at the University of Chicago. CELS is an interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, and other fields who are interested in the empirical analysis of law and legal institutions. Visit the conference website.

Questions about the submissions process should be directed to the conference organizers at: [email protected].

Data on Local Prosecutor Campaign Contributions & Electoral Outcomes

Building on prior descriptive work (here) that focuses on local prosecutor elections in the U.S., in a recent paper, Donating to the District Attorney, Carissa Hessick (UNC) et al., broaden their focus to assess how district attorney campaign contributions map onto electoral success. To inform the authors' interest in "the local politics of mass incarceration," the paper exploits a unique data set that links contribution and election results data. Merging these two data sets allows one to know whether a given recipient [local prosecutor candidate] was an "incumbent or challenger, the recipient’s vote share in both the primary and general election, and whether the election was contested in the primary election, the general election, or either election."

Such data provide a "more nuanced picture of prosecutorial politics, allowing us to make important distinctions between offices and interest groups, suggest new accountability measures, and better evaluate how elections can lead to reform." What the authors find propels them to conclude that "accountability mechanisms beyond elections are necessary to ensure prosecutors do not exercise their significant discretionary power unchecked." The paper's excerpted abstract follows.

“… The study offers a more complete empirical account of prosecutor accountability by analyzing contributions to local candidates as well as their election results. It details the amount of money in local prosecutor elections, including from interest groups, and the relationship between candidate fundraising and success. The stark differences across the country underscore that the more than two thousand local prosecutors are not a monolith; some offices are best understood as political, with contested elections and significant amounts of campaigning, while most appear more bureaucratic...”

Judicial Decisionmaking: Moving Beyond Case Outcomes

While conceding that "judges matter" as a general matter, a recent paper, Do Judges Matter?, considers a slightly narrower question: what about judges matters when it comes to the more mundane aspects incident to the business of judging?" More specifically, the authors, Yun-chien Chang (Cornell) and Geoffrey Miller (NYU), explore potential relationships "between a judge’s personal characteristics and the day-to-day business of courts – the writing of opinions, the use of citations, the frequency of dissent, and like factors."

Exploiting a hand-collected data set of 4,591 substantive decisions issued by 48 state supreme courts in 2003 (excluding Alaska and Hawaii), the paper finds that while state supreme courts themselves vary in how they carry out their judicial business (e.g., citation distributions, frequency of dissents), personal characteristics of judges, by contrast, have only a "negligible impact." The abstract follows.

"An extensive literature examines whether characteristics of judges correlate with votes on cases. These studies generally consider the judges’ votes on the merits of cases. Examining a data set of 4,591 decisions issued by 48 state supreme courts in 2003, we consider whether judges’ personal features affect their opinion writing. We find virtually no significant differences along any of the dimensions under review. Judicial characteristics matter only indirectly — ideological differences increase dissensus in public-law cases, which increases opinion length and citation numbers. Our study suggests that when carrying out the ordinary business of courts, judges are almost all the same."

Data on the Fourth Amendment's "Good Faaith" Exception's Application

While the Fourth Amendment is designed to protect criminal defendants against prosecutors' use of unlawfully obtained evidence, an important exception exists for evidence obtained by law enforcement officers who rely in good faith on existing legal authority. Thus, if police rely on a statute that turns out to be unconstitutional or invalid warrants or precedent, the evidence obtained is nonetheless admissible under the good faith exception.

Empirical analyses on how the good faith exception works "on the ground," however, are scant. To this end, a recent paper, The Reality of the Good Faith Exception, by Matthew Tokson (Utah) and Michael Gentithes (Akron), helps to fill a void.

Their paper draws from available published (and unpublished) judicial decisions (N=5,531) that mention the Fourth Amendment or "motion to suppress" across three separate months (July 2015, 2018, and 2021). Among the paper's findings includes a discernible uptick in the exceptions salience over time (see Figure 1, below).

Figure 1: Good Faith Exception (GFE) Prevalence Over Time - Proportion of Fourth Amendment Suppression Cases Involving the Good Faith Exception

The authors are correctly mindful that selection effects certainly influence their descriptive findings. The authors argue that selection effects in this context likely lead, on balance, to an under-counting of suppression hearings to the extent that "defendants and their attorneys likely decline to file suppression motions because of the good faith exception." (emphasis added)

Overall, the paper argues that "the [good faith] exception, originally confined to narrow circumstances, is turning into a protection for nearly any investigatory activity with an arguable connection to an existing precedent or statute" and that "[c]ourts rely on the exception with remarkable frequency, and a large proportion of courts invoking the exception avoid any substantive constitutional ruling, stunting the development of Fourth Amendment doctrine."

"Heatplots" for Correlation Coefficients Graphs

While I've never used one myself, I detect an upward trend in "heatplots" (example below) for basic descriptive correlation coefficient graphs.

A helpful discussion (along with coding suggestions) on what is possible with Stata's heatplot command is found here.

CELS 2023 -- Call for Papers

2023 Conference on Empirical Legal Studies

October 13-14, 2023

Call for Papers

University of Chicago Law School

The 2023 Conference on Empirical Legal Studies (CELS) is now accepting submissions. CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference).

The Paper Submission Deadline is Thursday, June 29, 2023 (11:59 PM Central Daylight Time). There is no charge for submissions.

To submit a paper, please visit the CELS 2023 website We encourage preregistration of all experiments, and we encourage all authors to make their data, scripts, and code publicly available upon publication to the extent your methodology permits. We welcome replication studies and papers that develop and demonstrate empirical methods.

Papers are selected through a peer review process. If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Discussion at the conference includes assigned commentators and audience questions. Accepted papers will be made available to all conference participants.

About: The University of Chicago Law School and the Society for Empirical Legal Studies (SELS) are pleased to host the 17th Annual Conference on Empirical Legal Studies (CELS) on October 13–14, 2023, in person at the University of Chicago. CELS is an interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, and other fields who are interested in the empirical analysis of law and legal institutions. To visit the conference website, click here.

If you have questions about the submissions process, please contact the conference organizers at [email protected].

Lay Understandings of Civil Settlements

When it comes to resolving formal civil disputes, popular imagination, partly fueled by the media and entertainment industry, is dominated by "courtroom drama" and jury trials. Empirical work over the decades, however, paints quite a different picture, and one dominated by the "disappearing trial." Thus, as Jennifer Robbennolt (Illinois) et al. make clear in Settlement Schemas: How Laypeople Understand Civil Settlement, "for all the attention paid to the real and imagined drama of the courtroom, these depictions overlook the reality of how many disputes are resolved: through any means but trial."

In the service of seeking a better understanding of how the public understands civil settlements, the authors surveyed a representative sample of more than 1,000 U.S. adults. Key findings include that while the public conveyed a widespread general understanding about civil settlements, evidence of misconceptions about the details, including the role of a jury and settlement scope, also emerged. The paper's abstract follows.

"What does the public think it means to ‘settle’ a civil case? Most legal disputes in the United States end in an agreement to settle, but little is known about what laypeople think about settlement. To fill this gap, we took a direct approach: we asked a nationally representative sample of U.S. adults – more than one thousand of them – basic questions about settlement. We found widespread understanding about the essential nature and frequency of settlement, but persistent, though not universal, misconceptions about the details, including the role of a jury and settlement scope. Because settlement is such a pervasive part of the U.S. legal system, the system’s legitimacy turns in part on how the public understands and views civil settlement. The survey reported here provides a foundational study of the understandings and framework – the schemas – that the public bring to settlement."

Importing Non-Stata Data Sets Into Stata

While perhaps old-hat for some (many), a quick new Stata tutorial video (click here) walks viewers through some of the basics incident to data importation. For old-time users (including those who may reflexively turn to StatTransfer programs for conversions), increasingly easy--and a wider array of--data transfer options remain much-welcome news.

Sentencing as a Product of Plea Agreements

When it comes to explaining criminal sentences, conventional wisdom emphasizes either judges or, given that criminal convictions largely flow from negotiated plea agreements, prosecutors. A closer look, however, reveals far more nuance and complexity and a dynamic process involving multiple parties concurrently rather than a static event dominated by one actor.

In a recent "closer look," Jeffrey Bellin (William & Mary) & Jenia Turner (SMU) explore 77 plea bargains in the first half of 2019. Noting that comprehensive empirical work on judicial responses to negotiated plea deals is sparse, In Sentencing in an Era of Plea Bargains makes a few helpful contributions. First, the authors' deep dive into 77 pleas bargains, though descriptive, revels some initial, general contours of these agreements. Notably, while judges ultimately approved all of the 77 plea bargains in the paper's sample, judges departed from the negotiated sentencing recommendation in about one-third of the cases. Also notable is that in virtually all instances where the judge departed from the negotiated sentence, she did so in a direction that favored greater leniency. Second, in separate analyses the authors examine 31 plea agreements that elicited judicial rejections. Somewhat perplexingly, judges rejected most plea agreements (74%) because they felt the recommended sentence was too lenient.

Obviously, descriptive studies that lack statistical power are limited in important ways. At the same time, however, such studies can help by pointing future researchers in potential helpful directions. The paper's (excerpted) abstract follows.

“The literature offers inconsistent answers to a question that is foundational to criminal law: Who imposes sentences? Traditional narratives place sentencing responsibility in the hands of the judge. Yet, in a country where 95 percent of criminal convictions come from guilty pleas (not trials), modern American scholars center prosecutors – who control plea terms – as the decider of punishment. This Article highlights and seeks to resolve the tension between these conflicting narratives by charting the pathways by which sentences are determined in a system dominated by plea bargains.

After reviewing the empirical literature on sentence variation, state and federal plea-bargaining rules and doctrines, and conducting some empirical analysis of our own, we conclude that neither of the competing narratives is correct. Sentencing in the United States has become a dynamic process with substantial contributions from multiple actors, not a static event controlled by any single actor.”

How Summary Judgment Changes Vary Across Subject Areas

The Supreme Court upended summary judgment doctrine with a trilogy of cases in 1986. At the time--and largely persisting into current prevailing wisdom--it was assumed that the 1986 changes made summary judgment comparatively more available and, thus, of greater benefit to defendants. While empirical evidence remains somewhat scant, emerging findings paint a far more complex and nuanced picture.

A recent contribution to this research literature comes from Jonathan Nash (Emory) and Daniel Sokol (USC). In The Summary Judgment Revolution that Wasn't, the authors lever six separate (though related) datasets that draw from federal court decisions in three discrete areas (antitrust, civil rights, and securities regulation) and exploit the 1986 trilogy with pre- and post-intervention comparisons. At bottom, the paper finds "no evidence of the trilogy influencing summary judgment decision making in securities regulation, and at best mixed evidence of such an effect in civil rights cases. In contrast, we find that the trilogy did have a strong statistically significant effect on decision making in antitrust cases, at both the district court and court of appeals levels." The abstract follows.

"The U.S. Supreme Court decided a trilogy of cases on summary judgment in 1986. Questions remain as to how much effect these cases have had on judicial decision making in terms of wins and losses for plaintiffs. Shifts in wins and losses and what cases get to decisions on the merits impact access to justice. We assemble novel datasets to examine this question empirically in three areas of law that are more likely to respond to shifts in the standard for summary judgment: antitrust, securities regulation, and civil rights. We find that the Supreme Court’s decisions had a statistically significant effect in antitrust, an ambiguous effect in civil rights cases, and no effect in securities regulation. We also find that, in the trilogy’s wake, antitrust appellate cases were far more likely to cite trilogy cases—and in particular the one trilogy case that was an antitrust case—than appellate cases in the other areas. This suggests that the lone trilogy case that arose in antitrust had an effect on decision making in that field, but that the trilogy had a limited effect across other substantive areas. This finding differs from Twombly and Iqbal where an antitrust decision ultimately reshaped the entire body of law around motions to dismiss."

Experimental Judicial Decisionmaking Studies & External Validity

For experimental judicial decisionmaking studies, the "gold-standard" research design includes using actual judges as participants. My Cornell colleague (and good friend) Jeff Rachlinski is a leading scholar in this space. For an array of reasons (including practical reasons), too much of the experimental research in this field is hampered by researchers' use of students (including law students) as the experimental participants. To the extant that researchers seek to generalize results from judicial decisionmaking studies to the "universe of judges," conventional  wisdom suggests that the use of students in these experiments raises important external validity questions.

Of course, such (and most) "conventional wisdoms" rest on empirical assumptions. And in a recent paper, Can Law Students Replace Judges in Experiments of Judicial-Making?, Holger Spamann (Harvard) and Lars Klohn (Humboldt) submit this question to an experiment. The answer to the question they pose in their paper's title?: "unfortunately, no." The paper's abstract follows.

"Experimental research on judicial decision-making is hampered by the difficulty of recruiting judges as experimental participants. Can students be used in judges’ stead? Unfortunately, no. We ran the same high-context 2×2 factorial experiment of judicial decision-making focused on legal reasoning with 31 U.S. federal judges and 91 elite U.S. law students. We obtained diametrically opposed results. Judges’ decisions were strongly associated with one factor (sympathy, i.e., bias) but not the other (law). For students, it was the other way around. Equality between the two groups is strongly rejected. Equality of document-view patterns—a proxy for thought processes—and written reasons is also strongly rejected."

A Quick Refresher on T- vs. F-Tests

While I haven't thought about this since grad school, a recent discussion on the StataList (here) provides a helpful, quick refresher on when F- and T-Tests converge and when they do not (and why).

Specific Performance, Expectations, and Elon Musk

For one brief moment in time in 2022, to some it looked as though a Delaware judge was poised to order Elon Musk to purchase Twitter, over his objections, due to a contract breach. Had that happened, as Tess Wilkinson-Ryan (Penn) et al. note in Expecting Specific Performance, it would have been "the highest-profile specific performance order in living memory."

The incident helped prompt Wilkinson-Ryan and colleagues to explore intuitions on specific performance through a series of mixed-method experiments. And what they find, on balance, are "fragile" intuitions. Levering a mix of survey and experimental data, the authors find that "ordinary people think that courts will give them exactly what they bargained for after breach of contract; in other words, specific performance is the expected contractual remedy." They go on to argue that "the default expectation of equitable relief is a widespread but malleable intuition—and that even a fragile legal intuition has practical consequences, individually and systemically." Indeed, updating subjects with the prospect of money damages for a contract breach loosens initial default expectations for specific performance. The paper's abstract follows.

“Using a series of surveys and experiments, we find that ordinary people think that courts will give them exactly what they bargained for after breach of contract; in other words, specific performance is the expected contractual remedy. This expectation is widespread even for the diverse array of deals where the legal remedy is traditionally limited to money damages. But for a significant fraction of people, the focus on equity seems to be a naïve belief that is open to updating. In the studies reported here, individuals were less likely to anticipate specific performance when they were briefly introduced to the possibility that courts sometimes award damages in contract disputes.

We argue that the default expectation of equitable relief is a widespread but malleable intuition—and that even a fragile legal intuition has practical consequences, individually and systemically. In a follow-up experiment, we show that subjects are more interested in the prospect of efficient breach when they know that money damages are a possible remedy. This finding suggests that the mismatch between what people assume the law will do (specific performance) and what it actually does (money damages) sometimes encourages performance. We consider the potential for exploitation of this tendency. Finally, we offer some suggestions about how law and psychologists should elicit folk beliefs about legal rules and remedies."

2023 Northwestern Main and Advanced Causal Inference Workshops

Co-organized by Bernie Black (Northwestern) and Scott Cunningham (Baylor—econ), the 12th annual workshop on Research Design for Causal Inference will be held at Northwestern Law School in Chicago, IL.

Main Workshop:  Monday – Friday, August 7-11, 2023; Advanced Workshop:  Monday – Thursday (midday), August 14-17, 2023; and Optional Machine Learning Primer:  Sunday afternoon, August 13, 2023.

Main Workshop Topics/Faculty: Introduction to Modern Methods for Causal Inference:  Donald B. Rubin (Harvard Univ.), Matching and Reweighting Designs and Difference-in-Differences: Jens Hainmueller (Stanford Univ.), Regression Discontinuity:  Matias Cattaneo (Princeton Univ.), and Instrumental variables:  Rocio Titiunik (Princeton Univ.).

Advanced Workshop Topics/Faculty: Applications of machine learning to causal inference:  Christian Hansen (Univ. Chicago), Advanced matching and balancing methods: Jeffrey Wooldridge (Michigan State Univ.), Advanced panel data methods:  Yiqing Xu (Stanford Univ.), and Empirical Bayes Methods:  Christopher Walters (UC Berkeley).

In person-registration is limited to 125 participants for each workshop. There will also be a Zoom option, but please come in person if you can.  The online experience is not the same.

For further information and registration: (click here)

ChatGPT & Statistical Analyses

I suspected discussions about this were inevitable; I wasn't disappointed. Where this may land over time in terms of coding, data, etc. is anyone's guess.

CELS 2023--Save the Date

2023 Conference on Empirical Legal Studies

Save the Date

October 13-14, 2023

The University of Chicago Law School

David Rubenstein Forum
1201 E 60th St
Chicago, IL 60637

19th Asian Law and Economics Association (AsLEA) Annual Meeting -- Call for Papers

The conference's central theme is "Rationality, Norms, and the Law." The in-person conference, scheduled for 8-9 August 2023, at the Academia Sinica, Taipei, Taiwan, invites papers regarding the economic analysis of laws and institutions or the relationship between rationality and norms in the legal system.

Professor Kaushik Basu (Department of Economics and SC Johnson Graduate School of Management, Cornell University) and Professor Todd Zywicki (Antonin Scalia School of Law, George Mason University) are the keynote speakers of the 2023 AsLEA Conference.

Please submit the manuscripts or abstracts of your article or a session proposal, with a short bio in PDF or Word to [email protected] by May 31, 2023. We will make early decisions on a rolling basis and inform applicants of the results as soon as possible. There are no submission or registration fees for the 2023 AsLEA Conference. For hotel information click here.

Please visit the Asian Law and Economics Association website (click here) for updated information. If you have any questions, please send an e-mail to [email protected] or contact the local organizer. Note that Participants should take care of booking hotels and arranging travels by themselves. Neither AsLEA nor the local organizer is able to provide stipends or reimburse expenses of participants.

Local Organizer: Tze-Shiou Chien (Deputy Director & Research Professor, Institutum Iurisprudentiae, Academia Sinica; Vice President, AsLEA. [email protected]).

Criminal Records & Accuracy

Given the outsized implications for individuals, to note that criminal record keeping requires accuracy is to note merely the obvious. This is especially true in the employment context where "94% of employers conduct background checks" on job applicants. Growing employer demand for background checks has, in turn, fueled a dramatic increase in private sector service providers in this space. While access to and uses of criminal records have expanded, concerns persist about "serious data quality problems" in various criminal justice databases.

To explore this issue, in a recent paper, The Problem With Criminal Records, Sarah Lageson (Rutgers--crim. just.) & Robert Stewart (Maryland--crim. just.) compare criminal record reports for 101 New Jersey participants, all of whom had at least one criminal change in their history, across three sources: (1) the official state record; (2) a consumer reporting agency; and (3) an open source, public records data aggregator. The researchers then asked each of the 101 participants to review and audit the information retrieved from the three sources.

As the figure above visually implies, what the researchers found includes errors, and lots of them. Among the 2,406 individual charges reported by the three separate sources, only 8.3% of the participants' criminal charges (200/2,406) were accurately reported by all three sources. Moreover, 60% and 50% of participants had at least one false positive error on their regulated and unregulated background checks, and nearly all (90% and 92% of participants, respectively) had at least one false negative error. As this study is limited to individuals with criminal histories, it remains unclear whether errors also implicate those with no criminal history. The paper's abstract follows.

"Criminal records are routinely used by employers and other institutional decision-makers, who rely on their presumed fidelity, to evaluate applicants. We analyze criminal records for a sample of 101 people, comparing official state reports, two sources of private sector background checks (one regulated and one unregulated), and qualitative interviews. Based on our analysis, private sector background checks are laden with false positive and false negative errors: 60% and 50% of participants had at least one false positive error on their regulated and unregulated background checks, and nearly all (90% and 92% of participants, respectively) had at least one false negative error. We define specific problems with private sector criminal records: mismatched data that create false negatives, missing case dispositions that create incomplete and misleading criminal records, and incorrect data that create false positives. The prevalence of error in private sector background checks limits access to social opportunities ranging from employment, education, and housing."

Call for Papers: 7th Annual Penn-NYU Empirical Contracts Workshop

As hints of our "pre-COVID" lives begin to emerge, including in-person conferences, I am delighted to pass along the following call for papers from a long-time ELS-blog reader:

The 7th Annual Penn-NYU Empirical Contracts Workshop is scheduled to place in-person at the New York University School of Law on 23 May 2023.

The Workshop is a forum for the presentation and discussion of early and mid-stage projects analyzing contract law and practice from a variety of empirical perspectives. Papers are selected through a peer review process.  Attendees of the conference are expected to cover their own travel and lodging costs, but there are no conference fees.

Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

Please note that the Workshop paper submission deadline is 1 March 2023.

Organizing committee:  Dave Hoffman (Penn); Florencia Marotta-Wurgler (NYU) and Tess Wilkinson-Ryan (Penn).  Please email submissions to: [email protected]   

Modeling Rare Events--SCOTUS Clerkships

Of the approximate 36,000 law school students who graduate each year, only 36 or so end up securing a SCOTUS clerkship. While a handful of biographical and anecdotal accounts exist, very few systematic data exist on the "1-in-1,000" clerkship lottery.

A recent paper by Tracey George (Vanderbilt), Albert Yoon (Toronto), and Mitu Gulati (Virginia), Some Are More Equal Than Others: U.S. Supreme Court Clerkship, digs into 40 years worth of data and provides rich granularity about this statistical rare event. In the main, while some unexpected twists emerge, what the authors find generally comports with perceived wisdoms. For example, a disproportionate number of clerks emerge from five elite law schools. One interesting change, however, is that while a SCOTUS clerkship once led "primarily to government and universities [law faculties]," former clerks now increasingly land at "law firms and corporations." An excerpted abstract follows.

"... Yet, we lack a clear story about who wins this clerkship lottery. This Essay seeks to provide that story. Our analysis relies on new datasets of all Court clerks who served between 1980 and 2020, including the details of their path to the Court and their road after. We amend and expand on theories of success in this important labor market. All clerks are highly qualified, yet the same is true for many not chosen. We find that educational pedigree, as opposed to academic performance or any other qualification, often distinguishes the winners from the also-rans. The Court clerkship selection process, like other elite labor markets, proves to be a blend of status and merit where status often prevails. Our analysis does not end there, however. We go on to look at where this forty-year cohort is currently working and confirm that once attained, a Court clerkship does lead to a bounty of opportunities including a return to the Court as a justice. Thus, the Court clerkship lottery is an important labor market not only to lawyers but also to society writ large. In the elite legal labor market, some people are, in fact, more equal than others."

More Evidence of a Racialized Death Penalty

Notably, race does not emerge as a central concern in the Court’s Furman decision, now more than fifty years old. In the decades since Furman, however, empirical evidence continues to mount documenting that "racial discrimination infects the administration of the death penalty." A recent paper by Jeffrey Fagan et al. (Columbia), Getting to Death: Race and the Paths of Capital Cases after Furman, represents a recent and important contribution to this literature. Researchers visited each judicial circuit in Georgia to record the details of every murder case that entered the courts from 1994–2005. Analyses of the 1,317 death-eligible murders suggest that “race continues to matter, and that the statutory aggravators that distinguish death cases from other killings are themselves racialized.” A more detailed description of the paper's core findings appears in the Abstract (below).

“Decades of research on the administration of the death penalty have recognized the persistent arbitrariness in its implementation and the racial inequality in the selection of defendants and cases for capital punishment. This Article provides new insights into the combined effects of these two constitutional challenges. We show how these features of post-Furman capital punishment operate at each stage of adjudication, from charging death-eligible cases to plea negotiations to the selection of eligible cases for execution and ultimately to the execution itself, and how their effects combine to sustain the constitutional violations first identified 50 years ago in Furman. Analyzing a dataset of 2,328 first- degree murder convictions in Georgia from 1995–2004 that produced 1,317 death eligible cases, we show that two features of these cases combine to produce a small group of persons facing execution: victim race and gender, and a set of case-specific features that are often correlated with race. We also show that these features explain which cases progress from the initial stages of charging to a death sentence, and which are removed from death eligibility at each stage through plea negotiations. Consistent with decades of death penalty research, we also show the special focus of prosecution on cases where Black defendants murder white victims. The evidence in the Georgia records suggests a regime marred less by overbreadth in its statute than capriciousness and randomness in the decision to seek death and to seek it in a racially disparate manner. These two dimensions of capital case adjudication combine to sustain the twin failures that produce the fatal lottery that is the death penalty.”

Data on Whether Lawyers Add Value in IPOs

A common complaint is that lawyers provide the "grit" while the I-bankers provide the "grease" when it comes to commercial transactions, especially IPOs. Existing research on lawyers' transactional roles focuses on two alternative perspectives: The skilled advice and the passive execution hypotheses. To test these two competing hypotheses, in a recent paper, Do Lawyers Matter in Initial Public Offerings?, Bates (ASU--Finance) et al. draw from the Thomson Financial Securities Data Company (SDC) new issues database to identify U.S. IPOs over the period of 1986–2016. After filtering, the study's final sample includes data on 7,460 IPOs.

What the paper finds may--or may not--surprise. Specifically, the results imply that issuer law firm fixed effects account for an "economically significant portion of the variation in IPO outcomes, even after accounting for underwriter fixed effects." Specifically, the paper notes that "legal advisor fixed effects explain approximately 16.7% of the variation in IPO underpricing, which is about 70% of the variation attributable to underwriters documented in this and prior studies." Thus, if these effects are real, the paper evidences "the economic importance of legal advisors in the IPO process." The abstract follows.

"This study provides novel evidence that law firms representing issuers have a material impact on economic outcomes in the IPO market. Fixed effects for legal counsel explain a significant portion of the variation in IPO underpricing, comparable to underwriter fixed effects documented in prior studies. Issuer law firms also exhibit differential skill in moderating the likelihood of IPO-related litigation, and have an idiosyncratic influence on the structure of IPO disclosures. Accounting for endogenous selection, we document that law firms associated with lower rates of IPO-related litigation are also associated with less first-day underpricing and receive higher fees from their clients."

An Online Resource of Note

For those new to ELS, and even those with more sophisticated questions, UCLA's statistical consulting site, festooned with an array of resource links (here), remains quite helpful. While the site continues to evolve over time, some well-organized and useful links will help orient and guide those engaged in ELS-related work. Examples include a quick reference chart on "choosing the correct statistical test" (which includes command syntax for SAS, Stata, SPSS, & R), illustrated examples of how to interpret annotated output for standard commands, and robust "FAQ" links that are, again, program-specific (complemented by a "general FAQ" link (here)).

Judges' Financial Holdings, Case Outcomes, and Recusal

Fundamental to the notion of a fair trial and the rule of law is the principle that a judge shall apply the law impartially and free from the influence of any personal biases. When real or perceived conflicts of interest emerge, judges are expected to recuse themselves. Despite such an expectation, however, recent Wall Street Journal reporting revealed many cases where judges violated U.S. law and judicial ethics (28 U.S.C. Section 455) by overseeing cases that involve companies in which they or their family-owned stock. Unsurprisingly, several of the cases with conflicts have been appealed.

While Chief Justice John Roberts downplayed the significance of the Wall Street Journal reporting, characterizing the judges' failures to recuse as "isolated violations" and "unintentional oversights," empirical explorations of the potential influence of judges' financial holdings on case outcomes remain rare. To address this scholarly gap, a recent paper, Judge Financial Holdings and Case Outcomes: Evidence from Judge Financial Disclosures, examines whether there is a relation between the financial holdings of judges and the outcomes of the trials over which they preside.

To construct the judges' investment data set, Harit (Univ. Texas-Dallas—mgmt.) et al. extracted judge financial disclosure data from the Free Law Project website. Their sample includes 33,726 annual disclosure reports involving 3,250 federal judges from 2000 to 2020. The authors lever the random assignment of cases to judges and examine whether a judge's rulings are related to his or her stock ownership in one or more of the parties in the case. The paper finds robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. It also notes that investors are positively surprised by these outcomes. The abstract follows.

"This paper examines the association between a judge's financial holdings and the outcome of legal cases. Using novel data on financial disclosures of judges and the civil case information of public firms across district courts in the United States between 2000 and 2021, we document multiple instances in which judges fail to recuse themselves in cases where they hold common stock. We find robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. We also find that investors are positively surprised by these outcomes, as reflected by stock returns to trial outcomes. Collectively, our results provide the first large-scale evidence of the relation between judges' financial holdings and case outcomes."

Interpreting DID Output

While one may quibble about how the questioner designed her DID estimation, the question itself--how to understand and interpret standard DID output (in Stata)--elicited some helpful replies in a recent StataList discussion (here).

The Two-Part Metascience 2023 Confernece(s)

At the request of a loyal Elsblog reader--who is also helping organize the Metascience 2023 conference--I pass along the following.

Metascience is an organization that promotes "global gathering for knowledge sharing, community building, and opportunities to define a roadmap of research and intervention priorities to accelerate science." To this end, "Metascience 2023" involves two separate--though related--conference components. The in-person component is scheduled for 9-10 May 2023, at the National Academy of Sciences in Washington, DC (USA). The organizing group is also mounting a free virtual component, set for "late April and early May," 2023.

For anyone interested in additional information, click here for general conference information, and here for the conferences' call for papers/proposals.

Civilian Complaints Against Police & Prosecutor "Upcharging"

Citizens’ engagement with police serves as a valuable resource. One particular form of citizen engagement—civil complaints—can impose a serious cost to the targeted police officer as well as the police department. To try to reduce the potential negative effects of a citizen complaint against law enforcement agents, other criminal justice actors could act to re-frame the encounter such that the optics look unfavorably on the complaining citizen. Since a defendant cannot seek monetary damages in civil court unless his/her conviction or sentence was declared invalid, prosecutors may increase the number or the severity of charges that the defendant faces to propel the defendant toward accepting a plea deal, which would prevent the defendant from seeking further financial reparations. In a new paper, Do Civilian Complaints Against Police Get Punished?, DeAngelo (Claremont Colleges) et al. pursue this question empirically.

To do so, the authors examine arrest, complaint and charging data from Cook County (Illinois). The results indicate that a positive, causal link between citizen complaints and an increased number of total charges filed against a defendant exists. The paper also notes that a civilian complaint yields a larger plea discount, likely to induce a plea agreement. Notably, the main findings are entirely driven by charges filed directly by the prosecutor’s office rather than directly by the police. Given that prosecutors are likely filing charges after civilian complaints have been filed against law enforcement, it appears that prosecutor behavior is leading to upcharging in response to civilian complaints and not police decisions. An excerpted abstract follows.

“… This paper empirically investigates one way in which criminal justice institutions respond to civilian complaints. Namely, criminal prosecutors can upcharge a defendant who files a civil complaint against law enforcement. By upcharging, the prosecutor can increase the likelihood that a defendant will accept a plea deal, thus preventing the defendant from seeking monetary damages in civil court (Heck vs. Humphrey, 1994). Using data on citizen complaints and criminal charge outcomes from Cook County (Illinois), we find a strong causal link between a citizen filing a complaint and the total number of charges filed.”

The Empirics of Sobriety Checkpoint Stops

Motor vehicle crashes remain the leading cause of death for Americans under age 25, and many of these crashes involve substance use. To help reduce the volume of impaired drivers, many--but not all--states allow for sobriety checkpoints (SC). While historically checkpoint stops were used to enforce drunk driving laws, they are increasingly being used to deter and prosecute drugged drivers, especially in states where marijuana is legal.

Although “no-cause” traffic stops remain controversial, comparatively little data exist that helps inform related debates. While the SCOTUS permits such stops as a general matter, courts in eleven states have ruled that state constitutional provisions preclude sobriety checkpoints. And those states that discontinued sobriety checkpoints provide a critical source of data that a recent paper, Sobriety Checkpoint Law, Fatal Car Crashes and Arrests, exploits. In their paper, Lauren Jones (OSU—public affairs) and C. Blain Moran (OSU—public affairs) compare trends in event frequencies “across states that did and did not turn off their SC laws, net of confounding factors.” The paper finds, at bottom, that when states make sobriety checkpoints illegal, annual fatal crashes and DUI arrests increase. The paper also notes a racial effect relating to DUI and cannabis arrests. The abstract follows.

“Sobriety checkpoints have been used since the early 1980s to enforce laws prohibiting driving while intoxicated (DWI), and to deter DWI. However, their constitutionality has been questioned, as they allow police to stop drivers without cause. As a result, eleven states have banned their use. Using a two-way fixed effects event study analysis, we estimate the effect of rendering sobriety checkpoints illegal on state-level counts of fatal car crashes, and arrests for driving under the influence of drugs or alcohol (DUI), and drug possession. We find evidence that when states make sobriety checkpoints illegal, the number of annual fatal crashes increases by 17 percent. We also find evidence that rendering sobriety checkpoints illegal leads to a 25 percent increase in the number of annual DUI arrests in a state, suggesting that the deterrent effect of checkpoints is much stronger than their enforcement impacts. We also find suggestive evidence that eliminating sobriety checkpoints increases racial disparities in DUI and cannabis possession arrests, with larger impacts on arrests among Black residents.”

Theory's Role in ELS

A not infrequent, though diminishing, canard is that "too many" empirical legal studies papers are "atheoretical." While the claim itself is an empirical one (and one that I side-step), a larger, broader point endures. Specifically, the best of ELS is certainly informed by well-developed theory. In a recent paper, The Role of Theory in Empirical Legal Studies, Tonja Jacobi (Emory) makes this case in a brief and widely accessible paper. While perhaps targeted to "early career" ELS scholars and non-technical consumers of empirical research, the paper helpfully reminds all of theory's essential role in empirical legal studies. The paper's abstract follows.

"The growing field of empirical legal studies is not an alternative to theory: within every empirical inquiry is an implicit formal theoretical model. Further, good empirical tests should be inspired by substantive theories of the phenomena being analyzed. This essay explains how theory is built into all empirical testing and illustrates why empirical tests are best informed by well-developed theories. It uses examples of empirical studies of both US Supreme Court and Australian High Court oral argument to illustrate how these issues apply to the central issues at the intersection of empirical legal studies and constitutional law: the role of judicial ideology and strategy, and predicting judicial behavior. The essay explores common errors in both conducting and consuming empirical work, and presents some solutions, including both methodological fixes and how to draw on substantive expertise. It aims to help the non-technical reader understand when counterintuitive results should be queried and when, in contrast, resistance is more reflective of priors of the reader rather than potential weaknesses of the work. And it acts as a reminder to technical readers and empirical practitioners that theory is built into every empirical test, and so must be taken seriously."

CELS 2023--Univ. of Chicago Law School

Coming on the heels of an exceptionally well-executed and successful CELS at Virginia, CELS 2023 co-organizers--Adam Chilton, Dhammika Dharmapala, and William Hubbard--announced that CELS 2023 is scheduled for 13-14 October 2023, at the University of Chicago Law School. More information will follow.

Case Outcomes & Judge Financial Conflicts

To assist with insuring that "justice is blind," a presiding judge is expected to recuse from any case where he/she is financially conflicted with any of the litigating parties. Evidence that this expectation is not being consistently met was widely reported in 2020-2021. Indeed, the public controversy elicited a response from Chief Justice John Roberts. In his 2021 annual report on the federal judiciary, and in direct response to the controversy surrounding numerous federal judges, Chief Justice Roberts downplayed the controversy’s significance, characterizing the incidents as "isolated violations" and "unintentional oversights."

Notwithstanding Justice Roberts' characterizations, given the widespread implications posed by judges' failures to recuse when proper it is important to understand whether these recent episodes are one-time incidents or point to a larger, systemic problem. To address this question, in a recent paper, Judge Financial Holdings and Case Outcomes: Evidence from Judge Financial Disclosures, Tuhin Harit (Univ. Dallas--Mgmt.) et al. examine whether there is a relation between judges' disclosed financial holdings and the outcomes of the trials over which they presided.

Drawing from a sample that includes cases from 2000–2021, the paper finds that “stock ownership by judges significantly improves the chances of a litigant's winning a case.” In cases where the presiding judge is financially conflicted with either the defendant or plaintiff, “their chance of a favorable decision improves by 3.3% and 4.3%, respectively, after controlling for judge, industry, district, case file year, and firm characteristics.” An excerpted abstract follows.

"... Using novel data on financial disclosures of judges and the civil case information of public firms across district courts in the United States between 2000 and 2021, we document multiple instances in which judges fail to recuse themselves in cases where they hold common stock. We find robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. We also find that investors are positively surprised by these outcomes, as reflected by stock returns to trial outcomes. Collectively, our results provide the first large-scale evidence of the relation between judges' financial holdings and case outcomes.”

Dodd-Frank, Accounting Fraud, and Whistleblowing

Owing to several notable financial frauds involving whistleblowers, regulators took steps to strengthen whistleblowing provisions. One prominent example is the Dodd-Frank whistleblower program which provides financial rewards to whistleblowers who report financial fraud to the SEC. While regulators may believe that the Dodd-Frank whistleblower law helps deter accounting fraud, whistleblower tips can be frivolous and processessing such tips is not cost-less. Similar to all too many public policies, little empirically is known about the the Dodd-Frank whistleblower provision's efficacy.

In a recent paper, Did the Dodd-Frank Whistleblower Provision Deter Accounting Fraud?, Philip Berger (Chicago--Booth Business) & Heemin Lee (Baruch College) examine whether the Dodd-Frank whistleblower law has a causal effect on the deterrence of fraud and quantify the size of the deterrence effect. Exploiting a control group of firms previously exposed to state-level false claims statutes, the paper sets out to assess the "underlying (or ex ante) likelihood of accounting fraud decreases for firms subject to Dodd-Frank and not previously exposed to the risk of whistleblowing" via analogous state laws. At least as it relates to Dodd-Frank's whistleblowing component, the paper a finds a decrease in the probability of fraud attributable to whistleblower provisions. The abstract follows.

“We examine the deterrence effect of the Dodd-Frank whistleblower provision on accounting fraud. To facilitate causal inference, we use state False Claims Acts (FCAs), under which whistleblowing about accounting fraud at a firm invested in by a state's pension fund can result in monetary rewards from that state’s government. We divide our sample into firms exposed and not exposed to whistleblowing risk from a state FCA during the 2008 – 2010 period that preceded the 2011 SEC implementation of the Dodd-Frank whistleblowing provision. We hypothesize that firms already exposed to a state FCA whistleblower law are less affected by the Dodd-Frank whistleblower provision. Using the companies exposed to a state FCA as control firms in our Dodd-Frank tests, the remaining firms constitute the treatment sample. We find that exposure to Dodd-Frank reduces the likelihood of accounting fraud of treatment firms by 12% to 22% relative to control firms, but do not find that it affects audit fees.”