Empirical Legal Studies

How Summary Judgment Changes Vary Across Subject Areas

The Supreme Court upended summary judgment doctrine with a trilogy of cases in 1986. At the time--and largely persisting into current prevailing wisdom--it was assumed that the 1986 changes made summary judgment comparatively more available and, thus, of greater benefit to defendants. While empirical evidence remains somewhat scant, emerging findings paint a far more complex and nuanced picture.

A recent contribution to this research literature comes from Jonathan Nash (Emory) and Daniel Sokol (USC). In The Summary Judgment Revolution that Wasn't, the authors lever six separate (though related) datasets that draw from federal court decisions in three discrete areas (antitrust, civil rights, and securities regulation) and exploit the 1986 trilogy with pre- and post-intervention comparisons. At bottom, the paper finds "no evidence of the trilogy influencing summary judgment decision making in securities regulation, and at best mixed evidence of such an effect in civil rights cases. In contrast, we find that the trilogy did have a strong statistically significant effect on decision making in antitrust cases, at both the district court and court of appeals levels." The abstract follows.

"The U.S. Supreme Court decided a trilogy of cases on summary judgment in 1986. Questions remain as to how much effect these cases have had on judicial decision making in terms of wins and losses for plaintiffs. Shifts in wins and losses and what cases get to decisions on the merits impact access to justice. We assemble novel datasets to examine this question empirically in three areas of law that are more likely to respond to shifts in the standard for summary judgment: antitrust, securities regulation, and civil rights. We find that the Supreme Court’s decisions had a statistically significant effect in antitrust, an ambiguous effect in civil rights cases, and no effect in securities regulation. We also find that, in the trilogy’s wake, antitrust appellate cases were far more likely to cite trilogy cases—and in particular the one trilogy case that was an antitrust case—than appellate cases in the other areas. This suggests that the lone trilogy case that arose in antitrust had an effect on decision making in that field, but that the trilogy had a limited effect across other substantive areas. This finding differs from Twombly and Iqbal where an antitrust decision ultimately reshaped the entire body of law around motions to dismiss."

Experimental Judicial Decisionmaking Studies & External Validity

For experimental judicial decisionmaking studies, the "gold-standard" research design includes using actual judges as participants. My Cornell colleague (and good friend) Jeff Rachlinski is a leading scholar in this space. For an array of reasons (including practical reasons), too much of the experimental research in this field is hampered by researchers' use of students (including law students) as the experimental participants. To the extant that researchers seek to generalize results from judicial decisionmaking studies to the "universe of judges," conventional  wisdom suggests that the use of students in these experiments raises important external validity questions.

Of course, such (and most) "conventional wisdoms" rest on empirical assumptions. And in a recent paper, Can Law Students Replace Judges in Experiments of Judicial-Making?, Holger Spamann (Harvard) and Lars Klohn (Humboldt) submit this question to an experiment. The answer to the question they pose in their paper's title?: "unfortunately, no." The paper's abstract follows.

"Experimental research on judicial decision-making is hampered by the difficulty of recruiting judges as experimental participants. Can students be used in judges’ stead? Unfortunately, no. We ran the same high-context 2×2 factorial experiment of judicial decision-making focused on legal reasoning with 31 U.S. federal judges and 91 elite U.S. law students. We obtained diametrically opposed results. Judges’ decisions were strongly associated with one factor (sympathy, i.e., bias) but not the other (law). For students, it was the other way around. Equality between the two groups is strongly rejected. Equality of document-view patterns—a proxy for thought processes—and written reasons is also strongly rejected."

A Quick Refresher on T- vs. F-Tests

While I haven't thought about this since grad school, a recent discussion on the StataList (here) provides a helpful, quick refresher on when F- and T-Tests converge and when they do not (and why).

Specific Performance, Expectations, and Elon Musk

For one brief moment in time in 2022, to some it looked as though a Delaware judge was poised to order Elon Musk to purchase Twitter, over his objections, due to a contract breach. Had that happened, as Tess Wilkinson-Ryan (Penn) et al. note in Expecting Specific Performance, it would have been "the highest-profile specific performance order in living memory."

The incident helped prompt Wilkinson-Ryan and colleagues to explore intuitions on specific performance through a series of mixed-method experiments. And what they find, on balance, are "fragile" intuitions. Levering a mix of survey and experimental data, the authors find that "ordinary people think that courts will give them exactly what they bargained for after breach of contract; in other words, specific performance is the expected contractual remedy." They go on to argue that "the default expectation of equitable relief is a widespread but malleable intuition—and that even a fragile legal intuition has practical consequences, individually and systemically." Indeed, updating subjects with the prospect of money damages for a contract breach loosens initial default expectations for specific performance. The paper's abstract follows.

“Using a series of surveys and experiments, we find that ordinary people think that courts will give them exactly what they bargained for after breach of contract; in other words, specific performance is the expected contractual remedy. This expectation is widespread even for the diverse array of deals where the legal remedy is traditionally limited to money damages. But for a significant fraction of people, the focus on equity seems to be a naïve belief that is open to updating. In the studies reported here, individuals were less likely to anticipate specific performance when they were briefly introduced to the possibility that courts sometimes award damages in contract disputes.

We argue that the default expectation of equitable relief is a widespread but malleable intuition—and that even a fragile legal intuition has practical consequences, individually and systemically. In a follow-up experiment, we show that subjects are more interested in the prospect of efficient breach when they know that money damages are a possible remedy. This finding suggests that the mismatch between what people assume the law will do (specific performance) and what it actually does (money damages) sometimes encourages performance. We consider the potential for exploitation of this tendency. Finally, we offer some suggestions about how law and psychologists should elicit folk beliefs about legal rules and remedies."

2023 Northwestern Main and Advanced Causal Inference Workshops

Co-organized by Bernie Black (Northwestern) and Scott Cunningham (Baylor—econ), the 12th annual workshop on Research Design for Causal Inference will be held at Northwestern Law School in Chicago, IL.

Main Workshop:  Monday – Friday, August 7-11, 2023; Advanced Workshop:  Monday – Thursday (midday), August 14-17, 2023; and Optional Machine Learning Primer:  Sunday afternoon, August 13, 2023.

Main Workshop Topics/Faculty: Introduction to Modern Methods for Causal Inference:  Donald B. Rubin (Harvard Univ.), Matching and Reweighting Designs and Difference-in-Differences: Jens Hainmueller (Stanford Univ.), Regression Discontinuity:  Matias Cattaneo (Princeton Univ.), and Instrumental variables:  Rocio Titiunik (Princeton Univ.).

Advanced Workshop Topics/Faculty: Applications of machine learning to causal inference:  Christian Hansen (Univ. Chicago), Advanced matching and balancing methods: Jeffrey Wooldridge (Michigan State Univ.), Advanced panel data methods:  Yiqing Xu (Stanford Univ.), and Empirical Bayes Methods:  Christopher Walters (UC Berkeley).

In person-registration is limited to 125 participants for each workshop. There will also be a Zoom option, but please come in person if you can.  The online experience is not the same.

For further information and registration: (click here)

ChatGPT & Statistical Analyses

I suspected discussions about this were inevitable; I wasn't disappointed. Where this may land over time in terms of coding, data, etc. is anyone's guess.

CELS 2023--Save the Date

2023 Conference on Empirical Legal Studies

Save the Date

October 13-14, 2023

The University of Chicago Law School

David Rubenstein Forum
1201 E 60th St
Chicago, IL 60637

19th Asian Law and Economics Association (AsLEA) Annual Meeting -- Call for Papers

The conference's central theme is "Rationality, Norms, and the Law." The in-person conference, scheduled for 8-9 August 2023, at the Academia Sinica, Taipei, Taiwan, invites papers regarding the economic analysis of laws and institutions or the relationship between rationality and norms in the legal system.

Professor Kaushik Basu (Department of Economics and SC Johnson Graduate School of Management, Cornell University) and Professor Todd Zywicki (Antonin Scalia School of Law, George Mason University) are the keynote speakers of the 2023 AsLEA Conference.

Please submit the manuscripts or abstracts of your article or a session proposal, with a short bio in PDF or Word to [email protected] by May 31, 2023. We will make early decisions on a rolling basis and inform applicants of the results as soon as possible. There are no submission or registration fees for the 2023 AsLEA Conference. For hotel information click here.

Please visit the Asian Law and Economics Association website (click here) for updated information. If you have any questions, please send an e-mail to [email protected] or contact the local organizer. Note that Participants should take care of booking hotels and arranging travels by themselves. Neither AsLEA nor the local organizer is able to provide stipends or reimburse expenses of participants.

Local Organizer: Tze-Shiou Chien (Deputy Director & Research Professor, Institutum Iurisprudentiae, Academia Sinica; Vice President, AsLEA. [email protected]).

Criminal Records & Accuracy

Given the outsized implications for individuals, to note that criminal record keeping requires accuracy is to note merely the obvious. This is especially true in the employment context where "94% of employers conduct background checks" on job applicants. Growing employer demand for background checks has, in turn, fueled a dramatic increase in private sector service providers in this space. While access to and uses of criminal records have expanded, concerns persist about "serious data quality problems" in various criminal justice databases.

To explore this issue, in a recent paper, The Problem With Criminal Records, Sarah Lageson (Rutgers--crim. just.) & Robert Stewart (Maryland--crim. just.) compare criminal record reports for 101 New Jersey participants, all of whom had at least one criminal change in their history, across three sources: (1) the official state record; (2) a consumer reporting agency; and (3) an open source, public records data aggregator. The researchers then asked each of the 101 participants to review and audit the information retrieved from the three sources.

As the figure above visually implies, what the researchers found includes errors, and lots of them. Among the 2,406 individual charges reported by the three separate sources, only 8.3% of the participants' criminal charges (200/2,406) were accurately reported by all three sources. Moreover, 60% and 50% of participants had at least one false positive error on their regulated and unregulated background checks, and nearly all (90% and 92% of participants, respectively) had at least one false negative error. As this study is limited to individuals with criminal histories, it remains unclear whether errors also implicate those with no criminal history. The paper's abstract follows.

"Criminal records are routinely used by employers and other institutional decision-makers, who rely on their presumed fidelity, to evaluate applicants. We analyze criminal records for a sample of 101 people, comparing official state reports, two sources of private sector background checks (one regulated and one unregulated), and qualitative interviews. Based on our analysis, private sector background checks are laden with false positive and false negative errors: 60% and 50% of participants had at least one false positive error on their regulated and unregulated background checks, and nearly all (90% and 92% of participants, respectively) had at least one false negative error. We define specific problems with private sector criminal records: mismatched data that create false negatives, missing case dispositions that create incomplete and misleading criminal records, and incorrect data that create false positives. The prevalence of error in private sector background checks limits access to social opportunities ranging from employment, education, and housing."

Call for Papers: 7th Annual Penn-NYU Empirical Contracts Workshop

As hints of our "pre-COVID" lives begin to emerge, including in-person conferences, I am delighted to pass along the following call for papers from a long-time ELS-blog reader:

The 7th Annual Penn-NYU Empirical Contracts Workshop is scheduled to place in-person at the New York University School of Law on 23 May 2023.

The Workshop is a forum for the presentation and discussion of early and mid-stage projects analyzing contract law and practice from a variety of empirical perspectives. Papers are selected through a peer review process.  Attendees of the conference are expected to cover their own travel and lodging costs, but there are no conference fees.

Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

Please note that the Workshop paper submission deadline is 1 March 2023.

Organizing committee:  Dave Hoffman (Penn); Florencia Marotta-Wurgler (NYU) and Tess Wilkinson-Ryan (Penn).  Please email submissions to: [email protected]   

Modeling Rare Events--SCOTUS Clerkships

Of the approximate 36,000 law school students who graduate each year, only 36 or so end up securing a SCOTUS clerkship. While a handful of biographical and anecdotal accounts exist, very few systematic data exist on the "1-in-1,000" clerkship lottery.

A recent paper by Tracey George (Vanderbilt), Albert Yoon (Toronto), and Mitu Gulati (Virginia), Some Are More Equal Than Others: U.S. Supreme Court Clerkship, digs into 40 years worth of data and provides rich granularity about this statistical rare event. In the main, while some unexpected twists emerge, what the authors find generally comports with perceived wisdoms. For example, a disproportionate number of clerks emerge from five elite law schools. One interesting change, however, is that while a SCOTUS clerkship once led "primarily to government and universities [law faculties]," former clerks now increasingly land at "law firms and corporations." An excerpted abstract follows.

"... Yet, we lack a clear story about who wins this clerkship lottery. This Essay seeks to provide that story. Our analysis relies on new datasets of all Court clerks who served between 1980 and 2020, including the details of their path to the Court and their road after. We amend and expand on theories of success in this important labor market. All clerks are highly qualified, yet the same is true for many not chosen. We find that educational pedigree, as opposed to academic performance or any other qualification, often distinguishes the winners from the also-rans. The Court clerkship selection process, like other elite labor markets, proves to be a blend of status and merit where status often prevails. Our analysis does not end there, however. We go on to look at where this forty-year cohort is currently working and confirm that once attained, a Court clerkship does lead to a bounty of opportunities including a return to the Court as a justice. Thus, the Court clerkship lottery is an important labor market not only to lawyers but also to society writ large. In the elite legal labor market, some people are, in fact, more equal than others."

More Evidence of a Racialized Death Penalty

Notably, race does not emerge as a central concern in the Court’s Furman decision, now more than fifty years old. In the decades since Furman, however, empirical evidence continues to mount documenting that "racial discrimination infects the administration of the death penalty." A recent paper by Jeffrey Fagan et al. (Columbia), Getting to Death: Race and the Paths of Capital Cases after Furman, represents a recent and important contribution to this literature. Researchers visited each judicial circuit in Georgia to record the details of every murder case that entered the courts from 1994–2005. Analyses of the 1,317 death-eligible murders suggest that “race continues to matter, and that the statutory aggravators that distinguish death cases from other killings are themselves racialized.” A more detailed description of the paper's core findings appears in the Abstract (below).

“Decades of research on the administration of the death penalty have recognized the persistent arbitrariness in its implementation and the racial inequality in the selection of defendants and cases for capital punishment. This Article provides new insights into the combined effects of these two constitutional challenges. We show how these features of post-Furman capital punishment operate at each stage of adjudication, from charging death-eligible cases to plea negotiations to the selection of eligible cases for execution and ultimately to the execution itself, and how their effects combine to sustain the constitutional violations first identified 50 years ago in Furman. Analyzing a dataset of 2,328 first- degree murder convictions in Georgia from 1995–2004 that produced 1,317 death eligible cases, we show that two features of these cases combine to produce a small group of persons facing execution: victim race and gender, and a set of case-specific features that are often correlated with race. We also show that these features explain which cases progress from the initial stages of charging to a death sentence, and which are removed from death eligibility at each stage through plea negotiations. Consistent with decades of death penalty research, we also show the special focus of prosecution on cases where Black defendants murder white victims. The evidence in the Georgia records suggests a regime marred less by overbreadth in its statute than capriciousness and randomness in the decision to seek death and to seek it in a racially disparate manner. These two dimensions of capital case adjudication combine to sustain the twin failures that produce the fatal lottery that is the death penalty.”

Data on Whether Lawyers Add Value in IPOs

A common complaint is that lawyers provide the "grit" while the I-bankers provide the "grease" when it comes to commercial transactions, especially IPOs. Existing research on lawyers' transactional roles focuses on two alternative perspectives: The skilled advice and the passive execution hypotheses. To test these two competing hypotheses, in a recent paper, Do Lawyers Matter in Initial Public Offerings?, Bates (ASU--Finance) et al. draw from the Thomson Financial Securities Data Company (SDC) new issues database to identify U.S. IPOs over the period of 1986–2016. After filtering, the study's final sample includes data on 7,460 IPOs.

What the paper finds may--or may not--surprise. Specifically, the results imply that issuer law firm fixed effects account for an "economically significant portion of the variation in IPO outcomes, even after accounting for underwriter fixed effects." Specifically, the paper notes that "legal advisor fixed effects explain approximately 16.7% of the variation in IPO underpricing, which is about 70% of the variation attributable to underwriters documented in this and prior studies." Thus, if these effects are real, the paper evidences "the economic importance of legal advisors in the IPO process." The abstract follows.

"This study provides novel evidence that law firms representing issuers have a material impact on economic outcomes in the IPO market. Fixed effects for legal counsel explain a significant portion of the variation in IPO underpricing, comparable to underwriter fixed effects documented in prior studies. Issuer law firms also exhibit differential skill in moderating the likelihood of IPO-related litigation, and have an idiosyncratic influence on the structure of IPO disclosures. Accounting for endogenous selection, we document that law firms associated with lower rates of IPO-related litigation are also associated with less first-day underpricing and receive higher fees from their clients."

An Online Resource of Note

For those new to ELS, and even those with more sophisticated questions, UCLA's statistical consulting site, festooned with an array of resource links (here), remains quite helpful. While the site continues to evolve over time, some well-organized and useful links will help orient and guide those engaged in ELS-related work. Examples include a quick reference chart on "choosing the correct statistical test" (which includes command syntax for SAS, Stata, SPSS, & R), illustrated examples of how to interpret annotated output for standard commands, and robust "FAQ" links that are, again, program-specific (complemented by a "general FAQ" link (here)).

Judges' Financial Holdings, Case Outcomes, and Recusal

Fundamental to the notion of a fair trial and the rule of law is the principle that a judge shall apply the law impartially and free from the influence of any personal biases. When real or perceived conflicts of interest emerge, judges are expected to recuse themselves. Despite such an expectation, however, recent Wall Street Journal reporting revealed many cases where judges violated U.S. law and judicial ethics (28 U.S.C. Section 455) by overseeing cases that involve companies in which they or their family-owned stock. Unsurprisingly, several of the cases with conflicts have been appealed.

While Chief Justice John Roberts downplayed the significance of the Wall Street Journal reporting, characterizing the judges' failures to recuse as "isolated violations" and "unintentional oversights," empirical explorations of the potential influence of judges' financial holdings on case outcomes remain rare. To address this scholarly gap, a recent paper, Judge Financial Holdings and Case Outcomes: Evidence from Judge Financial Disclosures, examines whether there is a relation between the financial holdings of judges and the outcomes of the trials over which they preside.

To construct the judges' investment data set, Harit (Univ. Texas-Dallas—mgmt.) et al. extracted judge financial disclosure data from the Free Law Project website. Their sample includes 33,726 annual disclosure reports involving 3,250 federal judges from 2000 to 2020. The authors lever the random assignment of cases to judges and examine whether a judge's rulings are related to his or her stock ownership in one or more of the parties in the case. The paper finds robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. It also notes that investors are positively surprised by these outcomes. The abstract follows.

"This paper examines the association between a judge's financial holdings and the outcome of legal cases. Using novel data on financial disclosures of judges and the civil case information of public firms across district courts in the United States between 2000 and 2021, we document multiple instances in which judges fail to recuse themselves in cases where they hold common stock. We find robust evidence that judges rule more favorably and take longer when deciding over conflicted cases as compared to unconflicted ones. We also find that investors are positively surprised by these outcomes, as reflected by stock returns to trial outcomes. Collectively, our results provide the first large-scale evidence of the relation between judges' financial holdings and case outcomes."