Empirical Legal Studies

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Levering AI in the Service of Contract Interpretation

Contract interpretation canons and theories continue to evolve. The latest foray seeks to lever AI.

In a recent article, Generative Interpretation, the authors, David Hoffman (Penn) and Yonathan Arbel (Alabama), advance a new approach designed to assist in determining contracting parties’ meaning. The authors' impulse behind generative interpretation, is, according to them, "simple:" they apply large language models (LLMs) to contractual texts and extrinsic evidence to predict what the parties would have said at contracting about what they meant. The paper's abstract follows.

“We introduce generative interpretation, a new approach to estimating contractual meaning using large language models. As AI triumphalism is the order of the day, we proceed by way of grounded case studies, each illustrating the capabilities of these novel tools in distinct ways. Taking well-known contracts opinions, and sourcing the actual agreements that they adjudicated, we show that AI models can help factfinders ascertain ordinary meaning in context, quantify ambiguity, and fill gaps in parties’ agreements. We also illustrate how models can calculate the probative value of individual pieces of extrinsic evidence.

After offering best practices for the use of these models given their limitations, we consider their implications for judicial practice and contract theory. Using LLMs permits courts to estimate what the parties intended cheaply and accurately, and as such generative interpretation unsettles the current interpretative stalemate. Their use responds to efficiency-minded textualists and justice-oriented contextualists, who argue about whether parties will prefer cost and certainty or accuracy and fairness. Parties—and courts—would prefer a middle path, in which adjudicators strive to predict what the contract really meant, admitting just enough context to approximate reality while avoiding unguided and biased assimilation of evidence. As generative interpretation offers this possibility, we argue it can become the new workhorse of contractual interpretation.”

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Fixed Effects vs. Random Effects

While debates persist, how fixed- and random-effect estimators differ and what they seek to accomplish, however, remain somewhat muddled in the various empirical literatures. A recent discussion on StataList, however, provides a rare moment of clarity (here) by nudging us to think about fixed- and-random effects as involving different tradeoffs (or preferences) between consistency and efficiency. As one commentator helpfully notes:

"The -fe- estimator is always consistent but inefficient if -re- is the way to go.
 The -re- estimator is inconsistent if -fe- is the way to go."

More Evidence of a Racialized Death Penalty

Notably, race does not emerge as a central concern in the Court’s Furman decision, now more than fifty years old. In the decades since Furman, however, empirical evidence continues to mount documenting that "racial discrimination infects the administration of the death penalty." A recent paper by Jeffrey Fagan et al. (Columbia), Getting to Death: Race and the Paths of Capital Cases after Furman, represents a recent and important contribution to this literature. Researchers visited each judicial circuit in Georgia to record the details of every murder case that entered the courts from 1994–2005. Analyses of the 1,317 death-eligible murders suggest that “race continues to matter, and that the statutory aggravators that distinguish death cases from other killings are themselves racialized.” A more detailed description of the paper's core findings appears in the Abstract (below).

“Decades of research on the administration of the death penalty have recognized the persistent arbitrariness in its implementation and the racial inequality in the selection of defendants and cases for capital punishment. This Article provides new insights into the combined effects of these two constitutional challenges. We show how these features of post-Furman capital punishment operate at each stage of adjudication, from charging death-eligible cases to plea negotiations to the selection of eligible cases for execution and ultimately to the execution itself, and how their effects combine to sustain the constitutional violations first identified 50 years ago in Furman. Analyzing a dataset of 2,328 first- degree murder convictions in Georgia from 1995–2004 that produced 1,317 death eligible cases, we show that two features of these cases combine to produce a small group of persons facing execution: victim race and gender, and a set of case-specific features that are often correlated with race. We also show that these features explain which cases progress from the initial stages of charging to a death sentence, and which are removed from death eligibility at each stage through plea negotiations. Consistent with decades of death penalty research, we also show the special focus of prosecution on cases where Black defendants murder white victims. The evidence in the Georgia records suggests a regime marred less by overbreadth in its statute than capriciousness and randomness in the decision to seek death and to seek it in a racially disparate manner. These two dimensions of capital case adjudication combine to sustain the twin failures that produce the fatal lottery that is the death penalty.”

Interpreting DID Output

While one may quibble about how the questioner designed her DID estimation, the question itself--how to understand and interpret standard DID output (in Stata)--elicited some helpful replies in a recent StataList discussion (here).

Law School Curriculum: Reform, Assessment, and Research Design

While age-old complaints persist about the basic structure of the first-year law school curriculum, few law schools have dared to experiment with meaningful reforms. Fewer still study these reforms coherently. The "Curriculum B" experiment at Georgetown Law School, as well as a recent paper by David Hyman et al. (Georgetown), Does the 1L Curriculum Make a Difference?, are important and interesting exceptions.

Since 1991, Georgetown Law School has offered an "alternative" Curriculum B along side a standard, more traditional first-year curriculum. Curriculum B was designed to provide “critical” perspectives on the law and respond to important changes in legal practice and theory in the years since the standard 1L curriculum was first developed decades earlier. At Georgetown, students elect into Curriculum B and, when demand exceeds supply, student assignment to Curriculum B is determined by lottery.

For assessment purposes, the quasi-randomized control trial aspect of student assignment to Curriculum B provides a way to overcome selection bias. That is, Hyman et al. compare student outcomes for those who opted into Curriculum B (the treatment group) with student outcomes from those students who wanted to receive the "treatment" (Curriculum B) but, due to excess demand and random selection (lottery), were instead assigned the "placebo" (that is, to the standard, traditional first-year curriculum).

Aside from careful attention to research design, a few of the paper's warrant note. When it comes to students' overall law school academic performances, for example, the authors find that "Curriculum B leads to a mean preserving contraction in students’ overall grades. At the same time, we find that it leads to diminished performance in two key business law electives (Corporations and Securities Regulation)."

Multi-Level Model "vs." Clustering

As this question (or something like it) is not an infrequent one, I found Jeff Wooldridge's response (No.2) in this StataList discussion (here) quite helpful. The bottom line is that decisions incident to estimating a multi-level model or/and clustering pivot on an array of considerations.

Simple Problems, Simple Solutions

While I should have known this years ago, better late than never.

As one becomes familiar with the broad descriptive contours of new data, the tab function is both well-known and ubiquitous. Finding analogous basic descriptive statistics for sub-samples, while certainly possible, was typically (for me, anyway) a bit cumbersome with the tab command.

A recent discussion on the StataList, however, introduced me to the simple (and code-saving) power of a related command, tabstat. As this discussion illustrates (click here), what the tab command accomplishes with multiple lines of coding and multiple output tables, the tabstat command, by contrast, accomplishes with a single line of code and a single, unified output table.

Comparing Regression Output Across Models

As I am seeing more papers trying to do so (as well as discussions about strategies on how to do so), I thought links to a recent StataList discussion as well as some web-based resources might be of general interest. The basic set-up follows:

Model A: Y = α1 + β1X1 + Σθ∙C + firm-fixed effect + year-fixed effect.

Model B: Y = α1 + β2X2 + Σθ∙C + firm-fixed effect + year-fixed effect.

Σθ∙C is list of control variables and are the same for both model A & B.

Question: How to compare the coefficients of β1 and β2?

One approach exploits a post-estimation command in Stata (suest), and then compares means with test (for a discussion and illustration, click here). Another illustrated example, courtesy of the UCLA Statistical Methods and Data Analytics site, is here.

Approaches Toward Missing Data

Missing data vex researchers, and there's no single "correct" way to deal with this not-infrequent real-world problem. A recent question posted on the StataList (here) visually illustrates what this can look like (in terms of N variation) across alternative specifications of a multilevel regression model drawing from a data set that contains missing data in some variables.

As this output illustrates, as one proceeds across models 1-4 what is clear is that the addition (and subtraction) of variables introduce missing data. How to approach this and what to do about it motivated the initial query. And in the discussion that follows identifies (some of the) various "standard" approaches.

Mistaking Correlation for Causation

Economists--and econometrics--have been involved in empirical legal studies for generations. This is particularly true for anti-trust doctrine, both its development and current application. Legal challenges to proposed mergers, and questions relating to anti-competitive spill-over effects, including market concentration, remain a staple for empirical analyses.

In the anti-trust context, one intuitive move involves assessing whether "prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods." Pursuing such a question can involve (and frequently does involve) something as simple (and potentially powerful) as regressing price on the HHI.

Despite whatever intuitive appeal such a move holds, in a recent brief essay--co-signed by many leading law and economics scholars (including, and I add in interest of full disclosure, my Cornell colleague George Hay), On the Misuse of Regressions of Price on the HHI in Merger Review, the authors caution against such a move as it invites one basic fundamental problem: Mistaking correlation for causation. And it is for the essay's description of just how such a basic mistake can arise (rather than it's specific application in the anti-trust context) that I recommend this essay, especially to younger, developing empirical legal scholars.

As it relates to the specific anti-trust point developed in the essay, the authors' argue that: "The underlying problem with regressions of price on the HHI is that the relationship between price and the HHI is not causal. Instead, both are equilibrium outcomes that are determined by demand, supply, and the factors that drive them. Thus, a regression of price on the HHI does not show the sort of causal effect that would be helpful in predicting the competitive effects of a merger." More specifically, as the "HHI can take values ranging from 0 to 10,000, with the former corresponding to a fragmented market with infinitesimal firms, and the latter to monopoly. All else equal, a decrease in the number of firms leads to a higher HHI. However, the HHI can increase if, for a given number of firms, the market shares of the larger firms increase."

The root problem, of course, flows from challenges posed by “endogeneity” or “simultaneity.” Such problems lurk "when a clear causal path between two variables is posited by the relevant economic theory but the causal factor (on the right side of the regression equation) is either correlated with unobserved factors or, more subtly, is simultaneously determined by the variables (in this case, price) on the left side of the regression. A famous example of simultaneity involves price and quantity, which are simultaneously determined in the classic economic model of supply and demand." The problems posed by "endogeneity” or “simultaneity," of course, are not limited to the anti-trust realm. An excerpted abstract follows.

"It might seem natural to determine whether prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods. This might be implemented by using a simple regression of price on the HHI. However, for reasons that we describe, regressions of price on the HHI should not be interpreted as establishing causation. That is, they do not inform how a change in concentration from a merger would affect prices. Courts and other policy-makers should not rely on regressions of price on the HHI for the purposes of antitrust merger review."

Writing Up Results From a Logit Model Using the Odds Ratio Option

A fairly innocuous question -- how best to interpret odds ratio results from a xtlogit model estimation -- unleashed a torrent of commentary on the StataList (click here). As this is a not uncommon model used in many ELS papers, I thought others may benefit from the (extended) discussion.

One of the (many) points that garners attention includes the numeric range for odds ratios: "There is no reason an odds ratio can't be greater than 2. On the other hand, an odds ratio can never be less than 0. The possible values for an odds ratio range from 0 to + infinity. And the interpretation does not depend on the size of the odds ratio. In your case, assuming the predictor is dichotomous, it means that the odds of a positive outcome when the predictor is 1 is 2.46 times the odds of a positive outcome when the predictor is 0. If the predictor is continuous, it means that a unit difference in the value of the predictor is associated with the odds of a positive outcome increasing by a factor of 2.46."

Logical Limits of the Randomized Controlled Trial Research Design

For many, and for obvious methodological reasons, the randomized control trial (“RCT”) is "the" (or "a") research design "gold standard" for those seeking data with possible causal implications. (For a nice quick description of RCTs click here.) In contrast, those interested in a clever, "tongue-in-cheek," "critique" of the pursuit of absolute research design "purity" might be interested in a 2003 "essay," Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials, by Gordon CS Smith (Cambridge--medicine) et al. In the "essay" the authors observe that they could not find any "published research presenting results from randomized controlled trials of parachute use." Consequently, the essay "argues" that the empirical basis for the parachute's efficacy is "purely observational" and, moreover, that existing "observational" data on the parachute's apparent efficacy could potentially be explained by a "healthy cohort" effect.