Empirical Legal Studies

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Do Patent Reforms Enhance Innovation?

Over the past two decades, academics and policy makers in the IP/patent space have focused on three specific developments: elimination of presumptive injunctive relief for victorious patent enforcers in eBay Inc., creation of the Patent Trial and Appeal Board (PTAB), and restriction of software’s eligibility for patent protection in the Alice Corp. decision. Despite general agreement on the importance of these three patent system changes, their individual and collective impact on innovation (construed in terms of firms’ patent filings and R&D investments) remains both contested and understudied.

A recent paper by Christian Helmers (Santa Clara—business) and Brian Love (Santa Clara), Patent Law Reform and Innovation: An Empirical Assessment of the Last 20 Years, jumps into this empirical void and seeks to assess these three reforms’ impact on innovation in the U.S. To do so, the paper constructs firm-level innovation measures and of exposure to each reform and submits them to various Dif-in-Dif specifications that compare relatively exposed and relatively unexposed firms. The paper finds that “Overall, our results suggest that these three reforms had a positive effect on innovation as measured by patent filings and R&D investment.” More detailed findings are described in the excerpted abstract.

“... We find: a positive association between eBay and R&D spending by firms that were relatively more exposed to patent litigation prior to the Court’s decision; a positive association between the introduction of PTAB proceedings and both R&D expenditures and patent filings by firms that innovate in tech classes where PTAB has been most active; and a positive association between Alice and both R&D spending by software firms and patenting by firms that held relatively more software patents prior to the Court’s opinion.”

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Side-By-Side vs. Overlayed Plots

At the risk of delving too deeply into Stata's graphics coding "weeds," two different ways to present the same graphs help illustrate graphing possibilities.

By convention, a "side-by-side" presentation of two distinct, though related, plots (with confidence intervals) in Stata looks like the following:

If the plots are snug enough in how they relate, however, additional coding can produce a single "overlayed" graph:

For those intrigued, the coding underneath these two possibilities is included here.