Empirical Legal Studies

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Federal Judicial Data: Greater Transparency or Opacity?

While not itself an empirical paper per se, The Necessary and Proper Stewardship of Judicial Data, by Zachary Clopton (Northwestern) and Aziz Huq (Chicago), engages in a discussion near-and-dear to most legal empiricists' heart. Specifically, their paper sets out to develop "the first comprehensive descriptive, doctrinal, and normative analysis of judicial data.”

As the authors note, judicial (especially Art. III courts) data are "valuable in governance and for those interested in understanding how they are being governed. In the absence of hurdles or walls, it can illuminate the operation of one branch of the national government. Constitutional lapses can be brought to light through retail or wholesale data." As such, data flowing from Art. III courts and institutions can promote various public goods by, in part, advancing a better understanding of, or improvements to, judicial institutions.

One critical anomaly is that, as compared to the legislative and Executive branches, judicial branch data remain "largely terra incognito for scholars and the general public alike. Instead, public attention rushes to high-profile cases of leaked Supreme Court opinions and private influence campaigns.” While why this is so remains unclear (to me, anyway), that it is so, however, is obvious to any empirical legal scholar who labors and writes in this space. And the stakes incident to the "transparency v. opacity" debate grow more acute as new scholarly tools emerge (e.g., natural language processessing, text-scrapping).

As the authors make clear, under current policy and practice judicial data "might be lost, dissipating into air, or kept under wraps, never yielding up their secrets." In contrast, a commitment to greater transparency invites characterizing judicial data as a "new public asset that can then be tapped to create public goods for the benefit of all."

New Stata Graphing "Toys"

While Stata's graphing capabilities are robust, coding remains complex in places. While the coding "load" has come down a bit over the years, the "bells & whistles" have remained relatively stagnant over the years. One recent change on the "bells & whistles" front involves an increased capacity to "colorize" standard graphs. While the example (below) illustrates how levering color in graphics can enhance (and/or clarify) what the data are telling you, I remain curious about how journals (and their publishers) will choose to deal with the increased production costs associated with color in published papers.

Any potential cost issues aside, a recent Stata newsletter (click here) explains and illustrates (with sample coding) how one can transform a standard twoway scatter plot from this:

To this:

Plotting Coefficients

As I'm seeing more examples of coefficient plots [coefplot command] in the literature (see, e.g., below), I thought the following links might interest some readers. For brief discussions and examples (including Stata coding) click here, here, here, and here.

How to Value Data

While a recent paper, Data Valuation and Law, is not itself an "empirical paper," it is certainly "ELS-adjacent" and, as such, I thought it might interest some. This is particularly so as both AI and "big data" become increasingly important.

Motivating Jordan Berry (USC) and D. Daniel Sokol's (USC) paper is the observation that "while data has become increasingly valuable and important, the law’s attempts to value data have lagged, remaining confused and underdeveloped." To improve matters in this regard, the paper advances a “real options” approach for framing data valuation not because the approach is perfect but, rather, because “it is the least bad alternative available.” The paper's abstract follows.

"Data has become an increasingly valuable asset. Numerous areas of law—including contracts, corporate law, IP, antitrust, tax, privacy, and bankruptcy—require parties and courts to determine the value of assets, including data. Unfortunately, data valuation has been hindered by a lack of clarity over what data is and why it is valuable. This lack of clarity also increases the chances of legal decisionmakers valuing data in inconsistent ways, which would create further confusion, inefficiencies, and opportunities for regulatory arbitrage.

This article proposes a unified framework for valuing data that will promote consistent valuations across fields of law. It begins by conceptualizing data as building blocks: It is of little value on its own. But when placed in skillful and creative hands, it can unlock choices for its holders—choices they would not otherwise have—that can generate tremendous profits. Thus, data constitutes what is known as a ‘real option.’ This article shows how using real options to value data can significantly improve upon existing data valuation practices.”

Plotting Kernal Density Estimates as Colored Areas

By default, kernel density estimates are plotted as lines, meaning curves:

According to recent StataList discussions (see here and here), however, plotting kernal density estimates as "areas" has become the standard in some fields. While for publication purposes the coding requires color, if such a possibility exists the resultant information conveyed is quite helpful as the following example illustrates.

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Skewed Data, Histograms & Axis "Breaks"

While skewed data distributions (e.g., a large proportion of zeros) invite an array of well-known methodological complications, they also complicated data presentation needs. For example, a standard histogram for a distribution that includes approx. 85% zeros looks something like the following:

Setting aside the methodological complications imposed by such skewed data, as it narrowly relates to descriptive data presentation needs one "dissenting" approach is to insert axis "breaks" (discussion here) that generates the following histogram:

Such a strategy may provide the optimal tradeoff between the visualization of "extreme" data and compression of the non-extreme data. This approach also has the virtue of keeping the underlying data in their raw rather than a transformed scale.

"Heatplots" for Correlation Coefficients Graphs

While I've never used one myself, I detect an upward trend in "heatplots" (example below) for basic descriptive correlation coefficient graphs.

A helpful discussion (along with coding suggestions) on what is possible with Stata's heatplot command is found here.

Importing Non-Stata Data Sets Into Stata

While perhaps old-hat for some (many), a quick new Stata tutorial video (click here) walks viewers through some of the basics incident to data importation. For old-time users (including those who may reflexively turn to StatTransfer programs for conversions), increasingly easy--and a wider array of--data transfer options remain much-welcome news.

A Small--But Important--Logistical Matter Incident to Dataset Merging

Those who have merged (or appended or crossed) datasets may have inadvertently bumped into a not uncommon problem--where the two separate datasets share a variable with the identical variable label name. For relatively smaller datasets, and where the researcher is aware of this hurdle ex ante, one can always, by hand, rename variables prior to combining the datasets. For larger datasets with many more variables, however, while this may not be practical it nevertheless remains critical.

A recent discussion of how to manage this nettlesome problem is found here. The comments' leading suggestion references Daniel Klein's -elabel- package, available from SSC (as it involves user-written code). Put briefly, the elable command allows a user to add a suffix (e.g., "_1") to every variable name in "dataset1." An analogous manipulation to datast2 can add another unique suffix (e.g., "_2") to every variable name in dataset2. And so forth. The end result includes datasets with distinct variable label names that can then be combined.

Business Litigants and the Roberts Court

Assessing “How pro business is the Roberts Court?” is both straightforward and complicated. Based on how business litigants fare in the Roberts Court versus non-business litigants over time, business does quite well. The win rate for business in the Roberts Court, 63.4 percent, is 15 percentage points higher than the next highest rate of business wins over the past century (the Rehnquist Court, at 48.3 percent). While it may be tempting for many to conclude that the current pro business tilt is purely a function of a Republican majority of justices on the Roberts Court, as Lee Epstein (USC) and Mitu Gulati (Virginia) argue in a recent paper, A Century of Business in the Supreme Court, 1920-2020, “this is only the beginning of the story.”

Epstein and Gulati bring data from the Supreme Court Database to the question. Starting with the 11,121 orally argued cases resulting in a signed opinion or judgment between the 1920-2020 terms, their paper focuses only on those 4,070 cases (35,173 votes) in which business was the petitioner (appellant) or respondent (appellee). That is, they excluded cases in which business was on both sides of a legal dispute.

Their core findings defy conventional wisdom and inject important complicating wrinkles. What they find includes: "(a) It is not just the Republicans on the Roberts Court who are more pro business than in prior Courts, but the Democrats as well; (b) The Government, through the SG’s office and across both Democratic and Republican administrations, has been much more supportive of business positions than in prior eras; and (c) An elite Supreme Court bar has emerged in recent years and businesses have hired them disproportionately so as to better influence the Court."

Defendant-Prosecutor Gender "Pairings" and Leniency

While various pockets on criminal law are increasingly subject to empirical analyses, one particular pocket--how prosecutors distribute discretion--is comparatively under-studied. This is so, partly, due to a comparative paucity of relevant data on the prosecutors themselves (and matching them to individual cases and dispositions). A recent paper, Gender Favoritism Among Criminal Prosecutors, by Stephanie Holmes Didwania (Wisc.), tries move the scholarly needle a bit by exploiting data on over 130,000 federal criminal defendants sentenced in 2002 through 2016 to assess possible "gender-based differences in how prosecutors exercise critical discretion."

The paper matches Sentencing Commission data and data from the Executive Office for United States Attorneys (EOUSA) which regularly publishes case data from the 93 USAOs located throughout the United States. Critically, the breadth of the LIONS data "is substantial—the data covers all cases in which a USAO was involved, including those cases that are the primary responsibility of another agency and those the USAO declines to prosecute." Despite its breath, the LIONS data set is not without important limitations including a comparatively non-trivial amount of missing data. After matching, the resultant data set data includes "131,207 federal criminal defendants sentenced between 2001 and 2016 in 63 of the 94 federal district courts."

Data set limitations aside, the paper's core finding intrigues: "That defendants receive more lenient charges when they match with their prosecutor on gender relative to when they are assigned a prosecutor of the opposite gender. Through this charging channel, prosecutor-defendant pairs that match on gender produce lower sentences relative to prosecutor-defendant pairs that mismatch." More specifically, "the leniency associated with gender-matching ultimately translates into significantly lower sentences: a defendant can expect to earn a roughly five-month shorter sentence when their prosecutor is their own gender compared to when their prosecutor is the opposite gender, which represents a roughly eight percent reduction in sentence length at the mean." The paper's abstract follows.

"Prosecutors enjoy wide discretion in the decisions they make but are largely unstudied by quantitative empirical scholars. This paper explores gender bias in prosecutorial decision-making. I find that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall but demonstrate relative leniency towards defendants of their own gender. Such favoritism at charging translates into a sentencing gap of roughly five months of incarceration for defendants who are paired with an own-gender prosecutor versus an opposite-gender prosecutor, which represents a roughly eight percent reduction in sentence length at the mean. The estimates do not appear to be driven by differences in case assignments for male and female prosecutors."

Moving From Raw Data to More Helpful Variables

For many research projects scholars have to manipulate existing raw data into more useful derived variables. One obvious approach is "hand-coding," or some related form of Excel file manipulation. That is fine for folks experienced with Excel, but for others (including experienced Excel users) such a task typically invites some degree of human error. Another approach is to rely on Stata coding for variable creation. A recent (and, I hope, building conversation on the State List) illustrates a not an atypical example.

The questioner begins with Current Population Survey data (from the Census Dept.) that include individual-level information on, e.g., year, state, and a dummy variable for employment status (0=employed, 1 = unemployed, 2 = not in labor force). What the CPS data set does not include, however, is a state-level unemployment rate by year and state variable. So, the question is how to most efficiently extract what is desired (the creation of a new state-level unemployment rate by year and state variable) from what exists in raw data form. The discussion thus far has elicited at least one very helpful step-by-step examples of how to accomplish this goal with three lines of code (click here).

To be sure, other options also exist. That said, clear explications of how to move from what one has in terms of data to what one wants in terms of a new variable are sometimes hard to find.