Empirical Legal Studies

From PACER to SCALES: Access to Federal Court Data

Even basic research questions on Art.III courts and federal litigation typically confront consequential barriers to needed data (e.g., federal court records). While the federal government's PACER platform and a few third-party operations provide some relief, both possess their own respective shortcomings and limitations (including, but not limited to, cost).

Initial fruits of a new collaborative effort based at Northwestern University, SCALES, have emerged recently. SCALES is an online platform that seeks to "assemble federal court records, systematically organize them and extract key information, and—most importantly—make them freely available to the public." What exists now includes access to information and records for all federal cases initiated in 2016 and 2017. As SCALES' mission includes "to make court records freely accessible to the public," the platform aspires to, in the future, expand coverage to include "all years."

In a recent paper, The SCALES Project: Making Federal Court Records Free, the authors, David Schwartz (Northwestern) et al., explain the SCALES initiative in greater detail. As well, the project also benefits from its own website (click here).

Improving Stata Performance When Processessing "Large" Amounts of Data

"Big Data" projects that involve processessing larger and larger data sets impose increased loads onto soft/hardware performance. When it comes to moving large amounts of data through various Stata commands, a recent discussion (here) identifies a few quite easy methods for improving computational performance.

High Court Data From 'Down Under'

Thanks to Zoe Robinson (ANU) and Patrick Leslie (ANU), a database of all cases decided by the High Court of Australia between 1995 and 2020, is now publicly-accessible (click here). The database was consciously designed to supply data on cases heard by the High Court of Australia as well as the justices that hear the cases and facilitate empirical analyses.

To this end, the database contains "over 315 variables for every case argued before a panel of at least two justices on the Australian High Court between 1995 and 2020, as well as variables on key identity information about the justices hearing the cases. Examples include justice votes, justice coalitions, legal issues in the case, the parties to the case, justices who granted special leave to appeal, justice gender, and justice ideology, amongst many others." Those intrigued might want to skim the dataset's codebook to get a more granular feel for what's included in the data.

Finally, and notably, the researchers strategically constructed the High Court database with an eye towards facilitating comparative work with the U.S. Supreme Court database in terms of variable categorization and naming.

Generating Lagged Variable Values

The need for lagged variables is not uncommon, particularly in the finance and statutory intervention areas or whenever the effective timing of any exogenous event of interest is unclear. How to manipulate an existing variable to generate lagged variants--as well as some common pitfalls--are discussed in a recent StataList discussion (here).

Various Tests for Fixed- v. Random-Effects Model Decision

When assessing whether a model warrants fixed- or random-effects, threats posed by, among other factors, autocorrelation and heteroscedasticity typically loom large. A recent discussion of this topic emerged on the StataList (click here) and considers various approaches.

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Federal Judicial Data: Greater Transparency or Opacity?

While not itself an empirical paper per se, The Necessary and Proper Stewardship of Judicial Data, by Zachary Clopton (Northwestern) and Aziz Huq (Chicago), engages in a discussion near-and-dear to most legal empiricists' heart. Specifically, their paper sets out to develop "the first comprehensive descriptive, doctrinal, and normative analysis of judicial data.”

As the authors note, judicial (especially Art. III courts) data are "valuable in governance and for those interested in understanding how they are being governed. In the absence of hurdles or walls, it can illuminate the operation of one branch of the national government. Constitutional lapses can be brought to light through retail or wholesale data." As such, data flowing from Art. III courts and institutions can promote various public goods by, in part, advancing a better understanding of, or improvements to, judicial institutions.

One critical anomaly is that, as compared to the legislative and Executive branches, judicial branch data remain "largely terra incognito for scholars and the general public alike. Instead, public attention rushes to high-profile cases of leaked Supreme Court opinions and private influence campaigns.” While why this is so remains unclear (to me, anyway), that it is so, however, is obvious to any empirical legal scholar who labors and writes in this space. And the stakes incident to the "transparency v. opacity" debate grow more acute as new scholarly tools emerge (e.g., natural language processessing, text-scrapping).

As the authors make clear, under current policy and practice judicial data "might be lost, dissipating into air, or kept under wraps, never yielding up their secrets." In contrast, a commitment to greater transparency invites characterizing judicial data as a "new public asset that can then be tapped to create public goods for the benefit of all."

New Stata Graphing "Toys"

While Stata's graphing capabilities are robust, coding remains complex in places. While the coding "load" has come down a bit over the years, the "bells & whistles" have remained relatively stagnant over the years. One recent change on the "bells & whistles" front involves an increased capacity to "colorize" standard graphs. While the example (below) illustrates how levering color in graphics can enhance (and/or clarify) what the data are telling you, I remain curious about how journals (and their publishers) will choose to deal with the increased production costs associated with color in published papers.

Any potential cost issues aside, a recent Stata newsletter (click here) explains and illustrates (with sample coding) how one can transform a standard twoway scatter plot from this:

To this:

Plotting Coefficients

As I'm seeing more examples of coefficient plots [coefplot command] in the literature (see, e.g., below), I thought the following links might interest some readers. For brief discussions and examples (including Stata coding) click here, here, here, and here.

How to Value Data

While a recent paper, Data Valuation and Law, is not itself an "empirical paper," it is certainly "ELS-adjacent" and, as such, I thought it might interest some. This is particularly so as both AI and "big data" become increasingly important.

Motivating Jordan Berry (USC) and D. Daniel Sokol's (USC) paper is the observation that "while data has become increasingly valuable and important, the law’s attempts to value data have lagged, remaining confused and underdeveloped." To improve matters in this regard, the paper advances a “real options” approach for framing data valuation not because the approach is perfect but, rather, because “it is the least bad alternative available.” The paper's abstract follows.

"Data has become an increasingly valuable asset. Numerous areas of law—including contracts, corporate law, IP, antitrust, tax, privacy, and bankruptcy—require parties and courts to determine the value of assets, including data. Unfortunately, data valuation has been hindered by a lack of clarity over what data is and why it is valuable. This lack of clarity also increases the chances of legal decisionmakers valuing data in inconsistent ways, which would create further confusion, inefficiencies, and opportunities for regulatory arbitrage.

This article proposes a unified framework for valuing data that will promote consistent valuations across fields of law. It begins by conceptualizing data as building blocks: It is of little value on its own. But when placed in skillful and creative hands, it can unlock choices for its holders—choices they would not otherwise have—that can generate tremendous profits. Thus, data constitutes what is known as a ‘real option.’ This article shows how using real options to value data can significantly improve upon existing data valuation practices.”

Plotting Kernal Density Estimates as Colored Areas

By default, kernel density estimates are plotted as lines, meaning curves:

According to recent StataList discussions (see here and here), however, plotting kernal density estimates as "areas" has become the standard in some fields. While for publication purposes the coding requires color, if such a possibility exists the resultant information conveyed is quite helpful as the following example illustrates.

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Skewed Data, Histograms & Axis "Breaks"

While skewed data distributions (e.g., a large proportion of zeros) invite an array of well-known methodological complications, they also complicated data presentation needs. For example, a standard histogram for a distribution that includes approx. 85% zeros looks something like the following:

Setting aside the methodological complications imposed by such skewed data, as it narrowly relates to descriptive data presentation needs one "dissenting" approach is to insert axis "breaks" (discussion here) that generates the following histogram:

Such a strategy may provide the optimal tradeoff between the visualization of "extreme" data and compression of the non-extreme data. This approach also has the virtue of keeping the underlying data in their raw rather than a transformed scale.

"Heatplots" for Correlation Coefficients Graphs

While I've never used one myself, I detect an upward trend in "heatplots" (example below) for basic descriptive correlation coefficient graphs.

A helpful discussion (along with coding suggestions) on what is possible with Stata's heatplot command is found here.