Empirical Legal Studies

Options for Small and Unbalanced ("Rare Events") Data Sets

While relogit is a standard "go-to" Stata command when a data set includes "rare events," a threat of bias is enhanced if rare events derive from a "small sample." While bootstraping may be an option for some, a recent Stata listserv discussion (here) emphasizes a pair of commands -- firthlogit and exlogit -- that might make sense for others. Relatedly, Prof. Williams (ND-sociology) helpfully delves into the challenge a bit more comprehensively (here).

SCOTUS Database--Updates

The Supreme Court Database, motivated in part by Prof. Harold Spaeth (1930-2017), remains a leading source for Court and judicial decisionmaking scholars. Indeed, "virtually all systematic analyses of the contemporary Supreme Court and its members have relied on it." Given the Database's scholarly prominence, I wanted to pass along a few, brief updates.

First, the Database, which includes information (247 variables) on every SCOTUS decision since the "beginning" (1791), has now been updated to include information on all SCOTUS cases through the 2023 Term. Second, the Database's "home" has migrated from Washington University and, starting with the current (Oct. Term 2024), will be archived at Penn State University (click here).

From PACER to SCALES: Access to Federal Court Data

Even basic research questions on Art.III courts and federal litigation typically confront consequential barriers to needed data (e.g., federal court records). While the federal government's PACER platform and a few third-party operations provide some relief, both possess their own respective shortcomings and limitations (including, but not limited to, cost).

Initial fruits of a new collaborative effort based at Northwestern University, SCALES, have emerged recently. SCALES is an online platform that seeks to "assemble federal court records, systematically organize them and extract key information, and—most importantly—make them freely available to the public." What exists now includes access to information and records for all federal cases initiated in 2016 and 2017. As SCALES' mission includes "to make court records freely accessible to the public," the platform aspires to, in the future, expand coverage to include "all years."

In a recent paper, The SCALES Project: Making Federal Court Records Free, the authors, David Schwartz (Northwestern) et al., explain the SCALES initiative in greater detail. As well, the project also benefits from its own website (click here).

Improving Stata Performance When Processessing "Large" Amounts of Data

"Big Data" projects that involve processessing larger and larger data sets impose increased loads onto soft/hardware performance. When it comes to moving large amounts of data through various Stata commands, a recent discussion (here) identifies a few quite easy methods for improving computational performance.

High Court Data From 'Down Under'

Thanks to Zoe Robinson (ANU) and Patrick Leslie (ANU), a database of all cases decided by the High Court of Australia between 1995 and 2020, is now publicly-accessible (click here). The database was consciously designed to supply data on cases heard by the High Court of Australia as well as the justices that hear the cases and facilitate empirical analyses.

To this end, the database contains "over 315 variables for every case argued before a panel of at least two justices on the Australian High Court between 1995 and 2020, as well as variables on key identity information about the justices hearing the cases. Examples include justice votes, justice coalitions, legal issues in the case, the parties to the case, justices who granted special leave to appeal, justice gender, and justice ideology, amongst many others." Those intrigued might want to skim the dataset's codebook to get a more granular feel for what's included in the data.

Finally, and notably, the researchers strategically constructed the High Court database with an eye towards facilitating comparative work with the U.S. Supreme Court database in terms of variable categorization and naming.

Generating Lagged Variable Values

The need for lagged variables is not uncommon, particularly in the finance and statutory intervention areas or whenever the effective timing of any exogenous event of interest is unclear. How to manipulate an existing variable to generate lagged variants--as well as some common pitfalls--are discussed in a recent StataList discussion (here).

Various Tests for Fixed- v. Random-Effects Model Decision

When assessing whether a model warrants fixed- or random-effects, threats posed by, among other factors, autocorrelation and heteroscedasticity typically loom large. A recent discussion of this topic emerged on the StataList (click here) and considers various approaches.

New Thinking About an Old Problem: Selection Effects

Keith Hylton (Boston Univ.) and Sanghoon Kim (Buffalo--econ.) distill their recent paper's challenge nicely. When it comes to much of empirical work on legal (and court) outcomes, the full potential of multiple regression "has not been realized, largely because of one reason: the information contained in court opinions comes from a selected sample."

For all the obvious reasons, much of empirical legal research exploits court outcomes as data. One challenge, however, is that the comparatively smaller subset of formal legal disputes that persist to a judicial outcome almost assuredly systematically differs from the much larger subset of legal disputes that privately settle. My former mentor and colleague, Ted Eisenberg, frequently reminded me that "one cannot over-state the importance of selection effects in empirical legal research."

While the challenge endures, solutions are not entirely obvious. In Trial Selection and Estimating Damages Equations, Hylton and Kim develop an interesting suggestion. Specifically, they articulate a structural econometric model that, they argue, better accounts for litigant selections in the appellate context. The paper's abstract follows.

"Many studies have employed regression analysis with data drawn from court opinions. For example, an analyst might use regression analysis to determine the factors that explain the size of damages awards or the factors that determine the probability that the plaintiff will prevail at trial or on appeal. However, the full potential of multiple regression analysis in legal research has not been realized, largely because of the sample selection problem. We propose a method for controlling for sample selection bias using data from court opinions."

Deploying Large Language Models in the Service of Empirical Legal Studies

The release of ChatGPT, among other large language models (LLMs), materially altered the natural language processessing landscape. Questions about whether and, more to the point, how, these tools could advance empirical legal studies quickly emerged and, unsurprisingly, persist.

In a recent paper, Language Model Interpretability and Empirical Legal Studies, Michael Livermore (UVa), et al., undertake two tasks: "(1) to present an empirical comparison of LLMs and non-LLM on a sampling of fundamental tasks, and (2) to examine the use of LLMs within the norms of ELS scholarship." The paper specifically seeks to place "the current capabilities and limitations of LLMs within the larger landscape of techniques used to convert law into data." The authors report that their comparisons yielded "mixed" results, noting that existing LLMs outperform non-LLM natural language processessing techniques in some contexts but not in others. The paper's abstract follows.

"Large language models (LLMs) now perform extremely well on many natural language processing tasks. Their ability to convert legal texts to data may offer empirical legal studies (ELS) scholars a low-cost alternative to research assistants in many contexts. However, less complex computational language models, such as topic modeling and sentiment analysis, are more interpretable than LLMs. In this paper we highlight these differences by comparing LLMs with less complex models on three ELS-related tasks. Our findings suggest that ELS research will—for the time being—benefit from combining LLMs with other techniques to optimize the strengths of each approach."

Federal Judicial Data: Greater Transparency or Opacity?

While not itself an empirical paper per se, The Necessary and Proper Stewardship of Judicial Data, by Zachary Clopton (Northwestern) and Aziz Huq (Chicago), engages in a discussion near-and-dear to most legal empiricists' heart. Specifically, their paper sets out to develop "the first comprehensive descriptive, doctrinal, and normative analysis of judicial data.”

As the authors note, judicial (especially Art. III courts) data are "valuable in governance and for those interested in understanding how they are being governed. In the absence of hurdles or walls, it can illuminate the operation of one branch of the national government. Constitutional lapses can be brought to light through retail or wholesale data." As such, data flowing from Art. III courts and institutions can promote various public goods by, in part, advancing a better understanding of, or improvements to, judicial institutions.

One critical anomaly is that, as compared to the legislative and Executive branches, judicial branch data remain "largely terra incognito for scholars and the general public alike. Instead, public attention rushes to high-profile cases of leaked Supreme Court opinions and private influence campaigns.” While why this is so remains unclear (to me, anyway), that it is so, however, is obvious to any empirical legal scholar who labors and writes in this space. And the stakes incident to the "transparency v. opacity" debate grow more acute as new scholarly tools emerge (e.g., natural language processessing, text-scrapping).

As the authors make clear, under current policy and practice judicial data "might be lost, dissipating into air, or kept under wraps, never yielding up their secrets." In contrast, a commitment to greater transparency invites characterizing judicial data as a "new public asset that can then be tapped to create public goods for the benefit of all."

New Stata Graphing "Toys"

While Stata's graphing capabilities are robust, coding remains complex in places. While the coding "load" has come down a bit over the years, the "bells & whistles" have remained relatively stagnant over the years. One recent change on the "bells & whistles" front involves an increased capacity to "colorize" standard graphs. While the example (below) illustrates how levering color in graphics can enhance (and/or clarify) what the data are telling you, I remain curious about how journals (and their publishers) will choose to deal with the increased production costs associated with color in published papers.

Any potential cost issues aside, a recent Stata newsletter (click here) explains and illustrates (with sample coding) how one can transform a standard twoway scatter plot from this:

To this:

Plotting Coefficients

As I'm seeing more examples of coefficient plots [coefplot command] in the literature (see, e.g., below), I thought the following links might interest some readers. For brief discussions and examples (including Stata coding) click here, here, here, and here.

How to Value Data

While a recent paper, Data Valuation and Law, is not itself an "empirical paper," it is certainly "ELS-adjacent" and, as such, I thought it might interest some. This is particularly so as both AI and "big data" become increasingly important.

Motivating Jordan Berry (USC) and D. Daniel Sokol's (USC) paper is the observation that "while data has become increasingly valuable and important, the law’s attempts to value data have lagged, remaining confused and underdeveloped." To improve matters in this regard, the paper advances a “real options” approach for framing data valuation not because the approach is perfect but, rather, because “it is the least bad alternative available.” The paper's abstract follows.

"Data has become an increasingly valuable asset. Numerous areas of law—including contracts, corporate law, IP, antitrust, tax, privacy, and bankruptcy—require parties and courts to determine the value of assets, including data. Unfortunately, data valuation has been hindered by a lack of clarity over what data is and why it is valuable. This lack of clarity also increases the chances of legal decisionmakers valuing data in inconsistent ways, which would create further confusion, inefficiencies, and opportunities for regulatory arbitrage.

This article proposes a unified framework for valuing data that will promote consistent valuations across fields of law. It begins by conceptualizing data as building blocks: It is of little value on its own. But when placed in skillful and creative hands, it can unlock choices for its holders—choices they would not otherwise have—that can generate tremendous profits. Thus, data constitutes what is known as a ‘real option.’ This article shows how using real options to value data can significantly improve upon existing data valuation practices.”

Plotting Kernal Density Estimates as Colored Areas

By default, kernel density estimates are plotted as lines, meaning curves:

According to recent StataList discussions (see here and here), however, plotting kernal density estimates as "areas" has become the standard in some fields. While for publication purposes the coding requires color, if such a possibility exists the resultant information conveyed is quite helpful as the following example illustrates.

"Anecdotes," Data, and Research Design

As I have neither an interest in nor inclination to engage substantively with an on-going scholarly "dust-up," I am content to let two papers that recently "crossed my desk" speak for themselves (compare Church & Williams with Baker & Bradt). That said, and independent of this exchange in particular and multidistrict litigation in general, a few points raised by Baker (Texas) and Bradt (Berkeley), in Anecdotes Versus Data in the Search for Truth about Multidistrict Litigation, have broader applicability.

One goal of empirical research, generally stated, is to generate results that are persuasively and adequately scaffolded by the underlying data and research design. Whether a "convenience sample" eliciting self-reports from an on-line survey can reflect an unbiased, random, and representative draw from an underlying population universe is not a given and requires further analyses. That is, for such a data-generating strategy the specter of selection bias lurks and must be addressed.

Moreover, even if one is comfortable that neither selection bias nor other data shortcomings pose methodological problems, additional research design issues persist. To be more specific, and as Baker and Bradt note, if a research question seeks to meaningfully engage with, e.g., plaintiffs' satisfaction with MDL litigation experience, data on respondents' (that is, plaintiffs in MDL litigation) satisfaction (or dissatisfaction) should be compared with an otherwise equally representative and unbiased sample of plaintiffs from non-MDL cases. Without an appropriate control group, the results lack a necessary reference point that facilitates interpreting the results.

Skewed Data, Histograms & Axis "Breaks"

While skewed data distributions (e.g., a large proportion of zeros) invite an array of well-known methodological complications, they also complicated data presentation needs. For example, a standard histogram for a distribution that includes approx. 85% zeros looks something like the following:

Setting aside the methodological complications imposed by such skewed data, as it narrowly relates to descriptive data presentation needs one "dissenting" approach is to insert axis "breaks" (discussion here) that generates the following histogram:

Such a strategy may provide the optimal tradeoff between the visualization of "extreme" data and compression of the non-extreme data. This approach also has the virtue of keeping the underlying data in their raw rather than a transformed scale.

"Heatplots" for Correlation Coefficients Graphs

While I've never used one myself, I detect an upward trend in "heatplots" (example below) for basic descriptive correlation coefficient graphs.

A helpful discussion (along with coding suggestions) on what is possible with Stata's heatplot command is found here.

Importing Non-Stata Data Sets Into Stata

While perhaps old-hat for some (many), a quick new Stata tutorial video (click here) walks viewers through some of the basics incident to data importation. For old-time users (including those who may reflexively turn to StatTransfer programs for conversions), increasingly easy--and a wider array of--data transfer options remain much-welcome news.

A Small--But Important--Logistical Matter Incident to Dataset Merging

Those who have merged (or appended or crossed) datasets may have inadvertently bumped into a not uncommon problem--where the two separate datasets share a variable with the identical variable label name. For relatively smaller datasets, and where the researcher is aware of this hurdle ex ante, one can always, by hand, rename variables prior to combining the datasets. For larger datasets with many more variables, however, while this may not be practical it nevertheless remains critical.

A recent discussion of how to manage this nettlesome problem is found here. The comments' leading suggestion references Daniel Klein's -elabel- package, available from SSC (as it involves user-written code). Put briefly, the elable command allows a user to add a suffix (e.g., "_1") to every variable name in "dataset1." An analogous manipulation to datast2 can add another unique suffix (e.g., "_2") to every variable name in dataset2. And so forth. The end result includes datasets with distinct variable label names that can then be combined.

Business Litigants and the Roberts Court

Assessing “How pro business is the Roberts Court?” is both straightforward and complicated. Based on how business litigants fare in the Roberts Court versus non-business litigants over time, business does quite well. The win rate for business in the Roberts Court, 63.4 percent, is 15 percentage points higher than the next highest rate of business wins over the past century (the Rehnquist Court, at 48.3 percent). While it may be tempting for many to conclude that the current pro business tilt is purely a function of a Republican majority of justices on the Roberts Court, as Lee Epstein (USC) and Mitu Gulati (Virginia) argue in a recent paper, A Century of Business in the Supreme Court, 1920-2020, “this is only the beginning of the story.”

Epstein and Gulati bring data from the Supreme Court Database to the question. Starting with the 11,121 orally argued cases resulting in a signed opinion or judgment between the 1920-2020 terms, their paper focuses only on those 4,070 cases (35,173 votes) in which business was the petitioner (appellant) or respondent (appellee). That is, they excluded cases in which business was on both sides of a legal dispute.

Their core findings defy conventional wisdom and inject important complicating wrinkles. What they find includes: "(a) It is not just the Republicans on the Roberts Court who are more pro business than in prior Courts, but the Democrats as well; (b) The Government, through the SG’s office and across both Democratic and Republican administrations, has been much more supportive of business positions than in prior eras; and (c) An elite Supreme Court bar has emerged in recent years and businesses have hired them disproportionately so as to better influence the Court."

Defendant-Prosecutor Gender "Pairings" and Leniency

While various pockets on criminal law are increasingly subject to empirical analyses, one particular pocket--how prosecutors distribute discretion--is comparatively under-studied. This is so, partly, due to a comparative paucity of relevant data on the prosecutors themselves (and matching them to individual cases and dispositions). A recent paper, Gender Favoritism Among Criminal Prosecutors, by Stephanie Holmes Didwania (Wisc.), tries move the scholarly needle a bit by exploiting data on over 130,000 federal criminal defendants sentenced in 2002 through 2016 to assess possible "gender-based differences in how prosecutors exercise critical discretion."

The paper matches Sentencing Commission data and data from the Executive Office for United States Attorneys (EOUSA) which regularly publishes case data from the 93 USAOs located throughout the United States. Critically, the breadth of the LIONS data "is substantial—the data covers all cases in which a USAO was involved, including those cases that are the primary responsibility of another agency and those the USAO declines to prosecute." Despite its breath, the LIONS data set is not without important limitations including a comparatively non-trivial amount of missing data. After matching, the resultant data set data includes "131,207 federal criminal defendants sentenced between 2001 and 2016 in 63 of the 94 federal district courts."

Data set limitations aside, the paper's core finding intrigues: "That defendants receive more lenient charges when they match with their prosecutor on gender relative to when they are assigned a prosecutor of the opposite gender. Through this charging channel, prosecutor-defendant pairs that match on gender produce lower sentences relative to prosecutor-defendant pairs that mismatch." More specifically, "the leniency associated with gender-matching ultimately translates into significantly lower sentences: a defendant can expect to earn a roughly five-month shorter sentence when their prosecutor is their own gender compared to when their prosecutor is the opposite gender, which represents a roughly eight percent reduction in sentence length at the mean." The paper's abstract follows.

"Prosecutors enjoy wide discretion in the decisions they make but are largely unstudied by quantitative empirical scholars. This paper explores gender bias in prosecutorial decision-making. I find that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall but demonstrate relative leniency towards defendants of their own gender. Such favoritism at charging translates into a sentencing gap of roughly five months of incarceration for defendants who are paired with an own-gender prosecutor versus an opposite-gender prosecutor, which represents a roughly eight percent reduction in sentence length at the mean. The estimates do not appear to be driven by differences in case assignments for male and female prosecutors."

Moving From Raw Data to More Helpful Variables

For many research projects scholars have to manipulate existing raw data into more useful derived variables. One obvious approach is "hand-coding," or some related form of Excel file manipulation. That is fine for folks experienced with Excel, but for others (including experienced Excel users) such a task typically invites some degree of human error. Another approach is to rely on Stata coding for variable creation. A recent (and, I hope, building conversation on the State List) illustrates a not an atypical example.

The questioner begins with Current Population Survey data (from the Census Dept.) that include individual-level information on, e.g., year, state, and a dummy variable for employment status (0=employed, 1 = unemployed, 2 = not in labor force). What the CPS data set does not include, however, is a state-level unemployment rate by year and state variable. So, the question is how to most efficiently extract what is desired (the creation of a new state-level unemployment rate by year and state variable) from what exists in raw data form. The discussion thus far has elicited at least one very helpful step-by-step examples of how to accomplish this goal with three lines of code (click here).

To be sure, other options also exist. That said, clear explications of how to move from what one has in terms of data to what one wants in terms of a new variable are sometimes hard to find.

The Federal Circuit Dataset project (v.3)

While Jason Rantanen's (Iowa) third update to The Federal Circuit Dataset Project will certainly interest empirical IP scholars, the Project's broader implications speak to empirical law scholars in all fields.

The third edition advances the existing Compendium of Federal Circuit Decisions in at least two main ways. First, the dataset now includes document numbers for all archived documents. Second, the dataset now facilitates increased understanding about what happens to appeals, including information about appeal pendency and origin.

Setting aside the dataset's utility to empirical IP scholars, the Project itself reflects a growing commitment to the hard and often under-rewarded work of dataset creation as well as an impulse to make such data (and related codebooks, etc.) publicly-available. Such impulses need to be rewarded, promoted, and nurtured. The paper's abstract follows.

"This paper introduces the third generation of the Federal Circuit Dataset Project. The third generation builds on the existing Compendium of Federal Circuit Decisions with the following enhancements:

• The addition of a dataset of all dockets for the Federal Circuit since 2000. These dockets are linked to the documents in the Compendium through the Appeal_Dockets field in both datasets.
• New fields containing additional information about Federal Circuit decisions.
• More complete data in existing fields that require manual coding.
• Parallel archiving of the document and docket datasets on the Dataverse.

In addition to describing the third generation of the Federal Circuit Dataset Project, this paper also provides updated descriptive statistics about Federal Circuit decisions and new statistics on appeal dockets."

An Updated Database of State Tort Law Reforms (7th)

Ronen Avraham (Tel Aviv Univ./Texas) recently alerted me to an update of one of the more helpful resources around for those who study torts and tort reform. Specifically, the Database of State Tort Law Reforms (7.0), seeks to create "one 'canonized' dataset will increase our understanding of tort reform’s impacts on our lives." A fuller (though excerpted) description follows.

"This This manuscript of the DSTLR (7th) updates the DSTLR (6th) and contains the most detailed, complete and comprehensive legal dataset of the most prevalent tort reforms in the United States between 1980 and 2020.... The dataset records state laws in all fifty states and the District of Columbia over the last several decades. For each reform we record the effective date, a short description of the reform, whether or not the jury is allowed to know about the reform, whether the reform was upheld or struck down by the states’ courts, as well as whether it was amended by the state legislator. "

Data Resources Courtesy of the APSA Law & Courts Section

Our good friends and political science colleagues over at the APSA's Law & Courts Section recently updated the Section's website (click here). What is especially helpful is a sub-page (here) that provides links to the leading court data sets (SCOTUS, CA, district courts, as well as state supreme courts) along with related data assets (e.g., Martin-Quinn Scores, Judicial Common Space Scores).

Bridge Ideal Points--2021 Update

Michael Bailey (Georgetown--Govt.) has updated the Bridge Ideal Points estimates (click here) to cover years 1950 through 2020. These Ideal point estimates include the Supreme Court, presidential, and congressional ideology estimates and are designed to facilitate inter-institutional and cross-temporal comparisons.

While the Bridge Ideal Points compete somewhat with the Martin-Quinn scores in the literature, the two sets of ideology estimates correlate reasonably highly with one another. Despite the overlap, these two ideology estimates differ in two important ways.First, "Bridge Ideal Points are designed to be intertemporally comparable. Martin and Quinn scores assume the ideological cutpoints of Supreme Court cases are drawn from a common distribution across years, which makes them less comparable across time when the case load changes." Second, while Martin and Quinn scores only cover the Supreme Court justices, Bridge Ideal Points for justices are comparable to Bridge Ideal Points for presidents and members of Congress as well.

Regardless of one's preference, efforts to estimate justice ideology remain a staple in the judicial decisionmaking literature.

Data Familiarization/Visualization

A recent post by Jessica Hullman (here) served as my most recent periodic reminder about the enduring need for folks (myself included) to take a few, basic, simple steps to become more familiar with the underlying structure of and relationships among critical data points. Tukey's elegant tables and figures notwithstanding, what is perhaps more important is what follows from Tukey's general impulse: Sometimes simple data plots can unlock key analytic insights. And even if such insights aren't necessary for the researcher, readers' (and journal editors') interests should not be taken for granted.

In trying to explain why researchers might ignore--or bypass--simple initial data visualization/familiarization steps, Hullman points to "latent beliefs that plotting data is somehow inferior to anything that feels more like math. And because making plots for the sake of checking what the data look like or discovering things you didn’t expect seems disconnected and even distracting from the ultimate goal of fitting some model." That such beliefs may be entirely understandable, however, does not detract from their potential harm. It remains awfully difficult to overestimate the value of simple, Tukey-inspired data familiarization/visualization efforts.

JELS' Research Notes and Datasets Section

If memory serves, a suggestion in 2016 (or so) from Bernie Black (Northwestern) and Marty Wells (Cornell-Statistics) to the JELS editors set into motion a chain of events that culminated in 2017 with the introduction of the "Research Notes and Datasets" section in JELS. The core motivation behind introducing a new paper "genre" for JELS was a fundamental belief that dataset construction, data sharing, and methods testing are a vital part of ELS' growth. We felt that a new section in JELS would help contribute to the realization of such beliefs. What we did not fully appreciate or anticipate, however, was the response. Since the Research Notes and Datasets section's launch in 2017 JELS has already published a number of important papers, including:

Patent Similarity Data and Innovation Metrics, by Ryan Whalen Alina Lungeanu Leslie DeChurch Noshir Contractor JELS 2020 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12261

Developing High‐Quality Data Infrastructure for Legal Analytics: Introducing the Israeli Supreme Court Database, by Keren Weinshall Lee Epstein JELS 2020 (June). https://onlinelibrary.wiley.com/doi/10.1111/jels.12250

The German Federal Courts Dataset 1950–2019: From Paper Archives to Linked Open Data, by Hanjo Hamann Sokol JELS 2019 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12230

Competition Law Gone Global: Introducing the Comparative Competition Law and Enforcement Datasets, by Anu Bradford Adam S. Chilton Christopher Megaw Nathaniel Sokol JELS 2019 (June). https://onlinelibrary.wiley.com/doi/10.1111/jels.12215

Accuracy and Fairness for Juvenile Justice Risk Assessments, by Richard Berk JELS 2019 (March). https://onlinelibrary.wiley.com/doi/10.1111/jels.12206

Text of Trade Agreements (ToTA)—A Structured Corpus for the Text‐as‐Data Analysis of Preferential Trade Agreements, by Wolfgang Alschner Julia Seiermann Dmitriy Skougarevskiy  JELS 2018 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12189

Medical Liability Insurance Premia: 1990–2016 Dataset, with Literature Review and Summary Information, by Bernard Black Jeanette W. Chung Jeffrey Traczynski Victoria Udalova Sonal Vats, JELS 2017 (March). https://onlinelibrary.wiley.com/doi/10.1111/jels.12146

A New Tool for Detecting Various Characters in Coding

While a new user-written command, chartab, (briefly described and discussed here) may be criticized as a "command looking for a problem to solve," but for anyone who has ever tried to "de-bug" code a quick tabulation of various "characters" can help identify, and thus solve, problems. These problems include, e.g., the letter l for the number 1; the letter O for the number 0 (zero). These problems typically arise in the context of importing data, especially if it involves string data transited via Microsoft Word or Excel. And anyone merging data based on string variables should probably invest some time in some degree of "Unicode due diligence" before hand to help get in front of any potential problems. To install this command:

ssc install chartab

State Supreme Court Elections Data Set

Judicial decisionmaking scholars comparing outcomes from "elected" and "non-elected" judges invariably turn to state court systems (if the analytic focus is domestic) for the necessary variation. Those turning to the array of state court systems--and the range of state supreme court election types--invariably confront data gathering challenges that materially differ from challenges associated with, e.g., Art. III judges. One of the leading state court scholars, Bert Kritzer (Minn.), continues to make state court election data gathering efforts a bit easier for the rest of us. Not only has Bert engaged in the often under-appreciated yet Herculean effort of original data set building, but he shares his data sets (and associated documentation, codebooks, etc.) with other interested scholars. Bert's most recent updating effort, curated on Harvard's Dataverse, includes a collection of state supreme court election results that now range from 1946 through 2020.

"The archive includes three core datasets plus state-level Excel spreadsheets for all states that employed elections at least part of the time since 1946 to select or retain members of the state supreme court (known New York state as the Court of Appeals) including for Texas and Oklahoma both the state supreme court and the Court of Criminal Appeals which are the final courts for criminal cases in those states."

Those interested in the data sets can click here.

SEC Enforcement Decisions Over Time

While one of the SEC's core duties includes the important task of enforcing various securities regulations, "relatively little empirical work exists assessing the choices the SEC makes on enforcement on an ongoing basis." A paucity of readily-available data likely helps stunt such assessments. Even though the SEC reports annual enforcement statistics, the granular, internal information upon which SEC enforcement decisions rest remains largely "opaque to outside observers."

A recent paper, Measuring the Impact of SEC Enforcement Decisions, by Stephen Choi (NYU), seeks to reduce the opacity by generating various SEC enforcement decision metrics that, in turn, might form the foundation for more sophisticated analyses in the future. To this end, Choi brings together 851 SEC enforcement "actions against public companies and subsidiaries of public companies" between 2005 and 2018, inclusive. From this data set Choi descriptively analyses such aspects as SEC enforcement actions and various perspectives on stock price reactions to these SEC actions. While Choi's presentation of data is descriptive, sometimes even the descriptive can identify rich targets for deeper analyses. To illustrate, Figure 3 (below) presents SEC actions, over time, and by month.

As Choi discusses, Figure 3 makes clear something of a "September Swell" and an "October Ebb." While what accounts for these odd ("hockey stick") distributions is not entirely obvious, a picture of the raw distributions across months and over time is necessary to identify the need for a deeper dive. The paper's abstract follows.

“This essay examines several metrics of Securities and Exchange Commission (SEC) decisionmaking that may be updated on a regular basis over time using publicly available data to give a picture of how SEC decisionmaking changes over time. I focus on SEC actions against public companies and subsidiaries of public companies and use data from the 2005 to 2018 period. The metrics include: the number of SEC actions per year and per month, the ratio of SEC actions to securities class actions by year, the mean abnormal return from an event study of the first public announcement of the problem that led to the eventual SEC enforcement action by year, the fraction of SEC actions with prior disclosure of the underlying violation by year, and the variability of the market response to the initiation of a SEC action by year. The metrics cannot demonstrate with certainty what motivates internal SEC enforcement decisions but may raise questions that guide future research.”

Data on Testacy and Intestacy in Florida

Filtered by my admitted soft spot for scholars who engage in the hard, unglamorous, and often under-appreciated tasks incident to data creation/collection, a recent paper taking an initial descriptive cut at some archival probate data caught my eye. In The Demographics of Intergenerational Transmission of Wealth: An Empirical Study of Testacy and Intestacy on Family Property, Danaya Wright (Florida), recognizing that "[e]veryone dies eventually, and the vast majority of us will die owning at least some property," digs into probate record archives to assess various trends. Exploiting all (N=408) estates that went through court-supervised probate in Alachua County (FL) in 2013, the paper focuses on such variables as wealth, age, number of children, race, sex and marital status, with an eye toward potential systematic differences between testate (293) and intestate (115) decedents. The paper reports "clear differences between testate and intestate decedents on the basis of race, sex, wealth, marital status, age, and even children, with testacy correlating to significantly more wealth than intestacy."

Appropriately mindful of data and research design limitations, the paper also conveys a welcome level of cautiousness and modesty in terms of its ability to scaffold firm conclusions. After all, the data used in the paper draw from one year (2013) in a single county in one state (FL). As well, while the paper benefits from all of estates submitted to probate, most (77.3%) of the deaths in Alachua County in 2013 did not trigger probate or were probated elsewhere. While important external validity and selection effects' questions linger, the paper nonetheless illustrates the scholarly potential of exploiting data that too often remain "hidden in plain sight." The paper's abstract follows.

“In this article, I analyze many of the differences between testate and intestate decedents and suggest areas of concern for law- and policy-makers, as well as practicing lawyers and financial planners. Identifying the right demographic at risk for wealth deterioration may allow for interventions that can slow or reverse the widening wealth gap and ultimately help the next few generations build economically stable lives. To that end, I discuss how a number of demographic differences (wealth, age, race, sex, and marital status) affect the downward trajectory of certain groups and the upward trajectory of others.”

"Quick-and-Dirty" Codebooks

While it's difficult to over-state the importance--indeed, necessity--of an accurate, readable codebook, what is also true is that they can be a "pain" to generate (especially if one waits until the very end of a project). Stata can generate a "bare-bones," though perfectly serviceable, codebook with relatively little effort. A helpful "how-to" is included in this discussion (here).

Assessing Normal Distributions

A nice illustrative discussion (here) on various approaches to assess whether "normal" distribution assumptions plausibly hold. Participants in the discussion provide a helpful illustration (below) on two perspectives (kernel density vs. qnorm) on a possible bi-modal distribution.

AI and Machine Learning: Algorithms and the Law

A pair of papers that crossed my desk recently, both circling around a similar topic and yet providing slightly different perspectives and analyses, reflects a palpable increase in scholarly attention to the various intersections involving AI, machine learning, algorithms, and legal doctrine. As both papers note, "predictive algorithms are increasingly being deployed in a variety of settings to determine legal status." To date, for example, "there are approximately 60 risk assessment tools deployed in the criminal justice system. These tools aim to differentiate between low-, medium-, and high-risk defendants and to increase the likelihood that only those who pose a risk to public safety or are likely to flee are detained."

"Proponents of actuarial tools claim that these tools meant to eliminate human biases and to rationalize the decision-making process by summarizing all relevant information in a more efficient way than the human brain. Opponents of such tools fear that in the name of science, actuarial tools reinforce human biases, harm defendants’ rights and increase racial disparities in the system. The gap between the two camps has widened in the last few years."

One paper, Dan Burk's (UC-Irving), Algorithmic Legal Metrics, seeks to: "link the the sociological and legal analysis of AI, highlighting the reflexive social processes that are engaged by algorithmic metrics. This paper examines these overlooked social effects of predictive legal algorithms, and contributes to the literature a vital fundamental but missing critique of such analytics. Specifically, this paper shows how the problematic social effects of algorithmic legal metrics extend far beyond the concerns about accuracy that have thus far dominated critiques of such metrics. Second, it demonstrates that corrective governance mechanisms such as enhanced due process or transparency will be inadequate to remedy such corrosive effects, and that some such remedies, such as transparency, may actually exacerbate the worst effects of algorithmic governmentality. Third, the paper shows that the application of algorithmic metrics to legal decisions aggravates the latent tensions between equity and autonomy in liberal institutions, undermining democratic values in a manner and on a scale not previously experienced by human societies. Illuminating these effects casts new light on the inherent social costs of AI metrics, particularly the perverse effects of deploying algorithms in legal systems."

Another paper, doaa Abu Elyounes' (Harvard—Berkman Center), Bail or Jail? Judicial Versus Algorithmic Decision-Making in the Pretrial System, delves a bit more concretely into the criminal law realm. The paper "examines the role that the technology play in this debate, and whether deploying AI in existing risk assessment tools realizes the fears hyped in the media or improves our criminal justice system? It focuses on the pretrial stage and examines in depth the seven most commonly used tools. Five of these tools are based on traditional regression analysis, and two have a certain machine-learning component. The paper concludes that, classifying pretrial risk assessment tools as AI-based tools creates the impression that sophisticated robots are taking over the courts and pushing judges from their jobs, but this is far from reality. Despite the hype, there are more similarities than differences between tools based on traditional regression analysis and tools based on machine learning. Robots have a long way to go before they can replace judges, and this is not the solution that this paper is arguing for."

Scatterplots With "Weighted" Markers

For those with the time and inclination to delve deeply into graph "presentation" issues, a (related) pair of discussion strings from the Statalist (here and here), discuss how to generate scatter plots with "weighted" markers to assist readers with "visualization for multidimensional data." The resultant output can look nice:

Graphing Counterfactual Regression Trend Lines

In a recent Statlist discussion (here), the question was posed about how to: "generate a graph to show how to the GDP per capita trend line would have continued if the conflict [or any discrete event] had not occurred." (emphasis added) Put slightly differently, the question involves how to over-lay two separate time-trend lines: one from actual data reflecting the influence of a discrete event and another trend line reflecting a counterfactual regression trend (that is, as if the discrete event did not happen).

To illustrate, the graph (above) plots and compares "the effect that the civil conflict in Mozambique and Rwanda have had on the GDP per capita growth in the long run" and over-lays a projected regression line reflecting GDP per capita assuming that the civil conflict did not emerge in these two countries.

While the Stata graphics coding can be tricky, the end result is a quickly interpretable graph.

"Zeros" Problems

While not unique to empirical legal scholarship, how to deal with a "zeros problem" is not uncommon, especially given the nature of many research designs and questions in legal scholarship. To take one obvious--and well-trod--example, efforts to model punitive damage awards need to account for their absence in the over-whelming majority of civil cases decided by courts. A recent discussion on the StataList (here) provides yet another take on the array of challenges posed by "zeros" in data sets. The resultant discussions emphasizes, in part, that researchers need to assess with accuracy and clarity, ex ante, precisely what a "zero" means in the context of their data sets. As well, for a recent take on differences between Heckman selection and two-part models, a 2015 JELS paper (here), by Eisenberg et al., remains current.

Ready-To-Use Data (and Codebooks) for Law & Education Research

One thing that the U.S. Department of Education does reasonably well is gather data--and make them publicly available. And as the number and complexity of law and education issues continue to increase over time, greater access to relevant data becomes even more important. To this end, the Dept. of Education's National Center for Education Statistics ("NCES") operates a landing spot (here) for direct access to ready-to-use public use data sets and codebooks. Users can select from an array of standard formats (Stata, SPSS, SAS, R, etc.) and tailor requests.

Comparative Competition Law & Datasets

Consistent with JELS' impulse to recently launch a new genre of papers--"Research Notes and Datasets"--a recent contribution to this initiative, Competition Law Gone Global: Introducing the Comparative Competition Law and Enforcement Datasets, by Anu Braford (Columbia), Adam Chilton (Chicago), Christopher Megaw (WilmerHale), and Nathaniel Sokol, introduces and discusses a pair of datasets that provide future researchers with "a new foundation for empirical research on the legal regimes used to regulate markets."

"Competition law has proliferated around the world. Due to data limitations, however, there is little systematic information about the substance and enforcement of these laws. In this article, we address that problem by introducing two new datasets on competition law regimes around the world. First, we introduce the Comparative Competition Law Dataset, which codes competition laws in 131 jurisdictions between 1889 to 2010. Second, we introduce the Comparative Competition Enforcement Dataset, which provides data on competition agencies’ resources and activities in 100 jurisdictions between 1990 and 2010. These datasets offer the most comprehensive picture of competition law yet assembled and provide a new foundation for empirical research on the legal regimes used to regulate markets."

So far, so good. But the authors recently undertook yet another step that warrants attention (and praise). Specifically, Bradford and Chilton created a new website (here) that already includes an array of ready-to-use datasets (and associated codebooks) germane to comparative competition law (see, e.g., here). According to the project's website, "this project generates novel data on competition regimes across time and countries. Using these data, the project studies competition law and enforcement around the world, including exploring the diffusion of these laws and the effects they have on market outcomes."

Frankly, the construction and public distribution of helpful and important datasets are among the most the most under-appreciated of endeavors. Future scholars (and scholarship) will gain from initiatives like this one.

MTurk and Evidence on Measurement Error

While questions surrounding the potential for systematic measurement error from MTurk samples cannot be easily resolved in a single paper, given the popularity of MTurk samples in many fields, notably psychology, careful attention scholarly is warranted. The initial "warning" about MTurk samples came in 2013 from Dan Kahan (Yale), particularly for studies seeking to test “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.” Since then we've seen a developing literature that sets out to test potential limitations to MTurk samples. A recent paper by Douglas Ahler (FSU-Poli Sci ) et al., The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk, focuses on the potential for measurement error, especially Type II errors, injected by, among other factors, the "presence of 'non-respondents' (bots) or non-serious respondents on the platform." The paper's core findings include:

"While we find no evidence of a 'bot epidemic,' we do find that a significant portion of survey respondents engaged in suspicious behavior. About 20% of respondents either circumvented location requirements or took the survey multiple times. In addition, at least 5-7% of participants likely engaged in 'trolling' or satisficing. Altogether, we find about a quarter of data collected on MTurk is potentially untrustworthy. Expectedly, we find response quality impacts experimental treatments. On average, low quality responses attenuate treatment effects by approximately 9%. We conclude by providing recommendations for collecting data on MTurk."

Balanced v. Unbalanced Panel Data; "reg" v. "xtreg" Commands--Asking the Right Question

An admittedly small but nonetheless important point emerged in a recent StataList discussion (here). While the original query wondered whether a decision between the "reg" and "xtreg" commands pivoted on whether panel data were balanced or unbalanced, a very helpful commenter quickly made clear the question's pivotal (and mistaken) assumption. To be sure, while the "reg" and "xtreg" commands may yield the same or similar results in certain conditions, they estimate different models and different parameters. In general, where panel data are involved (balanced or unbalanced), as the commenter notes,"-reg- is simply inappropriate."

Tips for More Readable Graphs

Over at Andrew Gelman's blog, John Kastellec (Princeton--Poli Sci) ignited a discussion (here) about "common but easily avoidable graphical mistakes" that hinder graphs' readability. While some suggestions are obvious, others are not. As well, the comments to John's post are as important and helpful as the post itself. When all is said and done, however, for my money the most important contribution to this topic remains Tufte's classic: The Visual Display of Quantitative Information (2nd ed.).

Advances in Data Selection: Evaluating Police Conduct

While agreement on a common set of metrics facilitates longitudinal studies as well as comparisons across studies, these benefits should not deter fresh new thinking about new sources of data that inform lingering research questions. A new paper by Aurelie Ouss (Penn--criminology) and John Rappaport (Chicago), Is Police Behavior Getting Worse? The Importance of Data Selection in Evaluating the Police, illustrates this point nicely. In it, the authors' analysis of police conduct draws from traditional data sources, including litigation levels and municipal settlement payouts, as well as a relatively new data source: liability insurance claims.

Exploiting 23 years worth of liability insurance data, involving 350 law enforcement agencies and a single insurance carrier, the paper finds that "while lawsuits and [settlement] payouts have trended upwards over the past decade, insurance claims have declined." What the authors conclude is that: "... police behavior is not getting worse; rather, public responses to policing harms are intensifying."

Stata Graphic "Trick" of the Day

For too many (myself included), Stata graphics can be among the more complicated coding tasks. Thus, concrete coding tips for some of the more "exotic" graphics capabilities invariably warrant attention (even if only to satisfy curiosity). In this spirit I pass along a brief discussion concerning coding requirements for a "fan" chart in Stata (here). In an example of what is possible (below), the presentation of a standard "line" chart is graphically enhanced with +/- 1, 2, and 3 standard error "cones."

Database of State Tort Law Reforms (v.6.1)

Ronen Avraham (Tel Aviv/Texas) recently alerted me to an update of one of the more helpful resources around for those who study torts and tort reform. Specifically, the (DSTLR v.6.1) "updates the DSTLR (5th) and contains the most detailed, complete and comprehensive legal dataset of the most prevalent tort reforms in the United States between 1980 and 2018. The DSTLR has been downloaded more than 2700 times and has become the standard tool in empirical research of tort reform. The dataset records state laws in all fifty states and the District of Columbia over the last several decades...." For direct access to the dataset (in Excel) click here.

Manipulating Date/Time Variables

An often needed--yet difficult to create--variable is a "duration" variable that seeks to capture time (or number of days) and is derived by "subtracting," e.g., a start-date from an end-date. This computation's degree of difficulty is enhanced by complexities incident to how dates (and time) are formatted in Stata. Those interested can find a recent step-by-step explanation (and illustration) here.

The Visual Display of Data--"Graphs v. Tables"

This enduring "chestnut" of an issue warrants periodic refreshing. An on-going discussion is found here. Those interested in the long-standing “classic” treatment of such issues, Edward Tufte’s, The Visual Display of Quantitative Information (2^nd ed., 2001) delves into the “theory and practice in the design of data graphics,” and includes illustrated examples of the “best (and a few of the worst) statistical graphics, with detailed analysis of how to display data for precise, effective, quick analysis.” Finally, in a 2011 essay Andrew Gelman (Columbia--Statistics) wades into these matters with a normative (and intentionally provocative) take favoring "tables over graphs." The Gelman paper's abstract follows.

"The statistical community is divided when it comes to graphical methods and models. Graphics researchers tend to disparage models and to focus on direct representations of data, mediated perhaps by research on perceptions but certainly not by probability distributions. From the other side, modelers tend to think of graphics as a cute toy for exploring raw data but not much help when it comes to the serious business of modeling. In order to better understand the benefits and limitations of graphs in statistical analysis, this article presents a series of criticisms of graphical methods in the voice of a hypothetical old-school analytical statistician or social scientist. We hope to elicit elaborations and extensions of these and other arguments on the limitations of graphics, along with responses from graphical researchers who might have different perceptions of these issues."

Data & Property Rights

While the array of issues incident to "data" and property rights (or "ownership") continues to grow both in number and complexity, many of the immediate, practical questions relate to the evolving relation between empirical scholars and their data sets and journal editors when it comes to the manuscript review process. A wider-angle lens, however, as well as an international perspective, are developed in a recent, brief working paper/report, Data Ownership. In it, Teresa Scassa (Ottawa) takes up broader questions concerning data "ownership," what ownership may mean, and how whatever it may mean can vary across jurisdictions. While certainly not the last word on these complex issues, an interesting take and perspective on a set of issues that will not likely go away anytime soon.

Importing (and "Destring"-ing) Data From Excel (Again)

Practical questions on how to import Excel data sets into Stata seem to follow a five-year (or so) cycle. While invaluable (and relatively inexpensive) data conversion software (e.g., StatTransfer) can mange most situations, how Stata "reads" imported Excel data that may include a blend of "alpha" and "numeric" data can trip folks up. A recent discussion on the Stata Listserv (here) on the utility of Stata's "destring" command may assist.