Empirical Legal Studies

"Heatplots" for Correlation Coefficients Graphs

While I've never used one myself, I detect an upward trend in "heatplots" (example below) for basic descriptive correlation coefficient graphs.

A helpful discussion (along with coding suggestions) on what is possible with Stata's heatplot command is found here.

Importing Non-Stata Data Sets Into Stata

While perhaps old-hat for some (many), a quick new Stata tutorial video (click here) walks viewers through some of the basics incident to data importation. For old-time users (including those who may reflexively turn to StatTransfer programs for conversions), increasingly easy--and a wider array of--data transfer options remain much-welcome news.

A Small--But Important--Logistical Matter Incident to Dataset Merging

Those who have merged (or appended or crossed) datasets may have inadvertently bumped into a not uncommon problem--where the two separate datasets share a variable with the identical variable label name. For relatively smaller datasets, and where the researcher is aware of this hurdle ex ante, one can always, by hand, rename variables prior to combining the datasets. For larger datasets with many more variables, however, while this may not be practical it nevertheless remains critical.

A recent discussion of how to manage this nettlesome problem is found here. The comments' leading suggestion references Daniel Klein's -elabel- package, available from SSC (as it involves user-written code). Put briefly, the elable command allows a user to add a suffix (e.g., "_1") to every variable name in "dataset1." An analogous manipulation to datast2 can add another unique suffix (e.g., "_2") to every variable name in dataset2. And so forth. The end result includes datasets with distinct variable label names that can then be combined.

Business Litigants and the Roberts Court

Assessing “How pro business is the Roberts Court?” is both straightforward and complicated. Based on how business litigants fare in the Roberts Court versus non-business litigants over time, business does quite well. The win rate for business in the Roberts Court, 63.4 percent, is 15 percentage points higher than the next highest rate of business wins over the past century (the Rehnquist Court, at 48.3 percent). While it may be tempting for many to conclude that the current pro business tilt is purely a function of a Republican majority of justices on the Roberts Court, as Lee Epstein (USC) and Mitu Gulati (Virginia) argue in a recent paper, A Century of Business in the Supreme Court, 1920-2020, “this is only the beginning of the story.”

Epstein and Gulati bring data from the Supreme Court Database to the question. Starting with the 11,121 orally argued cases resulting in a signed opinion or judgment between the 1920-2020 terms, their paper focuses only on those 4,070 cases (35,173 votes) in which business was the petitioner (appellant) or respondent (appellee). That is, they excluded cases in which business was on both sides of a legal dispute.

Their core findings defy conventional wisdom and inject important complicating wrinkles. What they find includes: "(a) It is not just the Republicans on the Roberts Court who are more pro business than in prior Courts, but the Democrats as well; (b) The Government, through the SG’s office and across both Democratic and Republican administrations, has been much more supportive of business positions than in prior eras; and (c) An elite Supreme Court bar has emerged in recent years and businesses have hired them disproportionately so as to better influence the Court."

Defendant-Prosecutor Gender "Pairings" and Leniency

While various pockets on criminal law are increasingly subject to empirical analyses, one particular pocket--how prosecutors distribute discretion--is comparatively under-studied. This is so, partly, due to a comparative paucity of relevant data on the prosecutors themselves (and matching them to individual cases and dispositions). A recent paper, Gender Favoritism Among Criminal Prosecutors, by Stephanie Holmes Didwania (Wisc.), tries move the scholarly needle a bit by exploiting data on over 130,000 federal criminal defendants sentenced in 2002 through 2016 to assess possible "gender-based differences in how prosecutors exercise critical discretion."

The paper matches Sentencing Commission data and data from the Executive Office for United States Attorneys (EOUSA) which regularly publishes case data from the 93 USAOs located throughout the United States. Critically, the breadth of the LIONS data "is substantial—the data covers all cases in which a USAO was involved, including those cases that are the primary responsibility of another agency and those the USAO declines to prosecute." Despite its breath, the LIONS data set is not without important limitations including a comparatively non-trivial amount of missing data. After matching, the resultant data set data includes "131,207 federal criminal defendants sentenced between 2001 and 2016 in 63 of the 94 federal district courts."

Data set limitations aside, the paper's core finding intrigues: "That defendants receive more lenient charges when they match with their prosecutor on gender relative to when they are assigned a prosecutor of the opposite gender. Through this charging channel, prosecutor-defendant pairs that match on gender produce lower sentences relative to prosecutor-defendant pairs that mismatch." More specifically, "the leniency associated with gender-matching ultimately translates into significantly lower sentences: a defendant can expect to earn a roughly five-month shorter sentence when their prosecutor is their own gender compared to when their prosecutor is the opposite gender, which represents a roughly eight percent reduction in sentence length at the mean." The paper's abstract follows.

"Prosecutors enjoy wide discretion in the decisions they make but are largely unstudied by quantitative empirical scholars. This paper explores gender bias in prosecutorial decision-making. I find that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall but demonstrate relative leniency towards defendants of their own gender. Such favoritism at charging translates into a sentencing gap of roughly five months of incarceration for defendants who are paired with an own-gender prosecutor versus an opposite-gender prosecutor, which represents a roughly eight percent reduction in sentence length at the mean. The estimates do not appear to be driven by differences in case assignments for male and female prosecutors."

Moving From Raw Data to More Helpful Variables

For many research projects scholars have to manipulate existing raw data into more useful derived variables. One obvious approach is "hand-coding," or some related form of Excel file manipulation. That is fine for folks experienced with Excel, but for others (including experienced Excel users) such a task typically invites some degree of human error. Another approach is to rely on Stata coding for variable creation. A recent (and, I hope, building conversation on the State List) illustrates a not an atypical example.

The questioner begins with Current Population Survey data (from the Census Dept.) that include individual-level information on, e.g., year, state, and a dummy variable for employment status (0=employed, 1 = unemployed, 2 = not in labor force). What the CPS data set does not include, however, is a state-level unemployment rate by year and state variable. So, the question is how to most efficiently extract what is desired (the creation of a new state-level unemployment rate by year and state variable) from what exists in raw data form. The discussion thus far has elicited at least one very helpful step-by-step examples of how to accomplish this goal with three lines of code (click here).

To be sure, other options also exist. That said, clear explications of how to move from what one has in terms of data to what one wants in terms of a new variable are sometimes hard to find.

The Federal Circuit Dataset project (v.3)

While Jason Rantanen's (Iowa) third update to The Federal Circuit Dataset Project will certainly interest empirical IP scholars, the Project's broader implications speak to empirical law scholars in all fields.

The third edition advances the existing Compendium of Federal Circuit Decisions in at least two main ways. First, the dataset now includes document numbers for all archived documents. Second, the dataset now facilitates increased understanding about what happens to appeals, including information about appeal pendency and origin.

Setting aside the dataset's utility to empirical IP scholars, the Project itself reflects a growing commitment to the hard and often under-rewarded work of dataset creation as well as an impulse to make such data (and related codebooks, etc.) publicly-available. Such impulses need to be rewarded, promoted, and nurtured. The paper's abstract follows.

"This paper introduces the third generation of the Federal Circuit Dataset Project. The third generation builds on the existing Compendium of Federal Circuit Decisions with the following enhancements:

• The addition of a dataset of all dockets for the Federal Circuit since 2000. These dockets are linked to the documents in the Compendium through the Appeal_Dockets field in both datasets.
• New fields containing additional information about Federal Circuit decisions.
• More complete data in existing fields that require manual coding.
• Parallel archiving of the document and docket datasets on the Dataverse.

In addition to describing the third generation of the Federal Circuit Dataset Project, this paper also provides updated descriptive statistics about Federal Circuit decisions and new statistics on appeal dockets."

An Updated Database of State Tort Law Reforms (7th)

Ronen Avraham (Tel Aviv Univ./Texas) recently alerted me to an update of one of the more helpful resources around for those who study torts and tort reform. Specifically, the Database of State Tort Law Reforms (7.0), seeks to create "one 'canonized' dataset will increase our understanding of tort reform’s impacts on our lives." A fuller (though excerpted) description follows.

"This This manuscript of the DSTLR (7th) updates the DSTLR (6th) and contains the most detailed, complete and comprehensive legal dataset of the most prevalent tort reforms in the United States between 1980 and 2020.... The dataset records state laws in all fifty states and the District of Columbia over the last several decades. For each reform we record the effective date, a short description of the reform, whether or not the jury is allowed to know about the reform, whether the reform was upheld or struck down by the states’ courts, as well as whether it was amended by the state legislator. "

Data Resources Courtesy of the APSA Law & Courts Section

Our good friends and political science colleagues over at the APSA's Law & Courts Section recently updated the Section's website (click here). What is especially helpful is a sub-page (here) that provides links to the leading court data sets (SCOTUS, CA, district courts, as well as state supreme courts) along with related data assets (e.g., Martin-Quinn Scores, Judicial Common Space Scores).

Bridge Ideal Points--2021 Update

Michael Bailey (Georgetown--Govt.) has updated the Bridge Ideal Points estimates (click here) to cover years 1950 through 2020. These Ideal point estimates include the Supreme Court, presidential, and congressional ideology estimates and are designed to facilitate inter-institutional and cross-temporal comparisons.

While the Bridge Ideal Points compete somewhat with the Martin-Quinn scores in the literature, the two sets of ideology estimates correlate reasonably highly with one another. Despite the overlap, these two ideology estimates differ in two important ways.First, "Bridge Ideal Points are designed to be intertemporally comparable. Martin and Quinn scores assume the ideological cutpoints of Supreme Court cases are drawn from a common distribution across years, which makes them less comparable across time when the case load changes." Second, while Martin and Quinn scores only cover the Supreme Court justices, Bridge Ideal Points for justices are comparable to Bridge Ideal Points for presidents and members of Congress as well.

Regardless of one's preference, efforts to estimate justice ideology remain a staple in the judicial decisionmaking literature.

Data Familiarization/Visualization

A recent post by Jessica Hullman (here) served as my most recent periodic reminder about the enduring need for folks (myself included) to take a few, basic, simple steps to become more familiar with the underlying structure of and relationships among critical data points. Tukey's elegant tables and figures notwithstanding, what is perhaps more important is what follows from Tukey's general impulse: Sometimes simple data plots can unlock key analytic insights. And even if such insights aren't necessary for the researcher, readers' (and journal editors') interests should not be taken for granted.

In trying to explain why researchers might ignore--or bypass--simple initial data visualization/familiarization steps, Hullman points to "latent beliefs that plotting data is somehow inferior to anything that feels more like math. And because making plots for the sake of checking what the data look like or discovering things you didn’t expect seems disconnected and even distracting from the ultimate goal of fitting some model." That such beliefs may be entirely understandable, however, does not detract from their potential harm. It remains awfully difficult to overestimate the value of simple, Tukey-inspired data familiarization/visualization efforts.

JELS' Research Notes and Datasets Section

If memory serves, a suggestion in 2016 (or so) from Bernie Black (Northwestern) and Marty Wells (Cornell-Statistics) to the JELS editors set into motion a chain of events that culminated in 2017 with the introduction of the "Research Notes and Datasets" section in JELS. The core motivation behind introducing a new paper "genre" for JELS was a fundamental belief that dataset construction, data sharing, and methods testing are a vital part of ELS' growth. We felt that a new section in JELS would help contribute to the realization of such beliefs. What we did not fully appreciate or anticipate, however, was the response. Since the Research Notes and Datasets section's launch in 2017 JELS has already published a number of important papers, including:

Patent Similarity Data and Innovation Metrics, by Ryan Whalen Alina Lungeanu Leslie DeChurch Noshir Contractor JELS 2020 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12261

Developing High‐Quality Data Infrastructure for Legal Analytics: Introducing the Israeli Supreme Court Database, by Keren Weinshall Lee Epstein JELS 2020 (June). https://onlinelibrary.wiley.com/doi/10.1111/jels.12250

The German Federal Courts Dataset 1950–2019: From Paper Archives to Linked Open Data, by Hanjo Hamann Sokol JELS 2019 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12230

Competition Law Gone Global: Introducing the Comparative Competition Law and Enforcement Datasets, by Anu Bradford Adam S. Chilton Christopher Megaw Nathaniel Sokol JELS 2019 (June). https://onlinelibrary.wiley.com/doi/10.1111/jels.12215

Accuracy and Fairness for Juvenile Justice Risk Assessments, by Richard Berk JELS 2019 (March). https://onlinelibrary.wiley.com/doi/10.1111/jels.12206

Text of Trade Agreements (ToTA)—A Structured Corpus for the Text‐as‐Data Analysis of Preferential Trade Agreements, by Wolfgang Alschner Julia Seiermann Dmitriy Skougarevskiy  JELS 2018 (September). https://onlinelibrary.wiley.com/doi/10.1111/jels.12189

Medical Liability Insurance Premia: 1990–2016 Dataset, with Literature Review and Summary Information, by Bernard Black Jeanette W. Chung Jeffrey Traczynski Victoria Udalova Sonal Vats, JELS 2017 (March). https://onlinelibrary.wiley.com/doi/10.1111/jels.12146

A New Tool for Detecting Various Characters in Coding

While a new user-written command, chartab, (briefly described and discussed here) may be criticized as a "command looking for a problem to solve," but for anyone who has ever tried to "de-bug" code a quick tabulation of various "characters" can help identify, and thus solve, problems. These problems include, e.g., the letter l for the number 1; the letter O for the number 0 (zero). These problems typically arise in the context of importing data, especially if it involves string data transited via Microsoft Word or Excel. And anyone merging data based on string variables should probably invest some time in some degree of "Unicode due diligence" before hand to help get in front of any potential problems. To install this command:

ssc install chartab

State Supreme Court Elections Data Set

Judicial decisionmaking scholars comparing outcomes from "elected" and "non-elected" judges invariably turn to state court systems (if the analytic focus is domestic) for the necessary variation. Those turning to the array of state court systems--and the range of state supreme court election types--invariably confront data gathering challenges that materially differ from challenges associated with, e.g., Art. III judges. One of the leading state court scholars, Bert Kritzer (Minn.), continues to make state court election data gathering efforts a bit easier for the rest of us. Not only has Bert engaged in the often under-appreciated yet Herculean effort of original data set building, but he shares his data sets (and associated documentation, codebooks, etc.) with other interested scholars. Bert's most recent updating effort, curated on Harvard's Dataverse, includes a collection of state supreme court election results that now range from 1946 through 2020.

"The archive includes three core datasets plus state-level Excel spreadsheets for all states that employed elections at least part of the time since 1946 to select or retain members of the state supreme court (known New York state as the Court of Appeals) including for Texas and Oklahoma both the state supreme court and the Court of Criminal Appeals which are the final courts for criminal cases in those states."

Those interested in the data sets can click here.

SEC Enforcement Decisions Over Time

While one of the SEC's core duties includes the important task of enforcing various securities regulations, "relatively little empirical work exists assessing the choices the SEC makes on enforcement on an ongoing basis." A paucity of readily-available data likely helps stunt such assessments. Even though the SEC reports annual enforcement statistics, the granular, internal information upon which SEC enforcement decisions rest remains largely "opaque to outside observers."

A recent paper, Measuring the Impact of SEC Enforcement Decisions, by Stephen Choi (NYU), seeks to reduce the opacity by generating various SEC enforcement decision metrics that, in turn, might form the foundation for more sophisticated analyses in the future. To this end, Choi brings together 851 SEC enforcement "actions against public companies and subsidiaries of public companies" between 2005 and 2018, inclusive. From this data set Choi descriptively analyses such aspects as SEC enforcement actions and various perspectives on stock price reactions to these SEC actions. While Choi's presentation of data is descriptive, sometimes even the descriptive can identify rich targets for deeper analyses. To illustrate, Figure 3 (below) presents SEC actions, over time, and by month.

As Choi discusses, Figure 3 makes clear something of a "September Swell" and an "October Ebb." While what accounts for these odd ("hockey stick") distributions is not entirely obvious, a picture of the raw distributions across months and over time is necessary to identify the need for a deeper dive. The paper's abstract follows.

“This essay examines several metrics of Securities and Exchange Commission (SEC) decisionmaking that may be updated on a regular basis over time using publicly available data to give a picture of how SEC decisionmaking changes over time. I focus on SEC actions against public companies and subsidiaries of public companies and use data from the 2005 to 2018 period. The metrics include: the number of SEC actions per year and per month, the ratio of SEC actions to securities class actions by year, the mean abnormal return from an event study of the first public announcement of the problem that led to the eventual SEC enforcement action by year, the fraction of SEC actions with prior disclosure of the underlying violation by year, and the variability of the market response to the initiation of a SEC action by year. The metrics cannot demonstrate with certainty what motivates internal SEC enforcement decisions but may raise questions that guide future research.”