Empirical Legal Studies

Some Stats Humor

(Thanks to: https://xkcd.com/2048/)

State Supreme Court Elections Data Set

Judicial decisionmaking scholars comparing outcomes from "elected" and "non-elected" judges invariably turn to state court systems (if the analytic focus is domestic) for the necessary variation. Those turning to the array of state court systems--and the range of state supreme court election types--invariably confront data gathering challenges that materially differ from challenges associated with, e.g., Art. III judges. One of the leading state court scholars, Bert Kritzer (Minn.), continues to make state court election data gathering efforts a bit easier for the rest of us. Not only has Bert engaged in the often under-appreciated yet Herculean effort of original data set building, but he shares his data sets (and associated documentation, codebooks, etc.) with other interested scholars. Bert's most recent updating effort, curated on Harvard's Dataverse, includes a collection of state supreme court election results that now range from 1946 through 2020.

"The archive includes three core datasets plus state-level Excel spreadsheets for all states that employed elections at least part of the time since 1946 to select or retain members of the state supreme court (known New York state as the Court of Appeals) including for Texas and Oklahoma both the state supreme court and the Court of Criminal Appeals which are the final courts for criminal cases in those states."

Those interested in the data sets can click here.

Regression Discontinuity "Agnostics"

One of Andrew Gelman's persistent "rants" (e.g., here and, more recently, here) involves his skepticism about regression discontinuity. Regardless of how one lands on Gelman's critiques, regression discontinuity designs remain a staple in many graduate economics programs and, regardless, increasingly appear in empirical journals that straddle a range of academic fields. As well, as various comments in the conversations that Gelman's posts ignited make clear, reasonable minds continue to differ a bit on regression discontinuity's efficacy (and proper use).

Nested in Gelman's more recent post on this issue, however, is a much more important, far-ranging, and, frankly, less contested point. Gelman notes: "To put it another way: If you want to make a big claim and convince me that you have evidence for it, I need that trail of breadcrumbs connecting data, model, and theory." Say what you will about the enduring contest over regression discontinuity and its potential to fuel overconfidence with a paper's causal identification strategy, Gelman's larger (background) point that researchers should avoid "making a strong claim that doesn’t make sense, is not supported by the data, and is unlikely to replicate" is an obvious one that warrants attention.

"Listening" to Data

Prior to embarking on any sophisticated analyses I invariably urge my students (and, frankly, colleagues as well) to take some time to get "underneath" and "a feel for" the data and how they distribute. For decades this typically included the standard suite of the related line, scatter-plot, box-plot, histograms, etc., commands.

While the original version has been around for a while, a user-written Stata command, stripplot, was recently updated. According to Nick Cox, "[t]he aim is to compare distributions and a variety of displays is possible, including linear and stacked dot and strip plots, with in conjunction box plots or confidence intervals." An example follows:

The example above includes a hybrid box and quantile plot, with the reference lines indicating medians. A more detailed description (with additional examples and coding) is found here.

COVID-19-Motivated Research

To observe that the COVID-19 pandemic has (frankly, predictably) influenced recent legal scholarship is to merely observe what's in plain sight. And external shocks to doctrinal and theoretical legal scholarship invariably emerge in empirical legal scholarship as well. As efforts to study a pandemic in the middle of said pandemic invite particular complications, scholarly care, prudence, and "humility" are at a premium.

One direct legislative consequence wrought by COVID-19 is the federal Paycheck Protection Program. Anytime professional politicians rush more than a half-trillion dollars "out-the-door" to more than five million businesses problems are inevitable. Exploiting data from the Small Business Administration, in a recent paper, Funding Crises: An Empirical Study of the Paycheck Protection Program, William Birdthistle (Chi-Kent) & Joshua Silver (Chicago--Sociology), set out the initial broad contours of some of the Program's central components with particular attention to the growing number of competing claims about the Program's efficacy. Specifically, the authors test various hypotheses on whether the Program funds' distribution "suffered from politics, corruption, and waste." While certainly far from the final word, a summary of the paper's key findings follows in the excerpted abstract.

"In this Article, we conduct one of the first empirical studies of the central component of the largest financial bailout in U.S. history. We examine quantitative data released by the Small Business Administration to answer various competing claims about the effect of the PPP. Critics have accused the program of being administered as a partisan political tool for the president’s reelection, as a corrupt slush fund for cronies of the Trump administration, and as an incompetent waste of money on undeserving recipients. We test these hypotheses to evaluate the distribution of funds and whether the disbursement materially suffered from politics, corruption, or waste. We find that the lending process not only suffered from high-profile failures, it failed to target the neediest areas, particularly early on. Other studies present mixed findings on whether the PPP successfully protected paychecks. We conclude that critical data does not yet exist, as the PPP’s greatest weakness was in failing to reach businesses unable to survive long enough to apply for or to receive loans."