Empirical Legal Studies

Regression Discontinuity "Agnostics"

One of Andrew Gelman's persistent "rants" (e.g., here and, more recently, here) involves his skepticism about regression discontinuity. Regardless of how one lands on Gelman's critiques, regression discontinuity designs remain a staple in many graduate economics programs and, regardless, increasingly appear in empirical journals that straddle a range of academic fields. As well, as various comments in the conversations that Gelman's posts ignited make clear, reasonable minds continue to differ a bit on regression discontinuity's efficacy (and proper use).

Nested in Gelman's more recent post on this issue, however, is a much more important, far-ranging, and, frankly, less contested point. Gelman notes: "To put it another way: If you want to make a big claim and convince me that you have evidence for it, I need that trail of breadcrumbs connecting data, model, and theory." Say what you will about the enduring contest over regression discontinuity and its potential to fuel overconfidence with a paper's causal identification strategy, Gelman's larger (background) point that researchers should avoid "making a strong claim that doesn’t make sense, is not supported by the data, and is unlikely to replicate" is an obvious one that warrants attention.

"Listening" to Data

Prior to embarking on any sophisticated analyses I invariably urge my students (and, frankly, colleagues as well) to take some time to get "underneath" and "a feel for" the data and how they distribute. For decades this typically included the standard suite of the related line, scatter-plot, box-plot, histograms, etc., commands.

While the original version has been around for a while, a user-written Stata command, stripplot, was recently updated. According to Nick Cox, "[t]he aim is to compare distributions and a variety of displays is possible, including linear and stacked dot and strip plots, with in conjunction box plots or confidence intervals." An example follows:

The example above includes a hybrid box and quantile plot, with the reference lines indicating medians. A more detailed description (with additional examples and coding) is found here.

COVID-19-Motivated Research

To observe that the COVID-19 pandemic has (frankly, predictably) influenced recent legal scholarship is to merely observe what's in plain sight. And external shocks to doctrinal and theoretical legal scholarship invariably emerge in empirical legal scholarship as well. As efforts to study a pandemic in the middle of said pandemic invite particular complications, scholarly care, prudence, and "humility" are at a premium.

One direct legislative consequence wrought by COVID-19 is the federal Paycheck Protection Program. Anytime professional politicians rush more than a half-trillion dollars "out-the-door" to more than five million businesses problems are inevitable. Exploiting data from the Small Business Administration, in a recent paper, Funding Crises: An Empirical Study of the Paycheck Protection Program, William Birdthistle (Chi-Kent) & Joshua Silver (Chicago--Sociology), set out the initial broad contours of some of the Program's central components with particular attention to the growing number of competing claims about the Program's efficacy. Specifically, the authors test various hypotheses on whether the Program funds' distribution "suffered from politics, corruption, and waste." While certainly far from the final word, a summary of the paper's key findings follows in the excerpted abstract.

"In this Article, we conduct one of the first empirical studies of the central component of the largest financial bailout in U.S. history. We examine quantitative data released by the Small Business Administration to answer various competing claims about the effect of the PPP. Critics have accused the program of being administered as a partisan political tool for the president’s reelection, as a corrupt slush fund for cronies of the Trump administration, and as an incompetent waste of money on undeserving recipients. We test these hypotheses to evaluate the distribution of funds and whether the disbursement materially suffered from politics, corruption, or waste. We find that the lending process not only suffered from high-profile failures, it failed to target the neediest areas, particularly early on. Other studies present mixed findings on whether the PPP successfully protected paychecks. We conclude that critical data does not yet exist, as the PPP’s greatest weakness was in failing to reach businesses unable to survive long enough to apply for or to receive loans."

ELS in the "Real World"

The growth (and import) of MDL proceedings in federal courts are, by now, beyond obvious. One practice that has emerged organically among some (many?) of the larger MDLs involves the use of "fact sheets." Fact sheets can involve a range of "party-negotiated questionnaires, directed at parties to the case (both plaintiff and defendant), [and] provide judges and attorneys with information about the scope of the proceeding." The rise of MDLs has also prompted calls for new rules for MDL proceedings. Perhaps not surprisingly, the Advisory Committee on Civil Rules (Advisory Committee) recently began considering "specific rules related to multidistrict litigation (MDL) proceedings," including rules relating to fact sheets.

As a recent paper by Margaret Williams (Fed. Judicial Center), An Empirical Evaluation of Proposed Civil Rules for Multidistrict Litigation, makes clear, "the question of whether or not rules for MDLs, specifically rules regarding fact sheets, are necessary would have to be answered by evaluating how fact sheets are currently used in MDL proceedings, if proposed rules would change that, and how." As is too often the case in legal reform, however, reform efforts go forward absent anything even resembling a necessary empirical footing. As Williams notes, "there is very little published empirical work examining the use of fact sheets in MDL proceedings." While the Williams paper makes an important initial, largely descriptive contribution to our knowledge base, the paper also makes clear that much more work remains on fact sheets' efficacy as a case management tool in the MDL context. An excerpted abstract follows.

"Using a sample of 116 mass tort proceedings (typically involving products liability) centralized through MDL between 2008 and 2018, we examine when fact sheets were ordered, what were the procedures for complying with the case management order, what information was collected, and what effect they have on the termination of the proceedings. The proceeding ranged between 3 and 40,533 actions and were open a minimum of 118 days and a maximum of 3,811 days. Actions terminated within the proceeding at least 98% of the time, but little information is available on how the actions terminated. Proceedings were centralized in 40 districts. We find that fact sheets are ordered more than half the time, and were most likely to be used in the largest proceedings. The information in fact sheets is used in several ways within the proceeding, including identifying cases for bellwether trials and winnowing cases, and the use of fact sheet processes leads to faster termination of the proceeding, all else being equal. Our sample of proceedings suggests judges use fact sheets to organize products liability proceedings when judges perceive they are merited, after considering the size of the proceeding or the nature of the litigation. The frequency with which judges already employ fact sheets and the variation in the use calls into question both the need for a rule and how to write one without tying the hands of transferee judges. Many issues regarding how fact sheets are used remain to be studied more in-depth."