Empirical Legal Studies

Empirical Conjectures About the Effects of Raising the Jurisdictional Amount

As my Civ Pro colleagues periodically remind me, we're likely overdue for an increase to jurisdictional thresholds. Even if they're correct, however, what might be the expected (and unexpected) consequences of various increases to jurisdictional thresholds? A recent paper by Steven Gensler (Oklahoma) and Roger Michalski (Oklahoma), The Million Dollar Diversity Docket, puts these counter-factual questions to data. And what they find should interest more than procedeuralists.

The authors hand-coded data from five (non-randomly selected) federal districts (Minnesota, Georgia Northern, New Jersey, Texas Northern, and Utah). From each of these districts they selected a random draw of diversity jurisdiction cases based on Federal Judicial Center (“FJC”) data on cases completed in 2018, resulting in 2,896 usable cases. The paper's methodological move involves substituting the temporal component in survival analysis with a measure of jurisdictional amount. That is, instead of asking on which day a machine fails, we ask at which jurisdictional amount a case fails. To do so, the authors estimated modified KaplanMeier survival functions and Cox Proportional Hazard Regression models.

The paper's main takeaway is that jurisdictional amounts "are not neutral throttles. They do not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others." The abstract follows.

"What would happen if Congress raised the jurisdictional amount in the diversity jurisdiction statute? Given that it has been almost 25 years since the last increase, we are probably overdue for another one. But to what amount? And with what effect? What would happen if Congress raised the jurisdictional amount from the current $75,000 to $250,000 or, say, $1 million?

Using modified Kaplan-Meier survival functions and Cox Proportional Hazard Regression models to analyze a novel hand-coded data set of pleadings in 2900 cases, we show that the jurisdictional amount is not a neutral throttle. It does not modulate all cases evenhandedly. Instead, different areas of law, different parts of the country, and different litigants are more affected by changes in the jurisdictional amount than others. As such, setting the amount in controversy at any level entails normative choices about which cases to bless and curse with a federal forum.

Our findings implicate not only policy discussions about diversity jurisdiction in general, but also debates currently underway in Congress to utilize modifications of the jurisdictional amount requirement to channel COVID-19 related litigation into federal courts. We hope our empirical work will inform these debates and enable a new wave of scholarship on the basic functions and functioning of the federal diversity docket. We outline some beginnings of this type of work by exploring radically different avenues of thought that develop and re-imagine the diversity docket anew. They range from incremental inflation-adjustments, to carve-outs, experiments with lotteries, and the radical replacement of an amount in controversy minimum with a maximum."

Some Stats Humor

(Thanks to: https://xkcd.com/2048/)

State Supreme Court Elections Data Set

Judicial decisionmaking scholars comparing outcomes from "elected" and "non-elected" judges invariably turn to state court systems (if the analytic focus is domestic) for the necessary variation. Those turning to the array of state court systems--and the range of state supreme court election types--invariably confront data gathering challenges that materially differ from challenges associated with, e.g., Art. III judges. One of the leading state court scholars, Bert Kritzer (Minn.), continues to make state court election data gathering efforts a bit easier for the rest of us. Not only has Bert engaged in the often under-appreciated yet Herculean effort of original data set building, but he shares his data sets (and associated documentation, codebooks, etc.) with other interested scholars. Bert's most recent updating effort, curated on Harvard's Dataverse, includes a collection of state supreme court election results that now range from 1946 through 2020.

"The archive includes three core datasets plus state-level Excel spreadsheets for all states that employed elections at least part of the time since 1946 to select or retain members of the state supreme court (known New York state as the Court of Appeals) including for Texas and Oklahoma both the state supreme court and the Court of Criminal Appeals which are the final courts for criminal cases in those states."

Those interested in the data sets can click here.

Regression Discontinuity "Agnostics"

One of Andrew Gelman's persistent "rants" (e.g., here and, more recently, here) involves his skepticism about regression discontinuity. Regardless of how one lands on Gelman's critiques, regression discontinuity designs remain a staple in many graduate economics programs and, regardless, increasingly appear in empirical journals that straddle a range of academic fields. As well, as various comments in the conversations that Gelman's posts ignited make clear, reasonable minds continue to differ a bit on regression discontinuity's efficacy (and proper use).

Nested in Gelman's more recent post on this issue, however, is a much more important, far-ranging, and, frankly, less contested point. Gelman notes: "To put it another way: If you want to make a big claim and convince me that you have evidence for it, I need that trail of breadcrumbs connecting data, model, and theory." Say what you will about the enduring contest over regression discontinuity and its potential to fuel overconfidence with a paper's causal identification strategy, Gelman's larger (background) point that researchers should avoid "making a strong claim that doesn’t make sense, is not supported by the data, and is unlikely to replicate" is an obvious one that warrants attention.

"Listening" to Data

Prior to embarking on any sophisticated analyses I invariably urge my students (and, frankly, colleagues as well) to take some time to get "underneath" and "a feel for" the data and how they distribute. For decades this typically included the standard suite of the related line, scatter-plot, box-plot, histograms, etc., commands.

While the original version has been around for a while, a user-written Stata command, stripplot, was recently updated. According to Nick Cox, "[t]he aim is to compare distributions and a variety of displays is possible, including linear and stacked dot and strip plots, with in conjunction box plots or confidence intervals." An example follows:

The example above includes a hybrid box and quantile plot, with the reference lines indicating medians. A more detailed description (with additional examples and coding) is found here.

COVID-19-Motivated Research

To observe that the COVID-19 pandemic has (frankly, predictably) influenced recent legal scholarship is to merely observe what's in plain sight. And external shocks to doctrinal and theoretical legal scholarship invariably emerge in empirical legal scholarship as well. As efforts to study a pandemic in the middle of said pandemic invite particular complications, scholarly care, prudence, and "humility" are at a premium.

One direct legislative consequence wrought by COVID-19 is the federal Paycheck Protection Program. Anytime professional politicians rush more than a half-trillion dollars "out-the-door" to more than five million businesses problems are inevitable. Exploiting data from the Small Business Administration, in a recent paper, Funding Crises: An Empirical Study of the Paycheck Protection Program, William Birdthistle (Chi-Kent) & Joshua Silver (Chicago--Sociology), set out the initial broad contours of some of the Program's central components with particular attention to the growing number of competing claims about the Program's efficacy. Specifically, the authors test various hypotheses on whether the Program funds' distribution "suffered from politics, corruption, and waste." While certainly far from the final word, a summary of the paper's key findings follows in the excerpted abstract.

"In this Article, we conduct one of the first empirical studies of the central component of the largest financial bailout in U.S. history. We examine quantitative data released by the Small Business Administration to answer various competing claims about the effect of the PPP. Critics have accused the program of being administered as a partisan political tool for the president’s reelection, as a corrupt slush fund for cronies of the Trump administration, and as an incompetent waste of money on undeserving recipients. We test these hypotheses to evaluate the distribution of funds and whether the disbursement materially suffered from politics, corruption, or waste. We find that the lending process not only suffered from high-profile failures, it failed to target the neediest areas, particularly early on. Other studies present mixed findings on whether the PPP successfully protected paychecks. We conclude that critical data does not yet exist, as the PPP’s greatest weakness was in failing to reach businesses unable to survive long enough to apply for or to receive loans."

ELS in the "Real World"

The growth (and import) of MDL proceedings in federal courts are, by now, beyond obvious. One practice that has emerged organically among some (many?) of the larger MDLs involves the use of "fact sheets." Fact sheets can involve a range of "party-negotiated questionnaires, directed at parties to the case (both plaintiff and defendant), [and] provide judges and attorneys with information about the scope of the proceeding." The rise of MDLs has also prompted calls for new rules for MDL proceedings. Perhaps not surprisingly, the Advisory Committee on Civil Rules (Advisory Committee) recently began considering "specific rules related to multidistrict litigation (MDL) proceedings," including rules relating to fact sheets.

As a recent paper by Margaret Williams (Fed. Judicial Center), An Empirical Evaluation of Proposed Civil Rules for Multidistrict Litigation, makes clear, "the question of whether or not rules for MDLs, specifically rules regarding fact sheets, are necessary would have to be answered by evaluating how fact sheets are currently used in MDL proceedings, if proposed rules would change that, and how." As is too often the case in legal reform, however, reform efforts go forward absent anything even resembling a necessary empirical footing. As Williams notes, "there is very little published empirical work examining the use of fact sheets in MDL proceedings." While the Williams paper makes an important initial, largely descriptive contribution to our knowledge base, the paper also makes clear that much more work remains on fact sheets' efficacy as a case management tool in the MDL context. An excerpted abstract follows.

"Using a sample of 116 mass tort proceedings (typically involving products liability) centralized through MDL between 2008 and 2018, we examine when fact sheets were ordered, what were the procedures for complying with the case management order, what information was collected, and what effect they have on the termination of the proceedings. The proceeding ranged between 3 and 40,533 actions and were open a minimum of 118 days and a maximum of 3,811 days. Actions terminated within the proceeding at least 98% of the time, but little information is available on how the actions terminated. Proceedings were centralized in 40 districts. We find that fact sheets are ordered more than half the time, and were most likely to be used in the largest proceedings. The information in fact sheets is used in several ways within the proceeding, including identifying cases for bellwether trials and winnowing cases, and the use of fact sheet processes leads to faster termination of the proceeding, all else being equal. Our sample of proceedings suggests judges use fact sheets to organize products liability proceedings when judges perceive they are merited, after considering the size of the proceeding or the nature of the litigation. The frequency with which judges already employ fact sheets and the variation in the use calls into question both the need for a rule and how to write one without tying the hands of transferee judges. Many issues regarding how fact sheets are used remain to be studied more in-depth."

Stata Graphics Resource

Given the "degree of difficulty" users confront when working with Stata's notoriously complex graphic commands--something I've long-complained about--it is difficult to overstate my delight at stumbling upon an extraordinarily helpful resource: Maarten Buis' (Univ. Konstanz) the "Graphics in Stata" website (click here). This is an exceedingly helpful resource that speaks to a wide array of common (and more sophisticated) practical Stata graphing challenges. What makes this resource especially useful is that it includes the actual coding (with public data) necessary to make concrete changes to graphs. For example, it walks readers through the coding needed to change these side-by-side graphs:

into this unified graph:

Criminal Sentencing Variations and Judge "Intersections"

While efforts to empirically assess the independent influence of, e.g., judge gender and age on an array of judicial outcomes are not uncommon in the judicial decisionmaking literature, a recent paper, The Intersectionality of Age and Gender on the Bench: Are Younger Female Judges Harsher with Serious Crimes?, by Morris Hoffman (Dist. Ct. Judge, St. of Colorado) et al., explores how judge age and gender may interact in ways that account for some of the observed variations in criminal sentencing. Or, more specifically, the paper considers whether "trial judges in their thirties sentence the same thirty years later, when they will likely still be on the bench.”

The paper benefits from an unusually rich data set: "2,995 individual state criminal sentences imposed in Colorado, covering 183 different types of crimes and 285 different judges—180 male and 105 female—over a sixteen-year time span (2001–2016)." Because the data set allows the authors to observe "the same judges sentencing similar crimes as they grow older," a longitudinal view of "the interactive effects of judicial age, gender, and crime level in determining sentences in actual criminal cases” becomes possible.

Some of the paper's core findings might surprise some. "We uncovered a three-way interaction not previously reported in the empirical or experimental literature: For high harm crimes, younger female judges sentenced convicted defendants more harshly than their male and older female colleagues. We controlled for the independent effect of judicial experience, leading to the conclusion that age—and not just experience on the bench—is driving the results.”

The analyses exclude, however, plea bargains, crimes where sufficient sentencing discretion does not exist (e.g., first-degree murder), and crimes which impede efforts to code for the crime's "harm" (e.g., drug possession). Despite any analytical attractiveness, such data exclusions invite questions about the results' generalizability. After all, a decision to exclude plea bargains, for example, removes a potentially sizable slice of criminal sentencing activity from the analyses. Data limitations aside, the paper's core results will likely interest judicial decisionmaking scholars and criminologists. The paper's abstract follows.

“We analyzed sentencing data from sixteen years of criminal trials in the State of Colorado, consisting of almost 3,000 individual sentences, and discovered an interaction effect of harm, gender, and age not reported in any of the empirical or experimental literature. Young female judges punished high harm crimes substantially more than their male and older female colleagues. These results, if confirmed, could have significant strategic and tactical implications for practicing lawyers. They may also inform policies surrounding judicial selection, education, training, and retirement.”

SEC Enforcement Decisions Over Time

While one of the SEC's core duties includes the important task of enforcing various securities regulations, "relatively little empirical work exists assessing the choices the SEC makes on enforcement on an ongoing basis." A paucity of readily-available data likely helps stunt such assessments. Even though the SEC reports annual enforcement statistics, the granular, internal information upon which SEC enforcement decisions rest remains largely "opaque to outside observers."

A recent paper, Measuring the Impact of SEC Enforcement Decisions, by Stephen Choi (NYU), seeks to reduce the opacity by generating various SEC enforcement decision metrics that, in turn, might form the foundation for more sophisticated analyses in the future. To this end, Choi brings together 851 SEC enforcement "actions against public companies and subsidiaries of public companies" between 2005 and 2018, inclusive. From this data set Choi descriptively analyses such aspects as SEC enforcement actions and various perspectives on stock price reactions to these SEC actions. While Choi's presentation of data is descriptive, sometimes even the descriptive can identify rich targets for deeper analyses. To illustrate, Figure 3 (below) presents SEC actions, over time, and by month.

As Choi discusses, Figure 3 makes clear something of a "September Swell" and an "October Ebb." While what accounts for these odd ("hockey stick") distributions is not entirely obvious, a picture of the raw distributions across months and over time is necessary to identify the need for a deeper dive. The paper's abstract follows.

“This essay examines several metrics of Securities and Exchange Commission (SEC) decisionmaking that may be updated on a regular basis over time using publicly available data to give a picture of how SEC decisionmaking changes over time. I focus on SEC actions against public companies and subsidiaries of public companies and use data from the 2005 to 2018 period. The metrics include: the number of SEC actions per year and per month, the ratio of SEC actions to securities class actions by year, the mean abnormal return from an event study of the first public announcement of the problem that led to the eventual SEC enforcement action by year, the fraction of SEC actions with prior disclosure of the underlying violation by year, and the variability of the market response to the initiation of a SEC action by year. The metrics cannot demonstrate with certainty what motivates internal SEC enforcement decisions but may raise questions that guide future research.”

A Deep Descriptive Dive Into Environmental Law

While the National Environmental Policy Act (NEPA) has been described as the “Magna Carta of environmental laws,” too many environmental laws, including NEPA, have escaped empirical analysis. In a recent paper, NEPA at 50: An Empirical Analysis of NEPA in the Courts, John Ruple (Utah) and Heather Tanana (Utah) take a descriptive dive into NEPA as the authors feel that "moving from anecdote to empiricism will help inform ongoing discussions of how to streamline NEPA compliance while ensuring that agencies satisfy their three-fold requirement to foster environmental stewardship, meaningfully engage with permit applicants and the public alike, and carefully consider the environmental impacts before taking action. Conversely, divorcing NEPA reform proposals from fact greatly increases the risk that changes to NEPA’s implementing procedures will result in significant unintended adverse consequences.”

To this end, and relying on public Environmental Protection Agency (EPA) data bases that track, among other things, annual Environmental Impact Statements (EIS) and Environmental Assessments (EA), the paper painstakingly marches through an array of descriptive trends. In the end, the paper conveys some degree of optimism about NEPA’s overall efficacy. “NEPA appears to be working better than many believe, with roughly 99% of all NEPA decisions avoiding the most rigorous analysis contained in an EIS. Yes, EISs are subject to litigation at fairly high rates, but that is not surprising considering that they are conducted for the 1% of projects that require the highest level of analysis and that involve the most significant environmental impacts. Overall, just 0.22% of NEPA decisions result in litigation and that, we believe, says more about NEPA than anomalous EISs mired in litigation.” The paper's abstract follows.

"The National Environmental Policy Act (NEPA), the groundbreaking 1970 statute that requires federal agencies to take a 'hard look' at the environmental impacts of their actions, turned 50 this year. In this anniversary year, and with NEPA revision efforts a hot topic in environmental law, we begin by quantifying the burden imposed by NEPA compliance. We then look back on approximately 1,500 court decisions to quantify the rate at which NEPA decisions are challenged, assess how those cases are resolved, and compare NEPA cases to other environmental litigation. We then discuss efforts to “streamline” NEPA and why we believe those efforts are likely to have unintended consequences."

A Quick "Margins" and "Marginsplot" Primer

Editors and referees (including, though with far less frequency, student law review editors) increasingly ask for more intuitive (and, frankly, visual) displays of empirical results. What presently dominates in the Stata world include the margins and marginsplot commands. And for anyone interested in seeing a few examples that illustrate how these two commands can, in Chuck Huber's words, "revolutionize" the presentation of regression (or other) results, I haven't come across anything more helpful than this quick video.

Data on Testacy and Intestacy in Florida

Filtered by my admitted soft spot for scholars who engage in the hard, unglamorous, and often under-appreciated tasks incident to data creation/collection, a recent paper taking an initial descriptive cut at some archival probate data caught my eye. In The Demographics of Intergenerational Transmission of Wealth: An Empirical Study of Testacy and Intestacy on Family Property, Danaya Wright (Florida), recognizing that "[e]veryone dies eventually, and the vast majority of us will die owning at least some property," digs into probate record archives to assess various trends. Exploiting all (N=408) estates that went through court-supervised probate in Alachua County (FL) in 2013, the paper focuses on such variables as wealth, age, number of children, race, sex and marital status, with an eye toward potential systematic differences between testate (293) and intestate (115) decedents. The paper reports "clear differences between testate and intestate decedents on the basis of race, sex, wealth, marital status, age, and even children, with testacy correlating to significantly more wealth than intestacy."

Appropriately mindful of data and research design limitations, the paper also conveys a welcome level of cautiousness and modesty in terms of its ability to scaffold firm conclusions. After all, the data used in the paper draw from one year (2013) in a single county in one state (FL). As well, while the paper benefits from all of estates submitted to probate, most (77.3%) of the deaths in Alachua County in 2013 did not trigger probate or were probated elsewhere. While important external validity and selection effects' questions linger, the paper nonetheless illustrates the scholarly potential of exploiting data that too often remain "hidden in plain sight." The paper's abstract follows.

“In this article, I analyze many of the differences between testate and intestate decedents and suggest areas of concern for law- and policy-makers, as well as practicing lawyers and financial planners. Identifying the right demographic at risk for wealth deterioration may allow for interventions that can slow or reverse the widening wealth gap and ultimately help the next few generations build economically stable lives. To that end, I discuss how a number of demographic differences (wealth, age, race, sex, and marital status) affect the downward trajectory of certain groups and the upward trajectory of others.”

Recent Update for User-Writen "stripplot"

"Longer-time" Stata users may recall (and have used) a user-written graphics command, stripplot. Old and new Stata users, however, should be aware that the stripplot command was recently updated and expanded to allow users to "compare distributions and a variety of displays is possible, including linear and stacked dot and strip plots, with in conjunction box plots or confidence intervals." To illustrate some possibilities, consider, e.g.,:

This facilitates, e.g., a side-by-side display of quantile plots combined with horizontal reference lines for means. For those interested, install with the ssc command. For more information, and examples, click here.

Data on Police Conduct and Civilian Oversight

Among the myriad of ideas for reforming police currently under debate, many call for greater oversight of police conduct in ways that increase civilian participation. Whether increased civilian oversight will generate the desired reductions in police misconduct without chilling effective policing, however, remains under-studied.

A recent paper by Kyle Rozema (Wash U) and Max Schanzenbch (Northwestern), Does Discipline Decrease Police Misconduct? Evidence from Chicago Civilian Allegations, makes a helpful contribution to the empirical void. Exploiting 1990-2015 officer-level data from Chicago, the paper implements "an officer-level event study to investigate how officer misconduct changes after a civilian allegation of misconduct is sustained." To this end the paper, presented at the 2019 CELS, implements a stacked event study approach to investigate how "officer misconduct changes after a civilian allegation of misconduct is sustained" and considers three possible mechanisms: (police) incapacitation, shirking, and improved conduct. What the authors find is that: "In terms of mechanisms, we found no evidence that this effect results from incapacitation, such as desk duty assignment, and no evidence that it results from chilling effective policing. Even though sustained allegations result in negligible suspensions, we found evidence that a sustained allegation has a detrimental impact on promotion and pay. We interpret the results as most consistent with improved officer conduct driven by specific or marginal deterrence. Consequently, our findings suggest that sustaining the marginal allegation will improve police conduct without affecting overall police effectiveness." The paper's abstract follows.

“Reformers are calling for greater oversight of police behavior, in part through enhanced use of civilian complaints. However, others counter that greater oversight could chill effective policing. We assess police officer response to administrative determinations of misconduct. Using Chicago data, we find strong evidence that a sustained allegation reduces that officer’s future misconduct. We find no evidence that this effect is driven by incapacitation, such as assignment to desk duty, or by officer disengagement with policing. We conclude that our findings are consistent with improved officer conduct, in part due to officer concerns over promotion, salary, and desirable assignments.”