Empirical Legal Studies

Blind Analysis

Robert MacCoun (Stanford) and Saul Perlmutter (Berkeley) recently published a fascinating comment in Nature, Blind analysis: Hide results to seek the truth, in which the authors urge empiricists to employ “blinding” at all levels of study, including data analysis. As the authors note:

“Many motivations distort what inferences we draw from data. These include the desire to support one's theory, to refute one's competitors, to be first to report a phenomenon, or simply to avoid publishing 'odd' results. Such biases can be conscious or unconscious. They can occur irrespective of whether choices are motivated by the search for truth, by the good mentor's desire to help their student write a strong PhD thesis, or just by naked self-interest.

We argue that blind analysis should be used more broadly in empirical research. Working blind while selecting data and developing and debugging analyses offers an important way to keep scientists from fooling themselves.”

Data on the "Teaching v. Research" Trade-Off Debate

A very interesting paper from Tom Ginsburg and Tom Miles, both at Chicago, explores "teaching v. research" debate. In The Teaching/Research Trade-Off in Law: Data From the Right Tail, the authors test the (long-assumed) "trade-off" hypothesis with the benefit of 10 years of data from the University of Chicago Law School. A summary of the paper's main take-away follows.

“There has been much debate about whether the performance of American law schools is failing, according to various metrics. Critics argue that U.S. law schools subsidize useless research, taking resources away from the core activity of educating students. Our own school is alleged to be a part of the problem, as it is part of a set of elite schools that ‘‘set the market’’ for the legal professorate. A key assumption of these critics is that a trade-off exists between teaching and scholarship: That the time and energy spent on research impair the quality of teaching or that professors proficient at scholarship (or scholarship that is not directly related to legal practice) are poor teachers of law…. The results of our study cast doubt on this assumption, at least at one elite school. Our analysis of 10 years of data encompassing nearly 50 faculty members over almost 500 courses reveals no strongly negative relationship between the volume of scholarship and the amount or quality of teaching.”

ELS and the Supreme Court

Empirical legal studies’ impact was on display last week in the Supreme Court’s death penalty decision, Glossip v. Gross. In particular, there is a lively back and forth among Breyer (joined by Ginsburg) in dissent, and the concurrences of Scalia and Thomas with no fewer than three JELS articles cited.

Suffice to say, Breyer and Ginsburg find the empirical evidence of problematic capital punishment application compelling, Thomas counters with questions about methods, and Scalia summarizes much of the scholarship as “abolitionist.” The link to the opinion is here. Cited article links follow.

“An Empirical Evaluation of the Connecticut Death Penalty System Since 1973, Are There Unlawful Racial, Gender, and Geographic Disparities?”  by John J. Donohue III in JELS 11:4

“Frequency and Predictors of False Conviction: Why We Know So Little, and New Data on Capital Cases” by Samuel R. Gross and Barbara O'Brien in JELS 5:4

“A Broken System: The Persistent Patterns of Reversals of Death Sentences in the United States” by Andrew Gelman, James S. Liebman, Valerie West, and Alexander Kiss in JELS 1:2 

Research Design & Endogeneity

A fascinating paper exploring the effect of partition on violence does a wonderful job illustrating how creative efforts (and rich data) can potentially dampen (though not eliminate) endogeneity concerns. In Which Side Are You On? Political Violence and Partition in Ireland 1920-1921, Elissa Berwick (MIT--Poli Sci) analyzes a quasi-natural experiment produced by the partition of Ireland in 1921. Exploiting a truly unique data set, Berwick finds that "although partition decreased violence against civilians on Northern Ireland's side of the border as compared to the Irish Free State side, violence against civilians in the border areas as a whole significantly increased."

Along with her interesting findings Berwick's research design also warrants note, especially on how it addresses endogeneity concerns. As Berwick concedes, "Partition is usually provoked by conflict, and yet its effect on conflict is the outcome of interest." Notwithstanding this tension, however, Berwick goes on to observe that:

"As for endogeneity, the initial proposal of partition came from a backbench Liberal in June 1914, years before the start of any violent civil conflict. Although the partition itself occurred in a context of civil war, its original justification was not to end conflict between northern Protestants and southern Catholics. Instead, British legislators were concerned by their inability to coerce Northern Unionists, a point driven home by the Curragh Mutiny of March 1914, in which the British Army refused to disarm the Ulster Volunteer militia. Thus the partition of Ireland was not intended to separate two sides, but instead to forestall action by a minority."

While whether Berwick's methodological optimism is warranted, of course, requires further study, her transparent and helpful discussion of the issue deserves praise.

Longitudinal Data on the Relation Between "Motherhood" Time and Child Outcomes

While admittedly only tenuously related to family law issues, broadly construed, the data set and results from a recent study, published in the Journal of Marriage and Family (77:355-72, Apr. 2015), caught my eye. In what is described as "the first large-scale longitudinal study of parent [here, mothers'] time" (more commentary here) on childhood outcomes, Does the Amount of Time Mothers Spend With Children or Adolescents Matter?, the authors (Melissa A. Milkie et al.) conclude that, overall and on balance, it matters far less than one might assume. An excerpted abstract follows.

"Using time diary and survey data from the Panel Study of Income Dynamics Child Development Supplement, the authors examined how the amount of time mothers spent with children ages 3–11 (N=1,605) and adolescents 12–18 (N=778) related to offspring behavioral, emotional, and academic outcomes and adolescent risky behavior. Both time mothers spent engaged with and accessible to offspring were assessed. In childhood and adolescence, the amount of maternal time did not matter for offspring behaviors, emotions, or academics, whereas social status factors were important. For adolescents, more engaged maternal time was related to fewer delinquent behaviors, and engaged time with parents together was related to better outcomes. Overall, the amount of mothers’ time mattered in nuanced ways, and, unexpectedly, only in adolescence."

Too Many Zeros: Empirical Strategies

The presence of "too many zeros" is a common challenge in empirical legal research. For example, "most" cases do not pursue appeals, the outcome of many civil trials (e.g., a finding of no liability) generates "zero" damages, etc. Thus, the distributions of outcome variables of interest are not infrequently skewed and this data skew warrants attention.

How to addresses this common feature in empirical legal research was the subject of one of Ted Eisenberg's (1947-2014) final scholarly papers (co-written with his son, Thomas, and Marty Wells and Min Zhang). Addressing the Zeros Problem: Regression Models for Outcomes with a Large Proportion of Zeros, with an Application to Trial Outcomes, recently published in JELS (12:1, Mar. 2015), reviews various empirical strategies and compares results from Tobit, Heckman selection, and two-part models. An excerpted abstract follows.

"Tobit models are often applied to deal with the excess number of zeros, but these are more appropriate in cases of true censoring (e.g., when all negative values are recorded as zeros) and less appropriate when zeros are in fact often observed as the amount awarded. Heckman selection models are another methodology that is applied in this setting, yet they were developed for potential outcomes rather than actual ones. Two‐part models account for actual outcomes and avoid the collinearity problems that often attend selection models. A two‐part hierarchical model is developed here that accounts for both the skewed, zero‐inflated nature of damages data and the fact that punitive damage awards may be correlated within case type, jurisdiction, or time. Inference is conducted using a Markov chain Monte Carlo sampling scheme. Tobit models, selection models, and two‐part models are fit to two punitive damage awards data sets and the results are compared. We illustrate that the nonsignificance of coefficients in a selection model can be a consequence of collinearity, whereas that does not occur with two‐part models."

Bad News, Good News

While drunk driving is an odious activity, a recent paper by Scott Adams et al., Seatbelt Use Among Drunk Drivers in Different Legislative Settings (or here), suggests that drunk drivers can be incented to buckle-up when driving.

The paper's hypothesis and research design are straight-forward. In states where local seatbelt laws are primarily enforced, law enforcement personnel can pull over a driver if he/she or a passenger is observed to be not wearing a seatbelt. The authors hypothesize that drunk drivers, to avoid such interaction with law enforcement, increase their compliance with seatbelt laws.

“We present evidence from the Fatality Analysis Reporting System and Behavioral Risk Factor Surveillance System that shows increased seatbelt use following the concurrent presence of stricter blood alcohol content thresholds and primarily enforced seat belt laws. This suggests that inebriated drivers may use their seat belts more judiciously to avoid being identified as a drunk driver by law enforcement. The interactive effect of stricter drunk driving laws and primary seatbelt laws are also shown to be more effective than either law passed in isolation in terms of reducing traffic fatalities.”

'Real World' Applications of ELS Research

One key assumption typically shared by many assessing federal circuit court decisions is that the three-judge panels that hear cases have been randomly configured. Indeed, scores of scholarly articles have noted this 'fact,' and it has been relied on heavily by empirical researchers.

Increasingly, however, scholars are putting this widely-shared assumption (or 'fact') to data. In a 2010 JELS article, Randomness Reconsidered: Modeling Random Judicial Assignment in the U.S. Courts of Appeals, Matthew Hall (St. Louis--Poli Sci) notes critical variation across circuits when it comes to panel assignments. Similarly, and more recetly, Adam Chilton (Chicago) and Marin Levy (Duke), in Challenging the Randomness of Panel Assignment in the Federal Courts of Appeals, push back on the widely-held assumption and find: "Our results show evidence of non-randomness in the majority of the federal courts of appeals. Moreover, our findings specifically reveal a lack of randomness along several key dimensions: the distribution of conservative judges, female judges, and minority judges."

How circuit panels are configured is no longer merely an academic question. Adam Liptak, in today's New York Times, reports on a legal challenge pivoting on how the Ninth Circuit assembled its panels to hear important same-sex appeals.

Epstein on Black & Spriggs' Supreme Court Precedent Depreciation

Over at PrawfsBlawg, Harold Wasserman (FL) notes that the latest JOTWELL Courts Law essay, by Lee Epstein (Wash U--Poli Sci), reviews Black & Spriggs' The Citation and Depreciation of U.S. Supreme Court Precedent (10 JELS 325 (2013)). The Black & Spriggs paper "examines how the use of precedent changes and depreciates over time." As Epstein notes, and Black & Spriggs find, "Supreme Court precedents don’t have an especially long shelf life: they depreciate by about 80% between years one and twenty. Interestingly, though, much of the depreciation occurs within the first couple of years." Given these findings, Epstein suggests that "because of its 'here today, gone tomorrow' quality, law professors and lawyers might (re)consider carefully the cases they emphasize in class and in the courtroom."

Immigration Enforcement and Crime

Setting aside the heated politics of immigration, much of the policy debate surrounding immigration reform simply assumes that immigration enforcement reduces crime rates. This central assumption, however, does not benefit from much data. In their paper, Does Immigration Enforcement Reduce Crime? Evidence from 'Secure Communities', Tom Miles (Chicago) and Adam Cox (NYU), bring much-needed data to this research question, exploit a natural experiment created by the Secure Communities program, and "provide the first empirical analysis of the most important deportation initiative to be rolled out in decades." What they find is that the Secure Communities program has led to "no meaningful reductions in the FBI index crime rate. Nor has it reduced rates of violent crime — homicide, rape, robbery, or aggravated assault. This evidence shows that the program has not served its central objective of making communities safer."

Scholarship on Mechanical Turk

Yanna Krupnikov (Stony Brook-Poli Sci) and Adam Seth Levine’s (Cornell-Govt) article, “Cross-Sample Comparisons and External Validity,” offers an interesting (and empirically welcome) contribution to the ongoing debate regarding the use of convenience samples drawn from MTurk. The takeaway, at least in part, is as the authors note in their conclusion: “Our results do serve to sound a note of caution when using MTurk to produce generalizable results for all but the simplest experimental designs.”

The article can be found in the inaugural issue of the Journal of Experimental Political Science.

"Empirical Legal Studies"

A while back, I was asked to write the entry for "empirical legal studies" that will be included in the Second Edition of Oxford's International Encyclopedia of the Social & Behavioral Sciences. I've (finally!) completed a draft, a copy of which is here (non-SSRN link). I'd love to hear feedback and suggestions for additions or changes.

Big Data, Law Enforcement, and the Fourth Amendment

Noting that "[t]he age of 'big data' has come to policing," in Policing By the Numbers: Big Data and the Fourth Amendment Elizabeth Joh (UC Davis) explores its impact in three discrete areas (mass surveillance, predictive policing, and DNA databanks) and considers the need to "draw new Fourth Amendment lines now that the government has the capability and desire to collect and manipulate large amounts of digitized information."

The Empirics of Litigating Privacy

The SELS blog includes a link to a recent paper in JELS (11:1, 2014) that explores data breach litigation. In Empirical Analysis of Data Breach Litigation, Romanosky (NYU) et al., take an initial look into a decade's worth of privacy litigation with a particular eye towards disposition differences. The abstract follows.

"In recent years, many lawsuits have been filed by individuals seeking legal redress for harms caused by the loss or theft of their personal information. However, very little is known about the drivers, mechanics, and outcomes of those lawsuits, making it difficult to assess the effectiveness of litigation at balancing organizations’ usage of personal data with individual privacy rights. Using a unique and manually collected database, we analyze court dockets for more than 230 federal data breach lawsuits from 2000 to 2010. We investigate two questions: Which data breaches are being litigated? And which data breach lawsuits are settling? Our results suggest that the odds of a firm being sued are 3.5 times greater when individuals suffer financial harm, but 6 times lower when the firm provides free credit monitoring. Moreover, defendants settle 30 percent more often when plaintiffs allege financial loss, or when faced with a certified class action suit. By providing the first comprehensive empirical analysis of data breach litigation, our findings offer insight into the debate over privacy litigation versus privacy regulation."

Empiricism, Experiments, and International Law

Beginning from the premise that "[s]cholarship on international law has undergone an empirical revolution," Adam S. Chilton (Chicago) and Dustin H. Tingley (Harvard--Govt.), in Why the Study of International Law Needs Experiments, go on to argue for more experiments in international law scholarship. Regardless of whether the authors' claims persuade, most agree that international law poses particular research design challenges and includes unusual barriers to reliable causal inference. The paper's full abstract follows.

"Scholarship on international law has undergone an empirical revolution. Throughout the revolution, however, shortcomings of the observational data that studies have used have posed serious barriers to reliable causal inference. During the same period, political scientists and legal scholars studying domestic law have increasingly employed experimental methods because they make it easier to make credible causal claims. Despite the simultaneous emergence of those trends, there have been relatively few attempts to use experimental methods to study international law. This should change. In this paper we present the first argument that the study of international law could uniquely benefit from the use of experimental research methods. To make this argument, we present data we have collected that illustrates why observational studies will often be unable to provide answers to many of the most important questions to legal scholars. After doing so, we provide guidance on how laboratory, survey, and field experiments can be used by legal scholars to research international law."