Empirical Legal Studies

ELS History

Schlegel's (1995) important and comprehensive treatment of American legal realism and "empirical social science" makes clear the importance of, among other factors, the "Brandeis Brief" in Muller v. Oregon (1908), Yale's comparatively under-appreciated Underhill Moore, and the Legal Realists, broadly understood, to today's empirical legal studies. Marty Wells (Cornell--Statistics) brought an additional fascinating piece of empirical legal studies history to my attention that warrants circulation as it also contributed to ELS's development. Specifically, in late 1912 Cornell professor (and American Statistical Association President) Walter Willcox's presidential address (subsequently published in 1913), entitled The Need of Social Statistics as an Aid to the Courts, notes: "It is the lack of convincing social statistics upon such problems which has made it impossible to answer with confidence many of the questions judges and legislatures have assumed or felt bound to answer."

Replication Problems in Psychology (& Beyond)

No doubt partly prompted by a recent spate of academic fraud, Science magazine published results from a study that set out to replicate 100 published psychology studies. An "international team of experts," however, could "reproduce only 36% of original findings." (News coverage here.) While I'm sure many in the psychology field will seek to explain such findings, an inability to replicate 74% of studies published in "top" psychology journals is, at the very least, jarring.

Update: From the 8.27.2015 NYT (here).

Update 2: In a recent editorial in Psychology Science (here), D. Stephen Lindsay responds to a replication effort, published in Science (described above), that signaled serious problems. As Gelman (Columbia--Statistics) notes, "Lindsay talks about replication problems and how researchers should do better. He warns about p-hacking, noise, and the difference between significance and non-significance not being itself statistically significant." While it's notable to see a leading psychology journal recognize a problem and undertake concrete editorial policy changes to address the problem, it is likely only a (necessary) first step. To be sure, the "replication issue" is certainly not confined to psychology and it's an issue that warrants continued and sustained scholarly attention.

Update 3: From the 3.4.2016 NYT (here). Reporting on a paper critiquing the 2015 Science paper challenging replicability of 100 published psychology studies. Additional commentary from Gelman (Columbia--Statistics) here, here, and here.

Update 4: Marty Wells (Cornell--Statistics) notes that, according to a recent report in Science, replication problems have expanded into experimental economics.

Property Law, Easement Modifications, and Land Conservation

While evident to most property folks, NPO activity on the conservation easement front during the past few decades has revolutionized land preservation norms. Recently, judges and policymakers have considered whether and, if so, how to modify conservation easements. A paucity of helpful data precluded meaningful analysis and, as a consequence, public debate typically generated more "heat than light." An Empirical Study of Modification and Termination of Conservation Easements: What the Data Suggest About Appropriate Legal Rules, by Gerry Korngold (NY Law) et al., brings much-needed and helpful data to this question. An excerpted abstract follows.

"This article provides and analyzes a previously uncollected dataset that offers guidance on the appropriate rules of law for conservation easement modification. It examines policy goals in light of the data to suggest various modification rules that would be more effective than current practice. The dataset represents a significant sample of easement modifications that have been made during a six year period (2008-2013) and indicates several findings: first, modifications have actually been taking place, despite claims that conservation easements are “perpetual,” apparently indicating that NPOs need flexibility in at least some areas; most of the changes have been “minor” and have been either conservation neutral or conservation positive, though one would expect pressure for more significant alterations over time due to shifts in the environment and human needs; there is a range of types and degree of modifications to this point, suggesting that there should be a spectrum of procedural and substantive requirements for the different varieties of modifications; and, a mandate for a stand-alone, state registry of conservation easements and modifications would allow for improved policymaking.

The article suggests that a doctrine that requires different procedures and substantive rules for various categories of modifications — a sliding scale — may yield the best, policy-based results. The work also identifies and analyzes existing doctrines — federal tax law, specific state statutes, charitable trust doctrine, standing rules, and director liability — that would need to be altered or clarified to adopt effective modification rules."

Blind Analysis

Robert MacCoun (Stanford) and Saul Perlmutter (Berkeley) recently published a fascinating comment in Nature, Blind analysis: Hide results to seek the truth, in which the authors urge empiricists to employ “blinding” at all levels of study, including data analysis. As the authors note:

“Many motivations distort what inferences we draw from data. These include the desire to support one's theory, to refute one's competitors, to be first to report a phenomenon, or simply to avoid publishing 'odd' results. Such biases can be conscious or unconscious. They can occur irrespective of whether choices are motivated by the search for truth, by the good mentor's desire to help their student write a strong PhD thesis, or just by naked self-interest.

We argue that blind analysis should be used more broadly in empirical research. Working blind while selecting data and developing and debugging analyses offers an important way to keep scientists from fooling themselves.”

Data on the "Teaching v. Research" Trade-Off Debate

A very interesting paper from Tom Ginsburg and Tom Miles, both at Chicago, explores "teaching v. research" debate. In The Teaching/Research Trade-Off in Law: Data From the Right Tail, the authors test the (long-assumed) "trade-off" hypothesis with the benefit of 10 years of data from the University of Chicago Law School. A summary of the paper's main take-away follows.

“There has been much debate about whether the performance of American law schools is failing, according to various metrics. Critics argue that U.S. law schools subsidize useless research, taking resources away from the core activity of educating students. Our own school is alleged to be a part of the problem, as it is part of a set of elite schools that ‘‘set the market’’ for the legal professorate. A key assumption of these critics is that a trade-off exists between teaching and scholarship: That the time and energy spent on research impair the quality of teaching or that professors proficient at scholarship (or scholarship that is not directly related to legal practice) are poor teachers of law…. The results of our study cast doubt on this assumption, at least at one elite school. Our analysis of 10 years of data encompassing nearly 50 faculty members over almost 500 courses reveals no strongly negative relationship between the volume of scholarship and the amount or quality of teaching.”

ELS and the Supreme Court

Empirical legal studies’ impact was on display last week in the Supreme Court’s death penalty decision, Glossip v. Gross. In particular, there is a lively back and forth among Breyer (joined by Ginsburg) in dissent, and the concurrences of Scalia and Thomas with no fewer than three JELS articles cited.

Suffice to say, Breyer and Ginsburg find the empirical evidence of problematic capital punishment application compelling, Thomas counters with questions about methods, and Scalia summarizes much of the scholarship as “abolitionist.” The link to the opinion is here. Cited article links follow.

“An Empirical Evaluation of the Connecticut Death Penalty System Since 1973, Are There Unlawful Racial, Gender, and Geographic Disparities?”  by John J. Donohue III in JELS 11:4

“Frequency and Predictors of False Conviction: Why We Know So Little, and New Data on Capital Cases” by Samuel R. Gross and Barbara O'Brien in JELS 5:4

“A Broken System: The Persistent Patterns of Reversals of Death Sentences in the United States” by Andrew Gelman, James S. Liebman, Valerie West, and Alexander Kiss in JELS 1:2 

Research Design & Endogeneity

A fascinating paper exploring the effect of partition on violence does a wonderful job illustrating how creative efforts (and rich data) can potentially dampen (though not eliminate) endogeneity concerns. In Which Side Are You On? Political Violence and Partition in Ireland 1920-1921, Elissa Berwick (MIT--Poli Sci) analyzes a quasi-natural experiment produced by the partition of Ireland in 1921. Exploiting a truly unique data set, Berwick finds that "although partition decreased violence against civilians on Northern Ireland's side of the border as compared to the Irish Free State side, violence against civilians in the border areas as a whole significantly increased."

Along with her interesting findings Berwick's research design also warrants note, especially on how it addresses endogeneity concerns. As Berwick concedes, "Partition is usually provoked by conflict, and yet its effect on conflict is the outcome of interest." Notwithstanding this tension, however, Berwick goes on to observe that:

"As for endogeneity, the initial proposal of partition came from a backbench Liberal in June 1914, years before the start of any violent civil conflict. Although the partition itself occurred in a context of civil war, its original justification was not to end conflict between northern Protestants and southern Catholics. Instead, British legislators were concerned by their inability to coerce Northern Unionists, a point driven home by the Curragh Mutiny of March 1914, in which the British Army refused to disarm the Ulster Volunteer militia. Thus the partition of Ireland was not intended to separate two sides, but instead to forestall action by a minority."

While whether Berwick's methodological optimism is warranted, of course, requires further study, her transparent and helpful discussion of the issue deserves praise.

Longitudinal Data on the Relation Between "Motherhood" Time and Child Outcomes

While admittedly only tenuously related to family law issues, broadly construed, the data set and results from a recent study, published in the Journal of Marriage and Family (77:355-72, Apr. 2015), caught my eye. In what is described as "the first large-scale longitudinal study of parent [here, mothers'] time" (more commentary here) on childhood outcomes, Does the Amount of Time Mothers Spend With Children or Adolescents Matter?, the authors (Melissa A. Milkie et al.) conclude that, overall and on balance, it matters far less than one might assume. An excerpted abstract follows.

"Using time diary and survey data from the Panel Study of Income Dynamics Child Development Supplement, the authors examined how the amount of time mothers spent with children ages 3–11 (N=1,605) and adolescents 12–18 (N=778) related to offspring behavioral, emotional, and academic outcomes and adolescent risky behavior. Both time mothers spent engaged with and accessible to offspring were assessed. In childhood and adolescence, the amount of maternal time did not matter for offspring behaviors, emotions, or academics, whereas social status factors were important. For adolescents, more engaged maternal time was related to fewer delinquent behaviors, and engaged time with parents together was related to better outcomes. Overall, the amount of mothers’ time mattered in nuanced ways, and, unexpectedly, only in adolescence."

Too Many Zeros: Empirical Strategies

The presence of "too many zeros" is a common challenge in empirical legal research. For example, "most" cases do not pursue appeals, the outcome of many civil trials (e.g., a finding of no liability) generates "zero" damages, etc. Thus, the distributions of outcome variables of interest are not infrequently skewed and this data skew warrants attention.

How to addresses this common feature in empirical legal research was the subject of one of Ted Eisenberg's (1947-2014) final scholarly papers (co-written with his son, Thomas, and Marty Wells and Min Zhang). Addressing the Zeros Problem: Regression Models for Outcomes with a Large Proportion of Zeros, with an Application to Trial Outcomes, recently published in JELS (12:1, Mar. 2015), reviews various empirical strategies and compares results from Tobit, Heckman selection, and two-part models. An excerpted abstract follows.

"Tobit models are often applied to deal with the excess number of zeros, but these are more appropriate in cases of true censoring (e.g., when all negative values are recorded as zeros) and less appropriate when zeros are in fact often observed as the amount awarded. Heckman selection models are another methodology that is applied in this setting, yet they were developed for potential outcomes rather than actual ones. Two‐part models account for actual outcomes and avoid the collinearity problems that often attend selection models. A two‐part hierarchical model is developed here that accounts for both the skewed, zero‐inflated nature of damages data and the fact that punitive damage awards may be correlated within case type, jurisdiction, or time. Inference is conducted using a Markov chain Monte Carlo sampling scheme. Tobit models, selection models, and two‐part models are fit to two punitive damage awards data sets and the results are compared. We illustrate that the nonsignificance of coefficients in a selection model can be a consequence of collinearity, whereas that does not occur with two‐part models."

Bad News, Good News

While drunk driving is an odious activity, a recent paper by Scott Adams et al., Seatbelt Use Among Drunk Drivers in Different Legislative Settings (or here), suggests that drunk drivers can be incented to buckle-up when driving.

The paper's hypothesis and research design are straight-forward. In states where local seatbelt laws are primarily enforced, law enforcement personnel can pull over a driver if he/she or a passenger is observed to be not wearing a seatbelt. The authors hypothesize that drunk drivers, to avoid such interaction with law enforcement, increase their compliance with seatbelt laws.

“We present evidence from the Fatality Analysis Reporting System and Behavioral Risk Factor Surveillance System that shows increased seatbelt use following the concurrent presence of stricter blood alcohol content thresholds and primarily enforced seat belt laws. This suggests that inebriated drivers may use their seat belts more judiciously to avoid being identified as a drunk driver by law enforcement. The interactive effect of stricter drunk driving laws and primary seatbelt laws are also shown to be more effective than either law passed in isolation in terms of reducing traffic fatalities.”

'Real World' Applications of ELS Research

One key assumption typically shared by many assessing federal circuit court decisions is that the three-judge panels that hear cases have been randomly configured. Indeed, scores of scholarly articles have noted this 'fact,' and it has been relied on heavily by empirical researchers.

Increasingly, however, scholars are putting this widely-shared assumption (or 'fact') to data. In a 2010 JELS article, Randomness Reconsidered: Modeling Random Judicial Assignment in the U.S. Courts of Appeals, Matthew Hall (St. Louis--Poli Sci) notes critical variation across circuits when it comes to panel assignments. Similarly, and more recetly, Adam Chilton (Chicago) and Marin Levy (Duke), in Challenging the Randomness of Panel Assignment in the Federal Courts of Appeals, push back on the widely-held assumption and find: "Our results show evidence of non-randomness in the majority of the federal courts of appeals. Moreover, our findings specifically reveal a lack of randomness along several key dimensions: the distribution of conservative judges, female judges, and minority judges."

How circuit panels are configured is no longer merely an academic question. Adam Liptak, in today's New York Times, reports on a legal challenge pivoting on how the Ninth Circuit assembled its panels to hear important same-sex appeals.

Epstein on Black & Spriggs' Supreme Court Precedent Depreciation

Over at PrawfsBlawg, Harold Wasserman (FL) notes that the latest JOTWELL Courts Law essay, by Lee Epstein (Wash U--Poli Sci), reviews Black & Spriggs' The Citation and Depreciation of U.S. Supreme Court Precedent (10 JELS 325 (2013)). The Black & Spriggs paper "examines how the use of precedent changes and depreciates over time." As Epstein notes, and Black & Spriggs find, "Supreme Court precedents don’t have an especially long shelf life: they depreciate by about 80% between years one and twenty. Interestingly, though, much of the depreciation occurs within the first couple of years." Given these findings, Epstein suggests that "because of its 'here today, gone tomorrow' quality, law professors and lawyers might (re)consider carefully the cases they emphasize in class and in the courtroom."

Immigration Enforcement and Crime

Setting aside the heated politics of immigration, much of the policy debate surrounding immigration reform simply assumes that immigration enforcement reduces crime rates. This central assumption, however, does not benefit from much data. In their paper, Does Immigration Enforcement Reduce Crime? Evidence from 'Secure Communities', Tom Miles (Chicago) and Adam Cox (NYU), bring much-needed data to this research question, exploit a natural experiment created by the Secure Communities program, and "provide the first empirical analysis of the most important deportation initiative to be rolled out in decades." What they find is that the Secure Communities program has led to "no meaningful reductions in the FBI index crime rate. Nor has it reduced rates of violent crime — homicide, rape, robbery, or aggravated assault. This evidence shows that the program has not served its central objective of making communities safer."

Scholarship on Mechanical Turk

Yanna Krupnikov (Stony Brook-Poli Sci) and Adam Seth Levine’s (Cornell-Govt) article, “Cross-Sample Comparisons and External Validity,” offers an interesting (and empirically welcome) contribution to the ongoing debate regarding the use of convenience samples drawn from MTurk. The takeaway, at least in part, is as the authors note in their conclusion: “Our results do serve to sound a note of caution when using MTurk to produce generalizable results for all but the simplest experimental designs.”

The article can be found in the inaugural issue of the Journal of Experimental Political Science.

"Empirical Legal Studies"

A while back, I was asked to write the entry for "empirical legal studies" that will be included in the Second Edition of Oxford's International Encyclopedia of the Social & Behavioral Sciences. I've (finally!) completed a draft, a copy of which is here (non-SSRN link). I'd love to hear feedback and suggestions for additions or changes.