Empirical Legal Studies

"Zeros" Problems

While not unique to empirical legal scholarship, how to deal with a "zeros problem" is not uncommon, especially given the nature of many research designs and questions in legal scholarship. To take one obvious--and well-trod--example, efforts to model punitive damage awards need to account for their absence in the over-whelming majority of civil cases decided by courts. A recent discussion on the StataList (here) provides yet another take on the array of challenges posed by "zeros" in data sets. The resultant discussions emphasizes, in part, that researchers need to assess with accuracy and clarity, ex ante, precisely what a "zero" means in the context of their data sets. As well, for a recent take on differences between Heckman selection and two-part models, a 2015 JELS paper (here), by Eisenberg et al., remains current.

The Harvard College Admissions Decision, The Import of Statistical Evidence, and Competing Modeling Assumptions

Focusing solely on the methodological aspect of the recent Harvard College admissions decision, one under-appreciated take-away from the district court's opinion (click here for the full decision) is the prominent role that traditional statistical analyses (basically, standard logistic models) played in the trial. If nothing else, the prominent role that statistical evidence bearing on the plaintiffs' claims, and Harvard's defense, played in the trial makes palpably clear, once again, the extent to which sophisticated litigation now--and almost routinely so in some case types--deploys empirical evidence. Indeed, the court's decision devotes more than 40 pages (pp.50-92) of its 130-page opinion to statistical matters.

In the Harvard admissions case, both parties to the litigation formally engaged experts, Professors David Card (Berkeley) and Peter Arcidiacono (Duke), to analyze a shared data set--six years worth of Harvard College admissions decisions. Despite a shared data set the two experts reached opposing conclusions on the critical question about the unique role that applicants' race/ethnicity played in Harvard's admissions decisions. Much of the experts' disagreement pivoted on one key modeling decision--whether to include or exclude admissions data germane to the "ALDC" applications. ("ALDC" applications are those involving: "recruited athletes, legacies, Dean's 'interest list' and children of Harvard faculty and staff.")

In the face of conflicting conclusions from the two experts, Judge Burroughs wrote that while she found both models “'econometrically defensible' and preferred different aspects of each model," in the end Judge Burroughs' decision largely preferenced Card's modeling assumptions over Arcidiacono's. Independent of whether one feels Judge Burroughs' assessment of the competing modeling decisions is persuasive, what is objectively clear is that she took the empirical and statistical evidence seriously and that it played a consequential--and potentially pivotal--role in the trial.

"Paired" v. "Unpaired" T-Test

Efforts to compare the means from two sample populations with a standard t-test invariably bump into a "paired" versus and "unpaired" decision. Whether a "paired" or "unpaired" comparison is appropriate pivots on a subtle distinction that can be difficult to describe. A recent Statalist discussion might provide a helpful way to think about this distinction.

The discussion (here) began with the following (edited) question: "If there is economic growth data about two countries over the same thirty years, should it be considered as paired or unpaired data for two sample t-test?"

The various responses included the following: "For unpaired tests imagine 30 men and 30 women completely unrelated and compare their heights. You are comparing the mean male height and mean female height and the context is the variability of male heights and of female heights. For paired tests imagine 30 heterosexual married couples and compare height for each husband with each wife."

With this in mind, the answer to the question posed is: "Your panel example is still paired because there is pairing by years."

Log Transformations & the "Zeros Problem"

Strategies to address normal distribution assumption concerns can quickly lead to the log transformation of variables. However, what to do with "zeros" in a variable requiring log transformation persists as an under-studied issue. And one well-known example, punitive damage awards, makes clear that such "zeros problems" are not uncommon in many empirical legal research projects.

In Dealing With the Log of Zero in Regression Models, the authors, Christophe Bellego (Center for Research in Economics and Statistics) and Louis-Danile Pape (Center for Research in Economics and Statistics), engage with this nettlesome issue head-on. The brief (15 pp.) paper's abstract follows.

"Log-linear and log-log regressions are one of the most used statistical model. However, handling zeros in the dependent and independent variable has remained obscure despite the prevalence of the situation. In this paper, we discuss how to deal with this issue. We show that using Pseudo-Poisson Maximum Likelihood (PPML) is a good practice compared to other approximate solutions. We then introduce a new complementary solution to deal with zeros consisting in adding a positive value specific to each observation that avoids some numerical issues faced by the former."

Ready-To-Use Data (and Codebooks) for Law & Education Research

One thing that the U.S. Department of Education does reasonably well is gather data--and make them publicly available. And as the number and complexity of law and education issues continue to increase over time, greater access to relevant data becomes even more important. To this end, the Dept. of Education's National Center for Education Statistics ("NCES") operates a landing spot (here) for direct access to ready-to-use public use data sets and codebooks. Users can select from an array of standard formats (Stata, SPSS, SAS, R, etc.) and tailor requests.

Instrumental Variables Regression--A Helpful Intro

For a brief (5+ min) introduction to--or a very quick refresher on--the basics of instrumental variables regression, including some post-estimation commands, one could do far worse than this Stata video. The video walks viewers through the basics on how to fit an instrumental variables model for endogenous covariates (using the "ivregress" command).

Comparative Competition Law & Datasets

Consistent with JELS' impulse to recently launch a new genre of papers--"Research Notes and Datasets"--a recent contribution to this initiative, Competition Law Gone Global: Introducing the Comparative Competition Law and Enforcement Datasets, by Anu Braford (Columbia), Adam Chilton (Chicago), Christopher Megaw (WilmerHale), and Nathaniel Sokol, introduces and discusses a pair of datasets that provide future researchers with "a new foundation for empirical research on the legal regimes used to regulate markets."

"Competition law has proliferated around the world. Due to data limitations, however, there is little systematic information about the substance and enforcement of these laws. In this article, we address that problem by introducing two new datasets on competition law regimes around the world. First, we introduce the Comparative Competition Law Dataset, which codes competition laws in 131 jurisdictions between 1889 to 2010. Second, we introduce the Comparative Competition Enforcement Dataset, which provides data on competition agencies’ resources and activities in 100 jurisdictions between 1990 and 2010. These datasets offer the most comprehensive picture of competition law yet assembled and provide a new foundation for empirical research on the legal regimes used to regulate markets."

So far, so good. But the authors recently undertook yet another step that warrants attention (and praise). Specifically, Bradford and Chilton created a new website (here) that already includes an array of ready-to-use datasets (and associated codebooks) germane to comparative competition law (see, e.g., here). According to the project's website, "this project generates novel data on competition regimes across time and countries. Using these data, the project studies competition law and enforcement around the world, including exploring the diffusion of these laws and the effects they have on market outcomes."

Frankly, the construction and public distribution of helpful and important datasets are among the most the most under-appreciated of endeavors. Future scholars (and scholarship) will gain from initiatives like this one.

"p-Hacking" explained

One consequence of the on-going "replication" crisis in the social (and, increasingly, hard) sciences is a spike in the salience of the term "p-Hacking." The term "p-Hacking," as defined by Robert MacCoun (Stanford) in his recent essay, p-Hacking: A Strategic Analysis, “joins a growing taxonomy for describing the myriad ways in which ‘cold cognitive’ and ‘hot motivational’ biases can creep into the interpretation and use of research findings.” In his brief essay MacCoun dives deeply into the issue and proposes “four different models of p-hacking behavior and examining what kinds of aggregate patterns they might be expected to produce.” The abstract follows.

"The phenomenon of p-hacking occurs when researchers engage in questionable practices that enable them to report findings as being statistically significant. I offer four models of p-hacking behavior – unconditional, strategic, greedy, and restrained – and explore the implications of each model. I then discuss the implications of recent reforms (routine replication, pre-registration, and blinded data analysis) with respect to these models."

Law Students, Role-Induced Bias, and the Inefficacy of De-Biasing Intervention Efforts

Those who teach in (as well as, frankly, attend) law schools may find a recent paper by Holger Spamann (Harvard) interesting. In Lawyers' Role-Induced Bias Arises Fast and Persists Despite Intervention, Spamann presents results from his experiment with Harvard Law students testing for evidence of role-induced bias and the efficacy of de-biasing efforts.

The experiment draws on students in "the basic class on corporations/business organizations taught by the same instructor with identical materials at two top ten U.S. law schools in six of the seven semesters from spring 2016 through spring 2019.”  Class enrollments (N) ranged from 43 to 104. “The purpose of the exercise is to engage students in a cutting-edge topic of insider trading law, while hopefully demonstrating to students the existence of own-side bias through a post-survey review of their debriefing survey.” A summary of the paper's key findings follows.

“Lawyers’ role-induced bias arises fast and persists despite a simple intervention, at least in an experiment with law students. After only 40 minutes or less of engagement with a case, law students exhibit substantial role-induced bias towards the side they have been randomly assigned to represent in the classroom. Of students randomly assigned to represent petitioner or respondent in a mock Supreme Court oral argument of U.S. v. Newman , over 50% of those assigned petitioner’s counsel’s role predicted petitioner victory, as opposed to less than 20% of those assigned respondent’s counsel’s role, with randomly assigned justices in the middle. These differences in prediction were actually starker when the role assignment was performed a mere 30 minutes before the 10-minute mock argument than when the assignment was performed five days prior, before the students ever started reading about the relevant law. Reminding students before making their prediction that their role in the oral argument had been “that of an advocate, whereas we are now stepping back out and […] asking your opinion as a neutral, sober observer” has no detectable effect on their biased predictions.” Moreover, similar to earlier research, “the bias occurs almost evenly in male and female participants, who also report equal confidence in their predictions.”

Criminal Misdemeanors in State Courts

As greater legal and policy attention turns to the intersection of criminal law--in particular, public-order policing--and low-income and communities of color, the need for well-executed empirical work increases. While much of the public attention tends to focus on high-profile federal criminal activity, criminal misdemeanors in state courts represent three-quarters of all criminal activity. Consequently, to gain a more accurate picture of overall criminal law activity the misdemeanor context is an obvious one to examine. To this end, in a recent paper, Misdemeanors by the Numbers, Mayson (Georgia) and Stevenson (GMU) take a deep dive into data from eight specific jurisdictions between 2011 and 2016. The paper's core findings are described in the excerpted abstract:

"The resulting portrait reveals a system that disproportionately impacts poor people and people of color. Between 2011 and 2016, each jurisdiction studied relied on monetary bail, which resulted in high rates of pretrial detention even at relatively low amounts, and imposed court costs upon conviction. There were substantial racial disparities in case-filing rates across locales and offense categories. The data also, however, highlight profound jurisdictional heterogeneity in how misdemeanors are defined and prosecuted. The variation suggests that misdemeanor adjudication systems may have fundamentally different characters, and serve different functions, from place to place. It thus presents a major challenge to efforts to describe and theorize the contemporary landscape of misdemeanor justice. At the most fundamental level, the variation calls into question the coherence of the very concept of a misdemeanor, or of misdemeanor criminal justice. As appreciation for the significance of lowlevel law enforcement builds, we urge scholars and policymakers to attend carefully to the complexity of this sub-felony world."

Everything You Ever Wanted to Know About Copyright Registrations Over Time

A recent paper reports results from the authors' effort to build what now has to be the single largest U.S. Copyright registration data set. In An Empirical Study of 225 Years of Copyright Registrations, Zvi Rosen and Richard Schwinn, report on what they find from observing more than two centuries of registration data. "This empirical analysis highlights two new insights that our analysis provides: The first is that the copyright term is, to a point, highly significant to registrations. In other words, a longer copyright term leads to more copyright registrations being filed. This is suggestive but not conclusive of a connection between a longer term and the creation of new works. Our analysis also shows that the incentive effect of a longer term offers diminishing returns above a certain point, and is no longer significant at a term of about 70 years. Secondly, we observe for the first time that the passage of a complete revision to the Copyright Law in 1909 which made registration (theoretically) optional without losing copyright protection represented a sharp paradigm shift for the relationship of copyright to economic progress.”

Equally interesting however, is the data set itself. As the authors note, “Until now, scholars have been dependent on limited tallies offered by the Annual Reports of the Register of Copyrights, which have provided the basis for all previous studies. In contrast, this study utilizes the more detailed statistics from the Catalog of Copyright Entries (“CCE”), which also have the benefit of corresponding to calendar year instead of the federal government’s fiscal year. This is also the first study ever to provide yearly estimates of copyright registrations per year prior to 1870 in any form – prior to this study the only figure used was 150,000 for the entire 80 year period… By taking account of changes in agency practice at the U.S. Copyright Office and its predecessors, we can harmonize our data over the entire series of 1790 to 2015.”

MTurk and Evidence on Measurement Error

While questions surrounding the potential for systematic measurement error from MTurk samples cannot be easily resolved in a single paper, given the popularity of MTurk samples in many fields, notably psychology, careful attention scholarly is warranted. The initial "warning" about MTurk samples came in 2013 from Dan Kahan (Yale), particularly for studies seeking to test “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.” Since then we've seen a developing literature that sets out to test potential limitations to MTurk samples. A recent paper by Douglas Ahler (FSU-Poli Sci ) et al., The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk, focuses on the potential for measurement error, especially Type II errors, injected by, among other factors, the "presence of 'non-respondents' (bots) or non-serious respondents on the platform." The paper's core findings include:

"While we find no evidence of a 'bot epidemic,' we do find that a significant portion of survey respondents engaged in suspicious behavior. About 20% of respondents either circumvented location requirements or took the survey multiple times. In addition, at least 5-7% of participants likely engaged in 'trolling' or satisficing. Altogether, we find about a quarter of data collected on MTurk is potentially untrustworthy. Expectedly, we find response quality impacts experimental treatments. On average, low quality responses attenuate treatment effects by approximately 9%. We conclude by providing recommendations for collecting data on MTurk."

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

When--and How--To Use Kruskal-Wallis

Helpful discussion, and example, of how to execute the Kruskal-Wallis ("K-W") test as well as an explanation on why one would turn to it (click here). In general (and in brief), folks typically turn to the K-W test when you have one nominal variable (e.g., gender) and one measurement variable (e.g., test scores). While one-way anova is used when the measurement variable distributes normally, K-W is used when the measurement variable does not.

Balanced v. Unbalanced Panel Data; "reg" v. "xtreg" Commands--Asking the Right Question

An admittedly small but nonetheless important point emerged in a recent StataList discussion (here). While the original query wondered whether a decision between the "reg" and "xtreg" commands pivoted on whether panel data were balanced or unbalanced, a very helpful commenter quickly made clear the question's pivotal (and mistaken) assumption. To be sure, while the "reg" and "xtreg" commands may yield the same or similar results in certain conditions, they estimate different models and different parameters. In general, where panel data are involved (balanced or unbalanced), as the commenter notes,"-reg- is simply inappropriate."