Empirical Legal Studies

Outliers

While there are no obvious "hard-line" rules when it comes to identifying outliers in a data set--let alone what to do with any that are identified--a recent wide-ranging discussion is found here. Moreover, Professor Williams (Notre Dame--sociology) has put together an extremely helpful and rich resource document (here). Finally, another possibly useful listserv discussion (albeit a dated one) is found here.

Call For Papers: Conference on Empirical Legal Studies–Europe 2020

Evidencing the growing internationalization of empirical legal studies, the 3rd Conference on Empirical Legal Studies in Europe (CELSE) will be held at the University of Oslo, 11-12 June 2020.

"CELSE is an interdisciplinary gathering that draws researchers from across Europe and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. CELSE will consider empirical papers across all areas of law, although priority will be given to papers with a European focus. Empirical analysis is understood to encompass any systematic approach to quantitative or qualitative data analysis, including statistical analysis, machine learning, experiments, text-mining and network analysis techniques.

Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants."

The deadline for submissions is 15 February 2020. Anyone with additional questions or seeking further information should click here.

Judicial Citation Patterns and NY County Surrogate's Courts

As ELS scholars continue to develop judge citation studies--typically from the "top-down" (that is, we know comparatively far more about judge citation patterns in the SCOTUS and other federal courts than non-federal judicial contexts)--comparatively far less is know about judge citation patterns in various lower and state court systems. A recent paper by Bridget Crawford (Pace), What Probate Courts Cite: Lessons from the New York County Surrogate's Court 2017-2018, contributes to a scholarly gap by describing the basic contours of judge citation patterns in New York County Surrogate's Court during 2017 and 2018. New York County's Surrogate's courts serve the most densely populated county in the United States. An excerpted abstract follows.

"... This Article makes three principal claims — one empirical, one interpretative, and one normative. This Article demonstrates through data, derived from a study of all decrees and orders issued by the New York County Surrogate’s Court in the years 2017 and 2018, that the probate court located in the most densely populated county in the United States cites fewer authorities less often than almost any other court (of any level) for which data is available.

There are a variety of factors that may explain this low rate of citation by the New York County Surrogate’s Court including docket size, the size and composition of the court’s staff, a judicial perception that the application of the law is a relatively mechanical process, or a subjective determination that speed in processing the court’s docket outweighs any public interest in citation-replete decrees and orders. Yet by increasing its engagement with a range of authorities, the New York County Surrogate’s Court (and indeed any probate court) may increase public confidence in the judiciary while also enhancing understanding of trusts and estates as a complex and dynamic area of law."

NYT, MTurk, and Data

Yesterday's New York Times ran a lengthy piece (here) on Amazon's Mechanical Turk ("MTurk") platform. The author joined the ranks of "Turkers" for a brief period and did some "Turking" to supplement his background research for the article. While the piece reflects a decidedly labor-economic focus, insofar as "MTurk"-created data remain a staple for an increasing slice of experimental psychological research (despite earlier concerns raised years ago by, among others, Dan Kahan (see, e.g., here and here)), the NYT''s article warrants a quick read.

CELS 2019--Conference Schedule

Co-sponsored by Claremont McKenna College and the Society for Empirical Legal Studies (SELS), the 14th annual CELS 2019 conference schedule is here.

ComplianceNet 2020 Conference--Call for Submissions

Yuval Feldman (Bar-Ilan, Israel) asked us to circulation information about the forthcoming ComplianceNet 2020 conference and we are, of course, delighted to do just that. Those interested should note the 1/24/2020 submission deadline.

The ComplianceNet 2020 conference will take place on June 29 – 30, 2020, and will be hosted by the University College London (UCL) School of Public Policy/Department of Political Science and the UCL Global Governance Institute. We are currently accepting panel or paper submissions, with a submission deadline of January 24, 2020.

ComplianceNet conferences seek to bring together scholars from a range of different disciplines to study the interaction between rules and human behaviour, broadly defined. The first two conferences were highly successful and brought together academics from business, criminology, economics, law, political science, psychology and sociology, among other fields. ComplianceNet’s website includes information about its structure and goals (click here) as well as the 2020 conference (click here).

QuantLaw 2020--Univ. Arizona College of Law--Call for Abstracts

As a fan--and proponent--of various genres of the "micro-conference" (that is, a 1-2 day topically/thematically-focused conference marked by an emphasis on scholarship by a limited number of junior scholars, a low-ratio of senior-to-junior scholars, and a prevailing norm/expectation that all participants read all presented work prior to the conference), I am delighted to pass along the following (excerpted) information about the upcoming 2020 iteration of Arizona's successful QuantLaw conference from Jane Bambauer (Arizona).

"The 2020 QuantLaw conference will take place February 27-28 (Thursday and Friday) in Tucson, Arizona. QuantLaw is a unique conference for scholars who apply empirical methods to the study of law. The program is both rigorous and laid back. Part of the QuantLaw program involves standard presentations of works-in-progress by our keynote speakers, by our returning ["senior"] gurus, and by faculty members and graduate students from many different disciplines. The rest of the program is designed to be an incubator. We will have a structured conversation over dinner one night that focuses on a methodological or philosophical topic related to empirical legal studies, and we will also discuss brand new works-in-progress while hiking in the beautiful Sonoran Desert.

QuantLaw 2020 will feature programming around the theme of experiments: experimental methods, and how best to combine experimental and observational data to get a more accurate picture. Kathryn Zeller (Boston University) will present a keynote address.

If you would like to present new empirical work related to law or public policy, please prepare an extended abstract (approximately 500-1000 words in length) describing the topic, the tested hypothesis, the methods you plan to use, and whether you expect to have results to share at the conference. Send your extended abstract and a current CV to Prof. Jane Bambauer (click here for contact info) by November 30, 2019. We will give preference to projects using experimental methods given this year’s theme. We will also give preference to graduate students and junior faculty members. Applicants who are selected to present will receive boarding accommodations in Tucson."

Again, those with questions or seeking additional information should contact Prof. Bambauer directly.

Graphing Counterfactual Regression Trend Lines

In a recent Statlist discussion (here), the question was posed about how to: "generate a graph to show how to the GDP per capita trend line would have continued if the conflict [or any discrete event] had not occurred." (emphasis added) Put slightly differently, the question involves how to over-lay two separate time-trend lines: one from actual data reflecting the influence of a discrete event and another trend line reflecting a counterfactual regression trend (that is, as if the discrete event did not happen).

To illustrate, the graph (above) plots and compares "the effect that the civil conflict in Mozambique and Rwanda have had on the GDP per capita growth in the long run" and over-lays a projected regression line reflecting GDP per capita assuming that the civil conflict did not emerge in these two countries.

While the Stata graphics coding can be tricky, the end result is a quickly interpretable graph.

"Zeros" Problems

While not unique to empirical legal scholarship, how to deal with a "zeros problem" is not uncommon, especially given the nature of many research designs and questions in legal scholarship. To take one obvious--and well-trod--example, efforts to model punitive damage awards need to account for their absence in the over-whelming majority of civil cases decided by courts. A recent discussion on the StataList (here) provides yet another take on the array of challenges posed by "zeros" in data sets. The resultant discussions emphasizes, in part, that researchers need to assess with accuracy and clarity, ex ante, precisely what a "zero" means in the context of their data sets. As well, for a recent take on differences between Heckman selection and two-part models, a 2015 JELS paper (here), by Eisenberg et al., remains current.

The Harvard College Admissions Decision, The Import of Statistical Evidence, and Competing Modeling Assumptions

Focusing solely on the methodological aspect of the recent Harvard College admissions decision, one under-appreciated take-away from the district court's opinion (click here for the full decision) is the prominent role that traditional statistical analyses (basically, standard logistic models) played in the trial. If nothing else, the prominent role that statistical evidence bearing on the plaintiffs' claims, and Harvard's defense, played in the trial makes palpably clear, once again, the extent to which sophisticated litigation now--and almost routinely so in some case types--deploys empirical evidence. Indeed, the court's decision devotes more than 40 pages (pp.50-92) of its 130-page opinion to statistical matters.

In the Harvard admissions case, both parties to the litigation formally engaged experts, Professors David Card (Berkeley) and Peter Arcidiacono (Duke), to analyze a shared data set--six years worth of Harvard College admissions decisions. Despite a shared data set the two experts reached opposing conclusions on the critical question about the unique role that applicants' race/ethnicity played in Harvard's admissions decisions. Much of the experts' disagreement pivoted on one key modeling decision--whether to include or exclude admissions data germane to the "ALDC" applications. ("ALDC" applications are those involving: "recruited athletes, legacies, Dean's 'interest list' and children of Harvard faculty and staff.")

In the face of conflicting conclusions from the two experts, Judge Burroughs wrote that while she found both models “'econometrically defensible' and preferred different aspects of each model," in the end Judge Burroughs' decision largely preferenced Card's modeling assumptions over Arcidiacono's. Independent of whether one feels Judge Burroughs' assessment of the competing modeling decisions is persuasive, what is objectively clear is that she took the empirical and statistical evidence seriously and that it played a consequential--and potentially pivotal--role in the trial.

"Paired" v. "Unpaired" T-Test

Efforts to compare the means from two sample populations with a standard t-test invariably bump into a "paired" versus and "unpaired" decision. Whether a "paired" or "unpaired" comparison is appropriate pivots on a subtle distinction that can be difficult to describe. A recent Statalist discussion might provide a helpful way to think about this distinction.

The discussion (here) began with the following (edited) question: "If there is economic growth data about two countries over the same thirty years, should it be considered as paired or unpaired data for two sample t-test?"

The various responses included the following: "For unpaired tests imagine 30 men and 30 women completely unrelated and compare their heights. You are comparing the mean male height and mean female height and the context is the variability of male heights and of female heights. For paired tests imagine 30 heterosexual married couples and compare height for each husband with each wife."

With this in mind, the answer to the question posed is: "Your panel example is still paired because there is pairing by years."

Log Transformations & the "Zeros Problem"

Strategies to address normal distribution assumption concerns can quickly lead to the log transformation of variables. However, what to do with "zeros" in a variable requiring log transformation persists as an under-studied issue. And one well-known example, punitive damage awards, makes clear that such "zeros problems" are not uncommon in many empirical legal research projects.

In Dealing With the Log of Zero in Regression Models, the authors, Christophe Bellego (Center for Research in Economics and Statistics) and Louis-Danile Pape (Center for Research in Economics and Statistics), engage with this nettlesome issue head-on. The brief (15 pp.) paper's abstract follows.

"Log-linear and log-log regressions are one of the most used statistical model. However, handling zeros in the dependent and independent variable has remained obscure despite the prevalence of the situation. In this paper, we discuss how to deal with this issue. We show that using Pseudo-Poisson Maximum Likelihood (PPML) is a good practice compared to other approximate solutions. We then introduce a new complementary solution to deal with zeros consisting in adding a positive value specific to each observation that avoids some numerical issues faced by the former."

Ready-To-Use Data (and Codebooks) for Law & Education Research

One thing that the U.S. Department of Education does reasonably well is gather data--and make them publicly available. And as the number and complexity of law and education issues continue to increase over time, greater access to relevant data becomes even more important. To this end, the Dept. of Education's National Center for Education Statistics ("NCES") operates a landing spot (here) for direct access to ready-to-use public use data sets and codebooks. Users can select from an array of standard formats (Stata, SPSS, SAS, R, etc.) and tailor requests.

Instrumental Variables Regression--A Helpful Intro

For a brief (5+ min) introduction to--or a very quick refresher on--the basics of instrumental variables regression, including some post-estimation commands, one could do far worse than this Stata video. The video walks viewers through the basics on how to fit an instrumental variables model for endogenous covariates (using the "ivregress" command).

Comparative Competition Law & Datasets

Consistent with JELS' impulse to recently launch a new genre of papers--"Research Notes and Datasets"--a recent contribution to this initiative, Competition Law Gone Global: Introducing the Comparative Competition Law and Enforcement Datasets, by Anu Braford (Columbia), Adam Chilton (Chicago), Christopher Megaw (WilmerHale), and Nathaniel Sokol, introduces and discusses a pair of datasets that provide future researchers with "a new foundation for empirical research on the legal regimes used to regulate markets."

"Competition law has proliferated around the world. Due to data limitations, however, there is little systematic information about the substance and enforcement of these laws. In this article, we address that problem by introducing two new datasets on competition law regimes around the world. First, we introduce the Comparative Competition Law Dataset, which codes competition laws in 131 jurisdictions between 1889 to 2010. Second, we introduce the Comparative Competition Enforcement Dataset, which provides data on competition agencies’ resources and activities in 100 jurisdictions between 1990 and 2010. These datasets offer the most comprehensive picture of competition law yet assembled and provide a new foundation for empirical research on the legal regimes used to regulate markets."

So far, so good. But the authors recently undertook yet another step that warrants attention (and praise). Specifically, Bradford and Chilton created a new website (here) that already includes an array of ready-to-use datasets (and associated codebooks) germane to comparative competition law (see, e.g., here). According to the project's website, "this project generates novel data on competition regimes across time and countries. Using these data, the project studies competition law and enforcement around the world, including exploring the diffusion of these laws and the effects they have on market outcomes."

Frankly, the construction and public distribution of helpful and important datasets are among the most the most under-appreciated of endeavors. Future scholars (and scholarship) will gain from initiatives like this one.