Empirical Legal Studies

Law Students, Role-Induced Bias, and the Inefficacy of De-Biasing Intervention Efforts

Those who teach in (as well as, frankly, attend) law schools may find a recent paper by Holger Spamann (Harvard) interesting. In Lawyers' Role-Induced Bias Arises Fast and Persists Despite Intervention, Spamann presents results from his experiment with Harvard Law students testing for evidence of role-induced bias and the efficacy of de-biasing efforts.

The experiment draws on students in "the basic class on corporations/business organizations taught by the same instructor with identical materials at two top ten U.S. law schools in six of the seven semesters from spring 2016 through spring 2019.”  Class enrollments (N) ranged from 43 to 104. “The purpose of the exercise is to engage students in a cutting-edge topic of insider trading law, while hopefully demonstrating to students the existence of own-side bias through a post-survey review of their debriefing survey.” A summary of the paper's key findings follows.

“Lawyers’ role-induced bias arises fast and persists despite a simple intervention, at least in an experiment with law students. After only 40 minutes or less of engagement with a case, law students exhibit substantial role-induced bias towards the side they have been randomly assigned to represent in the classroom. Of students randomly assigned to represent petitioner or respondent in a mock Supreme Court oral argument of U.S. v. Newman , over 50% of those assigned petitioner’s counsel’s role predicted petitioner victory, as opposed to less than 20% of those assigned respondent’s counsel’s role, with randomly assigned justices in the middle. These differences in prediction were actually starker when the role assignment was performed a mere 30 minutes before the 10-minute mock argument than when the assignment was performed five days prior, before the students ever started reading about the relevant law. Reminding students before making their prediction that their role in the oral argument had been “that of an advocate, whereas we are now stepping back out and […] asking your opinion as a neutral, sober observer” has no detectable effect on their biased predictions.” Moreover, similar to earlier research, “the bias occurs almost evenly in male and female participants, who also report equal confidence in their predictions.”

Criminal Misdemeanors in State Courts

As greater legal and policy attention turns to the intersection of criminal law--in particular, public-order policing--and low-income and communities of color, the need for well-executed empirical work increases. While much of the public attention tends to focus on high-profile federal criminal activity, criminal misdemeanors in state courts represent three-quarters of all criminal activity. Consequently, to gain a more accurate picture of overall criminal law activity the misdemeanor context is an obvious one to examine. To this end, in a recent paper, Misdemeanors by the Numbers, Mayson (Georgia) and Stevenson (GMU) take a deep dive into data from eight specific jurisdictions between 2011 and 2016. The paper's core findings are described in the excerpted abstract:

"The resulting portrait reveals a system that disproportionately impacts poor people and people of color. Between 2011 and 2016, each jurisdiction studied relied on monetary bail, which resulted in high rates of pretrial detention even at relatively low amounts, and imposed court costs upon conviction. There were substantial racial disparities in case-filing rates across locales and offense categories. The data also, however, highlight profound jurisdictional heterogeneity in how misdemeanors are defined and prosecuted. The variation suggests that misdemeanor adjudication systems may have fundamentally different characters, and serve different functions, from place to place. It thus presents a major challenge to efforts to describe and theorize the contemporary landscape of misdemeanor justice. At the most fundamental level, the variation calls into question the coherence of the very concept of a misdemeanor, or of misdemeanor criminal justice. As appreciation for the significance of lowlevel law enforcement builds, we urge scholars and policymakers to attend carefully to the complexity of this sub-felony world."

Everything You Ever Wanted to Know About Copyright Registrations Over Time

A recent paper reports results from the authors' effort to build what now has to be the single largest U.S. Copyright registration data set. In An Empirical Study of 225 Years of Copyright Registrations, Zvi Rosen and Richard Schwinn, report on what they find from observing more than two centuries of registration data. "This empirical analysis highlights two new insights that our analysis provides: The first is that the copyright term is, to a point, highly significant to registrations. In other words, a longer copyright term leads to more copyright registrations being filed. This is suggestive but not conclusive of a connection between a longer term and the creation of new works. Our analysis also shows that the incentive effect of a longer term offers diminishing returns above a certain point, and is no longer significant at a term of about 70 years. Secondly, we observe for the first time that the passage of a complete revision to the Copyright Law in 1909 which made registration (theoretically) optional without losing copyright protection represented a sharp paradigm shift for the relationship of copyright to economic progress.”

Equally interesting however, is the data set itself. As the authors note, “Until now, scholars have been dependent on limited tallies offered by the Annual Reports of the Register of Copyrights, which have provided the basis for all previous studies. In contrast, this study utilizes the more detailed statistics from the Catalog of Copyright Entries (“CCE”), which also have the benefit of corresponding to calendar year instead of the federal government’s fiscal year. This is also the first study ever to provide yearly estimates of copyright registrations per year prior to 1870 in any form – prior to this study the only figure used was 150,000 for the entire 80 year period… By taking account of changes in agency practice at the U.S. Copyright Office and its predecessors, we can harmonize our data over the entire series of 1790 to 2015.”

MTurk and Evidence on Measurement Error

While questions surrounding the potential for systematic measurement error from MTurk samples cannot be easily resolved in a single paper, given the popularity of MTurk samples in many fields, notably psychology, careful attention scholarly is warranted. The initial "warning" about MTurk samples came in 2013 from Dan Kahan (Yale), particularly for studies seeking to test “hypotheses about cognition and political conflict over societal risks and other policy-relevant facts.” Since then we've seen a developing literature that sets out to test potential limitations to MTurk samples. A recent paper by Douglas Ahler (FSU-Poli Sci ) et al., The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk, focuses on the potential for measurement error, especially Type II errors, injected by, among other factors, the "presence of 'non-respondents' (bots) or non-serious respondents on the platform." The paper's core findings include:

"While we find no evidence of a 'bot epidemic,' we do find that a significant portion of survey respondents engaged in suspicious behavior. About 20% of respondents either circumvented location requirements or took the survey multiple times. In addition, at least 5-7% of participants likely engaged in 'trolling' or satisficing. Altogether, we find about a quarter of data collected on MTurk is potentially untrustworthy. Expectedly, we find response quality impacts experimental treatments. On average, low quality responses attenuate treatment effects by approximately 9%. We conclude by providing recommendations for collecting data on MTurk."

Cognitive Bias, Judges, and Evidence Admissibility Contests

While research on individuals' susceptibility to various cognitive biases has blossomed over the years, the application of this particular research to judges and their decisions is less well-developed. A recent paper by Jeffrey Segal et al. (Stony Brook--Poli Sci), The 'Murder Scene Exception'—Myth or Reality? Empirically Testing the Influence of Crime Severity in Federal Search-and-Seizure Cases, contributes to the literature by assessing whether variation in crime severity informs judges' resolutions of contests over evidence admission. The paper draws from a sample (N=495) of federal appellate search-and-seizure cases from 1972 to 1986 and the excerpted abstract (below) summarizes its core findings.

"Using legislative criminal penalties to measure crime severity, our analysis shows that as crime severity increases, judges become significantly less likely to exclude challenged evidence on Fourth Amendment grounds—even though crime severity is not a doctrinally relevant consideration. Another legally extrinsic factor, the ideology of the opinion-writing judge, is also found to exert an influence, but only in the most serious criminal cases that involve a life sentence or the death penalty. In these particularly high-stakes decisions, conservative-leaning judges are more likely to uphold the admissibility of challenged evidence, while liberal-leaning judges are more likely to suppress it. Our data also indicate that the intrusiveness of the challenged police search, a doctrinally relevant factor, independently influences admissibility judgments."

When--and How--To Use Kruskal-Wallis

Helpful discussion, and example, of how to execute the Kruskal-Wallis ("K-W") test as well as an explanation on why one would turn to it (click here). In general (and in brief), folks typically turn to the K-W test when you have one nominal variable (e.g., gender) and one measurement variable (e.g., test scores). While one-way anova is used when the measurement variable distributes normally, K-W is used when the measurement variable does not.

Balanced v. Unbalanced Panel Data; "reg" v. "xtreg" Commands--Asking the Right Question

An admittedly small but nonetheless important point emerged in a recent StataList discussion (here). While the original query wondered whether a decision between the "reg" and "xtreg" commands pivoted on whether panel data were balanced or unbalanced, a very helpful commenter quickly made clear the question's pivotal (and mistaken) assumption. To be sure, while the "reg" and "xtreg" commands may yield the same or similar results in certain conditions, they estimate different models and different parameters. In general, where panel data are involved (balanced or unbalanced), as the commenter notes,"-reg- is simply inappropriate."

CELS 2019 Submission Deadline--Extended to June 28, 2019 (Midnight, Pacific Time)

Claremont McKenna College, led by conference co-organizers Eric Helland, David Bjerk, and Daniel Krauss, and the Society for Empirical Legal Studies (SELS) are pleased to announce the 14th Annual Conference on Empirical Legal Studies (CELS) on November 15-16, 2019, in Claremont, California.

CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

The paper submission deadline has been extended to June 28, 2019 (midnight, Pacific Time). To submit a paper or for more information about CELS 2019, including conference registration, please go to the CELS 2019 conference website (here).

Testing Police Officers' "Intuition" About "High-Crime" Areas

In a fascinating paper, The End of Intuition-Based High-Crime Areas, Ben Grunwald (Duke) and Jeffrey Fagan (Columbia) engage an important criminal procedure question with the benefit of a massive data set. The paper begins with the Supreme Court's conclusion in Wardlow that a "suspect’s presence in a 'high-crime area' is relevant in determining whether an officer has reasonable suspicion to conduct an investigative stop." Perhaps not surprisingly, the Court's opinion in 2000 provides little guidance about what constitutes a "high-crime" area. The paper sets out to assess how the doctrine was operationalized by examining data on more than two million investigative stops conducted by the NYC police from 2007 to 2012.

The data are especially rich in that in NYC police officers who conduct an investigative stop are required to complete a form after each one. "The forms collect rich information on suspect demographics and the precise geographic location of each stop. The data also contain anonymized officer identifiers, which allow us to observe how the same officer behaves in different areas, and how different officers behave in the same areas. And, most important for our purposes, the forms require officers to check off a series of roughly twenty boxes, indicating the bases of suspicion that justified the stop. Fortunately, one of those boxes is for high-crime areas. We merged this dataset with crime statistics and racial and demographic information on small geographic areas in New York City.”

As the paper makes clear, the data cast uneasy light on the Wardlow decision. "Our results suggest that Wardlow may have been wrongly decided. Specifically, we find evidence that officers often assess whether areas are high crime using a very broad geographic lens; that they call almost every block in the city high crime; that their assessments of whether an area is high crime are nearly uncorrelated with actual crime rates; that the suspect’s race predicts whether an officer calls an area high crime as well as the actual crime rate; that the racial composition of the area and the identity of the officer are stronger predictors of whether an officer calls an area high crime than the crime rate itself; and that stops are less or as likely to result in the detection of contraband when an officer invokes high-crime area as a basis of a stop. We conclude with several policy proposals for courts, police departments, and scholars to help address these problems in the doctrine.”

Equally interesting is that the authors point the way for fruitful follow-up empirical work on this topic. "In future empirical work, legal scholars can use body camera footage or SSO to validate the most common factors invoked by officers to establish reasonable suspicion. They can do so by assessing the factual accuracy of police officers’ claims about the presence of these factors and by examining whether they are in fact predictive of criminal behavior. Armed with that information, scholars and courts will have a clearer picture of what Fourth Amendment factors provide a reliable basis for reasonable suspicion and meaningful protection against unreasonable intrusions on personal privacy."

A New Legal Cloud Over Statistical Analyses of Judicial Decisions in France

In a move that will strike many court scholars (and, likely, others as well) as verging on the "unbelievable" (in the literal sense of the word), according to emerging news reports (e.g, here) the French Government has now "banned the publication of statistical information about judges’ decisions–with a five year prison sentence set as the maximum punishment for anyone who breaks the new law." The new french law (Art. 33 of France's Justice Reform Act) makes clear that "[t]he identity data of magistrates and members of the judiciary cannot be reused with the purpose or effect of evaluating, analysing, comparing or predicting their actual or alleged professional practices.” (translated; emphasis added)

According to news accounts, the new french law targets "legal tech companies focused on litigation prediction and analytics–from publicly revealing the pattern of judges’ behaviour in relation to court decisions." That said, the implications for empirical legal scholars seeking to model judicial decisions, certainly at the individual judge-level, in France are obvious.

While empirical legal scholars seeking to analyze U.S. judicial decisions face bothersome "hurdles" when it comes to linking judge names and their published decisions (e.g., a long-standing AO policy that strips judge names/IDs from most publicly-available AO data sets), I am unaware of anything in the U.S. that comes close to what the French Government seeks to accomplish with its new law.

Data on Immigration Judge Decisions Over Time

Public (and scholarly) debate over immigration laws and their enforcement all too often takes place in a data vacuum. A recent paper by Catherine Kim (UNC) and Amy Semet (Columbia), An Empirical Study of Political Control Over Immigration Adjudication, sets out to reduce this data paucity. Exploiting a large data set that includes every immigration removal proceeding decision from January 2001 through November 2018 (N=800,000+), the paper endeavors to isolate and assess the potential influence of an Administration's political preferences. The excerpted abstract, below, summarizes the paper's key findings.

"This Article presents the first comprehensive empirical assessment of the extent to which Immigration Judges (IJs), the administrative officials charged with adjudicating whether a given noncitizen will be deported from the country, decide cases on the basis of an Administration’s political preferences rather than an independent assessment of the legal merits of a given case. 

We constructed an original dataset of over 800,000 removal proceedings decided between January 2001 and November 2018. First, we found that every Administration—not just the current one—disproportionately appoints IJs with backgrounds in the former Immigration and Naturalization Service (INS), the Department of Homeland Security (DHS), or the Department of Justice (DOJ), agencies with responsibilities in prosecuting noncitizens. Second, using logistical regression to control for over a dozen variables that might impact a decision to remove, we found that the identity of the Administration that appointed an IJ was not a statistically significant predictor of the likelihood of ordering removal. That is, after controlling for other variables, Trump appointees were no more likely to order removal than Obama or Bush appointees.

Finally, using logistical regression and controlling for other variables, we found that the Administration in control at the time of decision is a statistically significant predictor of removal rates. For example, Bush appointees who served during the Bush, Obama, and Trump Eras were far more likely to order removal during the Trump Era. Specifically, when all other variables are held constant, they were 14% less likely to order removal during the Obama Era than during the current Era, and 12% less likely to order removal during the Bush Era than during the Trump Era. These results suggest that the current Executive-in-Chief exercises a profound influence over removal decisions, undermining the assumption of independence among administrative adjudicators."

Tips for More Readable Graphs

Over at Andrew Gelman's blog, John Kastellec (Princeton--Poli Sci) ignited a discussion (here) about "common but easily avoidable graphical mistakes" that hinder graphs' readability. While some suggestions are obvious, others are not. As well, the comments to John's post are as important and helpful as the post itself. When all is said and done, however, for my money the most important contribution to this topic remains Tufte's classic: The Visual Display of Quantitative Information (2nd ed.).

"georoute"--Measuring Distance and Time Between Two Points

While requiring a user-download, a new--albeit quirky--command, "georoute", allows researchers to compute "travel distance and travel time between two points defined by their addresses or their geographical coordinates." Or, more specifically:

"calculates the georouting distance between two addresses or two geographical points identified by their coordinates. It uses the HERE API (https://developer.here.com) to retrieve distances in two steps. In the first step, addresses are geocoded and their geographical coordinates (latitude and longitude) are obtained. In the second step, the georouting distance between the two points is obtained. The user can also provide directly geographical coordinates, which will bypass the first step."

Such a program may prove useful, e.g., to researchers exploring various "access to justice" research papers that seek to exploit/explore data on physical distance. A brief discussion is found here.

For those interested, to download this program, from within Stata, type: ssc install georoute

Incorporation & the "Delaware Trap"

In The Delaware Trap: An Empirical Analysis of Incorporation Decisions, Robert Anderson (Pepperdine) engages with, as he notes, "One of the most significant debates in corporate law," namely "whether the United States system of corporate law federalism leads to a race to the bottom or a race to the top. Race-to-the-bottom theorists argue that because insiders of companies must initiate incorporation decisions, jurisdictions compete to provide legal rules that favor insiders, allowing them to extract private benefits at the expense of the corporation or its shareholders. Race to-the-top theorists argue that market constraints prevent insiders from favoring such jurisdictions and that jurisdictions actually compete to provide efficient legal rules that enhance shareholder value. Although the dichotomous framing as a 'race' to the 'top' or 'bottom' is a bit of an oversimplification of a more nuanced debate, that version of the debate has dominated discussions of corporate law for decades.” And to this classic research question Anderson brings a fresh set of under-explored data: "filings made with the Securities and Exchange Commission under Regulation D. Regulation D is probably the most commonly used exemption from the Securities Act of 1933; it is used for private capital-raising and is relied on by over 10,000 public and private companies each year.”

Anderson's findings emphasize "“demographic markers of sophistication, such as choice of law firm and headquarters location, predict the jurisdictional choice about as well as state law or the business attributes of companies. Companies with more demographic markers of sophistication tend to choose Delaware incorporation, and companies with fewer demographic markers of sophistication tend to choose home-state incorporation.” And these findings persist after including many of the obvious controls.

Anderson interprets these results to imply that "the choice of legal representation is an important missing variable in models of incorporation decisions. The fact that the choice of law firms drives the jurisdictional choice has far broader implications. If law firms drive the jurisdictional choice they may steer companies toward states that serve the law firms’ own interests without regard to the quality of legal rules or the needs of the client. When the state chosen is Delaware, as it often is, there are few alternative jurisdictions that shareholders and managers can agree on. As a result, companies inadvertently fall into a 'governance trap' from which reincorporation out of state is nearly impossible. This interpretation would suggest that Delaware’s carefully calibrated positioning in the charter market has largely eliminated meaningful competition among the states for the quality of corporate law."

Prof. Frank R. Cross (1955-2019)

It is with sadness and a widely-shared sense of loss that I pass along the following note (from Prof. Stephen Griffin) on Frank Cross' passing.

"I wanted to record the passing of my wonderful friend Frank Cross, a longtime faculty member at the University of Texas, Austin. Other obituaries will recall that he was one of the best (NDT) college debaters of the 1970s and a gifted teacher. I wish to highlight the tremendous loss to academia and legal scholarship. Frank contributed enormously by continually challenging himself. Already a UT faculty member, he embarked on a course of study in statistics that transformed his work and gave us new insights. He began an extraordinary run of publication in leading law reviews with an article that questioned work that downplayed the contribution of lawyers to the economy. He published many noteworthy books including Decision Making in the U.S. Courts of Appeals (2007); Measuring Judicial Activism (2009)(with Stefanie A. Lindquist); The Failed Promise of Originalism (2012); The Theory and Practice of Statutory Interpretation (2012); and Constitutions and Religious Freedom (2015). Frank contributed as long as he could. Robert Prentice, his friend on the UT faculty, said in his obituary: 'Frank bravely battled adrenoleukodystrophy his entire adult life. This cruel disease slowly robbed Frank of his ability to move, but never took away his love of life or sense of humor. ALD ultimately defeated Frank, but it did not define him.' Everyone who knew Frank will miss him. He had this rare gift: an independent mind and an independent heart."