Empirical Legal Studies

Re-Scheduled CELS--From 2020 to 2021

It is with a heavy heart that I pass along the following CELS update from the Society for Empirical Legal Studies' Board of Directors:

Empirical Efforts to Assess Recidivism Risk--Is There a Role For Algorithms?

In assessing recidivism risk, policymakers increasingly turn to algorithms. This turn remains contested. Many of these contests pivot on claims that algorithms themselves incorporate various biases. More specifically, "some critics oppose the use of risk assessment instruments to scaffold criminal justice reform efforts, based on concerns that algorithms are racially biased and could result in inequitable outcomes for already disadvantaged groups."

Of course, and as made clear in a recent paper by Jennifer Skeem (Berkeley--public policy) and Christopher Lowenkamp (AO, US Courts), Using Algorithms to Address Trade-Offs Inherent in Predicting Recidivism, "concerns about the legitimacy of specific risk factors and the potential discriminatory effects of using risk assessment instruments are not new—and can be examined empirically." This paper takes up the challenge by comparing "how alternative strategies for 'debiasing' risk assessment algorithms affect these tradeoffs, based on a large sample of Black and White federal probationers assessed with the Post Conviction Risk Assessment [PCRA]." The abstract follows.

"Although risk assessment has increasingly been used as a tool to help reform the criminal justice system, some stakeholders are adamantly opposed to using algorithms. The principal concern is that any benefits achieved by safely reducing rates of incarceration will be offset by costs to racial justice claimed to be inherent in the algorithms themselves. But fairness tradeoffs are inherent to the task of predicting recidivism, whether the prediction is made by an algorithm or human. Based on a matched sample of 67,784 Black and White federal supervisees assessed with the Post Conviction Risk Assessment (PCRA), we compare how three alternative strategies for “debiasing” algorithms affect these tradeoffs, using arrest for a violent crime as the criterion. These candidate algorithms all strongly predict violent re-offending (AUCs=.71-72), but vary in their association with race (r= .00-.21) and shift tradeoffs between balance in positive predictive value and false positive rates. Providing algorithms with access to race (rather than omitting race or ‘blinding’ its effects) can maximize calibration and minimize imbalanced error rates. Implications for policymakers with value preferences for efficiency vs. equity are discussed."

All About Multilevel Modelling Resources

With an admitted bias for "one-stop" resource sites, I recently came across a very thorough (and thoroughly helpful) site sponsored by the University of Bristol's (UK) Centre for Multilevel Modeling (here). Multilevel modelling's usefulness for many ELS research projects is as acute as it is obvious. The site teems with an array of focused resources, including a deep pool of useful online instructional videos (here). For anyone interested in a brief overview of multilevel modelling, this quick explanatory video (here) might be one place to start; a series of PowerPoint presentations (found here) is another.

Interrupted Time Series Regression

I note this recent discussion in Statalist (click here) only because an interrupted time series research design is not infrequently helpful in empirical legal studies. Note also comment No.7's point about the assumption that the external shock (the "interruption") applying equally across all the firms in the panel.

Local Prosecutor Elections, Prosecutorial Reforms, and Geography

A growing movement of local prosecutor candidates running for office with clear "reform" agendas (e.g., decriminalization of various drug laws through express non-enforcement, deincarceration, etc.) assumes the efficacy of the ballot box as an instrument of policy reform. Such a movement, however, risks a collision with received academic wisdom that local prosecutor elections are little more than "empty exercises" owing to an enormous "incumbency advantage." In a recent paper, Picking Prosecutors, Carissa Hessick (UNC) and Michael Morse (Harvard--Govt.) submit the received academic assumption to data. The data include prosecutor election results drawn from 2,314 separate districts across 45 states during the 2014 (or 2016) election cycle.

While largely descriptive, what they find includes the expected as well as the unexpected. Consistent with the "received wisdom," Hessick and Morse find that "incumbent prosecutors rarely face challengers and almost always win." One complicating wrinkle, however, is that the "degree of [prosecutor] incumbent entrenchment" is a function of an incumbent's time in office. Finally, one surprise is the salience of geography: or, more specifically, "a stark divide between rural and urban prosecution." A summary of this point is described in the excerpted abstract (below).

"Urban areas are more likely to hold a contested election than rural areas. Rural areas, in which very few lawyers live, rarely hold contested elections and sometimes are not able to field even a single candidate for a prosecutor election. The results suggest that the nascent movement to use prosecutor elections as a source of criminal justice reform may have success, at least in the short term. But elections are, as of now, not a likely source of reform in rural areas—the very areas where incarceration rates continue to rise."

Assessing Normal Distributions

A nice illustrative discussion (here) on various approaches to assess whether "normal" distribution assumptions plausibly hold. Participants in the discussion provide a helpful illustration (below) on two perspectives (kernel density vs. qnorm) on a possible bi-modal distribution.

CELS 2020--Call For Papers

Spearheaded by CELS 2020 co-presidents Andrew Green, Anthony Niblett, and Albert Yoon, the University of Toronto Faculty of Law and the Society for Empirical Legal Studies (SELS) are pleased to announce the 15th Annual Conference on Empirical Legal Studies (CELS) on October 2-3, 2020, in Toronto, ON, Canada.

Paper Submission Deadline: June 5, 2020 (midnight, Eastern Time).

To submit a paper, please go to the 2020 CELS website (click here) .

CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

We are of course monitoring the progress of the COVID-19 pandemic. At this point we are hopeful the conference will proceed as planned. If the SELS Board deems it necessary to change our plans, we will let you know as soon as possible.

If you have additional questions about the submissions process, please contact cels2020.law@utoronto.ca

There is no charge for paper submissions.

A Research Request

Prompted partly by an earlier post (here) remarking on how the COVID-19 crisis "and the near-universal (and simultaneous) reaction of U.S. law schools to turn to on-line instruction may provide quasi-experimental research design opportunities," two loyal ElsBlog readers, John Bliss (Denver) and David Sandomierski (Western Ontario), who are launching a research project relating to law school grading policy changes, asked that I share the following survey research request, which I am happy to do:

"We write to ask for your help filling out a brief, anonymous research survey. Many law schools have recently changed their grading schemes in response to COVID-19, suspending their normal grading system in favor of pass/fail or honors/pass/fail systems. Our survey asks what considerations went into your choice of grading scheme and how you expect this change to impact your students.

The survey is anonymous and will be reported anonymously. You may complete the survey by clicking on this link: https://uwo.eu.qualtrics.com/jfe/form/SV_9H4HYsRjqVq3LUN?Source=pers&ID=JB.

We thank you for considering this and look forward to sharing the results!

Many thanks,"

Professor John Bliss (University of Denver Sturm College of Law)

Professor David Sandomierski (University of Western Ontario Faculty of Law)

Mandatory Minimums, Sentencing, and Endogenity

A recent paper, (How Much) Do Mandatory Minimums Matter?, by Stephanie Holmes Didwania (Temple), takes another shot at a persistent challenge in the growing sentencing literature. As the paper's abstract makes clear, "Understanding the relationship between mandatory minimums and sentencing outcomes is often difficult due to the endogeneity of mandatory minimum charging." To parse this challenge the paper exploits a quasi-natural experiment generated by a sweeping policy change imposed by an August 2013 directive from then-U.S. Attorney General Eric Holder that instructed all U.S. attorneys to cease charging mandatory minimums in drug cases for certain low-level offenders. As the paper makes clear, however, Holder's charging policy change "did not work." More specifically, while Holder's policy change did, in fact, succeed in reducing mandatory minimum charging, when it came to actual sentencing outcomes, however, the effect was "modest."

What explains the "curious" result? The paper hypothesizes that the targeted beneficiaries of the Holder policy change in 2013--certain low-level federal drug offenders--were, in effect, already receiving sentence reductions through prosecutors' use of "safety-valve and substantial assistance reductions" prior to 2013. To better achieve the goal of reducing actual sentences for certain low-level federal drug defendants, the paper suggests that "advocates ... should expand their efforts to promoting interventions that reach more serious offenders and on reforming the U.S. Sentencing Guidelines.”

Parsing Police Oversight's Influence on Crime and Police Complaints

A recent paper by Bocar Ba (Penn) and Roman Rivera (Columbia), The Effect of Police Oversight on Crime and Allegations of Misconduct: Evidence from Chicago, considers whether increased police oversight mechanisms contribute to increased crime levels as well as citizens' complaints about police conduct. In addition to what the paper finds, how the authors execute their research design also warrants note.

“[W]e do not find evidence that crime increases when [police] self-monitoring increases. We do find evidence that rates of property crime, murder, and robbery increase significantly after the scandal. Our preferred specification indicates that the OIS Scandal is associated with 18.4 additional murders in Chicago per month (p>0.01) in the year following the scandal. For comparison, self-monitoring is associated with fewer than two extra murders per month, but this is not statistically significant from zero.”

Aside from plumbing a rich source of data, this paper sets out to adjust for what it terms “simultaneity bias” that arises because “[police department] scandals induce officers to increase self-monitoring as a form of self-protection against misconduct accusations and further scandal. However, as a result of the civil unrest and civilian distrust following a scandal, civilians simultaneously become more critical of officer actions and less likely to obey laws and authorities.” To adjust for “simultaneity bias” the paper exploits events that only influence police monitoring incentives and not civilian incentives (e.g., civilian complaints about policing). Two factors are critical to the paper’s success. One involves the plausibility of such events performing as assumed. Another pertains to the generalizability of Chicago’s (admittedly enviable) granular data aggregated at the police beat-level.

The Empirics of Legal Education: Australia 2000-2016

A forthcoming book chapter by Alex Steel (Univ. New South Wales), Imperatives for Legal Education: Then, Now and Tomorrow, surveys empirical studies focusing on various aspects of legal education in Australia. In so doing, the chapter also sheds indirect light on two other broader themes. One is the growing internationalization of empirical legal studies, broadly understood. A second is an increasingly self-conscious turn to submit legal education to rigorous empirical study. It is one thing for empirical legal scholars to explore legal theory, doctrine, and institutions from an empirical perspective. Deploying this perspective inward--and assessing whether and, if so, how, legal education achieves its various objectives--warrants note. On a related--though distinct--point, my sense is that the external shock imposed by the COVID-19 crisis and the near-universal (and simultaneous) reaction of U.S. law schools to turn to on-line instruction may provide quasi-experimental research design opportunities. The paper's abstract follows.

"Recent decades have seen a growing understanding that legal education can be improved and informed by empirical analysis of curriculum design, teaching approaches, student learning strategies and broader contextual factors. However, access to that analysis has been impeded by a lack of broad awareness of its existence and difficulty in locating publications.

This chapter provides a review of empirically based studies into Australian legal education from 2000–2017. The analysis demonstrates a significant rise in empirically based publications and an increasing sophistication with empirical methods. The chapter highlights trends in subjects examined empirically, the most commonly used methods, and publications where such analyses are most commonly found.

In order to make this research more available, in appendixes the chapter also lists the full database of empirical legal research publications on which the analysis was based."

UPDATE: Northwestern University Law Review—3rd Annual Empirical Legal Scholarship Issue: Call for Papers

"Given the COVID-19 pandemic, the Northwestern University Law Review has extended the exclusive submissions deadline for the Empirical Issue. The deadline for all empirical submissions is now May 1, 2020."

Please note that "the submission process and guidelines described in the original Call for Papers (here) remain the same. The Northwestern University Law Review considers empirical submissions on an exclusive basis only. To be considered for the Empirical Issue, pull your piece from the Scholastica pool and from consideration at all other schools, and submit the manuscript, your CV, and data/data paths via email at nulrempirical@gmail.com." For additional details on this update click here.

AI and Machine Learning: Algorithms and the Law

A pair of papers that crossed my desk recently, both circling around a similar topic and yet providing slightly different perspectives and analyses, reflects a palpable increase in scholarly attention to the various intersections involving AI, machine learning, algorithms, and legal doctrine. As both papers note, "predictive algorithms are increasingly being deployed in a variety of settings to determine legal status." To date, for example, "there are approximately 60 risk assessment tools deployed in the criminal justice system. These tools aim to differentiate between low-, medium-, and high-risk defendants and to increase the likelihood that only those who pose a risk to public safety or are likely to flee are detained."

"Proponents of actuarial tools claim that these tools meant to eliminate human biases and to rationalize the decision-making process by summarizing all relevant information in a more efficient way than the human brain. Opponents of such tools fear that in the name of science, actuarial tools reinforce human biases, harm defendants’ rights and increase racial disparities in the system. The gap between the two camps has widened in the last few years."

One paper, Dan Burk's (UC-Irving), Algorithmic Legal Metrics, seeks to: "link the the sociological and legal analysis of AI, highlighting the reflexive social processes that are engaged by algorithmic metrics. This paper examines these overlooked social effects of predictive legal algorithms, and contributes to the literature a vital fundamental but missing critique of such analytics. Specifically, this paper shows how the problematic social effects of algorithmic legal metrics extend far beyond the concerns about accuracy that have thus far dominated critiques of such metrics. Second, it demonstrates that corrective governance mechanisms such as enhanced due process or transparency will be inadequate to remedy such corrosive effects, and that some such remedies, such as transparency, may actually exacerbate the worst effects of algorithmic governmentality. Third, the paper shows that the application of algorithmic metrics to legal decisions aggravates the latent tensions between equity and autonomy in liberal institutions, undermining democratic values in a manner and on a scale not previously experienced by human societies. Illuminating these effects casts new light on the inherent social costs of AI metrics, particularly the perverse effects of deploying algorithms in legal systems."

Another paper, doaa Abu Elyounes' (Harvard—Berkman Center), Bail or Jail? Judicial Versus Algorithmic Decision-Making in the Pretrial System, delves a bit more concretely into the criminal law realm. The paper "examines the role that the technology play in this debate, and whether deploying AI in existing risk assessment tools realizes the fears hyped in the media or improves our criminal justice system? It focuses on the pretrial stage and examines in depth the seven most commonly used tools. Five of these tools are based on traditional regression analysis, and two have a certain machine-learning component. The paper concludes that, classifying pretrial risk assessment tools as AI-based tools creates the impression that sophisticated robots are taking over the courts and pushing judges from their jobs, but this is far from reality. Despite the hype, there are more similarities than differences between tools based on traditional regression analysis and tools based on machine learning. Robots have a long way to go before they can replace judges, and this is not the solution that this paper is arguing for."

Scatterplots With "Weighted" Markers

For those with the time and inclination to delve deeply into graph "presentation" issues, a (related) pair of discussion strings from the Statalist (here and here), discuss how to generate scatter plots with "weighted" markers to assist readers with "visualization for multidimensional data." The resultant output can look nice:

Submitting Bail Reforms to Data: Competing Interpretations in Cook County, Illinois

Various bail reforms are quickly emerging across jurisdictions. Proponents argue, at bottom, that such reforms will not impose undue increased risk on public safety. Such a claim is, of course, an empirical one. And, as is far too common in the “legal world,” such claims are made without the benefit of careful empirical study.

Enter bail reform initiatives in Cook County, Ill (functionally, Chicago) that were implemented in 2017 and studied in 2019. What has ensued—and is the focus of a recent paper by Paul Cassell (Utah) and Richard Fowles (Utah—econ), Does Bail Reform Increase Crime? An Empirical Assessment of the Public Safety Implications of Bail Reform in Cook County, Illinois—is a dispute over how to best interpret the study’s (that is, the “Bail Reform Study,” supervised by the Cook County Circuit Court) findings.

While the Bail Reform Study’s authors landed on far more “optimistic”  conclusions, the Cassell and Fowles paper, by contrast, has “serious doubts about its upbeat conclusions.” Their paper drills down on two main points. First, “the [Bail Reform] Study fails to recognize that, given that more defendants are being released after the reforms, even a 'stable' rate of community safety will inexorably lead to more crimes. That stable rate of safety—and, inversely, the stable rate of failure or public safety danger—applies across a larger pool of released defendants, which necessarily means that the public suffers additional crimes. In other words, at least in Cook County, more bail reform apparently means more crimes." Second, “we find that, contrary to the Study’s suggestion of stability, the number of crimes committed by pretrial releasees appears to have significantly increased.” A slightly excerpted abstract follows.

“Recently bail reform issues have been in the news across the country, as concerns about fair treatment of defendants and possible public safety risks from expanding pretrial release have collided. These issues involve important empirical questions, including whether releasing more defendants before trial leads to additional crimes. An opportunity to investigate this public safety issue has developed in Chicago, our nation’s third largest city. There, the Office of the Chief Judge of the Cook County Courts adopted new bail reform measures in September 2017 and reviewed them empirically in May 2019. Cook County’s Bail Reform Study concluded that the new procedures had released many more defendants before trial without any concomitant increase in crime. This article disputes the Study’s conclusions. This article explains that, contrary to the Study’s assertions, the new changes to pretrial release procedures appear to have led to a substantial increase in crimes committed by pretrial releasees in Cook County. Properly measured and estimated, after more generous release procedures were put in place, the number of released defendants charged with committing new crimes increased by 45%. And, more concerning, the number of pretrial releasees charged with committing new violent crimes increased by an estimated 33%. In addition, as reported by the Chicago Tribune, the Study’s data appears to undercount the number of releasees charged with new violent crimes; and a substantial number of aggravated domestic violence prosecutions prosecutors dropped after the changes, presumably because batterers were able to more frequently obtain release and intimidate their victims into not pursuing charges. These public safety concerns call into question whether the bail “reform” measures implemented in Cook County were cost-beneficial.”