Empirical Legal Studies

The Empirics of Legal Education: Australia 2000-2016

A forthcoming book chapter by Alex Steel (Univ. New South Wales), Imperatives for Legal Education: Then, Now and Tomorrow, surveys empirical studies focusing on various aspects of legal education in Australia. In so doing, the chapter also sheds indirect light on two other broader themes. One is the growing internationalization of empirical legal studies, broadly understood. A second is an increasingly self-conscious turn to submit legal education to rigorous empirical study. It is one thing for empirical legal scholars to explore legal theory, doctrine, and institutions from an empirical perspective. Deploying this perspective inward--and assessing whether and, if so, how, legal education achieves its various objectives--warrants note. On a related--though distinct--point, my sense is that the external shock imposed by the COVID-19 crisis and the near-universal (and simultaneous) reaction of U.S. law schools to turn to on-line instruction may provide quasi-experimental research design opportunities. The paper's abstract follows.

"Recent decades have seen a growing understanding that legal education can be improved and informed by empirical analysis of curriculum design, teaching approaches, student learning strategies and broader contextual factors. However, access to that analysis has been impeded by a lack of broad awareness of its existence and difficulty in locating publications.

This chapter provides a review of empirically based studies into Australian legal education from 2000–2017. The analysis demonstrates a significant rise in empirically based publications and an increasing sophistication with empirical methods. The chapter highlights trends in subjects examined empirically, the most commonly used methods, and publications where such analyses are most commonly found.

In order to make this research more available, in appendixes the chapter also lists the full database of empirical legal research publications on which the analysis was based."

UPDATE: Northwestern University Law Review—3rd Annual Empirical Legal Scholarship Issue: Call for Papers

"Given the COVID-19 pandemic, the Northwestern University Law Review has extended the exclusive submissions deadline for the Empirical Issue. The deadline for all empirical submissions is now May 1, 2020."

Please note that "the submission process and guidelines described in the original Call for Papers (here) remain the same. The Northwestern University Law Review considers empirical submissions on an exclusive basis only. To be considered for the Empirical Issue, pull your piece from the Scholastica pool and from consideration at all other schools, and submit the manuscript, your CV, and data/data paths via email at nulrempirical@gmail.com." For additional details on this update click here.

AI and Machine Learning: Algorithms and the Law

A pair of papers that crossed my desk recently, both circling around a similar topic and yet providing slightly different perspectives and analyses, reflects a palpable increase in scholarly attention to the various intersections involving AI, machine learning, algorithms, and legal doctrine. As both papers note, "predictive algorithms are increasingly being deployed in a variety of settings to determine legal status." To date, for example, "there are approximately 60 risk assessment tools deployed in the criminal justice system. These tools aim to differentiate between low-, medium-, and high-risk defendants and to increase the likelihood that only those who pose a risk to public safety or are likely to flee are detained."

"Proponents of actuarial tools claim that these tools meant to eliminate human biases and to rationalize the decision-making process by summarizing all relevant information in a more efficient way than the human brain. Opponents of such tools fear that in the name of science, actuarial tools reinforce human biases, harm defendants’ rights and increase racial disparities in the system. The gap between the two camps has widened in the last few years."

One paper, Dan Burk's (UC-Irving), Algorithmic Legal Metrics, seeks to: "link the the sociological and legal analysis of AI, highlighting the reflexive social processes that are engaged by algorithmic metrics. This paper examines these overlooked social effects of predictive legal algorithms, and contributes to the literature a vital fundamental but missing critique of such analytics. Specifically, this paper shows how the problematic social effects of algorithmic legal metrics extend far beyond the concerns about accuracy that have thus far dominated critiques of such metrics. Second, it demonstrates that corrective governance mechanisms such as enhanced due process or transparency will be inadequate to remedy such corrosive effects, and that some such remedies, such as transparency, may actually exacerbate the worst effects of algorithmic governmentality. Third, the paper shows that the application of algorithmic metrics to legal decisions aggravates the latent tensions between equity and autonomy in liberal institutions, undermining democratic values in a manner and on a scale not previously experienced by human societies. Illuminating these effects casts new light on the inherent social costs of AI metrics, particularly the perverse effects of deploying algorithms in legal systems."

Another paper, doaa Abu Elyounes' (Harvard—Berkman Center), Bail or Jail? Judicial Versus Algorithmic Decision-Making in the Pretrial System, delves a bit more concretely into the criminal law realm. The paper "examines the role that the technology play in this debate, and whether deploying AI in existing risk assessment tools realizes the fears hyped in the media or improves our criminal justice system? It focuses on the pretrial stage and examines in depth the seven most commonly used tools. Five of these tools are based on traditional regression analysis, and two have a certain machine-learning component. The paper concludes that, classifying pretrial risk assessment tools as AI-based tools creates the impression that sophisticated robots are taking over the courts and pushing judges from their jobs, but this is far from reality. Despite the hype, there are more similarities than differences between tools based on traditional regression analysis and tools based on machine learning. Robots have a long way to go before they can replace judges, and this is not the solution that this paper is arguing for."

Scatterplots With "Weighted" Markers

For those with the time and inclination to delve deeply into graph "presentation" issues, a (related) pair of discussion strings from the Statalist (here and here), discuss how to generate scatter plots with "weighted" markers to assist readers with "visualization for multidimensional data." The resultant output can look nice:

Submitting Bail Reforms to Data: Competing Interpretations in Cook County, Illinois

Various bail reforms are quickly emerging across jurisdictions. Proponents argue, at bottom, that such reforms will not impose undue increased risk on public safety. Such a claim is, of course, an empirical one. And, as is far too common in the “legal world,” such claims are made without the benefit of careful empirical study.

Enter bail reform initiatives in Cook County, Ill (functionally, Chicago) that were implemented in 2017 and studied in 2019. What has ensued—and is the focus of a recent paper by Paul Cassell (Utah) and Richard Fowles (Utah—econ), Does Bail Reform Increase Crime? An Empirical Assessment of the Public Safety Implications of Bail Reform in Cook County, Illinois—is a dispute over how to best interpret the study’s (that is, the “Bail Reform Study,” supervised by the Cook County Circuit Court) findings.

While the Bail Reform Study’s authors landed on far more “optimistic”  conclusions, the Cassell and Fowles paper, by contrast, has “serious doubts about its upbeat conclusions.” Their paper drills down on two main points. First, “the [Bail Reform] Study fails to recognize that, given that more defendants are being released after the reforms, even a 'stable' rate of community safety will inexorably lead to more crimes. That stable rate of safety—and, inversely, the stable rate of failure or public safety danger—applies across a larger pool of released defendants, which necessarily means that the public suffers additional crimes. In other words, at least in Cook County, more bail reform apparently means more crimes." Second, “we find that, contrary to the Study’s suggestion of stability, the number of crimes committed by pretrial releasees appears to have significantly increased.” A slightly excerpted abstract follows.

“Recently bail reform issues have been in the news across the country, as concerns about fair treatment of defendants and possible public safety risks from expanding pretrial release have collided. These issues involve important empirical questions, including whether releasing more defendants before trial leads to additional crimes. An opportunity to investigate this public safety issue has developed in Chicago, our nation’s third largest city. There, the Office of the Chief Judge of the Cook County Courts adopted new bail reform measures in September 2017 and reviewed them empirically in May 2019. Cook County’s Bail Reform Study concluded that the new procedures had released many more defendants before trial without any concomitant increase in crime. This article disputes the Study’s conclusions. This article explains that, contrary to the Study’s assertions, the new changes to pretrial release procedures appear to have led to a substantial increase in crimes committed by pretrial releasees in Cook County. Properly measured and estimated, after more generous release procedures were put in place, the number of released defendants charged with committing new crimes increased by 45%. And, more concerning, the number of pretrial releasees charged with committing new violent crimes increased by an estimated 33%. In addition, as reported by the Chicago Tribune, the Study’s data appears to undercount the number of releasees charged with new violent crimes; and a substantial number of aggravated domestic violence prosecutions prosecutors dropped after the changes, presumably because batterers were able to more frequently obtain release and intimidate their victims into not pursuing charges. These public safety concerns call into question whether the bail “reform” measures implemented in Cook County were cost-beneficial.”