Empirical Legal Studies

Checks on Prosecutorial Power

In light of the significant authority that prosecutors wield, we want them to be, at once, both accountable and independent. Obviously, these twin goals are structurally at odds with each other. "Electing prosecutors ensures that they are accountable, but it limits their independence. In contrast, appointing prosecutors increases independence, but it does so at the expense of accountability." In the U.S., 45/50 states elect local prosecutors "who bear the primary responsibility for bringing felony prosecutions in their jurisdictions." However, as Carissa Hessick (UNC) et al. note in a recent paper, Understanding Uncontested Prosecutor Elections: "[prosecutor] elections in the United States generally fail to capture the accountability benefits of elections because so many prosecutor elections are largely uncontested and uncompetitive."

Drawing partly on data gathered by the Prosecutors and Politics Project, the paper models "contested" prosecutor elections. In a slight break from prior findings from related literatures, this paper's core findings emphasize the salience of whether an incumbent is running and the jurisdiction's population. The paper's (excerpted) abstract follows.

"... Using a logistic regression analysis, this Article estimates the individual effects of a number of variables on prosecutor elections. It finds that several factors that have been previously identified as contributing to uncontested election are not, in fact, what drives uncontested elections for prosecutors. Instead, the factors with the largest effect are whether an incumbent runs and the population of the district. It also identifies two features of state election law that contribute to the dearth of contested elections. The Article concludes by noting that these factors suggest specific policy changes that could help to increase the number of contested and competitive elections—thus ensuring that voters can help guide important criminal justice decisions in their communities."

The Empirics of Choice of Law

In an effort to peer into judicial decisionmaking more generally, a recent paper by Dan Klerman (USC), Bias in Choice of Law: New Empirical and Experimental Evidence, levers both observational and experimental data drawn from the vehicle accident context to examine choice of law disputes. The paper considers three leading sources of choice of law bias: pro-resident, pro-plaintiff, and forum law. The findings intrigue and complicate prevailing conventional wisdoms.

Specifically, core findings include that while judges seem to favor the plaintiff, judge decisions do not tilt in favor of application of the law they are most familiar with (the law of the forum). While state court judges favor local residents, federal court judges do not. Although the traditional “place of the injury” choice-of-law rule was more constraining than modern American choice-of-law standards (such as the Restatement Second), even judges under the traditional rule favored the plaintiff more often than seemed warranted. The abstract follows.

"Scholars have argued that modern American choice-of-law is subject to three biases: (1) a bias in favor of residents of the forum state (pro-resident bias), (2) a bias in favor of plaintiffs (pro-plaintiff bias), and (3) a bias in favor of the law of the forum (forum law bias). This article brings new evidence about these biases from a comprehensive database of vehicular accident cases and from the first experiment related to choice of law. Overall, there is some evidence of a pro-resident bias in state (but not federal) court, some evidence of pro-plaintiff bias in both federal and state court, and no evidence of a pro-forum bias."

Assessing Primary and Secondary Effects of Sobriety Checkpoints

While police use of sobriety checkpoints has become increasingly common since the 1980s, legal discomfort, particularly Fourth Amendment concerns, persist. While the SCOTUS concluded in its Sitz decision (1990) that sobriety checkpoints' social benefits outweighed costs on individual liberties, states remain free to preclude their use. Indeed, 11 states have ruled sobriety checkpoints illegal.

Aside from their legal status, empirical questions about sobriety checkpoints' efficacy lurk. In an intriguing new contribution to the emerging empirical literature, Sobriety Checkpoint Laws, Fatal Cars Crashes and Arrests, Lauren Jones (OSU--Public Policy) and C. Blain Morin (OSU--Public Policy) explore the effects of a state removing sobriety checkpoints on fatal car crashes and arrests.

Leveraging data from the Fatality Analysis and Reporting System and Uniform Crime Reports, the authors execute a twoway fixed effects (TWFE), event study analysis. As the paper notes, the twoway fixed effects analysis extends a difference-in-difference specification, where changes in outcomes across pre-treatment and post-treatment time periods are compared, across a treated unit and a control unit. As the abstract (below) makes clear, the paper's key findings include an annual 17 percent increase in fatal crashes and a 25 percent increase in the number of DUI arrests in states that cease using sobriety checkpoints.

“Sobriety checkpoints have been used since the early 1980s to enforce laws prohibiting driving while intoxicated (DWI), and to deter DWI. However, their constitutionality has been questioned, as they allow police to stop drivers without cause. As a result, eleven states have banned their use. Using a two-way fixed effects event study analysis, we estimate the effect of rendering sobriety checkpoints illegal on state-level counts of fatal car crashes, and arrests for driving under the influence of drugs or alcohol (DUI), and drug possession. We find evidence that when states make sobriety checkpoints illegal, the number of annual fatal crashes increases by 17 percent. We also find evidence that rendering sobriety checkpoints illegal leads to a 25 percent increase in the number of annual DUI arrests in a state, suggesting that the deterrent effect of checkpoints is much stronger than their enforcement impacts. We also find suggestive evidence that eliminating sobriety checkpoints increases racial disparities in DUI and cannabis possession arrests, with larger impacts on arrests among Black residents.”

Law School Curriculum: Reform, Assessment, and Research Design

While age-old complaints persist about the basic structure of the first-year law school curriculum, few law schools have dared to experiment with meaningful reforms. Fewer still study these reforms coherently. The "Curriculum B" experiment at Georgetown Law School, as well as a recent paper by David Hyman et al. (Georgetown), Does the 1L Curriculum Make a Difference?, are important and interesting exceptions.

Since 1991, Georgetown Law School has offered an "alternative" Curriculum B along side a standard, more traditional first-year curriculum. Curriculum B was designed to provide “critical” perspectives on the law and respond to important changes in legal practice and theory in the years since the standard 1L curriculum was first developed decades earlier. At Georgetown, students elect into Curriculum B and, when demand exceeds supply, student assignment to Curriculum B is determined by lottery.

For assessment purposes, the quasi-randomized control trial aspect of student assignment to Curriculum B provides a way to overcome selection bias. That is, Hyman et al. compare student outcomes for those who opted into Curriculum B (the treatment group) with student outcomes from those students who wanted to receive the "treatment" (Curriculum B) but, due to excess demand and random selection (lottery), were instead assigned the "placebo" (that is, to the standard, traditional first-year curriculum).

Aside from careful attention to research design, a few of the paper's warrant note. When it comes to students' overall law school academic performances, for example, the authors find that "Curriculum B leads to a mean preserving contraction in students’ overall grades. At the same time, we find that it leads to diminished performance in two key business law electives (Corporations and Securities Regulation)."

Business Litigants and the Roberts Court

Assessing “How pro business is the Roberts Court?” is both straightforward and complicated. Based on how business litigants fare in the Roberts Court versus non-business litigants over time, business does quite well. The win rate for business in the Roberts Court, 63.4 percent, is 15 percentage points higher than the next highest rate of business wins over the past century (the Rehnquist Court, at 48.3 percent). While it may be tempting for many to conclude that the current pro business tilt is purely a function of a Republican majority of justices on the Roberts Court, as Lee Epstein (USC) and Mitu Gulati (Virginia) argue in a recent paper, A Century of Business in the Supreme Court, 1920-2020, “this is only the beginning of the story.”

Epstein and Gulati bring data from the Supreme Court Database to the question. Starting with the 11,121 orally argued cases resulting in a signed opinion or judgment between the 1920-2020 terms, their paper focuses only on those 4,070 cases (35,173 votes) in which business was the petitioner (appellant) or respondent (appellee). That is, they excluded cases in which business was on both sides of a legal dispute.

Their core findings defy conventional wisdom and inject important complicating wrinkles. What they find includes: "(a) It is not just the Republicans on the Roberts Court who are more pro business than in prior Courts, but the Democrats as well; (b) The Government, through the SG’s office and across both Democratic and Republican administrations, has been much more supportive of business positions than in prior eras; and (c) An elite Supreme Court bar has emerged in recent years and businesses have hired them disproportionately so as to better influence the Court."

Empirical Contours of the Political Question Doctrine

Amid the enduring and varied debates about the political question doctrine, motivated by the Supreme Court's 1962 Baker v. Carr decision, the scholarly commentary shares one key attribute. Specifically, it dwells on the Supreme Court. Such a focus is ironic as while the Court has only rarely applied the political question doctrine, lower federal courts, by contrast, have invoked the doctrine "as a basis for dismissal in hundreds of cases." Drawing on both qualitative and quantitative (descriptive) methods, in a recent paper, The Real Political Question Doctrine, Curtis Bradley (Chicago) and Eric Posner (Chicago) explore how the political question doctrine has operated in the lower courts since the Supreme Court's Baker decision.

The authors find that lower courts use the doctrine to "evaluate their own institutional capacity to resolve politically sensitive disputes. It is the lower courts’ more limited capacity compared to the Supreme Court that explains the lower courts’ heavier reliance on the doctrine." More specifically, "the political question doctrine is a screening mechanism that the lower courts use to take account of limits on their institutional capacity. The Supreme Court has less need for this mechanism, so most of the doctrine’s life must be found in the foothills rather than at the summit of Mount Olympus. From this standpoint, the doctrine is not moribund but full of life. It is not limited in the ways suggested by the 'classical' or 'functional' theories. The lower court practice is best captured by Bickel’s prudential theory." An excerpted abstract follows.

"... We provide the first empirical account of how the doctrine has operated in the lower courts since Baker. Our account is based on both a quantitative analysis of a sample of these decisions and on a qualitative review of these and other decisions. This account reveals a political question doctrine that is substantially different from the one described in most scholarship: it is more vibrant, heavily focused on foreign affairs, often applied in non-constitutional cases, more prudential, and not a permanent disallowance of judicial review...."

The Impact of Increased "Sunlight" on Police Traffic Stops

Complicated legal and policy issues incident to policing--and conflicts with citizens the police serve--persist and continue to motivate empirical work. One under-studied aspect, however, involves the degree to which, if any, adverse (digital) media coverage of instances of police brutality informs police conduct going forward. Policing the Police: Examining the Role of News
Reports on Racially-Biased Policing reflects a recent effort at this difficult research question.

In their paper, Uttara Ananthakrishnan (Wash.--Business) et al., set out to assess the impact of public media accounts of police misconduct/violence from January 2013 to May 2018 (N=6,026) on police-initiated traffic stop data drawn from 227 counties across 18 states. The authors execute a DiD model that, because police traffic stop outcomes are "naturally tied to counties," compares "the change in the number of police stops in counties exposed to news reports with the same change in control counties that did not have news reports about their policing incidents during the same period.” Among the paper's findings is that news reports on fatal uses of police force on minorities have a "negative impact on police stops of minorities but not on Whites, on average.” An excerpted abstract follows.

"... However, it is theoretically unclear whether the reporting of excessive use of police force on minorities can have a tangible impact on subsequent policing outcomes. In this paper, we aim to answer the question of whether and how digital news on police brutality is effective in shaping subsequent police actions. To address this question, we construct a cross-sectional dataset of news reports of police violence and police traffic stop records. Under a difference-indifference framework, we find that news reports on police brutality reduces police stops of minorities. Additionally, we find that news with sad frames are more effective in effectuating change in policing behaviors. Finally, we learn that the impact of news reports are less effective in minority-dominated areas and high-crime areas."

Data on Naturalization

Naturalization grants immigrants a host of new and important rights, privileges, and opportunities. While a growing empirical literature focuses on such issues as who is willing to naturalize and why, barriers to seeking naturalization, and the impact of obtaining citizenship on the social, economic, and political integration of immigrants, comparatively less is known about government determinations of who is approved or denied once a naturalization application is submitted to the U.S. Citizenship and Immigration Services (USCIS).

Drawing on new administrative data obtained from USCIS (N=2,687,101 non-military applications adjudicated between October 2014 and March 2018), in their paper, The Importance of Race, Gender, and Religion in Naturalization Adjudication in the United States, Emily Ryo (USC) and Reed Humphrey (Wash.—pub. policy) find significant group disparities in naturalization approvals based on applicants’ race/ethnicity, gender, and religion, controlling for individual applicant characteristics, adjudication years, and variation between field offices.

More specifically, the paper finds that “non-White applicants and Hispanic applicants are less likely to be approved than non-Hispanic White applicants, male applicants are less likely to be approved than female applicants, and applicants from Muslim-majority countries are less likely to be approved than applicants from other countries. In addition, race/ethnicity, gender, and religion interact to produce a certain group hierarchy in naturalization approvals. For example, the probability of approval for Black males is 5 percentage points smaller than that of White females. The probability of approval for Blacks from Muslim-majority countries is 9 percentage points smaller than that of Whites from other countries. The probability of approval for females from Muslim majority countries is 6 percentage points smaller than that of females from other countries. This study contributes to our understanding of the nature of inequalities present in agency decisionmaking in the naturalization process.”

Everything You Wanted to Know About Tables (in Stata)

Throughout the second-half of 2021, the Stata Blog ran a 7-part series on generating various tables. A single link to the 7-part series is here.

Multi-Level Model "vs." Clustering

As this question (or something like it) is not an infrequent one, I found Jeff Wooldridge's response (No.2) in this StataList discussion (here) quite helpful. The bottom line is that decisions incident to estimating a multi-level model or/and clustering pivot on an array of considerations.

CELS -- UVA -- 2022: Papers -- Deadline Extended

EXTENDED Paper Submission Deadline: Sunday, July 10, 2022 (11:59 PM Eastern Daylight Time).

To submit a paper, please go to the CELS UVA 2022 website (click here; submissions tab, click here). There is no charge for submissions.

Hotels: It is likely to be a crowded weekend in Charlottesville, so please be sure to reserve your hotel early by following the Hotels tab on the conference website.

About: The University of Virginia School of Law and the Society for Empirical Legal Studies (SELS) are pleased to host the 16th Annual Conference on Empirical Legal Studies (CELS) on November 4-5, 2022, in person at the University of Virginia in Charlottesville. CELS is a highly regarded interdisciplinary gathering that draws researchers from across the United States and the world, bringing together scholars in law, economics, political science, psychology, policy analysis, and other fields who are interested in the empirical analysis of law and legal institutions. Papers are selected through a peer review process, and discussion at the conference includes assigned commentators and audience questions.

Scope: CELS will consider empirical papers spanning all areas of empirical legal studies. Authors are encouraged to submit works-in-progress; however, submissions should be completed drafts that include principal results. Submitted papers must be unpublished (and expected to be unpublished at the time of the conference). If accepted, authors will have an opportunity to submit a revised draft prior to the conference for presentation and discussion. Please note that accepted papers will be made available to all conference participants.

Keynote: We are thrilled to announce that the keynote Friday dinner speaker will be Dr. Anne-Marie Slaughter, CEO of New America, Bert G. Kerstetter '66 University Professor Emerita of Politics and International Affairs at Princeton University, and former Dean of Princeton University's School of Public and International Affairs. There is a separate charge for attendance to this dinner event.

If you have questions about the submissions process, please contact the conference organizers at [email protected].

Simple Problems, Simple Solutions

While I should have known this years ago, better late than never.

As one becomes familiar with the broad descriptive contours of new data, the tab function is both well-known and ubiquitous. Finding analogous basic descriptive statistics for sub-samples, while certainly possible, was typically (for me, anyway) a bit cumbersome with the tab command.

A recent discussion on the StataList, however, introduced me to the simple (and code-saving) power of a related command, tabstat. As this discussion illustrates (click here), what the tab command accomplishes with multiple lines of coding and multiple output tables, the tabstat command, by contrast, accomplishes with a single line of code and a single, unified output table.

A "Guide" on how to Interpret Empirical Corporate Governance Scholarship

While empirical work continues to proliferate across an growing array of legal "silos," lawyers (and some law professors) increasingly need to engage with these studies to understand their implications for legal practice and policy implications. This is especially true for corporate governance scholars (and practitioners). As empirical corporate governance work reaches a wider audience, the work intersects with an ever-increasing number of individuals "not trained in econometrics and statistical methods."

This is a problem, and a recent paper endeavors to contribute to a solution. In A Lawyer's Guide to Empirical Corporate Governance, Ofer Eldar (Duke) sets out to provide a general guide that overviews "key empirical strategies for evaluating the relationship between governance and shareholder value, and to demonstrate to lawyers how they can engage with their weaknesses." Critically, the paper identifies a set of three key comparisons that dominate empirical corporate governance scholarship: "(1) comparing firms with and without the governance provisions, (2) evaluating the value of firms before and after adoptions or removals of governance provisions, and (3) evaluating what happens to shareholder value after a legal change."

While papers like these may not be standard empirical legal scholarship, they nonetheless contribute to the field more generally by helping to make empirical scholarship a bit more accessible to those lacking formal empirical training. The abstract follows.

"Empirical studies in corporate law have proliferated in the last two decades, and have had a major impact on legal practice and policy relating to corporate governance. As a result, lawyers increasingly engage with these studies in order to understand their implications for legal practice and policy. This is often challenging because most lawyers are not trained in econometrics and statistical methods, and thus, there is a risk that they misinterpret these studies and their implications. These studies, while critical for evaluating different corporate governance regimes, suffer from well-known weaknesses. The purposes of this guide are to provide an overview of the key empirical strategies for evaluating the relationship between governance and shareholder value, and to demonstrate to lawyers how they can engage with their weaknesses. The main theme of the guide is that largely all empirical strategies are based on three key intuitive comparisons: (1) comparing firms with and without the governance provisions, (2) evaluating the value of firms before and after adoptions or removals of governance provisions, and (3) evaluating what happens to shareholder value after a legal change. By identifying these basic comparisons and the roles they play in the key methodologies for evaluating shareholder value, lawyers will be better positioned to assess empirical studies and evaluate their implications for legal policy."

Law Firms, M&A Work, Gender, and the Uneven Credentials Burden

It remains well-understood that while women have made up approximately one-half of law school classes over the past decades as well as the ranks of entry-level law firm associates, attrition is uneven across genders throughout the path to law firm partnership. No doubt this uneven attrition is the product of many complex variables and factors, some more troubling than others. What is less well-understood, however, is how attrition distributes post-law firm partnership.

In a recent paper, Gender, Credentials and M&A, Tracey George (Vanderbilt), Albert Yoon (Toronto), and Mitu Gulati (Virginia) explore this topic with data from one specific practice area: M&A. The specific practice area (M&A) is important as "[r]elative to a number of other practice areas, such as the issuance of corporate or municipal bonds or managing trusts and estates, M&A tends to be a “feast or famine” area. Lawyers are either busy working around the clock (if they are trying to close a deal) or are twiddling their thumbs.” And this aspect is salient for gender comparisons as "a high degree of variability in the demands from work tends to hurt more those workers who have bigger fixed outside obligations that cannot easily be moved or postponed (e.g., caring for small children or elderly parents).”

The paper executes a two-stage empirical strategy. It first explores the "universe" of M&A attorneys. The second stage analyzes who becomes a "lead deal lawyer" (representing either the buyer or the seller). One key finding is that while at both stages lawyer credentials matter, they "matter more" for women (emphasis added). The abstract follows.

“For the past several decades, women have made up roughly half of law school classes and the ranks of entering law firm associates. Attrition between entry to law firms and partnership results in women comprising 20 and 25 percent of partners. But is there yet more attrition to the top of the partnership pyramid? Analyzing the past decade of data on publicly-filed M&A deals and detailed biographical information of M&A lawyers, we find that women make up fewer than 10 percent of deal leaders. When we look at the factors that determine who becomes a deal leader, we find that credentials – both educational and professional – matter. But they matter more for women. And one credential – attending a top law school – seems to matter a lot. Using conversations with senior lawyers, we try to get answers for why.”

Comparing Regression Output Across Models

As I am seeing more papers trying to do so (as well as discussions about strategies on how to do so), I thought links to a recent StataList discussion as well as some web-based resources might be of general interest. The basic set-up follows:

Model A: Y = α1 + β1X1 + Σθ∙C + firm-fixed effect + year-fixed effect.

Model B: Y = α1 + β2X2 + Σθ∙C + firm-fixed effect + year-fixed effect.

Σθ∙C is list of control variables and are the same for both model A & B.

Question: How to compare the coefficients of β1 and β2?

One approach exploits a post-estimation command in Stata (suest), and then compares means with test (for a discussion and illustration, click here). Another illustrated example, courtesy of the UCLA Statistical Methods and Data Analytics site, is here.

Moving From "Data to Policy Recommendations"

For many scholars and policymakers, “short-termism” behavior in the stock markets is assumed to be a problem. Unsurprisingly, researchers have empirically assessed this issue to better understand its dynamics, extent, and contours.

In a recent paper, Stock Market Short-Termism: What the Empirical Evidence Tells Policymakers, Mark Roe (Harvard) does not contribute more results. Instead, Roe takes a step back and presents a helpful example for empiricists in terms of how to interpret and contextualize results from multiple studies looking at the same general issue (here, stock market short-termism). That is to say, this paper’s contribution speaks to the discrete task of moving from “data to policy recommendations.”

Specifically, Roe considers “what the evidence directly tells us about stock-market-induced short-termism and how that evidence must be evaluated conceptually to reach an economy-wide assessment for policymakers.” What he finds is that for “stock market short-termism, it is harder to move from the data to policy recommendations than is usually thought.” Roe notes an array of factors contributing to the degree of difficulty, including evenly divided academic evidence and varied (and limited) research designs that invite external validity questions.

While empirical legal scholars appropriately focus on executing a coherent research design and generating stable, robust results, additional moves remain. And one of these, for some empirical legal scholars, involves a move from the “data to policy recommendations.” What Roe’s paper helpfully emphasizes is that these secondary and tertiary moves introduce an additional set of challenges and complications that warrant greater attention. An excerpted abstract follows.

“… The typical research effort seeks to measure whether a local treatment induces more local short-termism, not whether the economy-wide impact is severe. But evaluating short-termism’s economy-wide impact is essential for policymaking; policymakers must consider whether the economy is failing to invest or cutting back on R&D because of a stock market afflicted with a truncated time horizon. Local findings (of the impact on a subset of firms with a particular characteristic, such as short-vesting stock options, rapid stock market trading, or hedge fund activism) need not scale to economy-wide results; some local results will do so only serendipitously; and, finally, there are structural reasons why for stock market short-termism one should expect economy-wide results not to match local results. More than most corporate law issues, the short-termism problem faces a high failure-to scale hurdle.”

Data on K-12 School Suspension Reform Efforts

Incident to the federal Gun Free School Act (1994), all fifty states adopted an array of "zero-tolerance" policies, which include out-of-school suspensions ("OSS"), for various forms of student misconduct. As evidence of adverse consequences mounted, some states, including Rhode Island, limited school districts' use of out-of-school suspensions. In 2012, Rhode Island began prohibiting out-of-school suspensions for "low-level" student infractions and, in 2016, began requiring school district-level reviews of whether out-of-school suspensions raise any long-standing EPC concerns.

In a recent paper, Do School Suspension Reforms Work? Evidence from Rhode Island, Terry-Ann Craigie (Smith College--econ) "exploits student-level administrative data from the Rhode Island Department of Education (AY 2009–2010 to AY 2017–2018), along with a triple-difference (DDD) framework, to measure the impact of each suspension reform on OSS [out-of-school suspension] outcomes for treatment infractions in treatment schools (i.e., schools and within-school grades that used OSS to penalize treatment infractions, ex ante). The study also employs a quadruple-difference (DDDD) model to measure the extent to which racial-ethnic disparities in OSS changed for the treatment groups under each reform."

Results from the study imply that Rhode Island's initial (2012) reform "substantially lowers OSS for attendance-specific infractions in treatment schools." Results on Rhode Island's 2016 reform designed to address distributional concerns, however, are "mixed." The paper's abstract follows.

“In Rhode Island, out-of-school suspensions were excessively and disproportionately used to penalize low-level infractions. To address this problem, the Rhode Island General Assembly passed legislation, effective May 2012, prohibiting out-of-school suspensions for attendance-specific infractions. Four years later, they passed additional legislation to curb out-of-school suspensions for disruption-specific infractions. This study examines the impact of these suspension reforms on out-of-school suspension outcomes for treatment infractions and corresponding racial-ethnic disparities. To execute the analyses, the study uses student-level administrative data (AY 2009–2010 to AY 2017–2018) from the Rhode Island Department of Education, along with quasi-experimental estimation. The study finds that only the first reform lowers out-of-school suspension outcomes for attendance-specific infractions and corresponding racial-ethnic disparities.”

Algos and Parole Decisions

Many--if not most--parole decisions remain fraught with peril. As parole boards are tasked with, to some degree, "guessing into the future," some form of risk assessment has been used to inform parole decisions since the 1920s. Despite this legacy, a recent and growing "decarceration" push has heightened public and scholarly attention to criminal risk prediction instruments. Recently, New York state amended its parole law to "center [parole board] decisions on individual’s rehabilitation in prison and future risk if released." Despite this shift in law, critics continue to argue that parole boards, such as New York's, "continue to incarcerate individuals well beyond their minimum sentence, not out of concern for public safety, but because of the nature and severity of the offense that had sent them to prison."

While concerns grow, few empirical estimates of the issue exist. In a recent paper, An Algorithmic Assessment of Parole Decisions, the authors, Hannah Laqueur (UC Davis--medicine) and Ryan Copus (Univ. Mizzou--KC), endeavor to do just that and test a risk-prediction algorithm. To do so, the paper paper exploits New York parole hearings’ data from 2012-2018. The data include individual sex, race/ethnicity, commitment crime, housing facility, parole board interview type, and the interview decision. The authors also obtained criminal history records for all individuals who had a recorded parole hearing from 2012 through 2018. The authors' risk prediction algorithm trained on the 4,168 individuals who were released on parole between 2012 and 2015 (a total of 19,713 individuals had parole hearings during this period). The primary outcome variables include "any recorded arrest within three years post-release and any violent felony arrest within three years post-release."

The authors find that the New York parole board "could have granted parole more than twice as often without increasing either the overall or violent arrest rate, or it could have released the same number of people while approximately halving both overall and violent arrest rates. Further, we find that they could have achieved these gains while simultaneously eliminating racial disparities in release rates.” The authors take care to note that they "cannot determine whether the Board is simply not as good at determining risk as our risk prediction algorithm, or is simply prioritizing non-risk related factors such as the commitment offense." The paper's abstract follows.

"Objectives: Parole is an important mechanism for alleviating the extraordinary social and financial costs of mass incarceration. Yet parole boards can also present a major obstacle, denying parole to low-risk inmates who could safely be released from prison. We evaluate a major parole institution, the New York State Parole Board, quantifying the costs of suboptimal decision-making.

Methods: Using ensemble Machine Learning, we predict any arrest and any violent felony arrest within three years to generate criminal risk predictions for individuals released on parole in New York from 2012–2015. We quantify the social welfare loss of the Board’s sub-optimal decisions by rank ordering inmates by their predicted risk and estimating the crime rates that could be observed with counterfactual risk-based release decisions. We also estimate the release rates that could be achieved holding arrest rates constant. We attend to the “selective labels” problem in several ways, including by testing the validity of the algorithm for individuals who were denied parole but later released after the expiration of their sentence.

Results: We conservatively estimate that the Board could have more than doubled the release rate without increasing the total or violent felony arrest rate, and that they could have achieved these gains while simultaneously eliminating racial disparities in release rates.

Conclusions: This study demonstrates the use of algorithms for evaluating criminal justice decision-making. Our analyses suggest that many low risk individuals are being unnecessarily incarcerated, highlighting the need for major parole reform."

The Impact of the Court's Halo Decision on Willful Infringement and Enhanced Damage Claims

While legal formalists may generally defer to clear legal rules, others respond in different ways. For example, as Karen Sandrik (Wilamette) writes, “For decades, companies and attorneys have instructed teams of engineers, researchers, and computer scientists to ignore patents." Such strategic advice is offered, in part, to help defeat a patent holder's subsequent claim for enhanced damages.

In her recent paper, An Empirical Study: Willful Infringement & Enhanced Damages in Patent Law After Halo, Sandrik subjects this convention wisdom to data. Critically, the run of data (from 2010 through 2020) includes the Supreme Court's 2016 Halo decision where the Court wrote that enhanced damages are “designed as a ‘punitive’ or ‘vindictive’ sanction” for an infringer’s conduct that is “willful, wanton, malicious, bad-faith, deliberate, consciously wrongful, flagrant, or—indeed—characteristic of a pirate.”

The data set includes all district court decisions on willfulness decided between 2010 and 2020. Exclusions from the data set include appeals as well as (interestingly) successful motions to dismiss willful infringement claims. As it relates to the latter exclusion, the author notes that successful motions to dismiss “are inconsistent standards post-Halo on what is needed to successfully plead a claim for willful infringement.” Key findings from the resultant data set (N=260) on pre- and post-Halo outcomes include: "(1) willfulness findings have increased by 27.8%; (2) enhanced damages findings have increased by 8.7%; and (3) judges are significantly more likely to find willfulness (representing an 18.6% increase in willfulness findings)." An excerpted abstract follows.

"... In this empirical study with data spanning 2010 to 2020, I provide a data-driven answer to whether this deliberate ignorance strategy is effective. The answer, in short, is that reading patents, conducting patent clearance searches, and/or responding to cease-and-desist letters does not, in isolation, open the door to enhanced damages. Finally, by employing an original data set to seek this answer and potential solutions to deliberate patent ignorance, this study provides empirical statistics regarding willful infringement and enhanced damages. This includes empirical statistics illustrating the impact of the 2016 Supreme Court decision, Halo Electronics v. Pulse Electronics.”

Judges Without J.D.s: One First Step

In an intriguing paper, Judging Without a J.D., the authors, Sara Greene (Duke) and Kristen Renberg (Duke), emphasize an underappreciated point: In a surprising number of states lower-level state court judges do not possess the J.D. degree. Now, to be sure, even at the federal level, the J.D. degree is not required for an Art. III commission. That said, insofar as virtually every (if not every) Art.III judge today possesses a J.D. the necessary variation does not exist to support useful comparative studies.

Research not possible in the federal judicial space, however, is possible in the state judicial space. And this paper does a good job at identifying a potentially rich vein of future research possibilities. Implications from this paper's brief case study of North Carolina magistrates (where more than 80 percent do not possess the J.D. degree) hint at possible distributional concerns. And these concerns may be especially acute where non-JD magistrates intersect with pro se litigants (itself a growing trend). The paper's abstract follows.

"One of the most basic assumptions of our legal system is that when two parties face off in court, the case will be adjudicated before a judge who is trained in the law. This Essay begins by showing that empirically, the assumption that most judges have legal training does not hold true for many low-level state courts. Using data we compiled from all fifty states and the District of Columbia, we find that thirty-three states allow at least some low-level state court judges to adjudicate without a law degree, and in seventeen states, judges who adjudicate eviction cases are not required to have law degrees. Since most poor litigants are unrepresented in civil legal cases, this sets up an almost Kafkaesque scene in courtrooms across the country: Legal cases that have a profound effect on poor families, such as whether they will lose their home to eviction, are argued either in courtrooms where no one knows the law or only one party---the lawyer for the more powerful party---knows the law.

Considering data collected from a case study of North Carolina, where over 80% of magistrates do not have J.D.s, this Essay argues that allowing a system of nonlawyer judges perpetuates long-standing inequalities in our courts. It further argues that the phenomenon of lay judges is a symptom of a much larger problem in our justice system: the devaluation of the legal problems of the poor, who are disproportionately Black and Latine. This devaluation stems in part from an enduring cultural history in the United States of blaming the poor for their poverty and the associated problems of poverty. A change is in order, one that intentionally considers the expertise of judges and adopts creative solutions to incentivize specially qualified adjudicators to serve as low-level state court judges."

Approaches Toward Missing Data

Missing data vex researchers, and there's no single "correct" way to deal with this not-infrequent real-world problem. A recent question posted on the StataList (here) visually illustrates what this can look like (in terms of N variation) across alternative specifications of a multilevel regression model drawing from a data set that contains missing data in some variables.

As this output illustrates, as one proceeds across models 1-4 what is clear is that the addition (and subtraction) of variables introduce missing data. How to approach this and what to do about it motivated the initial query. And in the discussion that follows identifies (some of the) various "standard" approaches.

Data on Life Insurance Markets and Racial Discrimination

A recent paper, Racial Discrimination in Life Insurance, explores the impact of modern anti-racial discrimination statutes on life insurance coverage differences across racial groups. To do so, the paper's authors, William Gale (Brookings Inst.) et al., exploit data from the 2014 panel of the Survey of Income and Program Participation (SIPP) and then merge it with Avraham, Logue and Schwarcz's (2014) ordinal rankings of "how restrictive a political jurisdiction (each of the 50 states and Washington, DC) is in limiting the use of race as a determinant of life insurance premiums and payouts (as of 2012)." The data sample is restricted to "white and Black adults aged 25-64 in order to focus on the working age population most likely to purchase life insurance. We limit our analysis to data for white and Black individuals in the interest of simplicity and in order to focus on the type of racial discrimination that has been most prevalent historically in the life insurance market."

The paper's core findings include: 1) individuals in states with strong anti-discrimination laws tend to have lower life insurance coverage rates than those living in other states; and 2) Black individuals are somewhat more likely than white individuals to hold life insurance, controlling for observable factors.

While hardly the final word, this paper identifies a fruitful path forward for future research. For example, this paper considers only whether a life insurance policy exits at all, and not the amount (or the face value) of such policies. It remains possible that racial discrimination "could conceivably impact the level [amount] of life insurance even if it does not influence the choice of whether to purchase a policy at all." The paper's abstract follows.

“We examine the historical and statistical relationship between race and life insurance. Life insurance can play a central role in households’ financial security. Race has played an important and changing role in the provision of life insurance in the U.S. from slave insurance before the Civil War, to ‘Scientific Racism’ continuing into the 20th century, to policies that do not explicitly mention race in recent decades. In empirical work using new data, we confirm earlier work showing that Black individuals have higher life insurance coverage rates than white individuals, controlling for observable characteristics. We find no difference in the likelihood of purchasing coverage—for Black individuals versus white individuals—in states with strong versus weak anti-discrimination laws. We also find that the presence of strong antidiscrimination laws tends to reduce overall life insurance coverage – by about 3 percentage points. We present some evidence that this finding is due to a generally stronger regulatory stance in the state rather than the specific impact of the anti-discrimination life insurance law. This analysis bears on the presence of discrimination in the current life insurance industry as well as related issues like the financial status of minority households.”

Save The Dates--16th Annual CELS (In-Person)

 

 

The Society for Empirical Legal Studies’

16th Annual Meeting

will be held on

November 4th and 5th, 2022 (Friday and Saturday) at the

University of Virginia School of Law

More information here (https://www.cels2022.com/)

A call for papers will be coming soon, but please mark your calendars now.

Federal Courts and "Certification" of State Law Questions

When federal courts need to apply state law and no clear articular of state law exists, federal judges have few options. While generating an Erie "guess" remains one option, another involves certifying the legal question to the state's highest court. Proponents suggest that state certification promotes federal-state cooperation, reduces forum shopping incentives, and, on net, reduces costs and increases efficiencies. Critics, by contrast, emphasize the unidirectional federal-state cooperation, the persistence of forum shopping, and the associated costs and delay.

While the SCOTUS and Judicial Conference have expressed growing interest in the state certification alternative of late, helpful data are scant. A recent paper by Jason Cantone (George Mason--criminology) and Carly Griffin (Univ. Ill.--ag.), Certified Questions of State Law: An Empirical Examination of Use in Three U.S. Courts of Appeals, seeks to contribute by presenting results drawn from three federal courts of appeals. While the analyses are purely descriptive, temporally bound (from 2010 through 2018), and limited to three federal circuits, they nonetheless cast some helpful initial empirical light on some of the basic contours of the certification process.

Drawing on a sample (N=218) of certified question events across three federal circuits from 2010-2018, the paper explores: "how often the certified question procedure is utilized in three U.S. Courts of Appeals, how often the circuits certify a question, how often states grant the certified question, and how long the process takes." Notably, the paper finds important variation across the three circuits when it came to state certification rates and duration. The paper's abstract follows.

"Federal courts routinely need to apply state law, but this task becomes complicated when no definitive state law exists. In these instances, federal judges have a few options, including making an Erie guess, or certifying the question of state law to the highest state court. During the U.S. Supreme Court’s October 2020 term, two opinions endorsed the certification of questions of state law procedure. However, despite recent interest by the U.S. Supreme Court and the U.S. Judicial Conference Committee on Federal-State Jurisdiction, little empirical research has examined the perceived benefits and burdens of the procedure. This Article provides new empirical data to help fill that gap.

Using a sample of 218 certified-question events across the U.S. Courts of Appeals for the Ninth, Third, and Sixth Circuits from 2010 to 2018, we examine how certified questions originate, in what types of cases, how often U.S. courts of appeal certify the questions of state law, and whether the corresponding state supreme courts grant or deny the question. Further, to address the perceived time burden associated with certification, we examine how long each part of the process takes in the appellate court, from filing to certification to termination. We found significant variation between the three U.S. courts of appeals in certification rates and timing. While we encourage future studies regarding certification of state law questions, our data can be readily used to inform courts and academic debates regarding the benefits and burdens of the procedure."

ABF Fellowship

A long-time Els blog reader, Elizabeth Mertz (Wisc./ABF), asked me to pass along the following information about an ABF Fellowship opportunity (sponsored jointly with AccessLex Institute). I am happy to do so.

"We are writing to spread the word about a special opportunity at the American Bar Foundation, offered jointly with the AccessLex Institute.  We have funding for a researcher who is currently a candidate for an LLM, SJD, or PhD -- and who is doing empirical, interdisciplinary research on legal education or on legal aspects of higher education.  The baseline funding is for a fellowship of $35,000 a year, extendable up to 2 years, to be held in residence at the ABF in Chicago.  (Additional funding for relocation to Chicago, and for research and travel to conferences is available as well.) Please circulate this to any colleagues who might be interested -- or who might have students who could apply."

Anyone interested in further information should directly contact: Jothie Rajah ([email protected]), Laura Beth Nielsen ([email protected]), or Elizabeth Mertz ([email protected]).

CELS 2022--Toronto: Registration and More Information

Registration and related logistical information follow for the 15th Annual Conference on Empirical Legal Studies (online), 18-19 March 2022, organized by the Univ. Toronto Faculty of Law:

Registration: We invite you to register for CELS using the link (click here). Given that the conference is remote, the registration fees $50 for faculty and $20 for students. We ask that you kindly register at your earliest convenience. For participants, we ask that you register by March 13, 2022.

Keynote Speaker: This year's keynote speaker will be Prof. David Card, the Class of 1950 Professor of Economics at the University of California, Berkeley. In 2021, Prof. Card was awarded the Nobel Memorial Prize in Economic Sciences. A native of Guelph, Ontario, Prof. Card keynote talk is titled, ""Affirmative Action for Racial and Gender Diversity." The talk will take place on Friday, March 18, 2022 from 12:30pm - 1:30pm EST.

Training: We will be using the VVC conference platform. To avoid technical snafus on the day of the conference, VVC will be providing participants with an opportunity to try out the platform in advance of the conference. The training session will be on Wednesday, March 16, 2022, at 1pm. We specifically ask that the chair for each session (i.e., see below) join. We will circulate a link in the upcoming days.

Presentation Length: Each paper is scheduled for 30 minutes. That 30 minutes will be divided into three parts: 15 minutes for the presenter’s presentation, 8 minutes for the discussant’s discussion, and 7 minutes for general Q&A.

We ask presenters and discussants to respect these times to the best of their ability. The Chair of each session (described below) will be asked to keep time, but it is the responsibility of each speaker to abide by the time limits.

Session Chairs:The presenter of the last paper in each session is the Chair of that session, as designated by an asterisk on the program. The Chair’s only responsibility is timekeeping.

For additional conference information (click here).

Judicial Outcomes, "Industry Peer" Stock Ownership, and Recusal

While research on "outside" influences on judicial decisions is relatively well-developed, much of that research dwells on various political influences (judge political ideology, etc.). Comparatively less well-developed is the possibility that judges' stock holdings influence their decisions in cases that involve corporate parties from "industry-peer" firms. While recusal doctrine (and practices and norms) are designed to guard against direct judicial financial conflicts of interests, indirect financial conflicts, by contrast, fall outside of formal recusal boundaries. And one example of an "indirect" financial conflict includes when a presiding judge has a financial stake in a litigant's "industry peer."

Barriers to data access are among the reasons that account for a deficit in research on this topic. In Stock Ownership of Federal Judges and its Impact on Corporations, April Knill (Fl. St.--Business) et al., explore this question after cobbling together data from an array of sources. The key move on the data front involved matching, at the individual case-level, presiding judges and their financial holdings. The authors "manually download 21,044 individual financial disclosures from JudicialWatch.org file by file... . This process results in a sample comprising 11,447 unique reports for federal judges in the years 2007 – 2012.” And within their sample, the authors report that the sample contains "observations from 69,060 federal district court cases.… Judges own industry peer stock in 8.6% of the cases in our sample.”

It is worth underscoring that in these instances the presiding judges did not violate any formal recusal obligations as the possible conflicts of interests flowing from their "industry peer" stock holdings are indirect. That said, the paper's core results imply that judges are "being influenced by their stock ownership in industry peer firms of litigants – a practice that is completely legitimate – in cases over which they preside." The paper's abstract follows.

“This paper investigates whether and how litigant peer stock ownership by federal district judges affects characteristics of case outcomes for large corporate litigants. We find that industry-peer stock ownership by district judges is associated with the following outcomes for corporate litigants named in their assigned cases: 1) an increased likelihood of judgments for the corporate litigants, 2) a decrease in the amount received by the parties suing these corporate litigants, and 3) a decrease in the length of the litigation proceedings. The random assignment of district judges to cases provides exogenous variation in the judge stock ownership. We further identify the association outlined in our base results by examining appellate court reversals of district judgments, a triple difference analysis isolating large-stake investments, and outcomes in case types that should impact industries either cooperatively or competitively. Our results survive a falsification test as well as a battery of robustness tests. Our findings underscore the importance of mandates governing judge stock ownership, and more broadly, judge conflicts of interest.”

Mistaking Correlation for Causation

Economists--and econometrics--have been involved in empirical legal studies for generations. This is particularly true for anti-trust doctrine, both its development and current application. Legal challenges to proposed mergers, and questions relating to anti-competitive spill-over effects, including market concentration, remain a staple for empirical analyses.

In the anti-trust context, one intuitive move involves assessing whether "prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods." Pursuing such a question can involve (and frequently does involve) something as simple (and potentially powerful) as regressing price on the HHI.

Despite whatever intuitive appeal such a move holds, in a recent brief essay--co-signed by many leading law and economics scholars (including, and I add in interest of full disclosure, my Cornell colleague George Hay), On the Misuse of Regressions of Price on the HHI in Merger Review, the authors caution against such a move as it invites one basic fundamental problem: Mistaking correlation for causation. And it is for the essay's description of just how such a basic mistake can arise (rather than it's specific application in the anti-trust context) that I recommend this essay, especially to younger, developing empirical legal scholars.

As it relates to the specific anti-trust point developed in the essay, the authors' argue that: "The underlying problem with regressions of price on the HHI is that the relationship between price and the HHI is not causal. Instead, both are equilibrium outcomes that are determined by demand, supply, and the factors that drive them. Thus, a regression of price on the HHI does not show the sort of causal effect that would be helpful in predicting the competitive effects of a merger." More specifically, as the "HHI can take values ranging from 0 to 10,000, with the former corresponding to a fragmented market with infinitesimal firms, and the latter to monopoly. All else equal, a decrease in the number of firms leads to a higher HHI. However, the HHI can increase if, for a given number of firms, the market shares of the larger firms increase."

The root problem, of course, flows from challenges posed by “endogeneity” or “simultaneity.” Such problems lurk "when a clear causal path between two variables is posited by the relevant economic theory but the causal factor (on the right side of the regression equation) is either correlated with unobserved factors or, more subtly, is simultaneously determined by the variables (in this case, price) on the left side of the regression. A famous example of simultaneity involves price and quantity, which are simultaneously determined in the classic economic model of supply and demand." The problems posed by "endogeneity” or “simultaneity," of course, are not limited to the anti-trust realm. An excerpted abstract follows.

"It might seem natural to determine whether prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods. This might be implemented by using a simple regression of price on the HHI. However, for reasons that we describe, regressions of price on the HHI should not be interpreted as establishing causation. That is, they do not inform how a change in concentration from a merger would affect prices. Courts and other policy-makers should not rely on regressions of price on the HHI for the purposes of antitrust merger review."

An Empirical Assessment of What "Substantial Similiarity" Means to Courts in Copyright Litigation

Judicial assessments of copyright infringement claims remain dominated by the two-prong "substantial similarity" test. Despite this test's centrality to copyright doctrine, very little is known about how courts operationalize and apply it. In a recent paper, An Empirical Study of Copyright's Substantial Similarity Test, Clark Asay (BYU) seeks to sketch some of the test's general empirical contours. To do so, the paper, drawing from the universe of substantial similarity opinions (found on WestLaw), exploits a random sample of 1,005 opinions (district and appellate) between 1978 and 2020.

The study's key findings intrigue. First, and unsurprising to most practitioners in this space, the Second and Ninth Circuits dominate this litigation. Second, judicial analyses on the substantial similarity test tend to dwell on the second prong with comparably less judicial attention to the first. Third, judicial analyses focusing on prong two--assessing whether the defendant’s copying amounts to improper appropriation--evidence a wide diversity of approaches. As Asay notes, "[n]o dominant means exists for resolving this question, and, in any given opinion, a court typically uses multiple subtests and copyright limitations to decide this inquiry."

The paper concludes by recommending: "First, courts should maintain the two traditional prongs of the substantial similarity test as distinct and reinvigorate their assessments of similarity under the first prong of the substantial similarity test. Second, courts should make discussing and applying any relevant copyright limitations the heart of their prong two analyses to ensure that copyright law serves its constitutional purposes."

Defendant-Prosecutor Gender "Pairings" and Leniency

While various pockets on criminal law are increasingly subject to empirical analyses, one particular pocket--how prosecutors distribute discretion--is comparatively under-studied. This is so, partly, due to a comparative paucity of relevant data on the prosecutors themselves (and matching them to individual cases and dispositions). A recent paper, Gender Favoritism Among Criminal Prosecutors, by Stephanie Holmes Didwania (Wisc.), tries move the scholarly needle a bit by exploiting data on over 130,000 federal criminal defendants sentenced in 2002 through 2016 to assess possible "gender-based differences in how prosecutors exercise critical discretion."

The paper matches Sentencing Commission data and data from the Executive Office for United States Attorneys (EOUSA) which regularly publishes case data from the 93 USAOs located throughout the United States. Critically, the breadth of the LIONS data "is substantial—the data covers all cases in which a USAO was involved, including those cases that are the primary responsibility of another agency and those the USAO declines to prosecute." Despite its breath, the LIONS data set is not without important limitations including a comparatively non-trivial amount of missing data. After matching, the resultant data set data includes "131,207 federal criminal defendants sentenced between 2001 and 2016 in 63 of the 94 federal district courts."

Data set limitations aside, the paper's core finding intrigues: "That defendants receive more lenient charges when they match with their prosecutor on gender relative to when they are assigned a prosecutor of the opposite gender. Through this charging channel, prosecutor-defendant pairs that match on gender produce lower sentences relative to prosecutor-defendant pairs that mismatch." More specifically, "the leniency associated with gender-matching ultimately translates into significantly lower sentences: a defendant can expect to earn a roughly five-month shorter sentence when their prosecutor is their own gender compared to when their prosecutor is the opposite gender, which represents a roughly eight percent reduction in sentence length at the mean." The paper's abstract follows.

"Prosecutors enjoy wide discretion in the decisions they make but are largely unstudied by quantitative empirical scholars. This paper explores gender bias in prosecutorial decision-making. I find that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall but demonstrate relative leniency towards defendants of their own gender. Such favoritism at charging translates into a sentencing gap of roughly five months of incarceration for defendants who are paired with an own-gender prosecutor versus an opposite-gender prosecutor, which represents a roughly eight percent reduction in sentence length at the mean. The estimates do not appear to be driven by differences in case assignments for male and female prosecutors."

Judge Ideology and Pleadings Decisions in the Circuits

While the Twombly and Iqbal decisions separately and collectively helped galvanize empirical work in civil procedure in terms of access to federal courts, now that we're well into a post-Twombly and Iqbal world sufficient data exist to begin to test various normative claims regarding whether the two procedure decisions amplify the potential impact of judge ideology. And in a new paper, Politics, Identity, and Pleadings Decisions on the U.S. Courts of Appeals, Stephen Burbank (Penn) and Sean Farhang (Berkeley) set out to do just that.

To do so, the authors drew a random sample of 700 cases from the universe of cases post-Iqbal through 2019 that involved a Court of Appeals review of a district court decision on whether to dismiss for failure to state a claim under Rule 12(b)(6). The paper focuses on "the extent to which the party (of appointing president), gender, and racial composition of panels are associated with their disposition of 12(b)(6) appeals." Key findings include "that panels with women and non-white judges are substantially more likely to rule in favor of a plaintiff reaching discovery in other civil rights claims, an important and cross-cutting civil rights category amounting to a quarter of 12(b)(6) appeals in our data, but that race and gender are insignificant outside that substantive area. Party is insignificant across the board in the random sample."

CELS (Toronto)--Some Very Good News and Some Sub-Optimal News

I begin with the good news: CELS (Toronto), scheduled for March 18-19, 2022, will go forward. The sub-optimal wrinkle, however, is that organizers were forced to move the event to an on-line (remote) platform and format. As the organizers note: "Unfortunately, the surge in Omicron has led to travel restrictions into Canada, as well as restrictions on gatherings in the province of Ontario. Events at the University of Toronto are not permitted. It is likely that many of these restrictions will still be in force in March."

For more information about the conference click here.

Writing Up Results From a Logit Model Using the Odds Ratio Option

A fairly innocuous question -- how best to interpret odds ratio results from a xtlogit model estimation -- unleashed a torrent of commentary on the StataList (click here). As this is a not uncommon model used in many ELS papers, I thought others may benefit from the (extended) discussion.

One of the (many) points that garners attention includes the numeric range for odds ratios: "There is no reason an odds ratio can't be greater than 2. On the other hand, an odds ratio can never be less than 0. The possible values for an odds ratio range from 0 to + infinity. And the interpretation does not depend on the size of the odds ratio. In your case, assuming the predictor is dichotomous, it means that the odds of a positive outcome when the predictor is 1 is 2.46 times the odds of a positive outcome when the predictor is 0. If the predictor is continuous, it means that a unit difference in the value of the predictor is associated with the odds of a positive outcome increasing by a factor of 2.46."

The Empirics of Post-Conviction Plea Bargains

As a prosecutorial tool, the plea bargain exists in uncomfortable space. Any discomfort aside, however, the criminal justice system, as currently constituted, necessarily relies on it. As research on how prosecutors exercise their discretion--including how they deploy plea bargains--continues to develop, greater empirical clarity surrounding plea bargains will hopefully emerge.

While virtually all (of the small number of) plea bargaining studies typically focus on how prosecutors wield plea bargains after a defendants has been charged and prior to the defendant entering a formal plea, a recent paper explores plea bargaining at an entirely different moment in time. Specifically, in Plea Bargaining in the Shadow of a Retrial: Bargaining Away Innocence, Keith Findley (Wisc.) et al. set out to re-focus plea bargaining research by exploring what it looks like in the post-conviction appeals stage. And by exploiting only those post-conviction appeals brought by U.S.-based innocence organizations from 2010 through 2020, the authors try to focus on plea bargains for those defendants who have not-implausible claims of actual innocence.

To be sure, a dataset filtered by innocence organizations' case selections does not by any means insure that all of the (convicted) defendants are actually (or legally) innocent. Indeed, the authors admit as much when they note that "it is a virtual certainty that not all of the defendants in these cases are actually innocent." Over-inclusiveness aside, the authors feel that their dataset is "about as good as one can get for studying plea bargaining with innocent defendants."

Even after adjusting for an imprecise dataset, what the authors find intrigues. "In post-conviction litigation involving defendants with a high likelihood of being actually innocent and wrongly convicted, prosecutors offered plea bargains in 23 percent of the cases. Moreover, when prosecutors made plea offers, the plea concessions they offered were uniformly steep." If nothing else, this paper helps point the way for future research. An excerpted abstract follows.

"Critics of plea bargaining have long contended that it has an innocence problem—that the imbalanced and coercive nature of plea negotiations can induce even innocent defendants to plead guilty. While laboratory studies confirm that innocent individuals can indeed be induced to plead guilty, little real-world empirical evidence exists about the nature and extent of plea bargaining’s innocence problem.

Utilizing original empirical data, this article begins to fill that void. Looking at cases in a post-conviction context, we study the extent to which prosecutors in real cases utilize their plea-bargaining power to preserve convictions, even when the convictions appear to be deeply flawed and the chances the defendants are innocent are high. We also examine the degree to which innocence-claiming defendants succumb to those pressures and accept the deals. To address these questions, we collected a wide-range of data from U.S.-based member organizations of the Innocence Network about the cases they litigated from 2010-2020."

Data on Prosecutorial Discretion By Gender

While a robust judicial outcomes research literature continues to mature and evolve, a critical pre-judicial outcome stage--the exercise of prosecutorial discretion--remains relatively understudied. Understanding the prosecutor's role is crucial as they "exercise unfettered decision-making authority throughout criminal cases by selecting whom to charge, which charges to bring, and what plea bargains to accept." While prosecturial discretion increasingly garners critical public attention, scholarly attention remains hamstrung by a paucity of helpful, systematic data.

In a recent paper, Gender Favoritism Among Criminal Prosecutors, Stephanie Holmes Didwania (Wisconsin) makes some intriguing headway by exploring individualized evidence about prosecutorial decision-making with the benefit of data from more than 130,000 federal criminal defendants sentenced from 2002 through 2016.

The paper's individualized line prosecutor-level data are, of course, key, and the data set is the product of two separate data sources: U.S. Sentencing Commission data (for the defendants) and the Executive Office for United States Attorneys (EOUSA) that regularly publishes case-level data from the 93 USAOs located throughout the United States. These data originate from the Legal Information Office Network System (LIONS), which is the computer program that the EOUSA uses to track and manage case loads. After employing various filters, what emerged includes data on "131,207 defendants sentenced in 63 of the 94 federal district courts in the United States, comprising 172 unique courthouses, 1,498 unique prosecutors, and 1,589 unique courthouse-prosecutor combinations."

Despite the granular data, the paper's empirical execution is not without important limitations. One limiting factor is that line prosecutors are not randomly assigned to cases. Another is that the defendant's underlying criminal conduct is not observable to the researcher. As this paper explores gender-based in-group favoritism, the non-random case assignment is important as "cases in which prosecutors and defendants match on gender are systematically different from those in which prosecutor and defendant mismatch."

Limitations aside, the paper's key findings include some intriguing gender-based differences. While the paper finds that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall, evidence of relative leniency towards defendants of their own gender exists. And for the defendants, such gender-based in-group favoritism is not inconsequential. "The leniency associated with gender-matching ultimately translates into significantly lower sentences: a defendant can expect to earn a roughly five-month shorter sentence when their prosecutor is their own gender compared to when their prosecutor is the opposite gender, which represents a roughly eight percent reduction in sentence length at the mean." The paper's abstract follows.

"Prosecutors enjoy wide discretion in the decisions they make but are largely unstudied by quantitative empirical scholars. This paper explores gender bias in prosecutorial decision-making. I find that male and female prosecutors exhibit small and statistically insignificant differences in their treatment of defendants overall but demonstrate relative leniency towards defendants of their own gender. Such favoritism at charging translates into a sentencing gap of roughly five months of incarceration for defendants who are paired with an own-gender prosecutor versus an opposite-gender prosecutor, which represents a roughly eight percent reduction in sentence length at the mean. The estimates do not appear to be driven by differences in case assignments for male and female prosecutors."

Logical Limits of the Randomized Controlled Trial Research Design

For many, and for obvious methodological reasons, the randomized control trial (“RCT”) is "the" (or "a") research design "gold standard" for those seeking data with possible causal implications. (For a nice quick description of RCTs click here.) In contrast, those interested in a clever, "tongue-in-cheek," "critique" of the pursuit of absolute research design "purity" might be interested in a 2003 "essay," Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials, by Gordon CS Smith (Cambridge--medicine) et al. In the "essay" the authors observe that they could not find any "published research presenting results from randomized controlled trials of parachute use." Consequently, the essay "argues" that the empirical basis for the parachute's efficacy is "purely observational" and, moreover, that existing "observational" data on the parachute's apparent efficacy could potentially be explained by a "healthy cohort" effect.

The Fourth Amendment Post-Carpenter

The long-anticipated collision between Fourth Amendment rights and technology has largely arrived. On the one hand, traditional legal doctrine implies that "information shared with a third party like a bank or telephone company was not protected by the Fourth Amendment." On the other hand, the volume of information now exposed to third parties has "exploded in the internet age, encompassing virtually every kind of digital data generated by internet and cell phone users." In 2018, the Court weighed in and its Carpenter decision was deemed "revolutionary" by holding, in part, that "individuals can retain Fourth Amendment rights in information they disclose to third parties." Specifically, the Court ruled that government agents had to "obtain a warrant before collecting cell phone location data that showed virtually everywhere a suspect had travelled over a seven-day period."

How lower courts understand and have operationalized the Carpenter decision is the focus of The Aftermath of Carpenter: An Empirical Study of Fourth Amendment Law, 2018-2021, by Matthew Tokson (Utah). The paper examines all 857 federal and state court decisions that cite to Carpenter from the decision's 2018 publication through March 2021. Among the salient factors driving lower court outcomes include: "the revealing nature of the data at issue, the amount of data collected, and the automatic nature of data disclosure, and clearly do not include the number of persons affected." An excerpted abstract follows.

"[The paper] examines disagreements among lower courts about the scope and breadth of Carpenter, as some courts apply its concepts expansively while others attempt to narrow it from below. It explores how state courts apply federal constitutional law, blending federal and state interests in unique ways. And it analyzes the enormous practical impact of the “good faith exception” to the exclusionary rule, which permits the government to use unconstitutionally obtained evidence to convict defendants if such evidence was collected in reliance on prior law. Based on these findings, the Article suggests alternative factors and paradigms that courts can adopt to more effectively address Fourth Amendment questions going forward. In addition to its many contributions to the Fourth Amendment literature, the Article is the most comprehensive study to date of the jurisprudential impact of a Supreme Court case in the years following its publication."

Pro Se Civil Litigation in Federal Courts

Public and scholarly attention to pro se litigation is growing. In Mapping the Civil Justice Gap in Federal Court, Roger Michalski (Oklahoma) and Andrew Hammond (Oklahoma) provide a helpful empirical perspective.

Careful attention to pro se litigation is warranted. The sheer volume of pro se litigants will surprise many, if not most. According to the authors' calculations, last year 12 percent of the non-prisoner civil cases filed in federal district courts (more than 25,000 cases) were pro se. Aside from the scope of this issue, "pro se litigants raise unique practical, doctrinal, and normative concerns for the legal academy, the bar, and the federal bench.”

Not surprisingly, methodological challenges limit empirical assessments of pro se litigation. While the paper is appropriately mindful of these limitations, it nonetheless does not "let the perfect be the enemy of the good." As such, the paper analytically confines itself to describing "where pro se litigants live and identify the communities they inhabit." To be sure, while a modest contribution it nonetheless  adds to the knowledge base. That said, analytic limitations endure. "None of our claims speak to the characteristics of pro se litigants directly, only indirectly. This is an important limitation of this study and raises bias concerns. Bias might creep in because it is possible that pro se litigants are systematically atypical of the neighborhoods in which they live."

To accomplish its goals, and limitations aside, the paper exploits "a massive data set of 2.5 million federal dockets from a ten-year period, we obtained addresses of non-prisoner pro se litigants. We then geolocated these addresses and cross-referenced that information with demographic and economic Census data." After various data exclusions (e.g., prisoner suits and cases involving habeas, immigration/deportation cases, naturalization, and bankruptcy claims), what remained included approximately 20,000 federal civil cases per year.

The papers dwells on two core findings. One involves pro se litigants' "profound ordinariness." A second finding involves exceptions to the first. "Even after accounting for population and income, pro se litigants are more likely to reside in communities that are not homogenously white. And numerous rural communities feature fewer pro se litigants than expected.” An excerpted abstract follows.

"Unrepresented litigants make up a sizable and normatively important chunk of civil litigation in the federal courts. Despite their importance, we still know little about who these pro se litigants are. Debates about pro se litigation take place without sufficient empirical information. To help fill some of the gaps in our understanding of pro se litigants, this Article takes a new approach by mapping where pro se litigants live."

Graphics, Options, and Color

While Stata's graphics options (and related coding) have always struck me as unnecessarily complex, others seem to be inspired by them. While in the past "color options" involved monochromatic versus gray-scale, journals increasingly permit actual "color" options. Thus, to this end, I want to pass along a pair of related, though distinct, StataList discussions describing (with helpful coding)  various basic color options for box-plots and histograms.

Moving From Raw Data to More Helpful Variables

For many research projects scholars have to manipulate existing raw data into more useful derived variables. One obvious approach is "hand-coding," or some related form of Excel file manipulation. That is fine for folks experienced with Excel, but for others (including experienced Excel users) such a task typically invites some degree of human error. Another approach is to rely on Stata coding for variable creation. A recent (and, I hope, building conversation on the State List) illustrates a not an atypical example.

The questioner begins with Current Population Survey data (from the Census Dept.) that include individual-level information on, e.g., year, state, and a dummy variable for employment status (0=employed, 1 = unemployed, 2 = not in labor force). What the CPS data set does not include, however, is a state-level unemployment rate by year and state variable. So, the question is how to most efficiently extract what is desired (the creation of a new state-level unemployment rate by year and state variable) from what exists in raw data form. The discussion thus far has elicited at least one very helpful step-by-step examples of how to accomplish this goal with three lines of code (click here).

To be sure, other options also exist. That said, clear explications of how to move from what one has in terms of data to what one wants in terms of a new variable are sometimes hard to find.

Marginal Effects

While journals have traditionally focused on a key variable's direction and statistical significance, over time an increasing number of journal editors (and reviewers) ask for explications of a variable's substantive and "practical" influence. This trend is unlikely to abate anytime soon. And this is precisely where marginal effects (and Stata's margins command) come into play.

While no longer a new topic, I want to pass along the following resource, Using Stata’s Margins Command to Estimate and Interpret Adjusted Predictions and Marginal Effects, by Richard Williams (Notre Dame--sociology). The document is, at once, accessible, thorough, and yet not over-bearing as it walks readers through the basics of the  margins command. The presentation is greatly enhanced by step-by-step illustrations and the necessary coding. In all, quite a helpful resource for those seeking a quick tutorial.

Data on POTUS Pardons

Discomfort with POTUS pardon activity seems to arise anew as administrations (Dem and Rep) turn over and POTUS power transfers. And this discomfort has likely increased over time. While the Constitution’s language in Article II, section 2 regarding the scope of executive clemency and the limitations upon its exercise is quite brief, it bestows upon the POTUS functionally unchecked pardon authority. Insofar as judicial review of POTUS pardon authority is virtually non-existent, the exact legal contours of POTUS authority in this space remain opaque.

How POTUS pardon activity distributes in this zone of potentially nonrenewable Art. II authority is the subject of a recent (and lengthy) RAND report, Statistical Analysis of Presidential Pardons. Distilling their results to a single point, the authors converge on the following, perhaps unremarkable, finding. “If one had to describe a petitioner in our analysis case files more likely to receive a presidential grant of pardon than others (and to do so without controlling for other petitioner and evaluation process characteristics), Table S.2 suggests it would have been a non-Hispanic white male who was a U.S.-born citizen and was in his late twenties when the underlying offense (a white-collar crime such those involving tax violations, embezzlement, forgeries, or counterfeiting) was committed.” The report's abstract follows.

“This report describes a study of pardon petition evaluations that were performed by the Office of the Pardon Attorney (OPA), a unit in the U.S. Department of Justice that processes thousands of requests for executive clemency each year. A statistical examination was conducted on how OPA screens incoming petitions for technical compliance, investigates the facts underlying each petitioner’s request for clemency, and applies substantive standards for evaluating the merits of such petitions when crafting recommendations to the U.S. President to grant or deny relief. Data were analyzed for any patterns in OPA decisions that indicated statistically significant evidence of racial or ethnic bias.”

Testing the Difference Between Two Regression Coefficients Within Same Model

Over on the StataList, a very helpful discussion was launched by the following question that may interest folks here:

"I have one single regression model estimating the coefficients (b) of two independent variables (both are binary dummy variables). After the regression, I would like to do a t-test which calculates the difference between the two variables' coefficients (bIV1 - bIV2) and then gives a p-value for that (i.e. p-value: bIV1 = bIV2)."

To generate a Wald test (and associated p-value), all that is needed is the test command. If, however, one wants to see magnitude (and confidence interval) of the difference between two coefficients within a model, lincom is the appropriate command. For an helpful illustration of how this command works (with output), click here.

CELS "2020" -- Call For Papers Extension [New Deadline: 31 October 2021]

As previously mentioned (here and here), CELS "2020," re-scheduled for March 18-19, 2022, still plans on an in-person conference at the University of Toronto (Canada). At the same time, however, conference organizers note that: "We are of course monitoring the progress of the COVID-19 pandemic and the Delta variant. At this point we are hopeful the conference will proceed as planned. But if the Society For Empirical Legal Studies (SELS) deems it necessary to change plans, we will let you know as soon as possible."

To better accommodate the various disruptions incident to the pandemic, the conference organizers have extended the Call for Papers deadline to October 31, 2021.

Click here for additional conference information.

To submit a paper, please click here.

How Lower Courts Digest Supreme Court Decisions: Data on the Fourth Amendment After Carpenter (2018)

Technical advances and their implications for civil liberties collided anew in the Supreme Court's 2018 Carpenter v. U.S. decision. In that case, the Court made clear that "individuals can retain Fourth Amendment rights in information they disclose to a third party." The Court specifically ruled that the government needed a warrant before collecting cell phone location data that revealed a suspect's location over a seven-day period. In today's Internet age, Carpenter has the potential to expand privacy protections over a wide and deep swath of sensitive digital information. Insofar as the decision's practical contours and boundaries are, unsurprisingly, vague, "scholars and lower courts have tried to guess at what the law of Fourth Amendment searches will be going forward." Equally unsurprising, these guesses about Carpenter's meaning diverge.

In a recent paper, The Aftermath of Carpenter: An Empirical Study of Fourth Amendment Law, 2018-2021, Matthew Tokson (Utah) subjects various predictions and claims about Carpenter's implications to data. In particular, the paper analyzes "all 857 federal and state judgments citing Carpenter from its publication in June 2018 through March 2021. Relying on this unique, hand-coded database, the Article illuminates both the present and future of Fourth Amendment law.” Among the paper's findings include that: "factors that drive lower court decisions clearly include the revealing nature of the data at issue, the amount of data collected, and the automatic nature of data disclosure, and clearly do not include the number of persons affected." An excerpted abstract follows.

"[The paper] identifies the factors that drive modern Fourth Amendment search decisions—and those that fail to drive them. It examines disagreements among lower courts about the scope and breadth of Carpenter, as some courts apply its concepts expansively while others attempt to narrow it from below. It assesses how state courts apply federal constitutional law, blending federal and state interests in unique ways. And it analyzes the enormous practical impact of the 'good faith exception' to the exclusionary rule, which permits the government to use unconstitutionally obtained evidence to convict defendants if such evidence was collected in reliance on prior law. Based on these findings, the Article explores alternative directions that courts may take as they continue to refine Fourth Amendment law and address novel surveillance technologies. In addition to its many contributions to the Fourth Amendment literature, the Article is the most comprehensive empirical study to date of the jurisprudential impact of a Supreme Court case in the years following its publication."

US News Law School Rankings and "Scholarly Impact" Metrics: An Update

A loyal Els Blog reader passed along the following and asked that I share it. I am delighted to do so.

A while back the US News and World Report reported that it was contemplating a new metric designed to measure “scholarly impact” incident to its infamous annual law school rankings. Needless to say, given law schools’ focus on annual rankings US News’ contemplated change to its ranking formula generated considerable blow-back (from many different directions). The Law & Society Association, for example, worried that a scholarly impact measure drawing from the Hein database would have deleterious impacts on interdisciplinary scholars.

In the end (and for now, at least), the US News and World Report elected not to proceed with its proposed new scholarly impact metric. A statement from the Law & Society Association (here) nicely describes its concerns and expresses relief at US News’ recent decision.

Copyright Injunctions After eBay and the "Hysteresis Hypothesis"

Interest in a recent paper, The Hysteresis Thesis: An Empirical Study of Copyright Injunctions After eBay, by Matthew Sag (Loyola-Chi) and Pamela Samuelson (Berkeley), should extend beyond traditional IP/copyright borders. Here's why.

First, the IP/copyright implications. Prior to the Supreme Court’s landmark 2006 eBay decision, courts routinely issued injunctions in both patent and copyright cases upon a finding of actual or likely infringement of plaintiffs’ rights. Even though most scholars expected that the eBay decision would have an immediate and transformative effect on the grant of permanent injunctions in patent infringement cases, Jiari Liu's 2012 paper reported that in the first four years after the eBay ruling, courts rarely cited that decision and continued to grant injunctions at a very high rate. Sag and Samuelson's paper, however, concludes that Liu's main finding (and conventional wisdom) were "always somewhat erroneous and are now clearly outdated."

Second, the implications of the "how" and "why" of Sag and Samuelson's paper spill-over into empirical legal studies more broadly. Sag and Samuelson began their study with an eye towards "the hypothesis of hysteresis or delayed effect: the simplest way to think about this theory is that, if it is correct, we should see a greater effect in the third and fourth periods than in the 2007-2010 period immediately following eBay."

To do so, they exploited a data set of "336 reported U.S. federal court copyright injunction cases decided between January 1, 2000 and December 31, 2019. We begin with the pre-eBay era, from 2000 to 2006. Our remaining three eras are 2007-2010 (the immediate aftermath era), 2011-2014 (the Salinger-Flexible era), and 2015-2019 (the consensus era)." What they find, in part, is that: "Our evidence shows that the citation rate for eBay has increased and that the corresponding grant rate for copyright injunctions has decreased."

They attribute their findings of an Hysteresis effect to "the doctrinal conservatism of lower courts and the parochial nature of appellate court precedents, it is unsurprising that there might have been some delay in the adoption of even the most obvious implications of new Supreme Court precedents such as eBay for injunction grants in copyright cases." An excerpted abstract follows.

"... This Article tests these claims and finds statistically significant differences in the grant rate for both preliminary and permanent injunctions in copyright cases after the eBay decision. The impression of eBay lacking effect appears to be a product of hysteresis—a time lag between cause and effect, as lower courts initially delayed but eventually embraced eBay. This study reveals, moreover, that the earlier empirical study was flawed by biased sampling, most notably by including a large number of unreported cases and default judgments. This Article offers a corrective analysis and provides a more accurate understanding of the copyright injunction landscape, both pre- and post-eBay. It shows that eBay was in fact a landmark case for copyright law, both in terms of jurisprudence—it is highly cited—and real-world effect— it has a dramatic impact on case outcomes."

The Federal Circuit Dataset project (v.3)

While Jason Rantanen's (Iowa) third update to The Federal Circuit Dataset Project will certainly interest empirical IP scholars, the Project's broader implications speak to empirical law scholars in all fields.

The third edition advances the existing Compendium of Federal Circuit Decisions in at least two main ways. First, the dataset now includes document numbers for all archived documents. Second, the dataset now facilitates increased understanding about what happens to appeals, including information about appeal pendency and origin.

Setting aside the dataset's utility to empirical IP scholars, the Project itself reflects a growing commitment to the hard and often under-rewarded work of dataset creation as well as an impulse to make such data (and related codebooks, etc.) publicly-available. Such impulses need to be rewarded, promoted, and nurtured. The paper's abstract follows.

"This paper introduces the third generation of the Federal Circuit Dataset Project. The third generation builds on the existing Compendium of Federal Circuit Decisions with the following enhancements:

• The addition of a dataset of all dockets for the Federal Circuit since 2000. These dockets are linked to the documents in the Compendium through the Appeal_Dockets field in both datasets.
• New fields containing additional information about Federal Circuit decisions.
• More complete data in existing fields that require manual coding.
• Parallel archiving of the document and docket datasets on the Dataverse.

In addition to describing the third generation of the Federal Circuit Dataset Project, this paper also provides updated descriptive statistics about Federal Circuit decisions and new statistics on appeal dockets."

Technology and Legal Citation "Stickiness"

An interesting citation paper caught my eye, principally for its research design. In Citation Stickiness, Computer-Assisted Legal Research, and the Universe of Thinkable Thoughts, Aaron Kirschenfeld (UNC) and Alexa Chew (UNC), set out to explore how citation “stickiness” has survived the emergence of a “digital era” in legal research.

What the paper means by “stickiness” is that, in theory, the percentage of sticky citations   in   courts’   opinions   would   be   something   approaching 100%. That is, the  parties  would  discuss  the  relevant authorities in their briefs, and the court would rely  on  the  same  authorities  in  its  opinions.

Setting aside from the paper’s substantive contribution on “stickiness,” for me the paper’s main analytic move is to assess “stickiness” across dynamic legal research platforms over time. While the data are limited to a small handful of cases (N=325) from a single Circuit (CA4) and a handful of years (1957, 1987, and 2017), I anticipate that other research will employ a similar move to gain additional insights on how the emergence of the “digital era” informs law-related output. The abstract follows.

“Legal information has been available in widespread digital format for more than forty years. In that time, law librarians have wondered whether this digital switch has changed how law students and lawyers conduct research and, if so, what those changes are. Does legal research differ when conducted in print sources rather than computerized sources? What influence did the systems of organizing law in the print era have on the digital systems that followed?

While we cannot put these questions to rest, we hope to shed some light on those difference by studying the work of lawyers and courts from the print era, the transition-to-digital era, and the digital era. Using a metric called “citation stickiness,” we studied how often parties to an appeal and the judges hearing that appeal agreed on the cases relevant to resolve the issues on appeal. This, we hoped, would also show whether there were perceptible differences in coherence and stability of the legal information landscape

Citation stickiness works like this: a citation is “sticky” if it appears in at least one party’s brief and then again in the court opinion. In an initial study of 325 federal court cases from 2017, 49% of the 7,552 cases that were cited in the courts’ opinions had been cited by at least one party in a brief.

This study considers cases from the United States Court of Appeals for the Fourth Circuit decided in 1957, 1987, and 2017. After examining the citations in the briefs and opinions in our sample of cases, we compare our findings from the pre-digital era and from the digital era. What we learned surprised us.”

Adjusting for "Ambiguous" Zeros in an Ordered Logit Model

In the ELS silo, what to do with--and/or how to best interpret--"zeros" frequently persist as common challenges. In the world of standard count data, possible approaches include Zero-inflated Poisson Regression and Zero-inflated Negative Binomial Regression. Ordered logit analyses, however, require something slightly different.

To illustrate, a standard example is social science research (and explicated in this brief yet helpful video) includes, e.g., a four-siloed response to the question "How many cigarettes did you smoke each day, on average (0, 1-7, 8-12, >12)?" The possible ambiguity of the "0" response is that it may include responses from people who smoked less than one cigarette per day, on average, as well as those do not smoke cigarettes at all. And if the research question is designed to get a smokers' daily volume, the "0" response from non-smoker respondents is qualitatively different than a "0" response from smokers.

To address potential distortions to the "0" response introduced by non-smokers in this hypothetical, Stata's ziologit command fits a model for ordinal outcomes with an expected "excess" of "0" responses.