Empirical Legal Studies

More Evidence of a Racialized Death Penalty

Notably, race does not emerge as a central concern in the Court’s Furman decision, now more than fifty years old. In the decades since Furman, however, empirical evidence continues to mount documenting that "racial discrimination infects the administration of the death penalty." A recent paper by Jeffrey Fagan et al. (Columbia), Getting to Death: Race and the Paths of Capital Cases after Furman, represents a recent and important contribution to this literature. Researchers visited each judicial circuit in Georgia to record the details of every murder case that entered the courts from 1994–2005. Analyses of the 1,317 death-eligible murders suggest that “race continues to matter, and that the statutory aggravators that distinguish death cases from other killings are themselves racialized.” A more detailed description of the paper's core findings appears in the Abstract (below).

“Decades of research on the administration of the death penalty have recognized the persistent arbitrariness in its implementation and the racial inequality in the selection of defendants and cases for capital punishment. This Article provides new insights into the combined effects of these two constitutional challenges. We show how these features of post-Furman capital punishment operate at each stage of adjudication, from charging death-eligible cases to plea negotiations to the selection of eligible cases for execution and ultimately to the execution itself, and how their effects combine to sustain the constitutional violations first identified 50 years ago in Furman. Analyzing a dataset of 2,328 first- degree murder convictions in Georgia from 1995–2004 that produced 1,317 death eligible cases, we show that two features of these cases combine to produce a small group of persons facing execution: victim race and gender, and a set of case-specific features that are often correlated with race. We also show that these features explain which cases progress from the initial stages of charging to a death sentence, and which are removed from death eligibility at each stage through plea negotiations. Consistent with decades of death penalty research, we also show the special focus of prosecution on cases where Black defendants murder white victims. The evidence in the Georgia records suggests a regime marred less by overbreadth in its statute than capriciousness and randomness in the decision to seek death and to seek it in a racially disparate manner. These two dimensions of capital case adjudication combine to sustain the twin failures that produce the fatal lottery that is the death penalty.”

Interpreting DID Output

While one may quibble about how the questioner designed her DID estimation, the question itself--how to understand and interpret standard DID output (in Stata)--elicited some helpful replies in a recent StataList discussion (here).

Law School Curriculum: Reform, Assessment, and Research Design

While age-old complaints persist about the basic structure of the first-year law school curriculum, few law schools have dared to experiment with meaningful reforms. Fewer still study these reforms coherently. The "Curriculum B" experiment at Georgetown Law School, as well as a recent paper by David Hyman et al. (Georgetown), Does the 1L Curriculum Make a Difference?, are important and interesting exceptions.

Since 1991, Georgetown Law School has offered an "alternative" Curriculum B along side a standard, more traditional first-year curriculum. Curriculum B was designed to provide “critical” perspectives on the law and respond to important changes in legal practice and theory in the years since the standard 1L curriculum was first developed decades earlier. At Georgetown, students elect into Curriculum B and, when demand exceeds supply, student assignment to Curriculum B is determined by lottery.

For assessment purposes, the quasi-randomized control trial aspect of student assignment to Curriculum B provides a way to overcome selection bias. That is, Hyman et al. compare student outcomes for those who opted into Curriculum B (the treatment group) with student outcomes from those students who wanted to receive the "treatment" (Curriculum B) but, due to excess demand and random selection (lottery), were instead assigned the "placebo" (that is, to the standard, traditional first-year curriculum).

Aside from careful attention to research design, a few of the paper's warrant note. When it comes to students' overall law school academic performances, for example, the authors find that "Curriculum B leads to a mean preserving contraction in students’ overall grades. At the same time, we find that it leads to diminished performance in two key business law electives (Corporations and Securities Regulation)."

Multi-Level Model "vs." Clustering

As this question (or something like it) is not an infrequent one, I found Jeff Wooldridge's response (No.2) in this StataList discussion (here) quite helpful. The bottom line is that decisions incident to estimating a multi-level model or/and clustering pivot on an array of considerations.

Simple Problems, Simple Solutions

While I should have known this years ago, better late than never.

As one becomes familiar with the broad descriptive contours of new data, the tab function is both well-known and ubiquitous. Finding analogous basic descriptive statistics for sub-samples, while certainly possible, was typically (for me, anyway) a bit cumbersome with the tab command.

A recent discussion on the StataList, however, introduced me to the simple (and code-saving) power of a related command, tabstat. As this discussion illustrates (click here), what the tab command accomplishes with multiple lines of coding and multiple output tables, the tabstat command, by contrast, accomplishes with a single line of code and a single, unified output table.

Comparing Regression Output Across Models

As I am seeing more papers trying to do so (as well as discussions about strategies on how to do so), I thought links to a recent StataList discussion as well as some web-based resources might be of general interest. The basic set-up follows:

Model A: Y = α1 + β1X1 + Σθ∙C + firm-fixed effect + year-fixed effect.

Model B: Y = α1 + β2X2 + Σθ∙C + firm-fixed effect + year-fixed effect.

Σθ∙C is list of control variables and are the same for both model A & B.

Question: How to compare the coefficients of β1 and β2?

One approach exploits a post-estimation command in Stata (suest), and then compares means with test (for a discussion and illustration, click here). Another illustrated example, courtesy of the UCLA Statistical Methods and Data Analytics site, is here.

Approaches Toward Missing Data

Missing data vex researchers, and there's no single "correct" way to deal with this not-infrequent real-world problem. A recent question posted on the StataList (here) visually illustrates what this can look like (in terms of N variation) across alternative specifications of a multilevel regression model drawing from a data set that contains missing data in some variables.

As this output illustrates, as one proceeds across models 1-4 what is clear is that the addition (and subtraction) of variables introduce missing data. How to approach this and what to do about it motivated the initial query. And in the discussion that follows identifies (some of the) various "standard" approaches.

Mistaking Correlation for Causation

Economists--and econometrics--have been involved in empirical legal studies for generations. This is particularly true for anti-trust doctrine, both its development and current application. Legal challenges to proposed mergers, and questions relating to anti-competitive spill-over effects, including market concentration, remain a staple for empirical analyses.

In the anti-trust context, one intuitive move involves assessing whether "prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods." Pursuing such a question can involve (and frequently does involve) something as simple (and potentially powerful) as regressing price on the HHI.

Despite whatever intuitive appeal such a move holds, in a recent brief essay--co-signed by many leading law and economics scholars (including, and I add in interest of full disclosure, my Cornell colleague George Hay), On the Misuse of Regressions of Price on the HHI in Merger Review, the authors caution against such a move as it invites one basic fundamental problem: Mistaking correlation for causation. And it is for the essay's description of just how such a basic mistake can arise (rather than it's specific application in the anti-trust context) that I recommend this essay, especially to younger, developing empirical legal scholars.

As it relates to the specific anti-trust point developed in the essay, the authors' argue that: "The underlying problem with regressions of price on the HHI is that the relationship between price and the HHI is not causal. Instead, both are equilibrium outcomes that are determined by demand, supply, and the factors that drive them. Thus, a regression of price on the HHI does not show the sort of causal effect that would be helpful in predicting the competitive effects of a merger." More specifically, as the "HHI can take values ranging from 0 to 10,000, with the former corresponding to a fragmented market with infinitesimal firms, and the latter to monopoly. All else equal, a decrease in the number of firms leads to a higher HHI. However, the HHI can increase if, for a given number of firms, the market shares of the larger firms increase."

The root problem, of course, flows from challenges posed by “endogeneity” or “simultaneity.” Such problems lurk "when a clear causal path between two variables is posited by the relevant economic theory but the causal factor (on the right side of the regression equation) is either correlated with unobserved factors or, more subtly, is simultaneously determined by the variables (in this case, price) on the left side of the regression. A famous example of simultaneity involves price and quantity, which are simultaneously determined in the classic economic model of supply and demand." The problems posed by "endogeneity” or “simultaneity," of course, are not limited to the anti-trust realm. An excerpted abstract follows.

"It might seem natural to determine whether prices are positively related to measures of concentration, such as the Herfindahl-Hirschman Index (HHI), comparing across different geographic markets or time periods. This might be implemented by using a simple regression of price on the HHI. However, for reasons that we describe, regressions of price on the HHI should not be interpreted as establishing causation. That is, they do not inform how a change in concentration from a merger would affect prices. Courts and other policy-makers should not rely on regressions of price on the HHI for the purposes of antitrust merger review."

Writing Up Results From a Logit Model Using the Odds Ratio Option

A fairly innocuous question -- how best to interpret odds ratio results from a xtlogit model estimation -- unleashed a torrent of commentary on the StataList (click here). As this is a not uncommon model used in many ELS papers, I thought others may benefit from the (extended) discussion.

One of the (many) points that garners attention includes the numeric range for odds ratios: "There is no reason an odds ratio can't be greater than 2. On the other hand, an odds ratio can never be less than 0. The possible values for an odds ratio range from 0 to + infinity. And the interpretation does not depend on the size of the odds ratio. In your case, assuming the predictor is dichotomous, it means that the odds of a positive outcome when the predictor is 1 is 2.46 times the odds of a positive outcome when the predictor is 0. If the predictor is continuous, it means that a unit difference in the value of the predictor is associated with the odds of a positive outcome increasing by a factor of 2.46."

Logical Limits of the Randomized Controlled Trial Research Design

For many, and for obvious methodological reasons, the randomized control trial (“RCT”) is "the" (or "a") research design "gold standard" for those seeking data with possible causal implications. (For a nice quick description of RCTs click here.) In contrast, those interested in a clever, "tongue-in-cheek," "critique" of the pursuit of absolute research design "purity" might be interested in a 2003 "essay," Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials, by Gordon CS Smith (Cambridge--medicine) et al. In the "essay" the authors observe that they could not find any "published research presenting results from randomized controlled trials of parachute use." Consequently, the essay "argues" that the empirical basis for the parachute's efficacy is "purely observational" and, moreover, that existing "observational" data on the parachute's apparent efficacy could potentially be explained by a "healthy cohort" effect.

Marginal Effects

While journals have traditionally focused on a key variable's direction and statistical significance, over time an increasing number of journal editors (and reviewers) ask for explications of a variable's substantive and "practical" influence. This trend is unlikely to abate anytime soon. And this is precisely where marginal effects (and Stata's margins command) come into play.

While no longer a new topic, I want to pass along the following resource, Using Stata’s Margins Command to Estimate and Interpret Adjusted Predictions and Marginal Effects, by Richard Williams (Notre Dame--sociology). The document is, at once, accessible, thorough, and yet not over-bearing as it walks readers through the basics of the  margins command. The presentation is greatly enhanced by step-by-step illustrations and the necessary coding. In all, quite a helpful resource for those seeking a quick tutorial.

Testing the Difference Between Two Regression Coefficients Within Same Model

Over on the StataList, a very helpful discussion was launched by the following question that may interest folks here:

"I have one single regression model estimating the coefficients (b) of two independent variables (both are binary dummy variables). After the regression, I would like to do a t-test which calculates the difference between the two variables' coefficients (bIV1 - bIV2) and then gives a p-value for that (i.e. p-value: bIV1 = bIV2)."

To generate a Wald test (and associated p-value), all that is needed is the test command. If, however, one wants to see magnitude (and confidence interval) of the difference between two coefficients within a model, lincom is the appropriate command. For an helpful illustration of how this command works (with output), click here.

Technology and Legal Citation "Stickiness"

An interesting citation paper caught my eye, principally for its research design. In Citation Stickiness, Computer-Assisted Legal Research, and the Universe of Thinkable Thoughts, Aaron Kirschenfeld (UNC) and Alexa Chew (UNC), set out to explore how citation “stickiness” has survived the emergence of a “digital era” in legal research.

What the paper means by “stickiness” is that, in theory, the percentage of sticky citations   in   courts’   opinions   would   be   something   approaching 100%. That is, the  parties  would  discuss  the  relevant authorities in their briefs, and the court would rely  on  the  same  authorities  in  its  opinions.

Setting aside from the paper’s substantive contribution on “stickiness,” for me the paper’s main analytic move is to assess “stickiness” across dynamic legal research platforms over time. While the data are limited to a small handful of cases (N=325) from a single Circuit (CA4) and a handful of years (1957, 1987, and 2017), I anticipate that other research will employ a similar move to gain additional insights on how the emergence of the “digital era” informs law-related output. The abstract follows.

“Legal information has been available in widespread digital format for more than forty years. In that time, law librarians have wondered whether this digital switch has changed how law students and lawyers conduct research and, if so, what those changes are. Does legal research differ when conducted in print sources rather than computerized sources? What influence did the systems of organizing law in the print era have on the digital systems that followed?

While we cannot put these questions to rest, we hope to shed some light on those difference by studying the work of lawyers and courts from the print era, the transition-to-digital era, and the digital era. Using a metric called “citation stickiness,” we studied how often parties to an appeal and the judges hearing that appeal agreed on the cases relevant to resolve the issues on appeal. This, we hoped, would also show whether there were perceptible differences in coherence and stability of the legal information landscape

Citation stickiness works like this: a citation is “sticky” if it appears in at least one party’s brief and then again in the court opinion. In an initial study of 325 federal court cases from 2017, 49% of the 7,552 cases that were cited in the courts’ opinions had been cited by at least one party in a brief.

This study considers cases from the United States Court of Appeals for the Fourth Circuit decided in 1957, 1987, and 2017. After examining the citations in the briefs and opinions in our sample of cases, we compare our findings from the pre-digital era and from the digital era. What we learned surprised us.”

Adjusting for "Ambiguous" Zeros in an Ordered Logit Model

In the ELS silo, what to do with--and/or how to best interpret--"zeros" frequently persist as common challenges. In the world of standard count data, possible approaches include Zero-inflated Poisson Regression and Zero-inflated Negative Binomial Regression. Ordered logit analyses, however, require something slightly different.

To illustrate, a standard example is social science research (and explicated in this brief yet helpful video) includes, e.g., a four-siloed response to the question "How many cigarettes did you smoke each day, on average (0, 1-7, 8-12, >12)?" The possible ambiguity of the "0" response is that it may include responses from people who smoked less than one cigarette per day, on average, as well as those do not smoke cigarettes at all. And if the research question is designed to get a smokers' daily volume, the "0" response from non-smoker respondents is qualitatively different than a "0" response from smokers.

To address potential distortions to the "0" response introduced by non-smokers in this hypothetical, Stata's ziologit command fits a model for ordinal outcomes with an expected "excess" of "0" responses.

Pretrial Detention and Release Experiments

As I have an admitted soft-spot for (even quasi-) randomized control trials as a research design, a recent paper by Javier Osorio (Arizona-Govt. & Pub Pol’y) et al., Reducing Pretrial Detention: A Randomized Intervention with Public Defenders in El Salvador, caught my eye. In it, the authors exploit a "randomized intervention in El Salvador designed to improve public defenders’ capacities to secure pretrial release for defendants.”

To note that pretrial detention is a problem is to note merely the obvious. And in countries facing acute security challenges, such as El Salvador, there exists enormous pressure on criminal justice authorities to limit pretrial release. The authors contend that the overreliance on pretrial detention is driven in part by lack of information about defendants’ flight risks and the prospect of their committing further crimes while awaiting trial. And this information deficit is “the consequence of deficiencies in the public defender system, upon which economically disadvantaged citizens rely to ensure legal representation.”

To assess their hypothesis, the authors exploit an unusual (by U.S. standards, anyway) experiment involving a randomly assigned treatment to public defenders in four municipalities in El Salvador. “The treatment included specialized legal training, an improved interview protocol, and enhanced communication channels with the defendants, their relatives, and other institutions. These components sought to improve public defenders’ ability to gather evidence that would support pretrial release.”

The authors found that the treatment “increased public defenders’ requests for pretrial release by nearly 10%. The intervention also helped public defenders to secure pretrial release of their clients by 4.4%, although these results are weaker and sensitive to model specification.” The paper's abstract follows.

"Approximately one third of the global prison population is in pretrial detention, waiting for trial. Overreliance on pretrial detention exposes defendants to harsh conditions, exacerbates jail overcrowding, increases recidivism, and favors criminal governance. What policies can resource strapped countries implement to effectively address excessive pretrial detention? Based on a theoretical model focused on institutional-level efforts, we evaluate an experimental intervention implemented in El Salvador intended to increase pretrial release requests and reduce pretrial detention. The intervention randomly assigned public defenders to receive specialized legal training, an improved interview protocol, material resources, and increased communication channels. We find that this inexpensive, scalable program increased pretrial release requests from public defenders by nearly 10% (0.228 standard deviations) and increased the success in securing pretrial release by 4.4% (0.114 standard deviations). Heterogeneous treatment effect analyses suggest that the program increased strategic litigation among the most experienced public defenders and has distinct effects for those accused of minor and severe crimes. We find no evidence that the mechanism explaining our results involves changes in public defenders’ attitudes or perceptions about their work environments. Criminal justice programs focusing on pretrial detention may help reduce prison overcrowding in high crime countries."