Empirical Legal Studies

Efforts to Randomize & Efficacy of Representation

In a pair of related, though distinct, papers, Jim Greiner (Harvard Law), Cassandra Pattanayak (Harvard--Stats), and Jonathan Hennessy (Harvard--Stats) seek to exploit randomized selection in a challenging effort to explore the efficacy of legal representation. Findings from the two studies point in slightly different directions.

In Limits of Unbundled Legal Assistance: a Randomized Study In A Massachusetts District Court and Prospects for the Future and How Effective Are Limited Legal Assistance Programs? A Randomized Experiment in a Massachusetts Housing Court the authors present results from a studies that share a setting-- summary eviction in Massachusetts. Specifically, the authors "persuaded entities conducting two civil Gideon pilot programs to randomize which potential clients would receive offers of traditional attorney-client relationships from professional service provider staff attorneys and which would receive only limited (“unbundled”) assistance. In both pilot programs potential clients were defendants in housing eviction proceedings, and both programs were oversubscribed."

"In the District Court Study, almost all study-eligible eviction defendants received limited assistance in the form of help in filling out answer and discovery request forms, and most also attended an instructional session on the summary eviction process. After receiving this limited assistance, each member of a randomly selected treated group received an offer of a traditional attorney-client relationship from one of the legal services provider’s staff attorneys; each member of the corresponding randomly selected control group received no such offer.... At least for the clientele involved in this District Court Study, a clientele recruited and chosen by the service provider’s proactive, timely, specific, and selective outreach and intake system, an offer of full representation mattered."

"In the Housing Court Study, all study-eligible potential clients could (and most did) receive assistance in filling out answers and discovery requests. In addition, occupants not offered a traditional attorney-client relationship from the provider's staff attorneys, the control group, received a referral to that provider's lawyer for the day program. The lawyer for the day program provided same-day-only representation in hallway settlement negotiations and mediation sessions but not in court appearances or in filing motions.... We find no statistically significant evidence that the service provider's offer of full, as opposed to limited, representation had a large (or any) effect on the likelihood that the occupant would retain possession, on the financial consequences of the case, on judicial involvement in or attention to cases, or on any other litigation-related outcome of substantive import. To the contrary, treated and control group point estimates are close to one another."

The Constrained Court

That's the title of Michael Bailey and Forrest Maltzman's new book, on Princeton press. It's a good one, and it is getting quite a bit of well-deserved praise. 

What it has also done, if taken seriously, is toss a grenade into how we measure Court-related things, especially policy preferences. To the extent that measures like the Martin-Quinn scores are based on votes, and votes aren't just tapping ideology (a point made convincingly in the book), we're back in the difficult spot of having relatively little to go on if we want to measure the justices' policy preferences directly.

Of course, people will continue to use M-Q scores and the like as preference measures, just as they use things like NOMINATE in the Congressional realm. But Bailey and Maltzman's book also suggests that we should begin to think hard about other sources of data for this key construct.

The Influence of Race on eBay Auctions

Regardless of what one might think about the prospects for a "Post-Racial" society, one could plausibly assume that racial effects are less likely to emerge in an auction setting. Results from a clever study challenge that assumption, however. In Race Effects on Ebay, Ian Ayres (Yale), Mahzarin Banaji (Harvard--Psych), and Christine Jolls (Yale) leverage eBay, manipulate race, and assess racial effects on the price of baseball cards. The abstract follows.

"We investigate the impact of seller race in a field experiment involving baseball card auctions on eBay. Photographs showed the cards held by either a dark-skinned/African-American hand or a light-skinned/Caucasian hand. Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards. Our evidence of race differentials is important because the on-line environment is well controlled (with the absence of confounding tester effects) and because the results show that race effects can persist in a thick real-world market such as eBay."

Statistical v. Legal "Significance": Matrixx-Inspired Scholarship

In Matrixx Initiatives Inc. v. Siracusano, the Supreme Court considered “[w]hether a plaintiff can state a claim under § 10(b) of the Securities Exchange Act and SEC Rule 10b-5 based on a pharmaceutical company’s nondisclosure of adverse event reports even though the reports are not alleged to be statistically significant.”

The Court said "yes," and argued that a reasonable investor might want to know of such reports if they are sufficiently extensive and disturbing that they could prompt the FDA to take some action or might lead to costly lawsuits. Thus, even non-statistically significant information can be material and enough to trigger a legal duty to disclose.

In Trapped in the Matrixx: The U.S. Supreme Court and the Need for Statistical Significance, David Kaye (Penn St.) defends the Court's interpretation of the pleading rule but argues for a narrow reading of dicta in the opinion about proof of causation in product liability cases. Kaye's paper "suggests that the unanimous opinion conflates issues of study design with statistical significance. Disentangling these concepts shows that the Court’s remarks do not address the limited value of adverse event reports in establishing causation in toxic tort litigation."

A Critical Appraisal of Empirically "Judging Judges"

A while back we noted, among other papers in the "judging judges" genre, one by Stephen Choi (NYU), Mitu Galati (Duke), Mirya Holman (Duke/UNC), and Eric Posner (Chicago) that sought to hold claims about Justice Sotomayor’s now-infamous assertion, uttered prior to her Senate confirmation hearing, that female judges might be “better” than male judges. As noted in 2009, the Judging Women paper "estimate three measures of judicial output: opinion production, outside state citations, and co-partisan disagreements." The paper found that "male and female judges perform at about the same level."

The enterprise of empirically "judging judges," however, recently generated its own push-back. In The Costs of Judging Judges by the Numbers, the authors, Marin Levy (Duke), Kate Stith (Yale), and José Cabranes (CA2), advance some plausibly dramatic claims about the efficacy of "judging judges," at least empirically. The paper's abstract follows.

"This essay discredits current empirical models that are designed to “judge” or rank appellate judges, and then assesses the harms of propagating such models. First, the essay builds on the discussion of empirical models by arguing that (1) the judicial virtues that the legal empiricists set out to measure have little bearing on what actually makes for a good judge; and (2) even if they did, the empiricists’ chosen variables have not measured those virtues accurately. The essay then concludes that by generating unreliable claims about the relative quality of judges, these studies mislead both decision-makers and the public, degrade discussions of judging, and could, if taken seriously, detrimentally alter the behavior of judges themselves."