Empirical Legal Studies

Statistical v. Legal "Significance": Matrixx-Inspired Scholarship

In Matrixx Initiatives Inc. v. Siracusano, the Supreme Court considered “[w]hether a plaintiff can state a claim under § 10(b) of the Securities Exchange Act and SEC Rule 10b-5 based on a pharmaceutical company’s nondisclosure of adverse event reports even though the reports are not alleged to be statistically significant.”

The Court said "yes," and argued that a reasonable investor might want to know of such reports if they are sufficiently extensive and disturbing that they could prompt the FDA to take some action or might lead to costly lawsuits. Thus, even non-statistically significant information can be material and enough to trigger a legal duty to disclose.

In Trapped in the Matrixx: The U.S. Supreme Court and the Need for Statistical Significance, David Kaye (Penn St.) defends the Court's interpretation of the pleading rule but argues for a narrow reading of dicta in the opinion about proof of causation in product liability cases. Kaye's paper "suggests that the unanimous opinion conflates issues of study design with statistical significance. Disentangling these concepts shows that the Court’s remarks do not address the limited value of adverse event reports in establishing causation in toxic tort litigation."

A Critical Appraisal of Empirically "Judging Judges"

A while back we noted, among other papers in the "judging judges" genre, one by Stephen Choi (NYU), Mitu Galati (Duke), Mirya Holman (Duke/UNC), and Eric Posner (Chicago) that sought to hold claims about Justice Sotomayor’s now-infamous assertion, uttered prior to her Senate confirmation hearing, that female judges might be “better” than male judges. As noted in 2009, the Judging Women paper "estimate three measures of judicial output: opinion production, outside state citations, and co-partisan disagreements." The paper found that "male and female judges perform at about the same level."

The enterprise of empirically "judging judges," however, recently generated its own push-back. In The Costs of Judging Judges by the Numbers, the authors, Marin Levy (Duke), Kate Stith (Yale), and José Cabranes (CA2), advance some plausibly dramatic claims about the efficacy of "judging judges," at least empirically. The paper's abstract follows.

"This essay discredits current empirical models that are designed to “judge” or rank appellate judges, and then assesses the harms of propagating such models. First, the essay builds on the discussion of empirical models by arguing that (1) the judicial virtues that the legal empiricists set out to measure have little bearing on what actually makes for a good judge; and (2) even if they did, the empiricists’ chosen variables have not measured those virtues accurately. The essay then concludes that by generating unreliable claims about the relative quality of judges, these studies mislead both decision-makers and the public, degrade discussions of judging, and could, if taken seriously, detrimentally alter the behavior of judges themselves."

Competing Takes on ELS and "Complexity"

We previously posted Dave Hoffman's (Temple) thoughts on potential problems flowing from an increasingly technical turn in much of the ELS literatures. Corey Yung (John Marshall) pushes back a bit here.

While Yung "completely agree(s) with his (Hoffman's) conclusion that empirical legal studies should seek to be more accessible (which I always note at the end of my introduction of my empirical work), I disagree with his contention that empirical legal studies are facing widespread incomprehensibility due to growing complexity." In fact, according to Yung, in some areas (e.g., judicial decisionmaking) current empirical work has "barely scratched the surface." That is, such work needs to become more technical to get at the dynamics with greater accuracy. Anyway, despite both posts nested in the admittedly trendy "Moneyball" and sabremetrics contexts, they engage with an interesting set of issues.

More CSLS Goodness: The 2012-13 Visiting Scholars Program

Once again, the Center for the Study of Law and Society (CSLS) at Berkeley Law is inviting applications for visiting scholars for the 2012-13 academic year.  Details about the Visiting Scholars program and the application process can be found here; the application deadline is November 14, 2011. 

The Center is, to my thinking, the single best spot for the empirical study of law and legal phenomena, and I can think of no better place to spend a year as a visiting scholar.

50 Years of CSLS!

The Center for the Study of Law and Society at UC-Berkeley is celebrating its 50th anniversary this year.  The great folks at the Center have been promoting, supporting, and conducting first-rate empirical work since before many of us were born, and the Center played a key role in (inter alia) the development of the law and society movement, the mainstreaming of empirical work in the legal academy, and the recruitment and training of generations of empirically-oriented legal scholars.

In celebration of this momentus anniversary, the Center is hosting a 50th Anniversary conference on "The Future of Law and Society."  The conference will be held November 3-4, 2011 in the Great Hall of the Bancroft Hotel in Berkeley, and boasts a lineup that is a "who's who" of contemporary law and society scholars. 

The conference is free, but attendees are asked to register at the conference link above. Please join the Center affiliates and the entire ELS community at what is sure to be a stimulating and enjoyable conference.

Americans' Appetite for Litigation: An Empirical Perspective

Despite conventional wisdom surrounding the general notions about Americans' appetite for litigation, surprisingly few approach the question with empirical rigor. Those that do approach the question empirically quickly confront a paucity of helpful data. Eric Rasmusen (IU-Bloomington, Business) and Mark Ramseyer (Harvard) take up the question in Are Americans More Litigious? Some Quantitative Evidence. What they find is that "The data themselves indicate that American law’s notoriety does not result from how we handle routine disputes. Instead, it results from the peculiar and dysfunctional way American courts handle particular legal doctrines like class actions." The authors' recommendations for reform include: "Focus not on litigation as a whole; but on areas of law where America is unique."

Organizational Sentencing Guidelines' Unintended Consequences

While sentencing guidelines has stimulated a fair bit of empirical work, much of it focuses on violent and drug crimes. In The Failure of the Organizational Sentencing Guidelines, Jennifer Arlen (NYU) explores the comparatively underdeveloped topic of sentencing guidelines' efficacy in the white-collar and corporate criminal context. The paper's title gives away Arlen's take. An excerpted abstract follows:

"In the U.S., corporate criminal sanctions for most firms convicted of federal crimes are determined by the Organizational Sentencing Guidelines, which were intended to encourage firms to adopt effective compliance programs, self-report and cooperate through the promise of a reduced fine for firms engaging in this desired conduct, in some circumstances. This Article evaluates whether the mitigation provisions of the Organizational Sentencing Guidelines are effective at achieving this goal. It shows that they are not because the Guidelines provide too little mitigation. Moreover, and perversely, the Guidelines are particularly ineffective when corporate detection and investigation is most important: in cases involving crimes committed by managers of large firms. The Sentencing Commission should reform the Guidelines to better encourage corporate monitoring, reporting and cooperation."

More Musings on "What is Empirical?"

Although I tend to steer clear of such debates, over at Concurring Opinions David Fagundes (Southwestern) discusses the issue. Semantics aside, partly driving David's interesting post is his concern that:

“'Empirical' is not just a neutral term that happens to describe a particular methodology.  It may be understood to connote, rightly or not, a certain degree of rigorousness and exactitude that can set it apart from, and perhaps even above, other methodologies."

While I understand David's point, I'm not sure I buy the premise underneath it. My own view (admittedly, only a datum) is that good scholarship is good scholarship, regardless of the methodological approach. And I am reasonably confident that my take on this is shared by others.

Contested Views of "Tw-Iqbal"

Months ago, the Federal Judicial Center (FJC) released its well-publicized study of Twombly-Iqbal's (or "Tw-Iqbal" for civ pro aficionados) impact on motions practice.  The FJC study concluded that, on balance, the Supreme Court’s new pleadings rules changed relatively little. Responding to the FJC study, Lonny Hoffman's (Houston) paper questions the FJC paper's methods and conclusions. An excerpted abstract (of the Hoffman paper) follows.

"... Three primary assessments are made of the FJC’s study. First, there are reasons to be concerned that the study may be providing an incomplete picture of actual Rule 12(b)(6) activity. Even if the failure to capture all relevant motion activity was a non-biased error, the inclusiveness problem is consequential. Because the study was designed to compare over time the filing and grant rate of Rule 12(b)(6) motions, the size of the effect of the Court’s cases turns on the amount of activity found. Second, even if concerns are set aside that the collected data may be incomplete, it misreads of the FJC’s findings to conclude that the Court’s decisions are having no effect on dismissal practice. The FJC found that after Iqbal, a plaintiff is twice as likely to face a motion to dismiss. This sizeable increase in rate of Rule 12(b)(6) motion activity represents a marked departure from the steady filing rate observed over the last several decades and means, among other consequences, added costs for plaintiffs who have to defend more frequently against these motions. The data regarding orders resolving dismissal motions even more dramatically shows the consequential impacts of the Court’s cases. There were more orders granting dismissal with and without leave to amend, and for every case category examined. Moreover, the data show that after Iqbal it was much more likely that a motion to dismiss would be granted with leave to amend (as compared to being denied) both overall and in the three largest case categories examined (Civil Rights, Financial Instruments and Other). Employment Discrimination, Contract and Torts all show a trend of increasing grant rates. In sum, in every case type studied there was a higher likelihood after Iqbal that a motion to dismiss would be granted. Third, because of inherent limitations in doing empirical work of this nature, the cases may be having effects that the FJC researchers were unable to detect. Comparing how many motions were filed and granted pre-Twombly to post-Iqbal cannot tell us whether the Court’s cases are deterring some claims from being brought, whether they have increased dismissals of complaints on factual sufficiency grounds, or how many meritorious cases have been dismissed as a result of the Court’s stricter pleading filter. Ultimately, perhaps the most important lesson to take away from this last assessment of the FJC’s report is that empirical study cannot resolve all of the policy questions that Twombly and Iqbal raise."

Plaintiffs' Fate on Appeal

My colleague Ted Eisenberg, along with Henry Farber (Princeton), continue their collaborative work exploring the many puzzles incident to appellate civil litigation in the federal courts. Their recent effort, Why Do Plaintiffs Lose Appeals? Biased Trial Courts, Litigious Losers, or Low Trial Win Rates?, takes up why plaintiffs who lose at trial and subsequently appeal are less successful on appeal than are losing defendants who appeal. Ex ante, there is little to explain why such an asymmetry exists between plaintiff and defendants. By matching federal district court trial outcomes and appeals to the circuit courts, the paper develops (and tests) a model that bears not only on appellate outcomes but appellate rates as well. The abstract follows.

"Multiple studies find that plaintiffs who lose at trial and subsequently appeal are less successful on appeal than are losing defendants who appeal. The studies attribute this to a perception by appellate judges that trial courts are biased in favor of plaintiffs. However, at least two alternative explanations exist. First, losing plaintiffs may appeal at higher rates independent of the potential merits. Second, if plaintiffs tend to pursue to trial lawsuits where they should win on the merits less than half the time, then errors at trial will be more likely to adversely affect defendants. This study revisits the analysis of the appellate process with a theoretical model that has implications not only for appellate outcomes but for the rate of appeal. By tying together win rates at trial, appeals rates, and success rates on appeal, the model can distinguish the competing explanations for differential appellate success rates. We estimate this model using matched data on Federal District Court trials and appeals to the U. S. Circuit Courts of Appeal. We provide evidence that the lower plaintiff success rate on appeal is due to plaintiffs' pursuing lawsuits where they should win on the merits (which we define to be an outcome that will not be reversed or remanded on appeal) less than half the time. We also provide evidence against explaining asymmetric success on appeal being attributable to trial courts favoring plaintiffs and evidence against juries being favorable to plaintiffs compared to judges."