Empirical Legal Studies

The Empirics of Twombly and Iqbal

A paper now circulating in SSRN (and forthcoming in Judicature) explores the impact of Twombly and Iqbal on dismissal rates and does so with an interesting methodological twist. A New Look: Dismissal Rates in Federal Civil Cases, by Scott Dodson (Hastings), contributes to the growing empirical literature by coding at the individual claim--rather than the case--level. The abstract follows.

"In the wake of Twombly and Iqbal, a number of studies have been conducted to determine the decisions' effects on dismissal practice in federal civil cases. However, those studies have tended to code whole cases rather than claims -- leading to the ambiguous coding category of “mixed” dismissals and to problems in characterizing the nature of the dispute -- and have failed to distinguish between legal sufficiency and factual sufficiency, potentially masking important detail about the effects of the pleadings changes.

This paper begins to fill in that detail. I compiled an original dataset of district court opinions and coded each claim -- rather than whole case -- subject to an adjudicated Rule 12(b)(6) motion. For each claim, I also determined whether the court resolved the motion on grounds of legal or factual sufficiency. This methodology opened an unprecedented level of granularity in the data.

The data reveal statistically significant increases in the dismissal rate overall and in a number of subsets of claims. I also find an increase in the relative prevalence and efficacy of factual-insufficiency arguments for dismissal. Perhaps surprisingly, I find a decrease in the relative prevalence and efficacy of legal-insufficiency arguments for dismissal. These data and insights on the rationales of dismissals are new to the literature and suggest that Twombly and Iqbal are affecting both movant strategy and judicial reasoning."

Ranking Journal Rankings

Cornell colleagues, Ted Eisenberg and Marty Wells, empirically analyze leading ranking metrics for refereed law journals in their recent paper, Ranking Law Journals and the Limits of Journal Citation Reports. Their analysis of ranking outcomes emphasizes a pre-occupation with ordinal ranking and database bias. The abstract follows:

"Rankings of schools, scholars, and journals emphasize ordinal rank. Journal rankings published by Journal Citation Reports (JCR) are widely used to assess research quality, which influences important decisions by academic departments, universities, and countries. We study refereed law journal rankings by JCR, Washington and Lee Law Library (W&L), and the Australian Research Council (ARC). Both JCR’s and W&L’s multiple measures of journals can be represented by a single latent factor. Yet JCR’s rankings are uncorrelated with W&L’s. The differences appear to be attributable to underrepresentation of law journals in JCR’s database. We illustrate the effects of database bias on rankings through case studies of three elite journals, the Journal of Law & Economics, Supreme Court Review, and the American Law & Economics Review. Cluster analysis is a supplement to ordinal ranking and we report the results of a cluster analysis of law journals. The ARC does organize journals into four large groups and provides generally reasonable rankings of journals. But anomalies exist that could be avoided by checking the ARC groups against citation-based measures. Entities that rank should use their data to provide meaningful clusters rather than providing only ordinal ranks."

Description

Over at Concurring Opinions, Dave Hoffman's (Temple) interesting post, When is it OK to be 'descriptive'?, has ignited interesting commentary and responses.

CELS 2012

Dan Ho (Stanford) has put together an outstanding Program for this year's conference. Event's begin later this morning with Ted Eisenberg's (Cornell) 10-hour empirical training workshop. Insofar as the Workshop proved quite popular (registration closed quickly due to participant demand), workshops will be planned for future CELS. In addition, Ted is presenting a similar workshop at AALS in January.

Graphically Displaying Uncertainty

A nice pair of posts from Andrew Gelman's (Columbia--Poli Sci, Statistics) blog (here) illustrates and briefly discusses the virtues of visually conveying uncertainty in regression results. The initial post illustrates two competing visual approaches. A follow-up post includes (or links to) helpful coding.

Empirical Training Workshop at CELS

The Society for Empirical Legal Studies Executive Director, Dawn Chutkow, passed along the following that might interest those planning to attend the CELS at Stanford in November.

Ted Eisenberg (Cornell) will conduct an Empirical Training Workshop on November 8 - 9, in connection with CELS 2012 at Stanford Law School. Enrollment is limited. The workshop begins the day before the conference. A brief description and a link for more information follow.

The Empirical Training Workshop is intended for professors and students who seek an introduction to the statistical and programming skills needed to conduct quantitative empirical legal research. Professor Eisenberg will guide participants through an intensive 10-hour course on statistical analysis in the legal context. Pre-registration and a small fee are required.

More information on the Workshop is found here.

More on "More Guns, Less Crime"

In The Impact of Right to Carry Laws and the NRC Report: The Latest Lessons for the Empirical Evaluation of Law and Policy, John Donohue (Stanford), Abhay Aneja (Stanford), and Alexandria Zhang (Johns Hopkins), revisit and synthesize empirical evidence about the policy efficacy of controversial "right-to-carry" laws. Also notable is that the paper helpfully addresses important methodological concerns incident to panel data methods. Insofar as a contested history frames the empirical "right-to-carry" debate, some brief background is warranted, followed by an excerpted abstract.

In 2005, the National Research Council (NRC) evaluated the “More Guns, Less Crime” hypothesis using county-level crime data for the period 1977-2000. 17 of the 18 NRC panel members essentially concluded that the existing research was inconclusive on whether "right-to-carry" laws increased or decreased crime.

"We evaluate the NRC evidence, and improve and expand on the report’s county data analysis by analyzing an additional six years of county data as well as state panel data for the period 1977-2006. We also present evidence using both a more plausible version of the Lott and Mustard specification, as well as our own preferred specification (which, unlike the Lott and Mustard model used in the NRC report, does control for rates of incarceration and police). While we have considerable sympathy with the NRC’s majority view about the difficulty of drawing conclusions from simple panel data models, we disagree with the NRC report’s judgment that cluster adjustments to correct for serial correlation are not needed. Our randomization tests show that without such adjustments the Type 1 error soars to 44-75 percent. In addition, the conclusion of the dissenting panel member that RTC laws reduce murder has no statistical support.

Our paper highlights some important questions to consider when using panel data methods to resolve questions of law and policy effectiveness. Although we agree with the NRC’s cautious conclusion regarding the effects of RTC laws, we buttress this conclusion by showing how sensitive the estimated impact of RTC laws is to different data periods, the use of state versus county data, particular specifications, and the decision to control for state trends. Overall, the most consistent, albeit not uniform, finding to emerge from both the state and county panel data models conducted over the entire 1977-2006 period with and without state trends and using three different specifications is that aggravated assault rises when RTC laws are adopted."

Always More Room on the ELS "Bandwagon"

Interesting post (and related discussion) over at PrawfsBlawg involving Martin Pritikin's (Whittier) initial "jump" onto the ELS "bandwagon" and into ELS scholarship. Notable to me is Martin's willingness to undertake the "heroic" task of data creation (rather than secondary analysis of existing data sets). It is also hearting to hear reports from participants about the efficacy of the growing number of empirical legal studies workshops. Finally, the comments' helpfulness evidences how the ELS community stands ready to assist those willing to engage.

Stata Users' Favorite Commands

The folks over at The Stata Blog recently polled readers (obviously, a non-random selection of Stata users) on their favorite Stata command. While some may find the results (here) themselves interesting, others might find unfamiliar commands that could prove useful.

Double-Blind Research Design & Expensive Violins

While not particularly legal per se, results from a test of professional violinists' ability to identify music from a Stradivarius as opposed to other and newer, expensive violins, originally published in The Strad (Feb. 2007), were featured in a recent NPR segment (here). (The NPR segment includes two audio clips for anyone interested and desiring to test their own musical acumen.)

Notably (and certainly within the ELS Blog sweet-spot), the researchers employed a double-blind test. "Researchers gathered professional violinists in a hotel room in Indianapolis. They had six violins — two Strads, a Guarneri and three modern instruments. Everybody wore dark goggles so they couldn't see which violin was which." Ironically, "the only statistically obvious trend in the choices was that one of the Stradivarius violins was the least favorite, and one of the modern instruments was slightly favored."