Empirical Legal Studies

A History of "p" Levels

While most ELS Blog readers understand traditional significance ("p") levels, few understand how (or why) "0.05" emerged as the standard for statistical significance. In The Adoption of Significance Tests by the Scientific Community: An Empirical Analysis, David A. Gully (Columbia--Engineering) discusses the adoption of the 0.05 standard. A excerpted abstract follows:

"This paper adds to the literature by determining the timing and level of acceptance of common tests of statistical inference. Using the archives of the Royal Society, we examined 574 research studies published between 1926 and 1997, by which point adoption was virtually complete... We detect the presence of several influences on the rate of adoption, which may include prior custom, the nature of empirical research topics being reported, the increasing ease of computer processing, and possibly journal editorial policies. We find that confidence/significance testing has been adopted by a majority of the scientific community for over 50 years; the customary reliance on 95 percent confidence (five percent significance) is upheld by the data; and that confidence intervals and critical significance levels are both widely reported and often together in recent decades."

The ELS "Sandbox" & Boundary Line-Drawing

Brian Leiter recently triggered a mild dust-up (here and here) when he sought to identify loose boundaries for construing legal scholars as "empirical." Lior Strahilevitz (Chicago) exploits the frustration Leiter uncovers and re-directs it into a discussion over at PrawfsBlawg that "might be helpful to people who are skeptical about empiricist hiring in general on law faculties." Specifically, Lior thoughtfully opines about "how to hire entry-level quantitative empiricists with PhDs in disciplines like Political Science or Economics, as well as a coda about what many empiricists should be doing as the 'field' matures." Well worth a read.

Reporting from MPSA

That's the Midwest Political Science Association Annual Meeting, in Chicago.  Without disparaging any of the papers not mentioned, here are a few law- and courts-related papers I found interesting (and that are available on the conference website):

• Representative Justice? Public Opinion and the Criminal Justice System, by Michael Nelson.
• The Influence of Amicus Curiae Briefs on U.S. Supreme Court Opinion Content: Evidence from Federalist Citations, by Evans/Howard/Corley.
• A Mechanism of Influence: Testing U.S. Supreme Court Law Clerk Influence using Amici Briefs, by Chris Kromphardt.
• Explaining The Home Court Advantage, by Bethany Blackstone.

There were also a couple of fascinating panels on measurement and related issues, which had a distinctly "big data" flavor to them (maching learning, topic models, etc.).  Good stuff.

Emerging (and Limited) Data on Law Review Submissions

Many of those in the (seemingly) cyclical throes of law review article submission process (particularly folks on the tenure track) eventually wonder about, among other issues, submission timing. Not quite content to rely on mere anecdote (or "urban legend" passed down from more seasoned colleagues), Shima Baradaran (BYU) is in the early stages of a project that seeks, in part, to hold commonly-held perceptions up to data (albeit limited data involving a single law review--BYU L Rev). Shima's efforts (at this point, essentially an early-stage case study) have already generated multiple blog posts on PrawfsBlawg.

Shima's first post on timing includes the following take-away: "many law reviews give offers out in waves and submitting your article in the first wave or second wave may be advantageous." A second, follow-up post on timing implies that folks seeking to maximize possible attention to their submissions should: avoid submitting on March 1, and, more generally, on Thursdays. While I'm not at all sure what the results might mean in a larger sense, more data involving more law reviews will assist.

The ACLU's Effectiveness as Amicus Curiae

One of my honors advisees this term has just completed her thesis, an empirical examination of the ACLU's ability to influence the U.S. Supreme Court through filing amicus curiae briefs during the Burger and Rehnquist Courts.  It's a careful study, combining qualitative and quantitative methods. The thesis has a number of interesting findings; I won't go into all the details, but I wanted to share this:

It's a plot of the odds ratio for the presence of an ACLU brief (from a series of justice-specific logistic regressions that also included a host of controls) where the outcome variable is a pro-ACLU vote.  The vertical line at 1.0 corresponds to "no marginal association;" values less than one are negative associations, while values above one are positive. (I'm omitting the CIs because they make the plot very messy.)

The interesting thing, of course, is that justices widely perceived as more politically liberal also tend to be those that are most "influenced" by the ACLU's amicus briefs.

Also, to all those NYC firms out there: My advisee, Shannon Azzaro, will be starting law school at Fordham in the fall, with a focus on intellectual property.  I understand that her 2013 summer is still free...

Bias in ABA Judicial Performance Evaluations

UNLV's Rebecca Wood just posted an interesting (and potentially explosive) paper, "Judicial Performance Evaluations as Biased and Invalid Measures: Why the ABA Guidelines are Not Good Enough," on SSRN. The abstract:

Judicial performance evaluations (JPEs) are an important part of the judicial selection process in the states, particularly those using a version of the merit plan. All states that use JPEs follow the ABA’s Guidelines (1985), which claim to minimize the potential for unconscious bias through the use of behavior-based evaluation. But these measures have yet to be subjected to rigorous analysis. This analysis of the “Judging the Judges” survey of Nevada attorneys provides such an analysis. After controlling for objective measures of judicial performance, gender and race still contribute significantly to the scores on all of the behavior-based measures implemented in the Nevada poll. I find evidence of significant unconscious bias, as social cognition theory would predict. The analysis also cast serious doubt on the overall validity of these measures of judicial quality. This result raises serious questions about the validity and fairness of JPEs around the country.