Empirical Legal Studies

Law Blogs & the "Juice"

Not at all quite sure what to make of it, but Paul Caron touts a law blog "juice rating" here.  I note it only because our ELS blog made the list (at 26th).  Our ranking (such as it is) is all the more notable in that we're a relatively new blog, dwell on the substantive (save for this post), and focus on the discrete--though palpably growing--empirical legal studies field.

Hierarchy of Legal Scholarship

J.B. Ruhl has listed his hierarchy of legal scholarship, ranking types of articles from 0 to 10.  His ranking is below.  (Our readers will want to note #9 and #10.) 

0 – Blog posts
1 – Publication of what are essentially blog posts with footnotes
2 – Doctrinal review of the state of the law
3 – Doctrinal study of interesting questions of law
4 – Doctrinal synthesis of developments in law
5 – Normative policy analysis of law
6 – Normative policy analysis of law with substantial reform proposals
7 – Legal theory
8 – “Law and” interdisciplinary studies
9 – Empirical study of legal institutions
10 – Empirical study of law’s impact on society

Hat tip: Solove at Concurring Opinions

Give the People What They Want...

Do you want to increase your SSRN downloads? If so, you should consider publishing a law school ranking. (If not, congratulations, you are far healthier than most of your colleagues. Of course, it does prompt one to wonder why you’re reading this post.)

In April 2005, I participated in the Indiana Law Journal Symposium on “The Next Generation of Law School Rankings” organized by Paul Caron and Rafael Gely of Cincinnati and ELS blog’s own Bill Henderson and Jeff Stake of Indiana. Invited to discuss ranking methodologies, I opted to create my own ranking. For me, the choice of focus was obvious: Empirical Legal Scholarship (ELS). Why? An ELS Ranking identifies schools leading in this burgeoning field and perhaps incentivizes others to invest in it.

The paper ranked each of the top 41 schools in US News based on an ELS-score that I constructed. The score included three variables: professors with social science doctorates, professors holding secondary appointments in social science departments, and articles in ELS-oriented publications. First posted on SSRN on August 9, 2005, the paper has been viewed more than 4,000 times and downloaded more than 860 times. (By contrast, my paper with Jeff Berger on judicial entrepreneurs posted at the same time has been downloaded only a fraction of that number – help here!) UC-Berkeley and George Mason each boasted on its webpage that it tied for first. Legal Affairs and various law blogs all provided short discussions of the piece. Obviously, law professors and their groupies love law school rankings.

I have revised, updated, and expanded the ranking and will post the new results over each of the next four days. The ranking now includes the top-50 schools in US News and looks at faculty as of 2006-2007 (as opposed to 2003-2004 in the original). This week, I will describe each measure and the new results, focusing on educational background and training on Tuesday, secondary appointments on Wednesday, and empirical research output (the only measure that has been completely changed) on Thursday. And, the 2006 ELS Ranking will be available on Friday.

My purpose here is not simply to navel gaze but also to seek input on the measures. Operationalization is hard. And the power of any empirical results necessarily turns on the soundness of those measures. I invite comments on the value of rankings (particularly given the substantial research costs including time away from other work) as well as thoughts on the components of the ELS ranking. Thanks to Jason Czarnezki for inviting me to be a guest blogger and to Geoff Turvey for exceptional research assistance.

Where is the Moneyball at Moneylaw?

Over at Moneylaw, a new blog on "the art of winning an unfair academic game" (a reference to the subtitle of Moneyball), Jim Chen asserts that the legal academy overvalues pedigree and undervalues performance, thus thwarting the emergence of a true academic meritocracy.  Jim then writes:

For my part, I offer my own tentative list of core principles for evaluating academic talent, whether on an individual or an institutional basis:

• Pedigree never matters.  Performance always does.
• In gauging performance, you can measure anything you want except other professors' opinions.
• In other words, no academic ratings system is valid if it depends in whole or in part on a subjective survey of academic reputation.

What, exactly, is Jim's objective here?  To structure a more theoretically coherent system of merit that will crush all existing ranking systems by sheer force of reason?  And surely, performance cannot be "anything you want" if the point is to "win an unfair academic game"--you need some criteria to know your making progress (or "winning"). 

In baseball, there are balls and strikes, hits and outs, and ultimately wins and losses.  But in the legal academy, it is much harder to operationalize useful constructs of performance.  Assuming we throw out reputation, as Jim suggests, and focus on productivity (certainly under the rubric of "performance"), lots of measures seem relevant:  articles produced, citations by courts, citations by scholars, SSRN downloads, journal placement (using, say, the W&L impact measures).  And if we can agree that some or all of these are relevant in this "game" we are engaged in, then how shall they be weighted?  Surely, the folly of this enterprise is self-evident.

It is rare that a junior scholar has the opportunity to utter these words to a scholar of Jim Chen's stature, but here it goes:  Jim, you need a theory. 

For a coherent theory on how Moneyball principles can be fruitfully applied to the law school world, and for some interesting data, you are going to have to read past this jump.

Berman on Blogs ... and an Interesting Data Point

This post by Peter Lattman at the WSJ Law Blog caught my eye.  It is a short review of  Doug Berman's (OSU Law) "Scholarship in Action: The Power, Possibilities, and Pitfalls for Law Professor Blogs,"  which was originally presented  as part of the symposium on legal blogging sponsored by the Berkman Center for Internet & Society at Harvard Law School.   

Berman, who edits the Sentencing Law & Policy Blog, does not need to be remotely defensive on the relevance of blogging to scholarship.   His blog, which has garnered over 20 citations in judicial decisions (as of April 2006), has been endorsed by the President of the Federal Judges Associations and cited in testimony before the Senate Judiciary Committee (see this WSJ online story).  Berman's essay is a thoughtful reflection on the purposes of legal scholarship--which, Berman observes,  has become remarkably undertheorized due to a process of institutional drift over the last few decades.    Berman believes that blogging is a medium that provides large synergies between the three traditional dimensions of a law professor's job: teaching, scholarship, and service.  Thus, Berman concludes that blogging is destine to have an impact and may someday be "seen--along with 'serious legal scholarship'--as a preferred form of law professor activity."  I agree.

The reason that the Berman post caught my eye (and why I downloaded the paper) was a reference to statistics on the average length and number of footnotes in the Harvard Law Review in 1950, 1975, and 2000.  The following chart, from page 3 of Berman's paper, speaks for itself [click on chart to enlarge].

Sander, Affirmative Action, and a Manhattan Project for Legal Education

In a provocative post over at Truth on the Market, Josh Wright asks, “where do we stand on the empirics of affirmative action?”

The catalyst for Josh’s post is the ongoing fallout from Richard Sander’s two articles on articles on affirmative action: A Systemic Analysis of Affirmative Action in American Law Schools, 57 Stan. L. Rev. 367 (2004), which made an empirically based argument that racial preferences set in motion a pattern of "mismatched" credentials, hurting blacks in several ways; and The Racial Paradox of the Corporate Law Firm, 84 N.C. L. Rev. 1755 (2006), which suggests that the large attrition rates of young black associates at prestigious corporate law firms may be attributable to the reluctance of white partners to give important and challenging work to black lawyers, presumably because these young attorneys tend to have lower grades than their white counterparts. (Virtually all of Sander’s work in this area, including responses from critics, can be found here.)

As Josh notes, Paul Butler (GWU Law) recently made the claim that Sander’s work had been “persuasively debunked” by Devon Carbado (UCLA Law), among others. In fact, the relevant post by Carbado does not directly refute any of Sander’s arguments; rather, Carbado asks the broader question of what type of empirical evidence, beyond many of the bleak statistics cited by Sander, would be sufficient to establish that blacks have been hurt by racial preferences. That is certainly an important issue.

Josh Wright is skeptical of Carbado’s statement that it is hard to have an empirical discussion on a blog. After a worthwhile, cautious summary of the debate surrounding Sander’s work, Josh suggested that the ELS bloggers might want to weigh in on the issue—myself in particular because I participated in a Legal Affairs Debate Club forum with Sander in January of 2005.

Obviously, this is an academic minefield, especially for the untenured. Am I dumb enough to take the bait?

Should Law Students Take a Stats Course?

See one answer here.

The Bar Exam as a measure of "competency" and "achievement"

Over at Concurring Opinions, Dan Solove discusses his recent satirical essay on the Multistate Bar Examination.  Dan’s other blog posts make clear that he favors the abolition of the bar exam. “In lieu of the Bar,” Dan writes, “states should permit all students who graduate from an accredited law school to become members of the Bar after working a certain number of supervised pro bono hours.”

I am somewhat skeptical of this proposal because of the agency problems that plague legal education. What law professors find interesting (and thus worth teaching) may not be a good preparation for a career as a practicing lawyer. The bar exam is at least a reference point for core substantive knowledge.  If we think it is a poor gatekeeping device, then the solution is professional engagement with state bar examiners … But most legal academics are rewarded for scholarship rather than teaching or service to the profession.  So we are back to incentives and agency problems; we complain but nothing changes. Sustained complaining is what is required, and the opportunity cost is quite high.

That said, the bar exam itself rests on some rather extravagant theoretical and empirical claims. For example, in a 1976 monograph entitled “The Relationships among Law School Predictors, Law School Performance, and Bar Examination Results,” the authors (two psychometricians from ETS) described the bar examination as “the ultimate test of competency” (p. 1). The authors also observed, “The bar examination measures, both essay and MBE, are in effect achievement tests of some the skills and knowledge developed, and tested, in law school” (p. 5, emphasis added). 

Really? Few law professors (or practicing attorneys, for that matter) would sign off on these statements.   Perhaps this statement is more accurate:  "The bar exam and the MBE are flawed, imprecise measures of competency and achievement; but they have persisted for decades with little or no change because of institutional incentives among legal educators."   This situation can be fixed, but who is going to bear the cost?
 

The Law Review Market

Over at Concurring Opinions Dave Hoffman discusses Nominally Empirical Evidence of Unraveling in the Law Review Market.

Case Studies & Empirical Scholarship

For a discussion of Case Studies and Empirical Scholarship, see here.