Empirical Legal Studies

Empirics, Gender & ACL Injuries

Prompted by a relatively recent NYT Sunday Magazine cover story, Paul Cassell (Utah) ignited an interesting thread (and comments) over at the Volokh Conspiracy.  Presumably, a careful longitudinal study is surely underway, though the research design is tricky.

More accurate than the polls? A Model for Predicting IN and NC Primary Results

Wondering, like me, whether the polls in the Democratic primary can be trusted?  Over at the Social Science Statistics Blog, statistics whiz Kevin Bartz has posted two very interesting models for predicting results of the Indiana and North Carolina primary.   Bartz mines a huge amount of county-level data, including likely voter turnout, and his models purport to explain between 84% and 99% of prior results from Ohio, Pennsylvania, and Virginia.  (Because his S.E. is 5%, Bartz qualifies that the "results could be shaky at the county level.")

Here are Kevin's projections:

• Indiana: Clinton 53.5%, Obama 46.5%; turnout 950,000
• North Carolina: Obama 58%, Clinton 42%; turnout 1,200,000

16% seems like a huge margin for Obama--indeed, if the above numbers are even close to right, Obama (who I had for Con Law in the fall of 2000) will likely add to his lead in pledged delegates.  If so, then perhaps we can wind up a protracted primary season that is no longer serving the interests of the Democratic Party (granted, that is just my opinion).  Kevin explains the differences between his models and the polls:

How do my results stack up against the current polls? In Indiana, the RealClearPolitics average has Clinton +6%, only a point from my prediction. In North Carolina, the RCP average has Obama +8%, significantly below my predicted 16% victory. Two factors shed light on this discrepancy:

• In neighboring South Carolina, the polling average had Obama +11.6% and he won by 28.9%.
• In neighboring Virginia, the polling average had Obama +17.7% and he won by 28.2%.
• So perhaps my analysis isn't so crazy putting Obama above what the polls say in NC.

Here is what Kevin's predictions look like using GIS mapping:

Large Law Firm Hiring -- Introducing the "Funnel Cloud"

Why do law students follow U.S. News ranking?  In a story in last week's National Law Journal, Andrew Morriss and I claim that USNWR provides a rough guide to future employment prospects.   Yet, drawing upon data on hiring patterns from the NLJ 250 (the 250 largest firms based on number of attorneys) and employment outcomes from the ABA-LSAC Official Guide, Andy and I demonstrate the serious financial perils of overreliance on USNWR.

The story has a lot of data.  The chart below, however, ended up on the editing room floor.  I call it the "funnel cloud."  It shows the percentage of students at each law school that were hired by a NLJ 250 firm.  [Side note: Why is the NLJ 250 a good barometer?  Because the starting salaries make law school tuition look like a bargain.  Based on data made available to us through the Law Firms Working Group,  the weighted average (based on firm size) starting salary at an NLJ250 firm for 2007 was $152,424.]

In my opinion, the above chart has at least two takeaways: (1) the funnel cloud formation shows large law firm employment payoffs are non-linear and that the vast majority of schools offer similarly modest, but not insignificant, entree to this sector; (2) based on the volume of green and red at the top of the chart, most large firms prefer to recruit deeper into the class at a Top 20 school (and will pay a price premium of $160,000 per year) rather than shifting their model to lower ranked schoolss.    53% of all new NLJ250 jobs between 2005 and 2007 went to Top 20 biggest feeder schools from 2005.  See also Leigh Jones, Hiring More Deeply into Top Schools, Nat'l L. J., Apr. 14, 2007.  You would think that by 2007, the Top 20 would have been tapped out.  Not so.

The empirical findings of this 2007 article by Dinovitzer and Garth, which is based on After the JD data, suggest that this elitism has unintended negative consequences in terms of high attrition rates. 

Here are links to other graphics in our recent NLJ story:

Employment trends for law school grads: The top 100 out of 194 ABA-approved schools ranked by percentage of 2005 graduates at NLJ 250 firms.

By Region (defined by states containing Top 10 legal markets):

• New York
• California
• Illinois
• Texas
• Washington, D.C.
• Massachusetts
• Georgia
• Pennsylvania
• Alabama to Kentucky
• Louisiana to Nevada
• Ohio to Wyoming
• Composite report of all 11 regions
• Methodology

ELS, Senate Confirmation Hearings & the NYT Op-Ed Page

Today's NYT includes an op-ed commenting on an empirical piece, co-authored by Lori Ringhand (Kentucky--soon-to-be Georgia) along with our very own ELS blog editors Jason Czarnezki and William Ford, forthcoming in Constitutional Commentary. The authors report that:

"Our results indicate that confirmation hearings statements about a nominee's preferred interpretive methodologies provide very little information about future judicial behavior. Inquiries into specific issue areas - such as the rights of criminal defendants - may be slightly more informative."

The Changing Economics of the Legal Profession

Last year, Amir Efrati of the Wall Street Journal authored a widely read story on the stark and rapidly shifting economics of the legal profession.   See "Hard Case: Job Market Wanes for U.S. Lawyers" (Sept 27, 2007).  In a story that ran this Sunday, Greg Burns, Senior Correspondent for the Chicago Tribune, may have turned this journalistic theme into a full fledged genre.  See "Two Lawyers Walk into a Bar ..." (Apr. 6,2008). 

Burns consulted with the best empirical literature on the legal profession, including the Chicago Lawyers I and II studies, and in turn brings the underlying realities to life by weaving interesting personal stories with remarkably clear explanations on why the market is changing.   I would bet that, in hindsight,  these two stories will be viewed as pivotal journalism that changed public opinion about the career benefits of law school.  Those of us who work in this sector need to take notice. 

More On Law School "Class Warfare": Tamanaha & Interdisciplinarity

Brian Tamanaha's provocative post on why an aggressive interdisciplinarity may not be such a good idea at non-elite (his words) law schools, noted below in Chris Zorn's post, continues to stimulate interesting commentary across many blogs. For a sampling, click here, here, here, and here.

Update:  More discussion here (Leiter) and here (Solum).

Executions in Texas - JELS in the NYT

Today's New York Times looks at the death penalty in Texas and relies, in part, on an article in the Journal of Empirical Legal Studies by Ted Eisenberg, John Blume, and Martin Wells.

Indeed, according to a 2004 study by three professors of law and statistics at Cornell published in The Journal of Empirical Legal Studies, Texas prosecutors and juries were no more apt to seek and impose death sentences than those in the rest of the country.

“Texas’ reputation as a death-prone state should rest on its many murders and on its willingness to execute death-sentenced inmates,” the authors of the study, Theodore Eisenberg, John H. Blume and Martin T. Wells, wrote. “It should not rest on the false belief that Texas has a high rate of sentencing convicted murderers to death.”

The NYT article is here.

CEI Report on Entertainment Ratings

The Competitive Enterprise Institute released a report this week entitled, “Politically Determined Entertainment Ratings and How to Avoid Them” by Cord Blomquist and Eli Lehrer. The report examines the ratings systems for movies, comic books, music, television, radio, and video games. Its most basic conclusion is one I agree with: we don’t need the government running ratings systems for entertainment, whether for video games or some other medium. Their more specific conclusions are somewhat problematic. They conclude that “the best ratings systems have evolved in response to market forces.” The report doesn’t offer much support for this conclusion. In part, there is a shortage of examples of ratings systems that haven’t been influenced by the political process and/or threats of government regulation. As best I can tell from the report, satellite radio is the one complete exception where the government played no role in the development of a ratings system. B&L at 18. Music ratings are only a partial exception, since according to the report, several politicians’ wives prompted the creation of music labels. B&L at 13. Blomquist and Lehrer also conclude that as compared to other ratings systems, the one for video games is “the least government-influenced of the lot.” B&L at 24. The report actually suggests otherwise. I'll offer some thoughts on the weaknesses of what is an otherwise interesting report.

To start, compare the comic book and video game rating systems, i.e., the Comics Code and the Entertainment Software Ratings Board (ESRB) (the two systems I know best). Blomquist and Lehrer do not mention the 1948 Comics Code and the associated controversy that led to it. Instead, they start in 1954, claiming that “[t]he battle over comic book content began with Frederic Wertham’s 1954 book Seduction of the Innocent.” B&L at 9. Cf. Amy Kiste Nyberg, Seal of Approval: The History of the Comics Code 23-44 (University of Mississippi 1998) (discussing the controversy over comic book content in the 1940s). Blomquist and Lehrer describe the 1954 Comics Code, which was later modified in 1971 and 1989, as a failure. B&L at 2, 11. But they describe the ESRB’s system as one of the best ratings systems—perhaps the best ratings system. B&L at 1, 24. As mentioned above, the report concludes that government influence impairs the development of rating systems. One was bad (the Comics Code). One was good (the ESRB). To find out why, we need to be clear about how the two systems developed and the level of government involvement in both. There are, however, problems with the authors’ account of the history of these ratings systems.

Getting Data: My Experience with the California Bar

Today I am in California for a hearing before the Board of Governors, which is the elected body that oversees the state bar exam.   For the past 18 months, I have been part of a research team (with Richard Sander, Vik Amar, Doug Williams, and Stephen Klein) seeking California data to test a possible mismatch effect in law schools.  The substance of the hearing today (see agenda here) was to determine whether we would be permitted to obtain regression results, via the Bar's own longtime psychometrician, from the Bar's rich archive of historical data.

After hearing approximately 50 minutes of  prepared testimony from those for and against our study, the Board voted to deny our data request.  Based on the tenor of the meeting, which was set by a well-organized opposition led by Cheryl Harris (UCLA Law) and Michele Landis Dauber (Stanford Law), the vote turned on legal uncertainties surrounding candidate confidentiality and consent. 

Acceding to these concerns has, in my opinion, three major flaws: (a) 0ur research team would only have access to regression tables with results aggregated among groups of schools, so no individually identifiable information would ever be released to researchers or the public;  (b) we would, at all times, be subject to university IRB protocols; and (c) the broad construction given to consent in this context suggests that much of the research sponsored by the California Bar over the last 20 years has been unlawful; moreover, this decision casts doubt on the scope of any future research by the California Bar. 

[Note: Richard Lempert (Michigan Law and formerly chair of the NSF Law & Social Science Program), wrote this memo on the IRB issue, which he graciously forwarded to our research team in advance of the hearing. Rick's assessment, however, is quite different from my own experience in obtaining IRB approval at three different institutions for projects which raised, in my opinion, much more significant human subjects issues.  Regardless, no research would occur without IRB approval; that should be obvious to all parties involved.]

Of course, the Board of Governors is not an adjudicative body -- rather, in this case, it is the regulatory unit that would be dragged into court if it approved our proposal.  Thus, on one level, I can understand the institutional reasons for taking a cautious approach.  (For a snapshot of the public debate leading up to this decision, see this National Law Journal op-ed by Cheryl Harris and Walter Allen, and our NLJ response; in the spirit of transparency, all documents related to our projects are posted on this website.)

Although I am disappointed with the outcome, I am proud of the testimony we put on today.  The basic theory underlying the mismatch effect is that recipients of large preferences (regardless of race) tend to learn less because the classroom instruction is pitched toward a modal student with significantly higher entering credentials.  Honest researchers can argue over the methodology and inferences of Sander's Systemic Analysis study.  But existing data provide little comfort that our current system is working well.  Minority students are disproportionately clustered at the bottom of their class.  Since LGPA is the single best predictor of bar passage, it is not surprising that these same students struggle on the bar examination. Sander's first choice/second choice analysis (in his 2005 Stan. L. Rev. response essay) suggests that law school performance and bar passage prospects actually increase when a student opts for a less elite law school. In my opinion, if this theory is wrong, it is imperative that it be debunked empirically rather than through efforts to withhold data.

As a lifelong Democrat and ardent supporter of racial diversity, I don't need to apologize for raising these issues.  As a said in my prepared remarks, it does not follow that evidence of a mismatch effect requires the dismantling of racial preferences -- indeed, in my opinion, this would be a very bad idea. 

But if we fail to diagnosis factors that contribute to low minority bar passage, we have no basis to formulate effective policy or educational strategies.  Regardless of which way the data cut, our study would have guaranteed one of two favorable outcomes:

1. Using the more refined California Bar exam (i.e., a continuous dependent variable rather than pass/fail and schools put in analytically useful clusters, unlike the LSAC-BPS data), the mismatch theory would have little or no empirical support.  So a contentious academic theory would be put to bed, at least in the law school context.
2. A mismatch effect would be supported, which would pressure law schools to take concrete steps to help current or prospective students.  These might include: (a) disclosures that reveal bar passage prospects for past students with similar entering credentials; (b) creation of rigorous academic support programs (such as this one) that increase the bar passage for students in the bottom 1/2 of the class; and (c) identify curricular and teaching strategies that produce higher bar exam scores.

As legal educators, we should want more than the status quo.

WSJ Article on Legal Job Market, Transfer Students, and Alternative Law School Rankings

Wall Street Journal reporter Amir Efrati continues to make a name for himself by aggressively covering the legal education beat.  Today's WSJ page B1 story (How to Cut Debt, Boost Job Prospects from Law School) lays out some the calculus that prospective law students  should consider in making the decision to enroll in law school.  It also includes reference to the benefits of transferring to a national law school after a successful 1L year at a lower ranked institution; in my last post, I learned that this is a very sensitive topic.

The story contains a quote from me and a reference to an alternative law school ranking that I am working on with Andy Morriss (Illinois Law).   I would like to add two key pieces of context:

• Chances of landing a large law firm job.  In the story, I am quoted, "If you go to a school ranked 35th or 40th over one that's ranked 70th or 80th, you are by no means substantially increasing your chances of landing a high-paying job."  This is more than just my opinion.  When I made this statement, I was looking a data that took months to compile.  Further, the same analysis holds true when you compare, say, the 25th or 30th with the 50th or 60th.   My point is simple: a higher rank in US News does not necessarily provide students with an opportunity set that justifies a price premium.  That will be the central takeaway of our alternative ranking--from a student perspective, over-reliance on US News can be a big financial mistake.

• Alternative rankings.  The story notes that our alternative rankings will include large law firm employment.  Our project, however,  is actually much broader than that.  If a student wants to preserve maximum financial flexibility upon graduation in order to pursue a variety of interesting legal jobs (government, public interest, small firm practice) with little or no law school debt, we will show students how to do this calculus using actual law school data.

As I have said before, the future of legal education is driven by labor markets.