Empirical Legal Studies

Data on Legal Market and Law School Enrollment Patterns

Over at Balkinization, Brian Tamanaha (Wash U) assesses trends involving the market demand for new attorneys and law school enrollment (here). As Brian notes, "law schools now pump out about 45,000 graduates annually at a time when the Bureau of Labor Statistics projects about 28,000 new lawyer positions per year." Why? Brian argues that the current law school business model demands as much. Specifically, law faculties have gotten bigger during the past two decades and, as a result, "bigger faculties must be paid for through some combination of more bodies (J.D. and LL.M) and higher tuition." Not surprisingly, Brian predicts "tough times" for many law schools in the future.

UPDATE: Ted Seto's (Loyola-LA) alternative interpretation of the data also warrants note.

Sociology, Social Science & The Wal-Mart Litigation

Whether the American Sociological Association (ASA) "missed an opportunity to advance sound scientific methods and opinions in litigation when it failed to scrutinize the content of, and method behind, Dr. Bielby’s case-specific claims in the Wal-Mart case" is a contested proposition. Rather than pick a side, I instead urge interested readers to assess the two recent and competing papers. Gregory Mitchell (UVa) et al.'s paper criticizing the ASA is found here; Laura Beth Nielson (Northwestern) et al.'s rebuttal paper here. Comments welcome.

A String of Hires at USC Law

Although the ELS Blog typically refrains from noting individual personnel moves, a recent string of hires by Dean Bob Rasmussen at USC Law warrants note insofar as it involves leading ELS scholars. USC just announced hiring Lee Epstein and Nancy Staudt away from Northwestern (here and here, respectively). Epstein's hire is especially interesting as it complements USC Law's earlier hire of Mathew McCubbins, a leading positive political theorist. As Brian Leiter notes, these hires substantially bolster USC's presence in the law and politics field.

Northwestern Law

After initially hesitating to link to this post on Brian Leiter's blog owing to its "inside-baseball" character, now that it has become almost "viral" (within the law school world, anyway) I've decided that it would be almost odd to ignore it as the intersections with empirical legal scholarship, while not direct, are nonetheless obvious. Suffice it to say, Prof. Ron Allen's reflections on former NU Law Dean, David Van Zandt, are, if nothing else, interesting and speak for themselves.

UPDATE: Jay Koehler (NU Law) contributes perspective and facts to the discussion ignited by Ron Allen's reflections.

Online Symposium Over at Concurring Opinions Blog

Today is the first day of a planned two-day online symposium over at Concurring Opinions on the recent paper, What Difference Representation. Kudos to Dave Hoffman (Temple) for organizing the event (Dave's description of the symposium is here) that has already generated substantial commentary.

Data on the "Employed at Graduation" U.S. News Rankings Input

By Bill Henderson & Andrew Morriss

Yesterday, Bob Morse of U.S. News published a blog post in which he signaled a change in the law school rankings methodology, specifically with regard to employment.  The prevailing view on the law school administrator list-serves (which nearly a dozen people have forwarded to us) is that U.S. News will be increasing the weighting of "employed at graduation," presumably because U.S. News Editor Brian Kelly sent a letter to law school deans--reprinted in Bob's blog post--discussing the importance of employed-at-graduation as a metric.

We have zero inside information, but we are willing to bet a substantial sum that any methodology change will be in a completely different direction.  Here is why.  Over the last decade, fewer and fewer schools have been supplying U.S. News with employed-at-graduation data.  Employment at graduation is not a statistic required or collected by the ABA; as such, its accuracy cannot be checked through cross-reference to the annual ABA-LSAC Official Guide. 

But much more significantly, when a school fails to provide this data, U.S. News has--up until now--imputed the figure based on employment at 9 months.  (Kudos to Ted Seto for unraveling this mystery.  See Understanding the U.S. News Law School Rankings.)  Crudely speaking, the magazine applied a roughly 30% discount rate on the employed at 9 months figure.   Earlier this year, Paul Caron suggested that if a school's employed-at-graduation rate is more than 30% lower than its employed at 9 month rate, it is "rankings malpractice" to supply U.S. News with the data.

As readers can see from the above chart (generated by Paul Caron in his rankings malpractice post), a large proportion of law schools have figured out the payoffs.  Over the last decade, the percentage of non-reporting schools has skyrocketed. With this information in mind, Bob Morse's blog post may seem less cryptic:

In an effort to make our law school employment data more reflective of the current state of legal employment, U.S. News has modified how we calculate the employment rates that are used in the new law school rankings.  ...

U.S. News agrees with the efforts of Law School Transparency to improve employment information from law schools and make the data more widely available.

If the goal is (a) to utilize data that better reflect reality, and (b) provide greater transparency and access to such data, it makes no sense to increase the weight of an input (employed-at-graduation) that is either withheld by law schools or is heavily gamed.  This latter point is made by U.S. News Editor Brian Kelly in his letter to the deans:

[E]mployment after graduation is relevant data that prospective students and other consumers should be entitled to.  Many graduate business schools are meticulous about collecting such data, even having it audited. The entire law school sector is perceived to be less than candid because it does not pursue a similar, disciplined approach to data collection and reporting.

At U.S. News, we work to make meaningful and fair comparisons, based on industry-accepted data. ...

To eliminate some of the gaming that seems to be taking place, we have changed the way we compute employment rates for the rankings due out March 15.  In addition, we will also be publishing more career data than we have in the past in an effort to help students more completely understand the current state of legal employment. We think more still needs to be done.

So Kelly is saying that employed at graduation data are important, and the magazine is tired of being gamed.  Therefore, we think two methodogical changes have a good chance of being implemented:

• U.S. News is likely to heavily penalized schools that withhold the employed at graduation data.  Going forward,  the imputation may be far more negative than -30% off of employed at 9 months.  A drop in rankings will stop in its tracks the non-response problem.

• Regarding perceptions of gaming, it is possible that U.S. News has formulated a way to quantify how many jobs at graduation map onto full-time professional jobs that require a law degree.  For example, the ABA Official Guide provides lots of comparable data by practice setting.  Law firms, judicial clerks, and government jobs could be weighted more heavily than business or academic jobs.  Unknown may also be treated as 100% unemployed rather than the current 25% presumption of employment.  Such changes would have the law schools scrambling to report better numbers in higher weighted categories rather than just finding ways to goose up the employed-at-graduation and employed-at-9 months figures.  Remember that Bob Morse explicitly endorsed the Law School Transparency movement.

We would like to suggest to our colleagues in the legal academy that we are approaching an endgame.  Here is the reality:  prospective students are not being given an accurate picture of their future employment prospects.  Why? Because we are all focused on filling next year's class with as many high credential students as possible, thereby protecting our school's place in the pecking order.  Our focus is so shockingly narrow that, from the outside looking in, it appears that our intent is to deceive incoming students.  Brian Kelly's letter to the deans essentially makes that point--law schools fall short on candor and ethical behavior.

The numbers that get submitted to U.S. News include many graduates who are technically employed but often significantly underemployed, often at positions that don't require law degrees.  Finer grained data get reported to NALP, but they are never published on a school-by-school basis.  If these data were released, prospective students may not fully process the information--that is an argument that we often hear law professors make.  But that does not alter our duty to provide "basic consumer information ...  published in a fair and accurate manner reflective of actual practice."  ABA Accreditation Standard 509.

At some point, all our lawyerly rationalizations will come to a bad end because a governmental agency or a court is going to challenge our right to self-regulation, thus ushering in a truly disgraceful chapter in the history of American legal education.  

Now is one of the very few moments in our careers as academics where we have to make hard choices and demonstrate that we warrant the trust and respect of our tenured positions.   Through our governance organizations (ABA, LSAC, NALP, AALS), we need to implement a system of complete transparency on employment outcomes.  If the system has real teeth, it will force us all to work very hard to ensure we are delivering value commensurate with the tuition dollars we collect.

It's the end of the road.  We likely have one last chance to get it right.

Law Schools, Students, and "Legal Employment": Supply and Demand

Over at Balkinization Brian Tamanaha (Wash U) posted graphs endeavoring to describe law student enrollment and BLS data on legal employment trends (2001-09). Visually, the two graphs support Brian's assessment: "Law schools thus responded to the worst recession in the legal market in at least two decades by letting in more law students." As Brian Leiter (Chicago) notes, however, "One can't tell, though, from the second chart [and the underlying BLS data] what portion of the downturn in 'legal employment' is a reduction in the employment of attorneys as opposed to other law-related employees."

Ranking Correlations

Over at The Faculty Lounge Al Brophy (UNC) has fun with the recent spat of various rankings (here). As Al notes: "A couple of things stand out here -- there's a high correlation between the Leiter ranks and the Sisk ranks (.7) -- but there's a really high correlation (.96) between the Leiter ranks and the US News peer assessment data.  The Leiter voters and the US News voters are coming to very similar conclusions...."

Can Stanford Be #1 in the US News Rankings? The Data

In a recent story in the San Jose Mercury News, the dean of Stanford Law School, Larry Kramer, announced his intention to make Stanford the #1 law school in the country.  As noted by Brian Leiter, the story did not quote Kramer as saying that he wanted Stanford to be #1 in the U.S. News rankings.  The reporter, nonetheless, inevitably gravitated to this dominant measure. As noted by Dan Filler, most students, alumni, employers, and bystanders, cannot get their heads around the meaning or usefulness of alternative measures. 

Fortunately, Larry Kramer's comments have served up an opportunity to grasp some of the limitations and folly of the current dominant method of ranking law schools. (And by the way, I am certain that Larry Kramer knows most of what I am about to write--he definitely understands the math.  He was making comments for public consumption.  Like Kramer, I am trying to make Indiana Law #1 in the nation, albeit not in US News sense. Every faculty member at every law school ought to share this aspiration.)

The Research Question: What will it take for Stanford Law School to be ranked #1 in the U.S. News Law School rankings?

Andy Morriss and I annually construct a simulation model of the US News rankings.  It routinely explains 99.5% of the variance in the actual rankings, and it enables us to test endless "what if" scenarios.  So let's load the dice and make some really extravagant assumptions:

• Academic Reputation (25% of the input formula).  Increase Stanford's Academic Reputation from 4.7 to a perfect 5.0 (surpassing Harvard and Yale, who languish at 4.8). 
• Lawyer/Judge Reputation (15%). Increase the Lawyer/Judge Reputation from 4.8 to a perfect 5.0 (breaking their current 4.8 tie with Harvard and Yale).
• UGPA (10%).  Increase median UGPA from 3.88 to 4.0 (Harvard and Yale are at 3.89 and 3.9 respectively).
• LSAT (12.5%).  Increase the median LSAT from 170 to 180 (blowing past the 173 medians at Harvard and Yale).  This assumption is somewhat absurd because there are not enough perfect 180 scores to produce this median at any law school.

Surely, giving Stanford a perfect score on 62.5% of the US News  weighting formula will make Stanford #1, right?  It turns out, the answer is no.  Yale still beats out Stanford by a hair.  Stanford would, however, finally muscle ahead of Harvard for #2.

It is fair to ask, what is it about the US News ranking formula that produces these peculiar outcomes?  The answer is standardization of the underlying inputs. In order to make things like library books, LSAT scores, and employment figures commensurable, US News converts the underlying arrays into standard deviation units where the mean is roughly equal to zero and  approximately 66% of the input scores fall within one standard deviation above or below the mean.  The US News weighting formula is then applied to each USN input and the weighted inputs are summed to produce the final US News score.

Many of the underlying input distributions, however, do not reflect a typical normal distribution (i.e., bell curve).  They are skewed.  The reputation variables are roughly normal, and therefore they are unimportant--or, more accurately, unlikely to strongly influence school movement.  Likewise, UGPA and LSAT have a natural cap (4.0 and 180) that limits outlier scores.  And outliers are what skew distributions.

The biggest "uncapped" input variable is direct per student expenses at 9.75% of the input formula.  In theory, there is no limit to how expensive legal education can be.  Yale, it turns out, spends over $100,000 per student, not counting financial aid, which is a separate variable accounting for 1.5% of the input formula.  In contrast, Harvard and Stanford spend only $80K+ per year per student.  In standard deviation units, that translates into 5.8 for Yale versus 4.2 for Harvard and 4.5 for Stanford.  So Stanford has a 1.3 standard deviation gap to close.  Reputation, LSAT, and UGPA are not enough to overcome this deficit.

Implications

My back of the envelope calculations suggest that a check for $350 million ought to be enough to produce enough endowment income to eclipse Yale in the US News rankings.  This assumes that the money is used for things like books, more faculty, and higher salaries for everyone.  If the money is spent on student scholarships, however, Stanford would need a check for roughly $1.8 billion to be #1.  Again, these are the idiosyncrasies of the dominant method of law school rankings.

Stanford alumni are well heeled.  And, like all of us, they want to be #1. But here is my my advice: Before you write your check to give Stanford faculty a big pay raise and a lower teaching load, or give some rich kid with a 180 LSAT a free legal education, including living expenses, consider other investment opportunities. $350 million would go a long way to solving the AIDS epidemic an Africa.  $1.8 billion could provide life-altering educational opportunities for children mired in poverty.

The legal profession, especially our students, have some big problems at the moment.  And society's are even larger.  The best law school is one that prepares its students to solve these problems.  This requires a careful balance of innovative teaching and scholarship.  The U.S. News rankings don't capture these metrics.  In fact, they obscure them and create incentives for truly destructive behavior.  By and large the deans are trapped.  From my own perspective, I don't think even one law school in the US News Tier 1 has reached even 10% of its potential to educate and solve problems.  Too many one-professor silos.  Too much ego.

I am sorry to moralize.  But someone needed to say it.  Let's focus on some problems worth solving.  At the end of the day, it will be worth it.

ELS--From the Academy to the Bench

David Hoffman alerted me to the recent news concerning the ELS Blog's very own David Stras. According to news reports, this morning Minnesota Governor Tim Pawlenty appointed David (all of 35 years-old) to the Minnesota Supreme Court as an Associate Justice. Congratulations to David.