Vault, Inc, which is a publisher that collects industry information on various professions, recently released its listing of the Top 25 Most Underrated Law Schools. (Hat tip: Paul Caron & Volokh.) The findings are derived from a survey of 512 law firm recruiting managers, hiring partners, and corporate counsel, who were asked "to name law schools that, based on their experience as hiring managers, are underrated."
As I read the list, I wondered, what is the practical implication of a law school being underrated by the people who make hiring decisions for entry level lawyers? Presumably, it means that graduates of certain law schools tend to perform better than their school's U.S. News ranking would suggest; thus, legal employers are more likely to hire them.
If this is true, what is the source of superior performance? Here are two possibilities:
- Stronger Students. Some schools may enroll a stronger student body than their rank might suggest.
- Better Education. Some law schools may equip graduates with more or better skills than other schools of comparable rank
Based on some preliminary statistical analysis, there is fairly clear quantitative evidence for the first hypothesis. There is also some qualitative evidence for the second--enough to warrant some additional research.
These research questions are really important to the ranking debate because their answers could reveal a mechanism by which some students will discount USN. If an "underrated" school offers a better entree to coveted legal employers, why go to the higher ranked school, especially if the underrated school costs significantly less? Law schools that understand these dynamics are in a better overall competitive position.
After the jump, I elaborate on these findings, including methodology.
To create a relevant dataset, I added the Vault underrated data to a file containing the current U.S. News rankings, including academic reputation, lawyer/judge reputation, 25th percentile LSAT, and 75th LSAT. Vault has two measures of underrated, national and regional. Based on my earlier research, which showed that geographic proximity is very important in the hiring opportunities available to students at non-elite law schools, I placed more credence in the regional listings. (The national scores were also based an arbitrary 50/50 weighting of the national and regional survey results, similar to the USN overall composite rank.) I then created two variables: Vault national underrated (1 = yes, 0 = no), and Vault regional (same coding).
Stronger Students theory
If some law schools have attracted the attention of recruiters because they tend to enroll stronger students, "underrated" might show up in a disparity between overall USN rank and rank based purely on the 25th or 75th percentile LSAT statistics. I thus subtracted USN rank from LSAT rank. For example, regionally "underrated" Brooklyn Law School is ranked #58 in USN but is #36 when ranked by 75th percentile LSAT. If employers believe that raw aptitude is a good predictor of future job performance, then, from that perspective, Brooklyn is arguably "underrated" by #22 slots (36 - 58 = -22).
To test whether the LSAT rank/USN rank is associated with making the Vault regional "underrated" list, I ran a simple t-test comparing the means of "underrated" and "not underrated." (Note this was limited to the Top 100 schools because I needed an ordinal ranking rather than a tier.) The results were fairly clear.
- The mean disparity between 75th LSAT rank and USN rank was -6.75 ranking slots (negative sign denotes "underrated). The means were statistically difference at p = .015 (t stat = 2.463).
- The mean disparity between 25th LSAT rank and USN rank was -5.96 ranking slots. The p-value, however, was relatively low but above the .05 threshold (p = .127).
Arguably the 75th percentile is more relevant because many employers will not interview students at many schools who do not reach a certain grade cutoff. A better way to model this is dynamic, of course, is a logistic regression that adds some control variables or OLS in which we have the actual regional "underrated" scores. But that is for another day. The above analysis provides limited but fairly straightforward support for the Stronger Students theory.
For the record, just as I thought, the Vault national underrated variable produced wholly random and non-significant results (i.e., p > .800 for all analyses).
Better Education theory
The Better Education theory has larger potential implications for law schools because it suggests that some law schools may be adding more value than other peer institutions during three years of JD education and that legal employers are noticing. In other words, curriculum and/or quality of instruction are "moving the market."
If some law schools have gained a reputation among legal employers for better preparation for practice, it might show up in either the USN academic or lawyer/judge reputation scores. (Don't laugh: Andy Morriss and I have shown the lawyer/judge reputation score appears to be associated with better employed-at-9 months figures after controlling for LSAT, legal market size, and school OCI activity. See "Measuring Outcomes.")
Using the same methodology as above, the Rep rank / USN rank disparity had no statistically significant association with Vault underrated status: -1.97 mean disparity for Acad rep/ USN rank, p = .317; -0.59 mean disparity for Lawyer-Judge rep / USN, rank, p = .852). Of course, this may not be all that surprising since the Acad rep is clearly influenced by the prior year's overall rank. See Jeffrey Evans Stake, The Interplay between Law School Rankings, Reputations, and Resource Allocation, 81 Ind. L. J. 229 (2005).
Yet, notwithstanding these indeterminate results, many of the recruiter comments in the Vault survey suggest that law school training may be making a difference. For example, these comments on specific schools lend support to the Better Education theory:
- Baylor Law School. "Baylor has a top trial advocacy program that puts a Baylor graduate fives years ahead on the power curve." "Baylor has the top advocacy program in the country."
- Chicago-Kent and Depaul. "These schools have great training. Summer associates from these schools have been well prepared, particularly in writing, and have performed well as summer associates and associates."
- Lewis & Clark (Ore). "Lewis & Clark's specialty areas-particularly Environmental and Natural Resource Law-the varied legal clinics, externship and internship programs, and moot court selections are excellent opportunities."
- Northwestern. "Northwestern should be Top 5 -- best prepared graduates, curriculum, business focus, leadership development."
- University of NC Central (Tier 4) and Univ. of Montana (Tier 3). "With their clinical programs and emphasis on pro bono / public interest practice, these schools [NCCU and University of Montana] do an excellent job of training technically proficient citizen lawyers."
- Seattle University. "Seattle University has a very highly rated writing program."
- Southern Illinois University (Tier 3). "Southern is a newer school in a remote location, so does not have the connections other schools have, however, they have a mandatory attendance policy, strong legal writing and advocacy programs, great student-teacher ratio and probably one of the best value propositions going in terms of cost to attend. ... [I]n my experience, [these graduates ] are often better trained than grads from overrated schools you are competing with."
- University of Denver. "The school emphasizes preparing practice ready students, and they do an excellent job of that. DU students easily compete with those from top law schools across the country."
- University of Richmond. "The students that come out of UR seem to have more prepared for the practice of law than those from some other schools."
In the above excerpts, I have selected out comments that speak to quality of curriculum and training. But there are also a large number of comments that corroborate the Better Students theory. Here are just three:
- Boston College and Boston Univ. "Smarter than is ranked."
- Brooklyn and Fordham. "Both schools have entering classes with stellar credentials."
- UC Davis and UC Hastings. "Both of these public law schools have highly selective admissions policies ... "
Endogeneity
One unknown in the above analysis is which way the causal arrow points: Are better curricula and teaching attracting better students, or are better students causing recruiters to think more highly of a school's instruction? For example, many of the schools lauded for excellent training also are among the Top 10 in disparity between 75th LSAT and USN rank: Lewis & Clark (-41); Univ. of Rich. (-29); and Brooklyn (-22).
What is interesting, however, is that some recruiters appear to be paying attention to the nuts and bolts of legal education. Who are these folks? This is a real opportunity for some law schools to build strategic relationships and outperform their U.S. News rankings. In the long run, a school's USN ranking may go up as well.
I have been thinking about this issue a lot and I have another possible factor to suggest, but I do not have the statistical expertise to actually examine it myself. Could you look into the possibility that the underratedness of a school correlates with the percentage of students who stay in that city after graduation? While staying in the region may sometimes be a reflection that the school does not produce students strong enough to get a job anywhere, it could also be merely a reflection of the fact that a certain city is very attractive to students. For example, graduates of New York schools might all want to stay in the city because of its many opportunities. If this is the case, then very few people in other parts of the country will have contact with these graduates and the school's general reputation may lag because of its lack of visibility.
Posted by: Justin | 16 January 2009 at 09:42 AM
This is a very nice and interesting blog which provides great knowledge about Empirical Legal Studies.
-----------------
lauran
http://www.shepelskylaw.com
Posted by: lauran | 28 September 2008 at 11:47 PM
Search for in all major search engines simultaneously on the site http://www.iknowall.com.
Simultaneous search on Google, Yahoo and MSN Live Search.
Try http://www.iknowall.com
Posted by: iknowall | 01 June 2007 at 03:03 PM
Conor,
Great to hear from your. Your comment reflects the brilliance and realism of an IU grad! I hope things are going well for you. Good luck. bh.
Posted by: William Henderson | 28 March 2007 at 08:18 AM
Professor Henderson,
I read the Vault article and thought it was a great publicity piece for the schools involved. I am trying to figure out if it was good or bad that IU was left off!
I would say that lower ranked regional schools have the ability to offer a more "practical," but not necessarily "better" education. First, their students are going to predominantly practice law in the state or the city where their school is situated. This means that a school can teach more state law to a student body that presumably would desire to learn that over more theoretical or general material. Clinical work and internships during the school year will be more fruitful because they are based where you will practice and can lead to job offers and exposure in the community. Local schools can specialize. University of San Francisco School of Law offers a course on how to take the performance section of the CA bar. This most certainly helps their bar passage rate and should make them more desirable to employers. Is this a better education? A whole other debate.
A lower school that is obviously not attracting the top candidates has to offer something of value. You aren't playing the same game. I like your "moving the market" line to explain this. I think the USNWR rankings cannot and will not move fast enough to track innovations employed by law schools in the middle because the rankings have a vested interest in maintaining some continuity and stability. If they fluctuated too much, they would become irrelevant to employers. So that makes them ripe for underrating of schools.
On the other side, better students sometimes go to lower schools for scholarships, to be close to family, etc. So there are diamonds in the rough. Plus, hard work is not always an ethic instilled in the brilliant.
-cdg
Posted by: Conor Granahan, IU '05 | 27 March 2007 at 11:58 PM
I put my LSAT score on my resume when I first started out. I did it because (1) I had a very high LSAT score, and (2) I went to practice in SoCal, where nobody knew a thing about my non-California school. I put my LSAT so recruiters 'would know' that I was smarter than the anybody else applying, be they from Stanford or some other rip-off private school.
And I'm posting anonymously, under my web wide handle, because nobody likes someone who says they are smart (Makes you sound like Fredo from the Godfather).
Didn't make much of a difference. I got a job, but not a super high paying one. Looking back, I should have turned down the free ride at my law school, and taken out a huge loan to go to some top 10 school. That's what I tell pre-law students: unless you plan on practicing in the regional area of your school, you need to go to the highest ranked school you can get into, no matter the cost or location. Rankings matter more than anything for that first job out of school.
Posted by: wooga | 27 March 2007 at 06:25 PM
AnonstatsPhD,
I am happy to accept criticism of my analysis, but your comment does not provide much guidance.
Sure, it would be great to know the characteristics of who replied to the survey and how it matches up to the population. Unfortunately, we don't have that information.
Why analyze it anyway?
1) Vault has a decent track record as a publisher of industry information (I have used their data before and am familiar with the company). Why should we assume they are manufacturing information?
2) 512 respondents is a fair N when 200 law firms employ over 10% of all lawyers in the country, and those firms have a history of being surveyed by Vault.
3) Assuming the survey is representative (an open question), this data is intrinsically interesting and is unavailable from another source; it is expensive to collect. If hiring partners depart from ranking in their recruiting, it may be as significant as the rankings themselves.
4) If sample biased the results, what is the mechanism? Regardless, now it is worth our time to inquiry further on Vault's sample.
Finally, why post anonymously? When commenters reveal their identity, the tone and dialogue tends to be more constructive. Let's have a discussion and learn, not just score points. bh.
Posted by: William Henderson | 26 March 2007 at 07:33 PM
Your highly flawed analysis shows that there is a statistic to prove anything.
Before you start proving Vault's results, you might want to ask yourself if Vault’s results themselves are valid. Did you even ask them how they came up with their results? Consider the possibility that Vault made up the "study" solely to attract readers.
To prevent further abuse of statistics, I implore you to please put the excel spreadsheet and step away.
Posted by: anonstatsPhD | 26 March 2007 at 06:28 PM
Jeff,
This post is not intended to give prescriptive advice to students. I would be cautious about the inferences in both #1 and #2.
Under the Strong Student theory, the 75th LSAT number is associated with recruiter perceptions of being underrated. But in the individual school case, that may not be true at all. The relevant measure here is whether a school's placement outperforms it rank (data I am currently compiling); LSAT is at best a proxy for this.
Regarding students placing their LSAT on their resume, that raises some difficult signaling problems. If it higher than a recruiter might expect, one might wonder why candidate X is not higher in the class; if is lower than expected, it does the candidate no good. These dynamics work against this practice being commonplace.
More importantly, once in law school, grades are a better signal. The 75th percentile, in contrast, speaks to overall strength of the class; thus, it might be relevant to which school to interview at. But it provides no information on which students to hire. bh.
Posted by: William Henderson | 26 March 2007 at 11:18 AM
What does that mean?
1) To the extent that the "stronger students" theory explains the Vault rating, hiring decisions are made according to student aptitude as measured by LSAT. Therefore, students should not worry about the usn&wr ranking but about the LSAT of the school?
2) Students should put their LSATs on their vitas, and schools should stop discouraging students from putting their LSATs on their vitas?
Posted by: Jeff Stake | 26 March 2007 at 10:51 AM