Some critics of “Systemic Analysis” have suggested that even if the mismatch effect exists, it doesn’t affect elite schools. Exhibit A is Richard Lempert’s claim that 97% of black Michigan graduates pass the bar. Exhibit B is Rothstein & Yoon’s contention that while there is evidence of mismatch among the lowest four quintiles of black students (measured by incoming credentials), the differences vanish at the top quintile. I think both claims are flawed; in this entry I’ll walk through Lempert’s claim carefully.
Let’s start by looking at the Bar Passage Study (BPS) data to get an idea what outcomes are like for students at a Michigan-caliber law school. We don't know whether Michigan actually participated in the BPS study. What the BPS calls Cluster 5 (and I will call Tier 1) is the most elite, but it generally includes smaller, private schools. Cluster 4 (aka Tier 2) is generally less elite and includes larger, public schools. A school like Michigan -- large, public, but very elite -- could plausibly fit into either tier. Because the two tiers together represent roughly the top 30 law schools, it's reasonable to use them together as a benchmark.
Here are some summary BPS statistics on the combined total for these two tiers:
|Grads taking the bar||91.8%||92.5%||93.0%|
|First-time pass rate||70.2%||94.2%||91.2%|
|Ultimate pass rate||84.5%||97.7%||96.3%|
|Cohort graduating and
passing bar on first attempt
|Cohort becoming attorneys||70.7%||86.4%||84.8%|
|Grads becoming attorneys||77.6%||90.4%||89.4%|
Here are the results for the other tiers (131 schools) in the BPS:
|Grads taking the bar||90.7%||93.6%||93.2%|
|First-time pass rate||58.5%||91.2%||87.8%|
|Ultimate pass rate||75.5%||96.4%||94.4%|
|Cohort graduating and
passing bar on first attempt
|Cohort becoming attorneys||53.9%||82.2%||79.0%|
|Grads becoming attorneys||68.5%||90.2%||88.0%|
The outcomes for blacks at the elite schools are certainly better than for blacks at lower tier schools, but still very troubling. For all blacks in the BPS, the first-time bar passage rate for blacks is 61%, against 91% for whites. The 70.2% figure for blacks at elite schools is far closer to the overall black number than the white number.
Lempert and his colleagues, so far as I am aware, have not examined or discussed graduation and first-time bar passage rates among Michigan graduates. Lempert’s key rhetorical focus is on “ultimate bar passage”. He maintains that 97% of black Michigan graduates become lawyers. The corresponding figure from the BPS for elite schools (bottom row of top table) is 77.6%. This is quite a large difference; 77.6% is far lower than the white rate at even the lowest-tier schools. If Lempert is right about Michigan, then black Michigan graduates are a lot more successful than elite blacks generally (they’d be one-seventh as likely to never become lawyers). How can we account for such a remarkable discrepancy?
I do not have any race-specific bar data for Michigan’s law school (hereinafter UMLS) – no state bar in the country discloses data by race and school, so the only practical way to compile such data is with internal school records – essentially tracking the outcomes of graduates. Many, perhaps most schools do this, but I don’t know if UMLS does. What I’d like to consider instead is whether the available circumstantial evidence is consistent with Lempert’s claim.
Since Lempert says the ultimate white bar passage rate is also very high at UMLS (98%), one possibility is that Michigan has a far higher bar passage rate than other schools. I obtained bar-passage-by-school data from the State Bar of Michigan (where over a quarter of UMLS grads take the bar) and California (where about an eighth take the bar). (My tabulations can be found here.) From 1982 through 2005, 90.2% of UMLS graduates passed the Michigan bar on their first attempt – a little lower than the 91.2% national average for elite school grads. In California, from 1997 through 2005, the UMLS first-time bar passage rate was 82.5% -- lower than UCLA’s average (86%) over that period and lower than most of the other elite schools. This data is not conclusive, but there’s little reason to believe, based on the available evidence, that UMLS has an unusually high overall bar passage rate.
A second possibility is that Michigan lacks the large black-white disparities in grades that exist at other law schools and that drive disparities in bar performance. We have very good data on this. The Professional Development Survey (PDS) was a one-time alumni survey that Terry Adams and David Chambers conducted with Lempert in 1997-98, which formed the basis of The River Runs Through Law School. A much larger project has been the Alumni dataset, which both Adams and Chambers have worked on for many years, and which for more than a generation has surveyed UMLS alumni five and fifteen years after graduation.
According to the Michigan data, the average black student’s GPA is about one-and-one-half standard deviations below the mean for all students. In the 5-year Alumni dataset, the mean final (standardized) GPA of black respondents is -1.48; in the 15-year Alumni dataset, it’s the same. Interestingly, in the PDS dataset, the mean final GPA of black respondents is even lower, -1.75. These are very low figures – translated, they imply that over 60% of Michigan’s black students are in the bottom tenth of their classes. By contrast, at the elite BPS schools generally, the mean black final GPA is around -1.25, and about 50% of black students are in the bottom tenth of their class. So blacks at Michigan are even more clustered at the bottom of their classes than were students generally in the BPS. Given the very high correlation between law school grades and bar outcomes, it's likely that Michigan has even more severe racial disparities in bar outcomes than those we see at other elite schools in the BPS.
How, then, can we explain Lempert’s 97% figure? Let’s look at how he came up with it. The PDS asked a sample of alumni questions about their professional lives as attorneys. One asked, “Since leaving law school, how many states have admitted you to the practice of law?” Twelve black respondents said zero, while 322 reported one or more, yielding 322/(322+12) = 96.4%, the apparent source of Lempert’s 97% figure. However, another fifteen blacks skipped this question. If we assume these respondents also did not pass the bar, the proportion falls to 322/(322+12+15) = 92.2%. The comparable numbers for whites are 98% (not counting those who skipped the question) and 95% (counting skippers).
More telling, however, are the response rates. Only 51% of the blacks contacted by Lempert, Chambers and Adams responded to the survey, compared to 62% of the whites. Black response rates in such surveys are often a little lower than white rates, for various reasons, but this is a very large disparity – the typical black contacted was only 83% as likely as the average white contacted to respond. If we assume that alumni (of any race) who never pass the bar are much less likely to respond to a “Professional Development Survey” than those who are active in the legal profession, the PDS results change significantly.
Below is an elementary simulation to show the problem with overlooking the response rate issue. For simplicity, let's assume 1000 black alumni and 1000 white alumni were contacted for the PDS. Let’s assume that response rates for blacks are a little lower than they are for whites, but that alumni who never became lawyers have much lower response rates than alumni who did become lawyers. (The idea is that a law school alumnus who never passes the bar has many reasons not to respond to a “Professional Development Survey” from his law school.) My illustrative response rates are in the “Response Rate” column below. Finally, let’s assume that the BPS numbers describing how many black and white graduates become lawyers apply to Michigan – so out of 1000 black alumni, 775 became lawyers and 225 did not. Here’s what we get:
|Status||# contacted||Response rate||# responding||Response rate||Respondents who are lawyers|
The numbers in parentheses are the actual response rates in the PDS and (in the right column) the actual % of all PDS respondents who indicated they had passed a bar (adjusting Lempert’s figures for skippers).
To be clear, let me walk through the numbers in this table for black alumni. I first assume that the BPS numbers on elite schools do apply to Michigan, so that out of 1000 black alumni contacted, 225 are not lawyers. I then assume the black alumni who are lawyers would respond to a survey in much higher numbers (response rate 60%) than those who are not lawyers (response rate 20%). This means that the respondents will appear to be lopsidedly lawyers (465 of those who respond, or 91.2%). Making these assumptions explains the actual response rates to the PDS, and comes very close to the ratio of bar-passers to non-bar-passers among blacks in the PDS data, if we adjust (as Lempert did not) for those who skipped the bar question.
This simulation is, of course, hypothetical and illustrative, but it does explain how someone overlooking both the “question skipping” problem and the “response rate” problem could come up with a 97% success rate for blacks when the true number was closer to 77%.
Lempert and his coauthors wrote in The River Runs Through Law School that they checked their sample for sample bias. However, it appears they did not do the most obvious check of all – look at the actual bar passage statistics for their school to see how blacks did on the bar exam.
When he testified before the District Court in Grutter, Lempert described his results this way: “Our study finds that Michigan, just not to put too fine a point on it, Michigan graduates pass the bar. It doesn't matter, really, whether you're a minority or whether you're white. In one decade, in the 1980's, I'm not going to bother to look at the table, but I think there might have been a statistically significant difference favoring whites, but it was substantively sort of completely trivial. It was like 95 percent of minorities and 98 or something or 99 percent of whites…” (Transcript here, pp. 57-58)
Lempert’s testimony on this and other aspects of his research raises interesting questions for legal empiricists. Did he have an obligation to be more careful in discussing his results? Did he have an obligation to release the data on which his conclusions were based (especially after The River …. was published in 2000) so that others could evaluate his claims? Does he have an obligation now, having studied the BPS data closely, to modify his earlier claims? (Lempert recently reiterated the 97% claim in testimony before the U.S. Civil Rights Commission.)
What is completely, clear, I think, is that the claim that 97% of blacks at Michigan become lawyers is entitled to very little weight in our current discussions. Blacks at elite schools clearly have significant trouble with the bar.