Sander, Affirmative Action, and a Manhattan Project for Legal Education

In a provocative post over at Truth on the Market, Josh Wright asks, “where do we stand on the empirics of affirmative action?”

The catalyst for Josh’s post is the ongoing fallout from Richard Sander’s two articles on articles on affirmative action: A Systemic Analysis of Affirmative Action in American Law Schools, 57 Stan. L. Rev. 367 (2004), which made an empirically based argument that racial preferences set in motion a pattern of "mismatched" credentials, hurting blacks in several ways; and The Racial Paradox of the Corporate Law Firm, 84 N.C. L. Rev. 1755 (2006), which suggests that the large attrition rates of young black associates at prestigious corporate law firms may be attributable to the reluctance of white partners to give important and challenging work to black lawyers, presumably because these young attorneys tend to have lower grades than their white counterparts. (Virtually all of Sander’s work in this area, including responses from critics, can be found here.)

As Josh notes, Paul Butler (GWU Law) recently made the claim that Sander’s work had been “persuasively debunked” by Devon Carbado (UCLA Law), among others. In fact, the relevant post by Carbado does not directly refute any of Sander’s arguments; rather, Carbado asks the broader question of what type of empirical evidence, beyond many of the bleak statistics cited by Sander, would be sufficient to establish that blacks have been hurt by racial preferences. That is certainly an important issue.

Josh Wright is skeptical of Carbado’s statement that it is hard to have an empirical discussion on a blog. After a worthwhile, cautious summary of the debate surrounding Sander’s work, Josh suggested that the ELS bloggers might want to weigh in on the issue—myself in particular because I participated in a Legal Affairs Debate Club forum with Sander in January of 2005.

Obviously, this is an academic minefield, especially for the untenured. Am I dumb enough to take the bait?

Well, I guess I don’t like the way I phrased the question. Diversity in legal education and the legal profession are matters of great public importance. Based on my prior and current empirical research on testing and the legal profession, I probably have some original—and hopefully productive—thoughts on this topic. I am inclined to speak honestly and not indulge the belief that I am somehow rolling the dice.  If I get run out of the academy for engaging in sincere, thoughtful dialogue on matters of public importance and personal conscience, then this job is not a good long-term fit anyway.

In general, I agree with Devon Carbado’s point that blogs are not the best forum to settle debates on empirical issues. Blogs are a valuable resource for sharing and discussing ideas and information among a virtual community of scholars, but they are not scholarship. Moreover, special caution attaches to Richard Sander’s work because it challenges the efficacy of deeply held political values.  People can and will say things they’ll regret. Why preserve our indiscretions on the Internet?

Since the 2004 Law & Society Meeting, when I first heard Richard Sander present a draft of Systemic Analysis, I have been deeply troubled by the statistics that underlie his analysis. I subsequently read his draft paper, provided Sander with pre-publication comments (Rick was familiar with my work on the LSAT and law school testing), read the responses of Sander’s critics, and more recently followed the controversy over the Racial Paradox. With a few exceptions, I have been disappointed with the substance and tenor of the debate. Some points have been scored, but several specific and troubling empirical claims have not been directly challenged.

In an effort to create a forum for a more productive dialogue, I have been involved over the past several months with two separate efforts to impanel respected, independent scholars to review the work of Sander and his critics. Sander has enthusiastically cooperated with these efforts.  I am optimistic that one or both efforts will eventually bear fruit. Although I believe myself to be uncommitted on the ultimate veracity of Sander’s claims, I suspect that I might be viewed by some as a partisan, primarily because I admit that the theory and empirics woven together by Sander are at least plausible. In addition, the stakes here are extremely high. If Sander is even partially right, shallow, uninformed, or politically driven dismissals have the potential to perpetuate a serious social harm.  So I am worried.

One reason why I probably find Sander’s work less threatening or offensive than other liberals is my ongoing research on law school testing. During the last thirty years, there has been relatively little research and innovation in this area. In 1974, the De Funis decision ruled that a reverse discrimination claim by a white applicant to the University of Washington was moot because the plaintiff, Marco De Funis Jr., had already been admitted to UW following an order of a trial judge. In a dissenting opinion, Justice Douglas addressed the merits of the case and, among other points, favorably quoted this passage from the 1970 Toledo Law Review:

Law school admissions criteria have operated within a hermetically sealed system; it is now beginning to leak. The traditional combination of LSAT and GPA [undergraduate grade point average] may have provided acceptable predictors of likely performance in law school in the past. . . . [But] there is no clear evidence that the LSAT and GPA provide particularly good evaluators of the intrinsic or enriched ability of an individual to perform as a law student or lawyer in a functioning society undergoing change. Nor is there any clear evidence that grades and other evaluators of law school performance, and the bar examination, are particularly good predictors of competence or success as a lawyer.

416 U.S. at 330 (quoting Rosen, Equalizing Access to Legal Education: Special Programs for Law Students Who Are Not Admissible by Traditional Criteria, 1970 U. Tol. L. Rev. 321, 332-333).

The obvious effect of De Funis, which was a 5-4 decision, was to cast doubt on the long-term legality of standardized tests, such as the LSAT and the bar exam, that disproportionately screened out ethnic minorities yet had not been validated against objective measures of lawyer competency. Various stakeholders, such as the AALS, the LSAC, the NCBE, and ETS, subsequently embarked upon an ambitious social science research agenda, known as the “Becoming a Competent Lawyer Research Program.“ The long-term objective of this project was to formulate a basis for more realistic and accurate methods of testing. This initiative produced some very important, large-scale research. See, e.g., Carlson & Werts, Relationships Among Law School Predictors, Law School Performance, and Bar Examination Results (ETS 1976).

Although there was a clear consensus that lawyer competency encompassed much more than analytical reasoning, operationalizing additional measures of legal ability and competence was (and is) an extremely complex task.  And after the Bakke decision in 1978, which upheld the use of racial preferences in higher education, the sense of urgency behind this research gradually began to dissipate. As a result, the “science” of law school and bar exam testing has been relatively stagnant for 30 years.

One of Richard Sander’s principal critics, David Chambers (Michigan Law), recently articulated the dangers of inaction: “Grutter is going to make law schools complacent. The Supreme Court has given us permission to continue the admissions procedures we have developed over the past three decades. Because of it, we may devote too little attention to improving minority success at the stages before and after admission.” Athena D. Mutua, et al., Who Gets In? The Quest for Diversity After Grutter, 52 Buffalo L. Rev. 531, 575 (2004) (quoting Chambers’ remarks during a roundtable discussion).

Of course, Sander’s work makes it more difficult to enjoy a comfortable complacency. Here is the bleak chronology of hurdles that face black applicants who hope to enter the legal profession:

• Admission to an ABA-Approved Law School. Primarily because of the disparity in LSAT and UGPA credentials, the acceptance rate to at least one ABA-approved law school is 65.1% for whites, 43.7% for blacks, and 34.1% for Puerto Ricans. Gita Wilder, THE ROAD TO LAW SCHOOL AND BEYOND 17-18 & tbl. 13 (LSAC Research Rep. 02-01, Aug. 2003).
• Law School Performance. Over half of all black law school students are in the bottom ten percent of their class after the first year of law school. Systemic Analysis at 426-27 & tbl. 5.1. Sander attributes this outcome to a system of racial preferences that “cascades” down the law school hierarchy. Whatever the causal mechanism, the legal academy should be fully engaged in finding a solution.
• Law School Attrition and Graduation Rates. After the 1L year, 93.6% of white law school students return for the 2L year, compared with 83.5% for blacks and 81.4% for American Indians. Wilder, supra, at 20-21 & tbl. 18. In the aggregate, 58.1% of minority students cite academic reasons, compared to 35.3% of whites. Id. at 22 tbl. 19. For students in the LSAC Bar Passage Study (entering law school in fall 1991), 90.3% of white matriculants graduated during the observation period versus 78.1.0% of blacks. See Wightman, The Threat to Diversity in Legal Education, 72 NYU L. Rev. 1, 36 tbl. 7 (1997).
• Bar Passage. Of the 23,086 subjects in the LSAC BPS who graduated and went on to sit for the bar exam, the eventual passage for white applicants was 96.7% versus 77.6% for blacks, 79.7% for Puerto Ricans, and 82.2% for American Indians. A recent study of the Texas bar exam reported that only 51% of black applicants pass the bar on their first attempt and only 77% pass after a total of four tries (versus 84%/94% for whites and 68%/89% for Hispanics). Klein & Bolus, Initial and Eventual Passing Rates of July 2004 First Timers 5 tbl. 5 (June 2006).

Faced with these miserable statistics, Sander did what any social scientist of conscience should consider doing: He articulated a theory that attempted to fit all the available observational data.

In essence, Sander claims that bad outcomes that emerge after law school enrollment are exacerbated by a credentials mismatch in which the typical affirmative action recipient is attending law school classes geared toward students significantly better prepared for law school. Hence, these students learn less, drop out at higher rates, and struggle more on the bar exam. Moreover, in his Reply to Critics essay, 57 Stan. L. Rev. 1963 (2005), Sander presented additional evidence that black students who attend their second-choice law school (usually less elite) tend to have significantly better outcomes.

During the last two years, critics have made in-roads on various parts of Sander’s analysis (particularly his prospective model that suggests that ending racial preferences could actually increase the number of black lawyers, since the estimate required assumption that could be easily contested). Many of the critiques, unfortunately, look at the data narrowly, concluding that if they can find any result with a subset of the data that varies from Sander's they have done their job. I think the task is broader -- to look at all of the data Sander has advanced (including the compelling second-choice data) and assess whether the mismatch theory does a better job than the alternative explanations in accounting for dismal black outcomes. But to date, no one has offered an alternative theory that fits the bulk of the observational data. And that’s a problem.

I am an optimist and an idealist who looks to data for inspiration. So I will conclude with this interesting data point.

Last year, the Howard School of Law moot court team won a national ABA trial court competition, beating, among other law schools, Harvard. See Wall, Howard Law Students Defeat Harvard, Atlanta Inquirer (Apr. 25, 2005).  It is my understanding that Howard, along with several other historical black law schools, have a history of strong showings in these events. In the cited story, the competition was graded on “speech advocacy, opening statements, direct examination, cross examination, objections, closing arguments and the presentation and use of exhibits”—all skills highly relevant to the practice of law but wholly untested by the LSAT, the bar exam, and the bulk of law school courses. In addition to a clear K-12 pipeline problem, perhaps this story sheds some light on part of the white-minority performance gap.

I am tired of academics trying to score points at Sander’s expense without offering an alternative hypothesis that directly confronts the data. Faced with the bleak statistics on minorities and the legal profession, our time would be better spent embarking on a Manhattan Project for legal education that examines each nexus of the lawyer creation process, including law school pedagogy. A panel of respected, independent scholars to review the questions raised by Sander and his critics would be a step in the right direction.

Comments

[jamesd]

Boost your career

Taking courses online is also a great way to be able to do the class work on your own time rather than have time obligations and requirements for each course. It is now possible to have a fully booked life and go to school with ease. Also, you will want to consider all the money in gas you would be saving by not having to commute to school on a daily basis………….

special courses

[mjs]

What bothers me about the replies to Sander is their apparent assumption that weaknesses in his specific conclusions drawn from the analysis of his data trump the underlying analysis and empirical support his work provides for commonsense. It makes sense that someone who is less well prepared is going to have more problems than someone who is more prepared. This does not mean that on an individual level, a person with less preparation going in cannot do brilliantly. It merely means that on average, we should expect the outcomes we see.

Because Sander's analysis makes sense at a very commonsense level, it seems altogether likely that many of the more heated and dismissive comments come from folks who will support racial profiling (so long as it helps a favored race) largely irrelevant of the actual effects those preferences have on the bulk of students who receive them and will argue from anecdote or any available data that preferences don't really lead to the outcomes that one would naturally assume would follow if race were taken out of the equation. Stop, for a moment, and just take race out of the equation and take Sander's statistics on two groups of white students. Does anyone really believe that there would be the kind of vehement responses he has received? Of course not.

Thus, the questions Sander raises are not only legitimate, but necessary if we want to actually look at what happens and measure the side effects of our policies. We may decide for normative reasons that we wish to continue those policies, but there may be contervailing normative arguments as well (such as the philosophical problems of using race as a criteria for much of anthing, the societal problems group preferences have tended to create over time historically and the tribalism that is at least supported by viewing people and treating their rights depending on their race or ethnicity).

Further, from what I can see (of course, I only an understanding in empirical methodology from my coursework in methods and statistics in economics, political science and communications) Sander's analysis that racial preferences, if large enough, hurt students (outside of those at the top 10-15 law schools) seems largely correct. While the magnitude of the effect is arguable, there seems little doubt that for whatever reason, it does hurt their prospects. His most recent work on law firms may explain some of why...major law firm partners make affirmative action hires, but don't really think those hires are as good and treat them the way they treat all students who they hire with lower grades (until that person in some way shows he or she is a superstar). This is consistent with my experience in the way young lawyers I knew were treated in large law firms.

Finally, and perhaps this is the most serious problem with Sander's critics, most of Sander's critics perpetuate the status myths that come from graduating from large, prestigious law schools. Were Sander's critics to take a more direct approach to the elitism issue -- i.e., that "only hiring from Harvard" is racist, tends to perpetuate the plutocracy and silly on top of that, we might be able to realize that brilliant lawyering does not exist merely among those from the "upper tier," but exists among those who might have graduated from, say, the University of Wyoming, of even, god forbid, CUNY.

[visitor]

Whatever else they agree or disagree about, Sander would agree with these words from Lempert, correct?

"In particular, very low scores on these dimensions suggest a person has a high chance of either not liking or not succeeding in law school or not passing a similar test - the bar examination. Affirmative action does, thus, bear a relationship to the relatively high failure rates of black law students since it admits students to law schools who would not be admitted to any law school without affirmative action, and these students have a high chance of not graduating and passing the bar. It also admits students who would be admitted without affirmative action but would rank low among all law students on the success-related characteristics the LSAT/UGPA index measures and so have a higher than average chance of not succeeding."

[Joshua Wright]

"I cannot help but respond to the (no doubt unintended) implication in Josh Wright's comment that the critics of Sander's work have not taken his argument seriously."

My comment was certainly not intended to imply that the critics engaging Sander's empirical results on the merits were not taking his argument seriously. I had thought that the clear implication of my commment in the context of Bill's post and my own linked post was that the discussion SHOULD revolve around important the important empirical debate (i.e. including your work and the work of other critics). What is "not taking the work seriously" is to make claims that results have been debunked without reference to these issues. It was this type of rhetoric that motivated my post -- not real scholarly attempts to confront the empirical issues surrounding affirmative action, like Richard's and others.

To the extent that this was unclear from my comment, I hope this helps.

[Richard Lempert]

I cannot help but respond to the (no doubt unintended) implication in Josh Wright's comment that the critics of Sander's work have not taken his argument seriously. I and my coauthors in responding to Sander, took Sander's article very seriously. For me, this began when it was in draft. Then, acting as a colleague and without having done any analyses of my own, I sent Rick numbers of comments on what I had read and suggestions for further analysis, almost all of which apparently were ignored. (This was the summer before the Stanford piece was published so there was plenty of time for further analysis and revision.) [added by WDH: The published response essay is available here: http://www.equaljusticesociety.org/sanders_rebuttal_final.pdf ]

Though I had a more than full time job at the NSF, when it was clear what Sander would publish, I spent many evenngs from 10 to midnight or later and some weekends working with Sander's data and analyses. (Not how I had wanted to spend my time.) This experience was repeated when he published his reply, which like the original article does not withstand close analysis despite its clear and confident rhetoric.

This effort was just my effort; similar time and attention to the data, and critiques and checking of each other's analyses, were put in by my coauthors, David Chambers, Bill Kidder, and Tim Clydesdale. (Indeed on the original reply at least two of these people put in more time than I did.) We were not "shooting from the hip," saying anything to poke holes in Sander's work. We were looking closely at Sander's data and analysis, trying first to duplicate or come close to duplicating his findings to better appreciate his models and then to sort out where and why we thought the analysis was mistaken. Had our results supported his conclusions we would not have published our critique.

The errors we identified were often fundamental. Indeed, implicit in Sander's reply to us and Ayres and Brooks is an acknowledgement of serious methodological and data weaknesses in his original analysis. [added by WDH: Ayres & Brooks' response is available here: http://heinonline.org/HOL/PDF?handle=hein.journals/stflr57&id=1821&print=section&section=55&ext=.pdf ] Indeed, I don't think it is a stretch to say that without acknowledging it the reply repudiates almost every important aspect of Sander's original article except its conclusion.

It is, however, fair for readers to ask for an alternative theory explaining the low level of black student performance (in a cohort that began law school more than 15 years ago), though a flawed theory should not be accepted just because no other theory is offered. I do have the outlines of an explantory story. (Theory if you will though I prefer to reserve that term for more formal explanations.)

The first ingrediant, and in my view the one that is likely to explain the most variance, is that UGPAs and LSAT scores do predict law school success because they measure, albeit imperfectly, characteristics important in learning law. In particular, very low scores on these dimensions suggest a person has a high chance of either not liking or not succeeding in law school or not passing a similar test - the bar examination. Affirmative action does, thus, bear a relationship to the relatively high failure rates of black law students since it admits students to law schools who would not be admitted to any law school without affirmative action, and these students have a high chance of not graduating and passing the bar. It also admits students who would be admitted without affirmative action but would rank low among all law students on the success-related characteristics the LSAT/UGPA index measures and so have a higher than average chance of not succeeding. But like all statistical predictors LSAT and UGPA indices are not perfect, and numbers of students with low prior probabilities of bar passage do eventually become lawyers. Thus some black students today (those at the bottom of the admitted black student LSAT/UGPA index distribution) are in the same position as even Harvard law students were 70 years ago - or so legend has it: "1Ls - look to the left, look to the right, one of you won't be here 3 years from now." If index scores were perfect predictors of success there would be no problem, we would simply not admit any student whose score did not predict to success; but they are not. Some students with low scores succeed; some with high scores fail. Thus the issue is in part do we decide for minority students who want to become lawyers that their risk of failure is so great, we will not let them take the chance of failure associated with entering law school. Answering this question poses complex cost-benefit questions for individuals and for the institutions that must decide whether to admit them.

Note how this aspect of my explanation differs from Sander's, though we both explain some part (in Sander's case almost the entire part) of the BPS results for black students relative to whites to affirmative action. I argue that index LSAT/UGPA scores correlate (though not necessarily linearly) with risk of failure in law school, and because black law students have, due in part to affirmative action, lower average index scores than white students (with a far higher proportion having very low scores), black law school failure rates are higher than they are for whites. Sanders argues that if a student with a given index score were admitted not to the school that admitted him/her but to a school that was not nearly as prestigious, that student's chances of graduating and passing the bar as well as his/her career chances would rise substantially. It is this thesis for which I and my coauthors - and so far as I know everyone but Sanders who has looked closely at the data - do not find support for. Indeed, in the case of black students entering elite schools through affirmative action, we find the evidence is strongly to the contrary.

This difference in perspective is important for many reasons. In particular, Sander's perspective would argue - as Sander has - that students should sacrifice law school prestige in choosing law schools because they will be more likely to succeed in lower quality schools. It also suggests that law schools are making mistakes when they admit students with index scores well below the school's average. I dispute both these contentions, but I do believe that students with very low index scores should seriously consider whether attending law school is likely to pay off and that law schools should consider whether there is some index score threshhold below which they should be reluctant to admit students, absent some special reason why to think the student will succeed. In this connection, note, however, that there has been a substantial increase in the index credentials of blacks admitted to U.S. law schools. (The proportion of black law students with index scores below 500 by Sander's measure has in recent years dropped by about 2/3 from what it was in the BPS cohort Sander and we studied.) To the extent American law schools were admitting students wth high risk of failure, the situation should be considerably improved today. Sander argues that any improvement has been offset by raising the bar for passing the bar; this may to some extent be true but if so it does nothing to support Sander's mismatch hypothesis and suggests that the racial implications of raising bar standards should be investigated.

I should also note that I don't completely reject the idea that there may be a degree of mismatch so great that it has implications for law school success. (e.g. a student who cannot succeed at a top ten school might succeed at a far less competitive institution.) But Sander never shows this to be the situation, and I expect that this degree of mismatch, if it exists at all, does not characterize more than a small minority of blacks admitted to American law schools and, in particular, rarely if ever characterizes black students at the nation's most elite institutions, which tend to be the focus of attention when affirmative action is discussed. (Albert Yoon and Jesse Rothstein have a draft paper that is the best treatment to date of this possibility.) [added by WDH: Yoon & Rothstein paper available here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=881110 ]

Part two of my explanation for the BPS bar passage data are that black students are on average substantially needier than whites. (At Michigan law school we found that almost all blacks but only about 3/4 of whites required some form of need-based financial aid, and even though blacks may have received more gift scholarship money than whites they still graduated on average with substantially higher debts. The BPS data suggest this is true of blacks generally as compared to whites.) This accounts in some measure for higher failure to graduate rates among blacks than among whites, which in turn is an important constituent of the lower eventual black bar passage rates. (Financial reasons are more prominent than grade-related reasons when BPS respondents were asked why they left law school before graduating.) Not only is it likely that some people cannot afford to continue in law school, but it may also be the case that greater need means that even some who could afford to continue choose not to do so because as they gain a better informed view of what their degree is likely to be worth, they realize staying in school is unlikely to be worth the cost. This is likely to be particularly true of students at the lowest prestige law schools where drop out rates are greatest. Increased costs of law schools may be another factor that today offsets the likely greater bar passage chances of what appears to be a stronger cohort of black admittees.

The final ingrediant of my theory to explain the lower rate of bar passage is a vector of factors related to race and likely to depress black student performance. These include such thing as classroom atmosphere, stereotype threat, comfortableness in the community, etc. It is easy to dismiss these as make weights, but as cognitive neuroscience develops and fMRI imagine come more into play we are learning that some of these factors may have surprisingly strong influences on cognitive capacity and behavior.

Finally, I would point out the large number of black students who are lawyers over the last 4 decades; from about 4,000 to probably in excess of 45,000. This is incredible progress, and I believe a good portion of it has been enabled by affirmative action. The legal world is not just a more diverse world but in my view a more just one due to 3 plus decades of affirmative action.

Rick

[Josh Wright]

Ditto. The empirical work here is too important too not take seriously. Excellent post, Bill.

[Tom Ulen]

Dear Bill,
This is a wonderful, heartfelt, clear-eyed comment, which, knowing you, doesn't surprise me at all. I'm with you entirely on this. Let's do Rick Sander the courtesy of taking his work seriously. And let's further recognize that this empirical work is extremely important but not necessarily dispositive of the vexing issues of equality of opportunity that lie behind it. Keep up the great work.
TSU