A Reprise of the Mismatch Argument - Sander 3

Previously: Part 1, Part 2

A comment to Monday's post prompted me to draw up an outline of various claims made in Systemic Analysis, identifying which claims are controversial and the principal critiques. The post that follows summarizes the three central arguments for the mismatch effect -- all of them often ignored by critics.

Students receiving large admissions preferences enter law school with lower credentials (GPA, LSAT) than their fellow students. The central argument in “Systemic Analysis” and “A Reply to Critics” is that these “mismatched” students have worse outcomes – in terms of graduation, bar passage, and the post-school salaries – than if they had gone to a less elite school where their credentials better matched their fellow students. I hypothesize this is because students experiencing mismatch actually learn less than they would otherwise – though that’s just an inference from the data, not something directly observed from experiments.

This mismatch effect is not caused by race. It can affect any group that receives a large preference. (For example, one of my students discovered that older white students – who receive admissions preferences at many law schools – appear to experience mismatch effects.) But it correlates to race because racial minorities are the most common beneficiaries of preferences. It correlates most strongly to blacks as a group simply because in contemporary legal education, blacks receive large and nearly universal preferences (at all but historically minority schools, where there is little evidence of a mismatch effect).

I have advanced three theoretical/empirical arguments so far that support the mismatch conclusion. These are:

1) The “grades” argument. Preferences elevate students to more elite schools; the obvious tradeoff is that the students moved up will be less competitive at the more elite school, and will get lower grades – a result neither surprising nor necessarily problematic. If, however, the less competitive student actually learns less (in an absolute, not a relative sense) in this setting than he would at a less elite school, then we have a problem. That is the hypothesized mismatch effect.

Lacking direct measures of absolute learning during law school, how can we use the Bar Passage Study to observe the mismatch effect? My first approach was to ask whether grades or school eliteness better predict outcomes in graduation, bar passage, and jobs. Bar passage is particularly useful, because it attempts to measure what students have learned in law school. If grades are better predictors of bar outcomes than school eliteness, that implies that the amount of learning is far more dispersed within schools than across schools of differing eliteness. It means that being at the bottom of one’s class at any law school means relatively low levels of learning, and difficulty on the bar.

Two regressions in “Systemic Analysis” (Tables 5.6 and 6.1) suggest that grades are dramatically more powerful predictors of outcomes than school eliteness. In later research I have shown the same result in half-a-dozen different ways. Table 1 in the “Reply to Critics”, for example, shows how a student’s chances of becoming a lawyer fall precipitously in every tier of law school as they approach the bottom 15% of the class. We also know that two-thirds of black students end up in that bottom 15%. So if preferences push students towards the bottom of the class, that’s strong evidence that preferences are harmful.

2) The “disparity in outcomes” argument. The Bar Passage Study – our only comprehensive source of data on the relation between law school performance and bar outcomes – is a minefield of selection biases, mostly because schools are clumped into large “tiers” and we have no information on the undergraduate college of participants. If we compare two white students with identical LSAT scores, one in a “third tier” law school and one in a “fourth tier” school, we have no idea if the “third tier” student is really in a more elite school, or if he has stronger credentials, than the other student. Most of the critics, as we shall see tomorrow, have fallen into a variety of selection-bias traps. The widespread use of law school admissions preferences for blacks, however, creates a nice natural experiment that “Systemic Analysis” sought to exploit. For any two students, one black and one white, with similar credentials, we can be pretty sure that the black student is going to a substantially more elite school. We also know that the black students are mostly at schools where their credentials are lower than those of most of their classmates, and so are their grades. Comparing blacks as a whole with whites as a whole gives us a simple way of comparing the grades vs. prestige tradeoff directly. The comparisons, shown in Tables 6.2 and 5.7 in Systemic Analysis, are sobering. Blacks are dramatically more likely to fail the bar than whites with similar credentials, and are somewhat more likely to not graduate from law school. Again, the bar comparison in particular implies that being “mismatched” causes one to actually learn less in an absolute sense. Of course, this comparison would be invalid (or at least far less conclusive) if one of two things were true. First, could black underperformance account for the disparities? If so, then lower graduation and bar statistics could be due to something else related to race (e.g., discrimination) rather than the mismatch effect. Second, if we compare a black and white with similar LSATs and UGPAs, is there evidence that the white student has other credential advantages (e.g., a more elite school background)? I developed a test for the second concern (credentials disparity) and presented the results in Table 6 of my “Reply to Critics” – blacks and whites have demonstrably similar “unobserved” credentials, unless we constrain the comparison to those at the same or similar schools (a constraint I avoided). “Systemic Analysis” also has a series of tests for the first (underperformance) concern and, indeed, it seems to be generally conceded that blacks do not underperform in graduation and the bar. The only point in dispute seems to be whether black underperformance in law school grades is significant and important – an issue I’ll address in my next entry. This comparative approach is sometimes vulgarized in the debate into a claim I'm showing the blacks "can't cut it" in law school or on the bar. But in fact it demonstrates the opposite -- I'm explaining black outcomes as a function of credentials and, especially, preferences, and showing how little role race plays in these outcomes.

3) The “first choice/second choice” argument. The BPS dataset lets us compare students who went to their first-choice law school against similar students who were admitted to their first-choice school but decided to go elsewhere, usually for geographic or financial reasons. These second-choice schools are generally less elite than the first-choice schools, and we can estimate how much the typical second-choice student has reduced the credentials gap between him and his classmates by going to a second-choice school.
For my purposes, the beauty of this data is that I knew nothing about it until “Systemic Analysis” had gone to press; it thus provides a nice independent test of the mismatch theory. I have run perhaps twenty different analyses with the first-choice/second choice data; in every one, the second-choice black outcomes are better by amounts that equal or exceed the predictions of mismatch theory. True, some of the differences are not statistically significant, but in all such cases the results are predicted by the mismatch theory to not be statistically significant.

The odd and disappointing aspect of the debate on mismatch effects is the degree to which the critics have ignored these three arguments and sets of findings, each one of which makes a powerful case for the theory. Virtually every empirical critique has instead focused on spinning out predictions that don’t really follow from the mismatch theory and demonstrating that those predictions don’t stand up. This strawman approach is a poor substitute for real engagement. A meaningful critique needs to address the three arguments that strongly support the mismatch theory, and explain how they can be reconciled with an absence of mismatch effects.

I will next examine the state of the evidence on each of these three arguments in a little more detail.

Comments

[attorney at law]

Thanks for the (quite useful) chart.