Empirical Legal Studies

Can the market fix the mismatch? - Sander 7

Previously: Part 1, 2, 3, 4, 5, 6

Conservatives tend to dislike affirmative action and have therefore been more receptive to the empirical findings of Systemic Analysis. But I regularly hear one “free market” critique: No one is forcing black students to accept preferences. If they are worse off accepting preferences, why do they do so? Don’t their actions prove that, for them, preferences are on balance a good thing?

It’s an important point, but there are some pretty good answers. First, law schools and the legal education system don’t provide applicants or students with the information they need to make rational decisions. Schools don’t disclose the size of the preferences they use; they don’t identify (or provide data that would enable students to measure) grade gaps, graduation gaps, or bar passage gaps along racial lines. Indeed, many schools – perhaps most schools – actively misrepresent to their students the relationships between race, credentials, and outcomes. (Often they do this with the arguably benign motive of making everyone feel welcome.)

Of course, some information does get out. Blacks starting law school are often aware that blacks, in the aggregate, have lower bar passage rates. According to the Bar Passage Study (BPS), 39% of blacks starting law school in 1991 said they were “very concerned” about passing the bar, compared to 17% of whites. But blacks who expressed this concern were more likely to say that strong “academic reputation” was a key factor in their choice of law school.

Given the available information, these black responses are perfectly rational. These applicants are aware that black bar passage rates are lower; they are aware that elite schools have higher bar passage rates than non-elite schools. The logical strategy, then, is to maximize eliteness, which in their case means maximizing preferences and (I argue) maximizing the mismatch problem.

Grades and the mismatch effect - Sander 6

Previously: Part 1, 2, 3, 4, 5

In my third post, I summarized the “grades” argument: preferences cause black students to have much lower grades; low grades lead ineluctably to a variety of bad outcomes; and grades matter more than school prestige in predicting outcomes. See that post for more detail and the theoretical context of the argument. Here I’d like to give a few empirical examples and discuss the critics.

The two tables below provide a nice intuitive illustration of how low grades produce the mismatch. The first breaks out, for each law school tier, the relationship between law school grades and the odds of graduating and passing the bar on the first attempt. For example, 27% of the students in the bottom 5% of Tier 3 (Duodecile 20) graduated and passed the bar on their first attempt. It’s pretty obvious from this table that the bottom of one’s class is a lousy place to be.

Invitation to Attend Conference on Empirical Legal Studies

See here.

Do elite schools avoid the mismatch effect (1) - Sander 5

Previously: Part 1, Part 2, Part 3, Part 4

Some critics of “Systemic Analysis” have suggested that even if the mismatch effect exists, it doesn’t affect elite schools. Exhibit A is Richard Lempert’s claim that 97% of black Michigan graduates pass the bar. Exhibit B is Rothstein & Yoon’s contention that while there is evidence of mismatch among the lowest four quintiles of black students (measured by incoming credentials), the differences vanish at the top quintile. I think both claims are flawed; in this entry I’ll walk through Lempert’s claim carefully.

Let’s start by looking at the Bar Passage Study (BPS) data to get an idea what outcomes are like for students at a Michigan-caliber law school. We don't know whether Michigan actually participated in the BPS study. What the BPS calls Cluster 5 (and I will call Tier 1) is the most elite, but it generally includes smaller, private schools. Cluster 4 (aka Tier 2) is generally less elite and includes larger, public schools. A school like Michigan -- large, public, but very elite -- could plausibly fit into either tier. Because the two tiers together represent roughly the top 30 law schools, it's reasonable to use them together as a benchmark.

Here are some summary BPS statistics on the combined total for these two tiers:

	Blacks	Whites	Everyone
Total students	426	5559	7041
Graduating	91.1%	95.6%	94.9%
Grads taking the bar	91.8%	92.5%	93.0%
First-time pass rate	70.2%	94.2%	91.2%
Ultimate pass rate	84.5%	97.7%	96.3%
Cohort graduating and passing bar on first attempt	58.7%	83.3%	80.4%
Cohort becoming attorneys	70.7%	86.4%	84.8%
Grads becoming attorneys	77.6%	90.4%	89.4%

Here are the results for the other tiers (131 schools) in the BPS:

	Blacks	Whites	Everyone
Total students	1396	16,675	19,933
Graduating	78.7%	91.1%	89.8%
Grads taking the bar	90.7%	93.6%	93.2%
First-time pass rate	58.5%	91.2%	87.8%
Ultimate pass rate	75.5%	96.4%	94.4%
Cohort graduating and passing bar on first attempt	41.8%	77.8%	73.5%
Cohort becoming attorneys	53.9%	82.2%	79.0%
Grads becoming attorneys	68.5%	90.2%	88.0%

The outcomes for blacks at the elite schools are certainly better than for blacks at lower tier schools, but still very troubling. For all blacks in the BPS, the first-time bar passage rate for blacks is 61%, against 91% for whites. The 70.2% figure for blacks at elite schools is far closer to the overall black number than the white number.

Law School Course on Empirical Methods

At the Illinois College of Law, Tom Ulen, Jennifer Robbennolt, and Bob Lawless are co-teaching a new course on empirical methods for lawyers.  On the Law & Economics Blog, Tom has posted the course syllabus, which impressed me as very comprehensive.  They are looking for feedback from ELS types, so I encourage readers to check it out.

I know that John Donahue at Yale,  Rick Sander at UCLA, and Michael Heise at Cornell (and formerly while at Case) have all taught law school quantitative and/or empirical methods course that were very popular with students.  Obviously, this is a trend.  To build more momentum, I hope Tom (at L&E) and Bob (at Credit Slips) will post a summary of the semester.

First-Choice / Second-Choice Analysis - Sander 4

Previously: Part 1, Part 2, Part 3

In my last entry I summarized the three sets of evidence that support the law school mismatch hypothesis, and promised to examine each in more detail. I’ll start here with the third set – the first-choice/second-choice analysis – because it helps set up the others.

The BPS dataset includes some 1840 black law students who started law school in 1991. All completed a detailed survey shortly after matriculation, and somewhat more than half reported that they had been admitted to their “first choice” law school. Of these nearly 1100 students, about one-sixth reported that they had passed over their first choice school to go somewhere else. These 181 blacks are those I call the “second-choice” students. Although the study did not inquire in depth into students’ decision-making processes, most of these students indicated they had passed over their first-choice school for either financial or geographic reasons.

A black who passes up his first-choice school is still probably attending a school that used preferences in admitting him. But it’s plausible that, usually, such a student is passing up a more elite school to go to one somewhat less elite, and therefore he will be less “mismatched” than his peer who goes to his first-choice school. According to the BPS data, the average “second-choice” student has an academic index that’s 92 points lower than the average for students in his tier. The other eight hundred “first-choice” students have index scores that average 130 points lower than their tier mean. The BPS tiers are wide, of course, but this seems like a reasonable and unbiased estimate of how much “less” mismatched the second-choice students should be.

I noted yesterday that when we compare outcomes for black and white students with similar pre-law school credentials, black outcomes are substantially worse, and I attribute this gap to the mismatch effect. If we reduce the “mismatch” (i.e., the credentials gap), we should reduce the gap in outcomes. The simplest assumption is that the outcome gap will decline in proportion to the credentials gap. In other words, the second-choice blacks should shrink the outcome gap by (130-92) / 130 = 29% relative to the first-choice blacks.

Okay. Let’s look at some results. In each of the tables below I compare the first-choice blacks with a weighted sample of whites with matched entering credentials, and I compare the second-choice blacks with a similar weighted sample of matched whites. That illustrates the gap caused by the mismatch effect, and controls for any differences in entering credentials between the two groups. For example, consider first-year grades:

	First-choice blacks (n=819)	Comparable whites	Second-choice blacks (n=161)	Comparable whites
Mean first-year GPA (standardized by school)	-1.00	-0.30	-0.71	-0.20
Outcome gap (difference in black / white GPAs)	0.70		0.51
Proportionate reduction in outcome gap	(0.70 - 0.51) ÷ 0.70 = 27%
Predicted reduction in outcome gap	29%
Statistically significant?	Yes, p < .001

Where is the Moneyball at Moneylaw?

Over at Moneylaw, a new blog on "the art of winning an unfair academic game" (a reference to the subtitle of Moneyball), Jim Chen asserts that the legal academy overvalues pedigree and undervalues performance, thus thwarting the emergence of a true academic meritocracy.  Jim then writes:

For my part, I offer my own tentative list of core principles for evaluating academic talent, whether on an individual or an institutional basis:

• Pedigree never matters.  Performance always does.
• In gauging performance, you can measure anything you want except other professors' opinions.
• In other words, no academic ratings system is valid if it depends in whole or in part on a subjective survey of academic reputation.

What, exactly, is Jim's objective here?  To structure a more theoretically coherent system of merit that will crush all existing ranking systems by sheer force of reason?  And surely, performance cannot be "anything you want" if the point is to "win an unfair academic game"--you need some criteria to know your making progress (or "winning"). 

In baseball, there are balls and strikes, hits and outs, and ultimately wins and losses.  But in the legal academy, it is much harder to operationalize useful constructs of performance.  Assuming we throw out reputation, as Jim suggests, and focus on productivity (certainly under the rubric of "performance"), lots of measures seem relevant:  articles produced, citations by courts, citations by scholars, SSRN downloads, journal placement (using, say, the W&L impact measures).  And if we can agree that some or all of these are relevant in this "game" we are engaged in, then how shall they be weighted?  Surely, the folly of this enterprise is self-evident.

It is rare that a junior scholar has the opportunity to utter these words to a scholar of Jim Chen's stature, but here it goes:  Jim, you need a theory. 

For a coherent theory on how Moneyball principles can be fruitfully applied to the law school world, and for some interesting data, you are going to have to read past this jump.

The Phantom Philosophy?

Despite this blog's general aversion to self-promotion (unless we're soliciting ideas about our early research), I am very pleased to announce that the final version of The Phantom Philosophy? An Empirical Investigation of Legal Interpretation, forthcoming in the Maryland Law Review, has been re-posted on SSRN.  The Abstract:

"While articles and books offering theories of judicial philosophy or legal interpretation flourish, evidence of the costs and benefits of various interpretive approaches is sorely lacking. Indeed, although there is evidence that ideology explains a limited amount of judicial decisions, there is virtually no systematic evidence that judges' interpretive philosophies, at least as typically defined by academics, even matter in their decisions. While a variety of studies test the ideological or attitudinal model of judicial decision-making, this research often omits any measure of the legal model, that is, traditional interpretive approaches, such as formalism, textualism, and originalism. This omission is not an oversight; the role of interpretive philosophies is rarely tested because of the difficulty in operationalizing the relevant variables. This Article tests a model of judicial decision-making that incorporates elements of both the attitudinal model and the legal model, along with a measure of collegiality and other variables. We develop a measure of interpretive philosophy using two sources of data: judicial opinions, which we code for certain indicators of traditional interpretive approaches (i.e., the use of interpretive tools); and a survey of former judicial clerks. The critical question is whether judges with similar interpretive philosophies are more likely to agree with one another in deciding cases. Our general finding is that ideology and interpretive philosophy are not significant predictors of agreement. Instead, experience on the bench together is a significant predictor of agreement, supporting the conclusion that judging is more about solving problems pragmatically with an eye to maintaining a collegial work environment. While further testing of the importance of the legal model is certainly warranted, at the present time, interpretive theorists pursue their labors in the absence of any evidence that interpretive theory matters in the day-to-day activities of actual judging."

Some papers of interest

Brown and Candeub have written Ideology versus Partisanship: Regulatory Behavior and Cyclical Political Influence.  The Abstract:

"We employ a unique data set of 41,137 commissioner votes in 8,252 Federal Communications Commission (FCC) decisions and orders, reflecting every voted commission decision from 1976-2003, in order to examine those factors that determined commissioner voting at the FCC. We discover a cyclical relationship between commissioner party-line voting and the presidential election. We hypothesize a partisanship effect, tied to the presidential election cycle, which works separately from commissioners' ideological preferences."

Minna Kotkin (Brooklyn) has written Outing Outcomes: An Empirical Study of Confidential Employment Discrimination Settlements.  The Abstract is below the fold:

A Reprise of the Mismatch Argument - Sander 3

Previously: Part 1, Part 2

A comment to Monday's post prompted me to draw up an outline of various claims made in Systemic Analysis, identifying which claims are controversial and the principal critiques. The post that follows summarizes the three central arguments for the mismatch effect -- all of them often ignored by critics.

Students receiving large admissions preferences enter law school with lower credentials (GPA, LSAT) than their fellow students. The central argument in “Systemic Analysis” and “A Reply to Critics” is that these “mismatched” students have worse outcomes – in terms of graduation, bar passage, and the post-school salaries – than if they had gone to a less elite school where their credentials better matched their fellow students. I hypothesize this is because students experiencing mismatch actually learn less than they would otherwise – though that’s just an inference from the data, not something directly observed from experiments.

This mismatch effect is not caused by race. It can affect any group that receives a large preference. (For example, one of my students discovered that older white students – who receive admissions preferences at many law schools – appear to experience mismatch effects.) But it correlates to race because racial minorities are the most common beneficiaries of preferences. It correlates most strongly to blacks as a group simply because in contemporary legal education, blacks receive large and nearly universal preferences (at all but historically minority schools, where there is little evidence of a mismatch effect).

I have advanced three theoretical/empirical arguments so far that support the mismatch conclusion. These are: