Empirical Legal Studies

First-Choice / Second-Choice Analysis - Sander 4

Previously: Part 1, Part 2, Part 3

In my last entry I summarized the three sets of evidence that support the law school mismatch hypothesis, and promised to examine each in more detail. I’ll start here with the third set – the first-choice/second-choice analysis – because it helps set up the others.

The BPS dataset includes some 1840 black law students who started law school in 1991. All completed a detailed survey shortly after matriculation, and somewhat more than half reported that they had been admitted to their “first choice” law school. Of these nearly 1100 students, about one-sixth reported that they had passed over their first choice school to go somewhere else. These 181 blacks are those I call the “second-choice” students. Although the study did not inquire in depth into students’ decision-making processes, most of these students indicated they had passed over their first-choice school for either financial or geographic reasons.

A black who passes up his first-choice school is still probably attending a school that used preferences in admitting him. But it’s plausible that, usually, such a student is passing up a more elite school to go to one somewhat less elite, and therefore he will be less “mismatched” than his peer who goes to his first-choice school. According to the BPS data, the average “second-choice” student has an academic index that’s 92 points lower than the average for students in his tier. The other eight hundred “first-choice” students have index scores that average 130 points lower than their tier mean. The BPS tiers are wide, of course, but this seems like a reasonable and unbiased estimate of how much “less” mismatched the second-choice students should be.

I noted yesterday that when we compare outcomes for black and white students with similar pre-law school credentials, black outcomes are substantially worse, and I attribute this gap to the mismatch effect. If we reduce the “mismatch” (i.e., the credentials gap), we should reduce the gap in outcomes. The simplest assumption is that the outcome gap will decline in proportion to the credentials gap. In other words, the second-choice blacks should shrink the outcome gap by (130-92) / 130 = 29% relative to the first-choice blacks.

Okay. Let’s look at some results. In each of the tables below I compare the first-choice blacks with a weighted sample of whites with matched entering credentials, and I compare the second-choice blacks with a similar weighted sample of matched whites. That illustrates the gap caused by the mismatch effect, and controls for any differences in entering credentials between the two groups. For example, consider first-year grades:

	First-choice blacks (n=819)	Comparable whites	Second-choice blacks (n=161)	Comparable whites
Mean first-year GPA (standardized by school)	-1.00	-0.30	-0.71	-0.20
Outcome gap (difference in black / white GPAs)	0.70		0.51
Proportionate reduction in outcome gap	(0.70 - 0.51) ÷ 0.70 = 27%
Predicted reduction in outcome gap	29%
Statistically significant?	Yes, p < .001

Where is the Moneyball at Moneylaw?

Over at Moneylaw, a new blog on "the art of winning an unfair academic game" (a reference to the subtitle of Moneyball), Jim Chen asserts that the legal academy overvalues pedigree and undervalues performance, thus thwarting the emergence of a true academic meritocracy.  Jim then writes:

For my part, I offer my own tentative list of core principles for evaluating academic talent, whether on an individual or an institutional basis:

• Pedigree never matters.  Performance always does.
• In gauging performance, you can measure anything you want except other professors' opinions.
• In other words, no academic ratings system is valid if it depends in whole or in part on a subjective survey of academic reputation.

What, exactly, is Jim's objective here?  To structure a more theoretically coherent system of merit that will crush all existing ranking systems by sheer force of reason?  And surely, performance cannot be "anything you want" if the point is to "win an unfair academic game"--you need some criteria to know your making progress (or "winning"). 

In baseball, there are balls and strikes, hits and outs, and ultimately wins and losses.  But in the legal academy, it is much harder to operationalize useful constructs of performance.  Assuming we throw out reputation, as Jim suggests, and focus on productivity (certainly under the rubric of "performance"), lots of measures seem relevant:  articles produced, citations by courts, citations by scholars, SSRN downloads, journal placement (using, say, the W&L impact measures).  And if we can agree that some or all of these are relevant in this "game" we are engaged in, then how shall they be weighted?  Surely, the folly of this enterprise is self-evident.

It is rare that a junior scholar has the opportunity to utter these words to a scholar of Jim Chen's stature, but here it goes:  Jim, you need a theory. 

For a coherent theory on how Moneyball principles can be fruitfully applied to the law school world, and for some interesting data, you are going to have to read past this jump.

The Phantom Philosophy?

Despite this blog's general aversion to self-promotion (unless we're soliciting ideas about our early research), I am very pleased to announce that the final version of The Phantom Philosophy? An Empirical Investigation of Legal Interpretation, forthcoming in the Maryland Law Review, has been re-posted on SSRN.  The Abstract:

"While articles and books offering theories of judicial philosophy or legal interpretation flourish, evidence of the costs and benefits of various interpretive approaches is sorely lacking. Indeed, although there is evidence that ideology explains a limited amount of judicial decisions, there is virtually no systematic evidence that judges' interpretive philosophies, at least as typically defined by academics, even matter in their decisions. While a variety of studies test the ideological or attitudinal model of judicial decision-making, this research often omits any measure of the legal model, that is, traditional interpretive approaches, such as formalism, textualism, and originalism. This omission is not an oversight; the role of interpretive philosophies is rarely tested because of the difficulty in operationalizing the relevant variables. This Article tests a model of judicial decision-making that incorporates elements of both the attitudinal model and the legal model, along with a measure of collegiality and other variables. We develop a measure of interpretive philosophy using two sources of data: judicial opinions, which we code for certain indicators of traditional interpretive approaches (i.e., the use of interpretive tools); and a survey of former judicial clerks. The critical question is whether judges with similar interpretive philosophies are more likely to agree with one another in deciding cases. Our general finding is that ideology and interpretive philosophy are not significant predictors of agreement. Instead, experience on the bench together is a significant predictor of agreement, supporting the conclusion that judging is more about solving problems pragmatically with an eye to maintaining a collegial work environment. While further testing of the importance of the legal model is certainly warranted, at the present time, interpretive theorists pursue their labors in the absence of any evidence that interpretive theory matters in the day-to-day activities of actual judging."

Some papers of interest

Brown and Candeub have written Ideology versus Partisanship: Regulatory Behavior and Cyclical Political Influence.  The Abstract:

"We employ a unique data set of 41,137 commissioner votes in 8,252 Federal Communications Commission (FCC) decisions and orders, reflecting every voted commission decision from 1976-2003, in order to examine those factors that determined commissioner voting at the FCC. We discover a cyclical relationship between commissioner party-line voting and the presidential election. We hypothesize a partisanship effect, tied to the presidential election cycle, which works separately from commissioners' ideological preferences."

Minna Kotkin (Brooklyn) has written Outing Outcomes: An Empirical Study of Confidential Employment Discrimination Settlements.  The Abstract is below the fold:

A Reprise of the Mismatch Argument - Sander 3

Previously: Part 1, Part 2

A comment to Monday's post prompted me to draw up an outline of various claims made in Systemic Analysis, identifying which claims are controversial and the principal critiques. The post that follows summarizes the three central arguments for the mismatch effect -- all of them often ignored by critics.

Students receiving large admissions preferences enter law school with lower credentials (GPA, LSAT) than their fellow students. The central argument in “Systemic Analysis” and “A Reply to Critics” is that these “mismatched” students have worse outcomes – in terms of graduation, bar passage, and the post-school salaries – than if they had gone to a less elite school where their credentials better matched their fellow students. I hypothesize this is because students experiencing mismatch actually learn less than they would otherwise – though that’s just an inference from the data, not something directly observed from experiments.

This mismatch effect is not caused by race. It can affect any group that receives a large preference. (For example, one of my students discovered that older white students – who receive admissions preferences at many law schools – appear to experience mismatch effects.) But it correlates to race because racial minorities are the most common beneficiaries of preferences. It correlates most strongly to blacks as a group simply because in contemporary legal education, blacks receive large and nearly universal preferences (at all but historically minority schools, where there is little evidence of a mismatch effect).

I have advanced three theoretical/empirical arguments so far that support the mismatch conclusion. These are:

"Extreme" and "Non-Extreme" Awards: More on Punies and Comps

Because my Cornell colleagues, Ted Eisenberg, Valerie Hans, and Marty Wells, are all too modest to shamelessly self-promote, the task falls to me to do so on their behalf (with neither their consent nor knowledge).  Their recent paper, The Relation Between Punitive and Compensatory Awards: Combining Extreme Data with the Mass of Awards, is now available on SSRN.  As the excerpted abstract, below, suggests, this paper intrigues for methodological and substantive reasons.  On the methods front, the paper cleverly exploits three separate data sets by integrating them into one. This move facilitates studying the compensatory:punitive damage awards relation throughout a broad range of trial awards.   Substantively, what they find, in part, is that even after accounting for "extreme" punitive awards, compensatory awards still explain the bulk of variation in punitive awards.

"This article assesses the relation between punitive and compensatory damages by combining two data sets of extreme awards with state court data from the National Center for State Courts (NCSC) for 1992, 1996, and 2001. One data set of extreme awards consists of punitive damages awards in excess of $100 million from 1985 through 2003, gathered by Hersch and Viscusi (H-V); the other includes the National Law Journal's (NLJ) annual reports of the 100 largest trial verdicts from 2001 to 2004.... Combining the data sets assists in observing the punitive-compensatory relation throughout the full range of trial awards....  The models indicate that the compensatory award explains more than 50 percent of the variance in the punitive award. We also find no increase in punitive or compensatory awards over time in any of the three data sets."

Are Black/White Disparities in Graduation and the Bar Getting Better, or Worse? - Sander 2

Previously: Introduction - Sander 1

Before delving into the debate on the mismatch effect, it is worth considering a prior question – have the terrible racial disparities documented by the Bar Passage Study improved since the mid-1990s?

The dismal statistics documented by the BPS and explored in Systemic Analysis are fairly familiar now: only 45% of blacks starting law school graduated and passed the bar on the first attempt; only 57% became lawyers after multiple bar attempts (compared to 79% and 83% for whites). 61% of blacks actually taking the bar in the BPS passed on their first attempt, compared to 91% of whites.

These are by far the best hard numbers we have on black outcomes – the only ones based on a large national sample of bar results – but they have a couple of weaknesses. They are ten years old now; and they are tainted to a degree by some creaming of the BPS sample. The overall pass rate of all participants in the BPS was 88%; the actual national average bar passage rate at the time was 82.3%.

There have been some encouraging developments for blacks since the BPS. As I noted in Systemic Analysis, the credentials gap between white and black students narrowed a little between the time of the BPS study and recent years. Fewer blacks thus entered law school with LSATs under 140.

On the other hand, following an argument laid out by Jeff Stake, it may be that law schools have widened their intra-school gaps between the races by focusing more on numbers (especially the LSAT) in admissions (Stake 2006, pp. 309-313). If intra-school race gaps expand and professors teach to the median student, then the "mismatch effect" I hypothesize would be worsened.

So what has actually happened? In the BPS sample, the black graduation rate was about 80.7% (19.3% of blacks did not graduate). The only source I know of for national figures on graduation rates is the ABA. The ABA maintains enrollment and graduation statistics on its website. The ABA data does not track individuals to determine graduation rates; it simply reports enrollment by race in each year of law school, and the number of J.D.s awarded by race. 

If we assume that black students entering in Year 0 will mostly graduate in Year 3, but some will graduate in Year 4 and a few in Year 5, we can estimate graduation rates for particular cohorts. Using that method, I estimated that 80% of the black students entering law school in 1991 earned J.D. degrees – a figure very close to the BPS number. 

Using similar methods for more recent cohorts, I estimate current black graduation rates are 78-79%.  This slight drop – which is paralleled by a slight drop in white graduation rates – may simply reflect the addition of several new schools and a ten percent overall increase in enrollment over the past ten years; the new schools probably tend to be low-prestige and have higher flunk-out rates.

Determining what has happened to black bar passage rates is much harder; the states generally don’t even keep bar passage data by race, much less disclose it. We do know that the national first-time bar passage rate was 75% in 2004. We also have very good bar passage data by race for California, and more fragmentary data from Texas and Colorado. What do these tell us?

Pass rate for first-time bar takers

	California, 2003-05 (ABA schools only)	Texas, 2004	Colorado, 1990-2004
All races	67.3%	81%	81.5%
Blacks	39.9%	53%	52.8%

In all three of these states, the first-time bar passage rate for blacks is about 28 points below the overall rate. Simple interpolation would suggest that if the national first-time bar passage rate is 75%, the national black first-time bar passage rate is close to 47% – unless these three states are somehow unrepresentative of national patterns.

(Recall, BTW, that in Systemic Analysis I argue that about half the black-white gap in bar passage is due to preferences and concomitant mismatch effects. Many studies show that, controlling for credentials and law school grades, blacks do not underperform on the bar.)

If the black first-time bar passage rate is only 47%, that is a large decline from the 61% rate observed in the BPS study. (But recall that the BPS figures were probably somewhat higher than the true 1994 numbers, and partly reflect a real decline over the past decade) Again, a substantial decline in black bar passage rates over the past decade is consistent with the available data.  In Colorado, for example, the overall bar passage rate fell from 76% in 1990-94 to 70% in 2000-04; for blacks, it fell from 42.8% to 34.5% – a larger absolute and relative decline.

Let’s assemble the data so far.  If 78% of black law students in the Class of 2004 successfully graduated from law school, 93% of black graduates took the bar (estimated from BPS data), and only 47% of those who do take the bar pass on the first attempt, then only 34% of the overall cohort graduated and passed the bar on the first attempt. The corresponding numbers for whites, using the same estimation procedures, are 90% graduation, 94% take the bar, and 78% pass on the first attempt – that is, 66% of whites starting law school graduate and pass the bar on their first attempt.

Using the BPS, we can estimate how a particular success rate for a cohort of first-time bar takers translates into ultimate bar passage. Again, the method is crude, but the point of this exercise is to gain some initial insight into where things stand now. A first-time bar pass rate of 47% (my guess of the 2004 black rate) translates into an ultimate pass rate of around 65%. (That is, cohorts in the BPS that had first-time bar passage rates in the 45-50% had ultimate bar passage rates about 18 points higher.) A first-time bar pass rate of 78% (my guess of the 2004 white rate) translates, by the same method, into an ultimate pass rate of around 90%.  This yields the following totals:

	BPS (1994 era)		2004 Era (my estimates)
	Whites	Blacks	Whites	Blacks
% of entering law students who graduate	92%	81%	90%	78%
% of graduates who take the bar	94%	93%	94%	93%
% of bar takers who pass on first attempt	91%	61%	78%	47%
% of bar takers who ultimately pass	96.5%	78%	90%	65%
% of entering law students who graduate and pass bar on first attempt	78.7%	45.1%	66%	34%
% of entering law students who ultimately become lawyers	82.7%	57.1%	76%	47%

Clearly, both whites and blacks are having worse outcomes today than in the BPS – mostly because of the decline in bar passage rates. But what about relative outcomes? In one sense, blacks are doing relatively better; the absolute declines for blacks and whites on first-time bar passage are similar, so the ratio of black-to-white failure rates has fallen from around four to around three.

On the other hand, relative chances of success for blacks have fallen much more sharply than for whites. The proportion of blacks graduating and passing the bar on the first attempt has fallen something like one-fourth ((45.1-34)/45.1) and the proportion of the black cohort becoming lawyers has fallen something more than one-sixth ((57.1-47)/57.1); the comparable declines for whites are one-sixth and one-twelfth.

These are seat-of-the-pants estimates, based on limited available data. But they leave no doubt that the mere passage of time has not cured the problem of racial disparities in legal education. The need to understand the causes of those disparities is more urgent than ever.

Satisfied and Unsatisfied Clients

The theoretical framework for my Law Firms as a Business Organization class is the Heinz-Laumann two-hemisphere theory, which posits that lawyers  fall into one of two distinct "hemispheres": (1) lawyers who serve organizational clients, such as corporations, institutions, and labor unions; or (2) lawyers that provide personal services to individuals and small businesses.   See Chicago Lawyers (1982) (detailed empirical study of ~700 lawyers in greater Chicago).

Jack Heinz (NWU Law/ABF) and Edward Laumann (Chicago Sociology) created the hemisphere typology because their research documented that these two groups had very little in common; they attended different law schools, hailed from different socio-economic and religious backgrounds, belonged to different bar organizations, and otherwise traveled in completely separate social and professional circles.

For the next two weeks, my class is focusing on the personal services hemisphere.  And let's face it--virtually all law professors either worked for, or were groomed to work for, the corporate hemisphere.  We don't have a lot of exposure (at least as working adults) to the gritty world of personal injury, family law, workers compensation, and other mainstays of solo and small firm practice.  Since I teach the Law Firms course, I have compensated over the last few years by attending the Annual ISBA Solo & Small Firm Conferences.   It was there that I first learned about Jay Foonberg, the author of the ABA's all-time best selling book, How to Start and Build a Law Practice (5th ed. 2004).  Literally tens of thousand of small firm lawyers have built highly successful practices by following his simple nuts and bolts advice.

In law school, one gets the impression that success in the legal profession is primarily a function of analytical ability.  But this snippet from an ABA survey, republished in Foonberg's magnificant volume, suggests that client service (a function of organization and emotional intelligence rather than book smarts)  is the real make or break category:

Although the importance of these attributes, upon reflection, is obvious, they are rarely discussed in law school.  Moreover, they reflect a skill set (indeed, an awareness) that many law students have not yet acquired.  Foonberg is a welcomed antidote to this deficiency.

Sander (1) - Introduction

I am grateful to Jason, Bill, and their fellow hosts for inviting me to write a bit this week.

            
The ELS movement is exciting and important.  For thirty years, the legal academy has been taking tentative steps toward the social sciences.  Many of us hope that legal scholars can adopt the rigor and sophistication that characterize the best social science work, while retaining the engagement with judges and legislatures that has traditionally been our strength.  ELS has signified a growing maturity in the academy and a readiness to step closer to legitimate social science status.

The success of legal empiricism partly depends on its ability to help the legal academy – and society more generally – deal with sensitive issues where data has often been trumped by emotion and politics. My “Systemic Analysis” of affirmative action  (mentioned by Bill in his introduction) raises important questions about a significant American social policy. But it raises equally important ethical questions about our academic community.

            
“Systemic Analysis” carefully documented that the use of racial preferences in legal education is far more intense and widespread than previously believed.  It also showed that at individual schools there are enormous gaps in the bar passage rates of blacks and whites. The first of these claims seems to be (quietly) conceded by all; the second is disputed by a few, but shouldn’t be, for reasons I will spell out in the course of the week.   

Advanced Conducting Empirical Legal Scholarship Workshop

For those of you who attended the helpful Empirical Legal Scholarship Workshop geared for beginners or who want more advanced information on doing empirical work, I pleased to make the following announcement:

On Friday, February 9, 2007 through Sunday, February 11, 2007 Lee Epstein and Andrew Martin will be conducting an Advanced Conducting Empirical Legal Scholarship workshop.  This workshop will be held in St. Louis, and is co-sponsored by Washington University and Northwestern University. There is more information available about the workshop here: http://cerl.wustl.edu. The workshop is designed for those who have some experience with empirical legal research, and an understanding of elementary statistics (at the level taught in the introductory workshop). Topics to be covered will include: multiple regression, regression models for limited dependent variables, presenting results from non-linear models, data visualization and graphics, and matching methods for causal inference. All logistical details will be available in mid-October.  In the interim, please feel free to contact Andrew Martin ([email protected]) with substantive questions, or Karma Jenkins ([email protected]) with logistical questions.

New Book - Roosevelt on Judicial Activism

The Myth of Judicial Activism: Making Sense of Supreme Court Decisions

by Kermit Roosevelt (Yale University Press)

Availability: Amazon / B&N / Yale

From the Description: “This carefully considered book is a welcome addition to the debate over ‘judicial activism.’ Constitutional scholar Kermit Roosevelt III offers an elegantly simple way to resolve the heated discord between conservatives, who argue that the Constitution is immutable, and progressives, who insist that it is a living document that must be reinterpreted in new cultural contexts so that its meaning evolves. Roosevelt uses plain language and compelling examples to explain how the Constitution can be both a constant and an organic document.”

Guest Blogger: Richard Sander

This week's guest blogger is Richard Sander, who is an economist and law professor at UCLA School of Law.    As I noted in an earlier post, Sander is best known for his empirical work on racial preferences in legal education (Systemic Analysis of Affirmative Action in American Law Schools, 57 Stan L. Rev. 367 (2004)) and corporate law firms (The Racial Paradox of Corporate Law Firms, 84 N.C. L. Rev. 1755 (2006)). These articles raise the possibility that racial preferences in legal education may have significant, unintended consequences that may work to the detriment of many aspiring minority lawyers.  Obviously, this is a very troubling prospect for those of us who work within the legal academy, especially since most law schools and law professors strongly supported the Supreme Court's decision in Grutter. 

Not surprisingly, over the last two years, Sander has garnered no shortage of critics.  Yet, Richard Sander's earlier scholarship and professional activities belie the perception that his work is motivated by politics or ideology.  During the last 20 years, Sander has been a ardent advocate of fair housing laws and worked on the successful campaign and transition team of Harold Washington, the first black mayor of Chicago.  Since joining the UCLA faculty in 1989, Rick has served as President of the Fair Housing Congress of Southern California, founded the Fair Housing Institute, and helped the City of Los Angeles design and implement what was, at the time, the nation's most ambitious living wage law. He also participated in the effort to develop outreach programs that sharply increased the local use of the Earned Income Tax Credit, generating tens of millions of dollars annually for LA's poorest working families.  In addition to scholarship on housing integration and living wage laws, Sander has done important empirical work on legal education, including effective strategies in academic support programs and how law students perceive the law school experience.

At UCLA, Sander teaches courses in Property, Quantitative Methods, Urban Housing, and Policy Analysis.  I look forward to his postings over the coming week. 

Funding for Empirical Research Projects

As I noted several months ago when I wrote ELS on the Cheap (i.e., without Graduate School), there are many organizations that are willing to fund empirical research on legal topics and legal institutions.  I recently ran across this excellent list of potential funders, which was compiled by the Charles B. Sears Law Library at SUNY--Buffalo. 

Political Science Data Directory

A terrific benefit of the ELS Blog is that it provides links to a great many databases that are of interest to scholars doing empirical research on legal issues.  One very useful site that is not yet linked on the ELS Blog, however, is the Data Directory of the Law and Courts Section of the American Political Science Association.

This site, which can be accessed here, contains links to numerous data holdings, including online data resources, a number of important databases, and several public domain datasets that are available for the replication and extension of published research. Unlike other sites that focus on a particular topic or dataset, this site enables researchers to access a wide variety of information and data.  It is well worth browsing.

The Gender Wage Gap: Spouses with "Intense Jobs"

Here at Indiana Law, I teach a course called The Law Firm as a Business Organization. In a nutshell, the course provides students with an overview of the historical, economic, and sociological factors that have shaped—and continue to shape—the modern legal marketplace.   

This year, thanks to the dedication and hardwork of my colleagues, Ken Dau-Schmidt and Jeff Stake, my course includes extensive data on Indiana Law graduates; in other words, current IU students get to study the career trajectories of IU alumni.  This has been an interesting experience as both a teacher and a researcher.  The first study, which was explicitly designed to facilitate comparisons with the famous Michigan alumni dataset, is now available on SSRN: "The Pride of Indiana": An Empirical Study of the Law School Experience and Careers of Indiana University Law Alumni. (Ken and Jeff will also be presenting these results at the CELS conference in Oct.)

As we moved through some of the data last week, I noted some interesting breakdowns associated with the variable "% Spouse has intense job."   This item reflects the percentage of respondents who are married to professionals with demanding jobs (e.g., other lawyers, doctors, business executives).  Here is a summary of the "intense job" data along with some relevant additional variables:

As shown in the table above [click to enlarge], male and female Indiana Law graduates in both the five- and fifteen-year cohorts are fairly comparable on some dimensions (shown in blue): % married, size of families, % of spouses who are homemakers, and total household income.  But there are consistent statistically significant differences along three dimensions (shown in green):