Empirical Legal Studies

2006 ELS Ranking

The 2006 ELS Ranking is complete.  A quick review of the variables that were discussed in greater detail over the last three days:

• relative number of research faculty with social science doctorates
• relative number of research faculty with a secondary social science appointment
• per capita articles citing “statistic! /1 significan!” in Westlaw JLR since 1996

The results:

Rank	School	Rank	School
1	Northwestern	26-tie	BYU
2	George Mason	26-tie	Indiana-Bloomington
3-tie	Cornell	26-tie	Virginia
3-tie	Vanderbilt	26-tie	Wake Forest
3-tie	Yale	30-tie	Connecticut
6-tie	Chicago	30-tie	William & Mary
6-tie	Illinois	32-tie	Boston University
8-tie	Pennsylvania	32-tie	UC-Hastings
8-tie	UC-Berkeley	34-tie	George Washington
10-tie	Stanford	34-tie	UC-Davis
10-tie	USC	36	Iowa
12-tie	Duke	37-tie	Notre Dame
12-tie	UCLA	37-tie	SMU
14-tie	Michigan	39	Washington & Lee
14-tie	Wisconsin	40	Georgetown
16	NYU	41-tie	Arizona
17	Columbia	41-tie	Tulane
18	Harvard	43-tie	Boston College
19	Texas	43-tie	Maryland
20-tie	Emory	45-tie	Alabama
20-tie	WashU (St. Louis)	45-tie	Fordham
22	Ohio State	47-tie	American
23-tie	Florida	47-tie	Georgia
23-tie	Minnesota	49-tie	Colorado
23-tie	North Carolina	49-tie	Washington (Seattle)

How does the 2006 ELS Ranking compare to 2005 ELS Ranking (which looked at 2003-2004 faculty)?  The methodology and data have changed. The empirical research output measure is based on a search of Westlaw rather than a selected set of journals and covers more years.  The 2006 ELS Ranking is based on faculty as of August 1, 2006 whereas the 2005 Ranking was based on the 2003-2004 faculty. And the 2006 ranking includes all US News tier-1 schools. 

The 2006 Ranking is highly correlated with the 2005 Ranking (.82), but there is substantial movement.  Noteworthy moves up the ranking: Vanderbilt up 15 spots, BYU up 11 spots, and Duke and Wake Forest each up 9 spots.  Substantial losses:  Iowa falls 23 spots, Fordham falls 20  spots, and Georgia falls 16 spots.  Nine additional schools are in the 2006 Ranking.  Of those, Ohio State at 22nd and Florida at 23rd make the strongest showing.

How can I account for the dramatic rise of my home institution?  Well, all three schools in this study that have employed me did very well:  Northwestern (2001-2004), George Mason (visitor in 2000) and Vanderbilt (2004 to present).  The only reasonable inference to draw is that my presence reflects and/or encourages ELS.  Or, perhaps, I’m just very clever at constructing a measure that recognizes those places with which I’ve been affiliated.  Either way, it’s all about me!

But, in truth, Vanderbilt’s rise owes more to the Viscusi Effect.  Kip Viscusi, who was recently hired away from Harvard Law, alone published as many or more articles citing statistical significance than 17 schools (some with as many as 65 professors).  If Kip and his wife Joni Hirsch had not moved to Vanderbilt, then Vanderbilt would have been ranked 10th rather than 3rd.

I welcome your feedback, either as posted comments or emails to me ([email protected]).  Thanks to Jason and the other ELS Blog organizers.  Guestblogging has been fun.

Empirical Test of Contractualism in Bankruptcy Theory

A paper by Elizabeth Warren (Harvard) and Jay Westbrook (Texas), Contracting Out of Bankruptcy: An Empirical Intervention, assesses the efficacy of contractualism in the bankruptcy context.  What they find, in part, is that: 

"Data from the Business Bankruptcy Project show that, contrary to the assumptions of the contractualists, there are many claimants in business bankruptcy cases and many of them are poorly adjusting creditors who would be unable to negotiate or adjust their prices. These findings point toward substantial inefficiencies and costs arising from the contractualist proposals, and severely undermine the case for a contractual bankruptcy system."

A New Measure of ELS Output

My ELS Ranking includes a proxy for empirical research output.  The original ranking counted the number of articles by a law school’s faculty in 13 peer-reviewed, interdisciplinary journals associated with ELS (for years 2000-2004).  This measure drew by far the most criticism.  The complaints generally were: 1) the proxy favors economics-related work because a disproportionate number of the journals are in that field, 2) it ignored the primary outlet for ELS – and other law scholarship – general law reviews, and 3) it included articles that lacked quantitative empirical work.  I agreed that the measure was flawed, but the time required to create a new measure and collect relevant data was too great in light of the publication deadline that I faced.  The 2005 ELS Ranking couldn’t be changed.

An updated ELS Ranking gave me the chance to fix flaws, or at least to fix the one most glaring one: the empirical research output measure.  Mark Lemley suggested computing the number of works referring to “statistical significance.” Building on work by Robert Ellickson, I had used that search term in 2005 to look at the ELS trend as reflected in Westlaw’s JLR database.  That phrase (and its variants) may be the best proxy for ELS work because the expression is most likely to appear in articles that include quantitative analysis.  It still has shortcomings: ELS work may refer instead to the particular test statistic used.  Or the article may describe the concept in more meaningful terms for a lay audience and to avoid confusion with practical significance.  But, on the whole, it is a good measure, decreasing false positives.  And, it also should decrease false negatives by looking at all journals in JLR (see Frank Cross’s 9/25 comment).

The 2006 ELS Ranking measures empirical scholarship based on references to “statistic! /1 significan!” in Westlaw JLR articles published since 1996.  Each law school’s score is the sum of all articles by its research faculty divided by the total number of research faculty, giving a per capita ELS publication rate (reflected in parentheses). 

1. Cornell (2.34)

2. Chicago (2.15)

3. Vanderbilt (1.67)

4. Northwestern (1.57)

5. UCLA (1.23)

6. Illinois and Stanford (1.20)

8. Yale (1.16)

9. Texas (.98)

10. George Mason (.97)

11. Harvard and Pennsylvania (.93)

13. Duke (.87)

14. Ohio State (.81)

15. NYU (.78)

16. Columbia and Wake Forest (.70)

18. USC (.67)

19. BYU (.64)

20. UC-Berkeley (.62)

What do you think of this new measure?  It will impact the overall rankings as we’ll see tomorrow.  As a final note, I again want to thank my research assistant Geoff Turvey.

ELS Ranking Continued: Law Schools as part of the University

The ELS ranking evaluates the intellectual environment and output of an institution by  focusing on education, institutional roles, and publications of faculty.  Yesterday, I posted the results for my proxy for education:  professors with social science doctorates.  Today, I present my findings on institutional role:  professors with secondary appointments.

Law professors with social science secondary appointments would seem more likely to be engaged in ELS research.  Likewise, a law school with close ties to social science departments is more likely to support social science research.  Such joint appointments are a signal of interdisciplinary spirit at the school as well as increased prospects for cross-fertilization.  Of course, an appointment does not equate to activity in another department, nor does it require engagement in empirical scholarship.  But, as a proxy, this measure tells us something about a school.

Law professors are unlikely to hold secondary appointments in social science, as apparent from the list below of the top 10.  Of the remaining 40 schools, 15 have positive percentages below 5% and the remaining 25 have no secondary appointments in social science.

1.  USC (16.7%)

2.  Northwestern (14.3%)

3.  George Mason (9.7%)

4.  UC-Berkeley (9.5%)

5.  Vanderbilt (7.7%)

6.  Wisconsin (7.5%)

7.  Illinois (7.5%)

8.  Yale (7.3%)

9.  Duke (6.4%)

10. Cornell (5.3%)

This proxy is the weakest of the three, in my estimation.  How else can we measure a law school’s connections with social science departments? Law schools that extend secondary or courtesy appointments to social science professors also are more likely to be receptive to interdisciplinary work that includes empirical research and quantitative methods.  It is a far weaker one than primary appointments because it doesn't reflect the same commitment of resources.  Plus, as a practical matter, consistent identification is quite difficult.  Other ideas?

Hierarchy of Legal Scholarship

J.B. Ruhl has listed his hierarchy of legal scholarship, ranking types of articles from 0 to 10.  His ranking is below.  (Our readers will want to note #9 and #10.) 

0 – Blog posts
1 – Publication of what are essentially blog posts with footnotes
2 – Doctrinal review of the state of the law
3 – Doctrinal study of interesting questions of law
4 – Doctrinal synthesis of developments in law
5 – Normative policy analysis of law
6 – Normative policy analysis of law with substantial reform proposals
7 – Legal theory
8 – “Law and” interdisciplinary studies
9 – Empirical study of legal institutions
10 – Empirical study of law’s impact on society

Hat tip: Solove at Concurring Opinions

Doctor(s) in the House

The ability of law professors to produce ELS depends in large part on educational background and training.  I opted to include social science doctorates, as a percentage of faculty, as one measure of an institution’s likely role in and commitment to ELS. Social science Ph.D. programs typically require course work in methodology, statistical analysis, and research design. The presence of social scientists also should increase the opportunities for non-social science colleagues to engage in empirical work through informal or formal collaboration.

This proxy required three decisions: First, which professors to include? I wanted to focus on research faculty; therefore, the database includes any tenure-track, non-emeritus, full-time professor whose primary appointment is at the law school. Second, what constitutes a social science for this measure? Beyond the obvious, I added degrees that include the methods training associated with the social sciences. Third, where to locate the information? My research assistants and I looked at the AALS directory as well as each law school’s webpage.

The top 20 schools (with percentage in parentheses) are listed below:

1. Northwestern (36.7%)

2. George Mason (29.0%)

3. UC-Berkeley (28.6%)

4.  Pennsylvania (27.9%)

5. Stanford (23.5%)

6. Yale (21.8%)

7.  Chicago (18.5%)

8. Wisconsin and Michigan (17.5%)

10. USC (16.7%)

11. Cornell (15.8%)

12. Vanderbilt (15.4%)

13.  Illinois (15.0%)

14. UCLA (12.8%)

15. NYU (11.5%)

16. Columbia and Emory (11.1%)

18. North Carolina (10.3 %)

19. UC-Hastings (8.2%)

20.  Florida (8.1%)

Are these results what you expected? Which school did you expect to be higher given its interdisciplinary reputation?  Are you surprised that Harvard is missing from the top 20? Does its absence possibly indicate that a percentage is not the right measure here? (Harvard is 13th overall in the number of social scientists.) How will these numbers shift in the next few years? 

(If you believe a number is in error, please let me know which number and why.)

The Curvalinear Relationship between MBE Cut Scores and Law School LSAT Scores

Last month, I blogged about the importance of visually inspecting data to ensure that ordinary least squares (OLS) is an appropriate method for estimating a relationship between two variables.  As I noted, one pattern that you will occasionally observe is a monotone curvalinear relationship--i.e., an increasing or decreasing function in which the best-fitting line has an observable bend.  Here is the example I posted:

Over at Moneylaw, Al Brophy (Alabama Law) posted some interesting data on the strong correlation between the various U.S. News input variables and bar passage rates of the California law schools.  In a nutshell, almost every U.S. News input variable correlates with bar passage between .84 and .90.   To illustrate his point, Al posted this scatterplot:

A visual inspection of Al's chart (here, bar passage and the calculated LSAT mid-point) suggests a curvalinear relationship. (Note that a correlation coefficient reflects the best-fitting straight line.)  Since I don't have access to Al's data, I superimposed a rough fit line, which approximates the first graph.   

Fortunately, Gary Rosin (South Texas Law) recently posted a paper, "Unpacking the Bar: Academic Qualifications, Cut Scores, MBE Scaling and Law School Bar Passage Rates," provides a very nice chart that shows the interaction between MBE cut scores and a law school's mean LSAT.  It provides some insight on why the best fitting line in Al's scatterplot is probably curved. [See chart after the jump.]

Give the People What They Want...

Do you want to increase your SSRN downloads? If so, you should consider publishing a law school ranking. (If not, congratulations, you are far healthier than most of your colleagues. Of course, it does prompt one to wonder why you’re reading this post.)

In April 2005, I participated in the Indiana Law Journal Symposium on “The Next Generation of Law School Rankings” organized by Paul Caron and Rafael Gely of Cincinnati and ELS blog’s own Bill Henderson and Jeff Stake of Indiana. Invited to discuss ranking methodologies, I opted to create my own ranking. For me, the choice of focus was obvious: Empirical Legal Scholarship (ELS). Why? An ELS Ranking identifies schools leading in this burgeoning field and perhaps incentivizes others to invest in it.

The paper ranked each of the top 41 schools in US News based on an ELS-score that I constructed. The score included three variables: professors with social science doctorates, professors holding secondary appointments in social science departments, and articles in ELS-oriented publications. First posted on SSRN on August 9, 2005, the paper has been viewed more than 4,000 times and downloaded more than 860 times. (By contrast, my paper with Jeff Berger on judicial entrepreneurs posted at the same time has been downloaded only a fraction of that number – help here!) UC-Berkeley and George Mason each boasted on its webpage that it tied for first. Legal Affairs and various law blogs all provided short discussions of the piece. Obviously, law professors and their groupies love law school rankings.

I have revised, updated, and expanded the ranking and will post the new results over each of the next four days. The ranking now includes the top-50 schools in US News and looks at faculty as of 2006-2007 (as opposed to 2003-2004 in the original). This week, I will describe each measure and the new results, focusing on educational background and training on Tuesday, secondary appointments on Wednesday, and empirical research output (the only measure that has been completely changed) on Thursday. And, the 2006 ELS Ranking will be available on Friday.

My purpose here is not simply to navel gaze but also to seek input on the measures. Operationalization is hard. And the power of any empirical results necessarily turns on the soundness of those measures. I invite comments on the value of rankings (particularly given the substantial research costs including time away from other work) as well as thoughts on the components of the ELS ranking. Thanks to Jason Czarnezki for inviting me to be a guest blogger and to Geoff Turvey for exceptional research assistance.

Guest Blogger: Tracey George

This week's guest blogger is Tracey George, Professor of Law at Vanderbilt University Law School.  Tracey studies judicial behavior, judicial politics and judicial selection.  She previously served as professor of law at Northwestern University Law School , where she was also Faculty Associate at the Institute for Policy Research.  After graduating from Stanford Law School, Professor George clerked for Judge Francis D. Murnaghan, Jr., on the U.S. Court of Appeals for the Fourth Circuit and practiced law in Washington, D.C..  From 1996 through 2001, she served as an Associate Professor of Law and Adjunct Professor of Political Science at the University of Missouri School of Law. 

Some concluding thoughts - Sander 8

Previously:
1. Introduction
2. Are Black / White Disparities in Graduation and the Bar Getting Better, or Worse?
3. A Reprise of the Mismatch Argument
4. First-Choice / Second-Choice Analysis
5. Do Elite Schools Avoid the Mismatch Effect?
6. Grades and the Mismatch Effect
7. Can the Market Fix the Mismatch?

This week, I’ve been aided by two third-year students at UCLA – Patrick Anderson (who will coauthor my forthcoming book) and Matthew Butterick (whose independent work on the legal side of the mismatch was mentioned in Post 7). I’m grateful to them both.

My previous posts laid out the story of what is happening to black law students as a result of affirmative action. There are grey areas and we need better data to resolve several issues. But if this were a less controversial topic, I think many would agree that affirmative action in law schools may not be achieving many of its core goals, and that it is probably producing some extremely serious side effects. If affirmative action were not an entrenched social policy, but instead an experimental drug, could it possibly survive an FDA review? Not on the existing evidence.

As I suggested at the beginning of the week (and Bill Henderson suggested earlier), there has not yet been a complete critical engagement with these arguments and the underlying data. Critics have raised specific questions about my analysis, which in some cases have led me to new insights and improvements. But no critic of the mismatch hypothesis has woven the data into a consistent alternative theory.

I have responded to most of the critiques; rather than rehash those, I’d like to illustrate the problem by examining a recent critique by Rothstein & Yoon.

Jesse Rothstein and Albert Yoon have produced by far the strongest response to Systemic Analysis, and I know and admire Albert. Rothstein & Yoon recognize the problem of selection bias, and attempt to deal with it. They construct models for the data, and they report evidence of significant mismatch effects, though they think other factors might explain their results. All this is good. But Rothstein & Yoon also commit many of the common critical errors:

Rhetoric. Rothstein & Yoon repeatedly say they find no evidence of mismatch effects in the top 80% of the credential distribution of law school applicants. This is misleading. On closer inspection, one finds that the “top 80%” of all students includes only the top 20% of black students, so their mismatch findings apply to 80% of black students. Even if they were right that the top 20% of black students don’t suffer mismatch (but see below), it trivializes the problem to say it only affects the “bottom quintile”.

Ignoring relevant data. Given Rothstein & Yoon’s concern with selection bias, it would seem that a comparison of first- and second-choice blacks would be called for; no one has identified any meaningful bias in the first-choice / second-choice data. Moreover, one can split the first-choice / second-choice sample to compare the elite blacks (who supposedly do not suffer from mismatch effects) with other blacks. I’ve done this, and the effects seem indistinguishable – that is, blacks with high credentials seem to gain as much from going to second-choice schools as other blacks do. In other words, Rothstein & Yoon don't reconcile their findings with relevant data.

Shrinking “significance”. Law student outcomes (for all racial groups) get much better with rising credentials. Relatively few students of any race with an academic index above 700 fail to graduate and pass the bar. But that doesn’t mean the mismatch effect has declined, if by "mismatch" we mean how much relative learning occurs. For example, it could easily be that mismatched students with fairly high credentials are scoring much lower on bar exams than comparable, non-mismatched students, but both groups have mean scores far above the passing threshold. Indeed, mismatch effects in grades (a measure of learning) are not different at all for the most elite blacks.

Rothstein and Yoon compound this problem by focusing on “ultimate bar passage” as a key outcome measure. Since first-time bar failure rates are much higher than ultimate bar failure rates, looking only at a bivariate measure of ultimate bar failure throws out most of the relevant information. Ultimate bar passage is also not a sensible measure theoretically. The mismatch effect occurs during law school. If a student passes the bar on his fourth attempt, two years after law school, he may well have partially offset the mismatch effect by hiring tutors, taking a variety of bar-preparation courses, and other work aimed at learning what he didn’t learn in law school. First-time bar attempts are the best measure of what was actually learned in law school.

Critics, including Rothstein & Yoon, chronically favor measures that narrow apparent black-white differences or shrink sample sizes, in what looks like a quest to shrink the gaps to a point where one can proclaim "it's no longer statistically significant, and therefore not a problem."

Measuring mismatch in the job market. Rothstein & Yoon use BPS data to see if blacks are hurt in the job market by mismatch effects. This is commendable, but they miss the obvious test -- they do not compare the relative impact of school eliteness and law school grades on black earnings. If they had, Rothstein & Yoon would have found results highly consistent with those I found in Systemic Analysis. Rothstein & Yoon instead show that blacks right out of law school receive large preferences. That's really not relevant to whether mismatch effects hurt black chances in the job market. (And, as I've argued elsewhere, the preferences in the job market often backfire on black hires.)

Bill Henderson has done his own analyses of the BPS data on a few of these points, with results compatible with mine; interested readers can ask him about those.

Again, the point here is not to single out Rothstein & Yoon, whose paper has many strengths. Rather, it is to highlight how even the better critiques too often read like briefs on behalf of a sacred policy, rather then disinterested investigations into the pros and cons of a controversial program.

As I suggested at the outset, I see this debate as a sort of maturity test for the ELS community. There are several things we, as an academic community, can do to make progress on these issues, and to move away from what has become an overly politicized area of research:

Pressure state bars to make bar performance data available. Actual bar scores are dramatically better than pass/fail results to evaluate mismatch effects across schools.

Pressure the LSAC to study the issue. LSAC has data on where most applicants apply to law school, where they are accepted, where they matriculate, and how they perform in the first year. It would not be difficult to use this data as the foundation for a quasi-experimental dataset similar (but far superior to) the data on first- and second-choice students. Supervised by credible, independent researchers, a study of this type could be truly definitive.

Debate and engagement. More candid discussion of specific aspects of law school affirmative action, and greater involvement in the debate by those with no ax to grind, will necessarily make the discussion healthier and more robust.

My thanks again to the ELS hosts for creating this wonderful forum and letting me borrow the microphone.

Can the market fix the mismatch? - Sander 7

Previously: Part 1, 2, 3, 4, 5, 6

Conservatives tend to dislike affirmative action and have therefore been more receptive to the empirical findings of Systemic Analysis. But I regularly hear one “free market” critique: No one is forcing black students to accept preferences. If they are worse off accepting preferences, why do they do so? Don’t their actions prove that, for them, preferences are on balance a good thing?

It’s an important point, but there are some pretty good answers. First, law schools and the legal education system don’t provide applicants or students with the information they need to make rational decisions. Schools don’t disclose the size of the preferences they use; they don’t identify (or provide data that would enable students to measure) grade gaps, graduation gaps, or bar passage gaps along racial lines. Indeed, many schools – perhaps most schools – actively misrepresent to their students the relationships between race, credentials, and outcomes. (Often they do this with the arguably benign motive of making everyone feel welcome.)

Of course, some information does get out. Blacks starting law school are often aware that blacks, in the aggregate, have lower bar passage rates. According to the Bar Passage Study (BPS), 39% of blacks starting law school in 1991 said they were “very concerned” about passing the bar, compared to 17% of whites. But blacks who expressed this concern were more likely to say that strong “academic reputation” was a key factor in their choice of law school.

Given the available information, these black responses are perfectly rational. These applicants are aware that black bar passage rates are lower; they are aware that elite schools have higher bar passage rates than non-elite schools. The logical strategy, then, is to maximize eliteness, which in their case means maximizing preferences and (I argue) maximizing the mismatch problem.

Grades and the mismatch effect - Sander 6

Previously: Part 1, 2, 3, 4, 5

In my third post, I summarized the “grades” argument: preferences cause black students to have much lower grades; low grades lead ineluctably to a variety of bad outcomes; and grades matter more than school prestige in predicting outcomes. See that post for more detail and the theoretical context of the argument. Here I’d like to give a few empirical examples and discuss the critics.

The two tables below provide a nice intuitive illustration of how low grades produce the mismatch. The first breaks out, for each law school tier, the relationship between law school grades and the odds of graduating and passing the bar on the first attempt. For example, 27% of the students in the bottom 5% of Tier 3 (Duodecile 20) graduated and passed the bar on their first attempt. It’s pretty obvious from this table that the bottom of one’s class is a lousy place to be.

Invitation to Attend Conference on Empirical Legal Studies

See here.

Do elite schools avoid the mismatch effect (1) - Sander 5

Previously: Part 1, Part 2, Part 3, Part 4

Some critics of “Systemic Analysis” have suggested that even if the mismatch effect exists, it doesn’t affect elite schools. Exhibit A is Richard Lempert’s claim that 97% of black Michigan graduates pass the bar. Exhibit B is Rothstein & Yoon’s contention that while there is evidence of mismatch among the lowest four quintiles of black students (measured by incoming credentials), the differences vanish at the top quintile. I think both claims are flawed; in this entry I’ll walk through Lempert’s claim carefully.

Let’s start by looking at the Bar Passage Study (BPS) data to get an idea what outcomes are like for students at a Michigan-caliber law school. We don't know whether Michigan actually participated in the BPS study. What the BPS calls Cluster 5 (and I will call Tier 1) is the most elite, but it generally includes smaller, private schools. Cluster 4 (aka Tier 2) is generally less elite and includes larger, public schools. A school like Michigan -- large, public, but very elite -- could plausibly fit into either tier. Because the two tiers together represent roughly the top 30 law schools, it's reasonable to use them together as a benchmark.

Here are some summary BPS statistics on the combined total for these two tiers:

	Blacks	Whites	Everyone
Total students	426	5559	7041
Graduating	91.1%	95.6%	94.9%
Grads taking the bar	91.8%	92.5%	93.0%
First-time pass rate	70.2%	94.2%	91.2%
Ultimate pass rate	84.5%	97.7%	96.3%
Cohort graduating and passing bar on first attempt	58.7%	83.3%	80.4%
Cohort becoming attorneys	70.7%	86.4%	84.8%
Grads becoming attorneys	77.6%	90.4%	89.4%

Here are the results for the other tiers (131 schools) in the BPS:

	Blacks	Whites	Everyone
Total students	1396	16,675	19,933
Graduating	78.7%	91.1%	89.8%
Grads taking the bar	90.7%	93.6%	93.2%
First-time pass rate	58.5%	91.2%	87.8%
Ultimate pass rate	75.5%	96.4%	94.4%
Cohort graduating and passing bar on first attempt	41.8%	77.8%	73.5%
Cohort becoming attorneys	53.9%	82.2%	79.0%
Grads becoming attorneys	68.5%	90.2%	88.0%

The outcomes for blacks at the elite schools are certainly better than for blacks at lower tier schools, but still very troubling. For all blacks in the BPS, the first-time bar passage rate for blacks is 61%, against 91% for whites. The 70.2% figure for blacks at elite schools is far closer to the overall black number than the white number.

Law School Course on Empirical Methods

At the Illinois College of Law, Tom Ulen, Jennifer Robbennolt, and Bob Lawless are co-teaching a new course on empirical methods for lawyers.  On the Law & Economics Blog, Tom has posted the course syllabus, which impressed me as very comprehensive.  They are looking for feedback from ELS types, so I encourage readers to check it out.

I know that John Donahue at Yale,  Rick Sander at UCLA, and Michael Heise at Cornell (and formerly while at Case) have all taught law school quantitative and/or empirical methods course that were very popular with students.  Obviously, this is a trend.  To build more momentum, I hope Tom (at L&E) and Bob (at Credit Slips) will post a summary of the semester.

First-Choice / Second-Choice Analysis - Sander 4

Previously: Part 1, Part 2, Part 3

In my last entry I summarized the three sets of evidence that support the law school mismatch hypothesis, and promised to examine each in more detail. I’ll start here with the third set – the first-choice/second-choice analysis – because it helps set up the others.

The BPS dataset includes some 1840 black law students who started law school in 1991. All completed a detailed survey shortly after matriculation, and somewhat more than half reported that they had been admitted to their “first choice” law school. Of these nearly 1100 students, about one-sixth reported that they had passed over their first choice school to go somewhere else. These 181 blacks are those I call the “second-choice” students. Although the study did not inquire in depth into students’ decision-making processes, most of these students indicated they had passed over their first-choice school for either financial or geographic reasons.

A black who passes up his first-choice school is still probably attending a school that used preferences in admitting him. But it’s plausible that, usually, such a student is passing up a more elite school to go to one somewhat less elite, and therefore he will be less “mismatched” than his peer who goes to his first-choice school. According to the BPS data, the average “second-choice” student has an academic index that’s 92 points lower than the average for students in his tier. The other eight hundred “first-choice” students have index scores that average 130 points lower than their tier mean. The BPS tiers are wide, of course, but this seems like a reasonable and unbiased estimate of how much “less” mismatched the second-choice students should be.

I noted yesterday that when we compare outcomes for black and white students with similar pre-law school credentials, black outcomes are substantially worse, and I attribute this gap to the mismatch effect. If we reduce the “mismatch” (i.e., the credentials gap), we should reduce the gap in outcomes. The simplest assumption is that the outcome gap will decline in proportion to the credentials gap. In other words, the second-choice blacks should shrink the outcome gap by (130-92) / 130 = 29% relative to the first-choice blacks.

Okay. Let’s look at some results. In each of the tables below I compare the first-choice blacks with a weighted sample of whites with matched entering credentials, and I compare the second-choice blacks with a similar weighted sample of matched whites. That illustrates the gap caused by the mismatch effect, and controls for any differences in entering credentials between the two groups. For example, consider first-year grades:

	First-choice blacks (n=819)	Comparable whites	Second-choice blacks (n=161)	Comparable whites
Mean first-year GPA (standardized by school)	-1.00	-0.30	-0.71	-0.20
Outcome gap (difference in black / white GPAs)	0.70		0.51
Proportionate reduction in outcome gap	(0.70 - 0.51) ÷ 0.70 = 27%
Predicted reduction in outcome gap	29%
Statistically significant?	Yes, p < .001

Where is the Moneyball at Moneylaw?

Over at Moneylaw, a new blog on "the art of winning an unfair academic game" (a reference to the subtitle of Moneyball), Jim Chen asserts that the legal academy overvalues pedigree and undervalues performance, thus thwarting the emergence of a true academic meritocracy.  Jim then writes:

For my part, I offer my own tentative list of core principles for evaluating academic talent, whether on an individual or an institutional basis:

• Pedigree never matters.  Performance always does.
• In gauging performance, you can measure anything you want except other professors' opinions.
• In other words, no academic ratings system is valid if it depends in whole or in part on a subjective survey of academic reputation.

What, exactly, is Jim's objective here?  To structure a more theoretically coherent system of merit that will crush all existing ranking systems by sheer force of reason?  And surely, performance cannot be "anything you want" if the point is to "win an unfair academic game"--you need some criteria to know your making progress (or "winning"). 

In baseball, there are balls and strikes, hits and outs, and ultimately wins and losses.  But in the legal academy, it is much harder to operationalize useful constructs of performance.  Assuming we throw out reputation, as Jim suggests, and focus on productivity (certainly under the rubric of "performance"), lots of measures seem relevant:  articles produced, citations by courts, citations by scholars, SSRN downloads, journal placement (using, say, the W&L impact measures).  And if we can agree that some or all of these are relevant in this "game" we are engaged in, then how shall they be weighted?  Surely, the folly of this enterprise is self-evident.

It is rare that a junior scholar has the opportunity to utter these words to a scholar of Jim Chen's stature, but here it goes:  Jim, you need a theory. 

For a coherent theory on how Moneyball principles can be fruitfully applied to the law school world, and for some interesting data, you are going to have to read past this jump.

The Phantom Philosophy?

Despite this blog's general aversion to self-promotion (unless we're soliciting ideas about our early research), I am very pleased to announce that the final version of The Phantom Philosophy? An Empirical Investigation of Legal Interpretation, forthcoming in the Maryland Law Review, has been re-posted on SSRN.  The Abstract:

"While articles and books offering theories of judicial philosophy or legal interpretation flourish, evidence of the costs and benefits of various interpretive approaches is sorely lacking. Indeed, although there is evidence that ideology explains a limited amount of judicial decisions, there is virtually no systematic evidence that judges' interpretive philosophies, at least as typically defined by academics, even matter in their decisions. While a variety of studies test the ideological or attitudinal model of judicial decision-making, this research often omits any measure of the legal model, that is, traditional interpretive approaches, such as formalism, textualism, and originalism. This omission is not an oversight; the role of interpretive philosophies is rarely tested because of the difficulty in operationalizing the relevant variables. This Article tests a model of judicial decision-making that incorporates elements of both the attitudinal model and the legal model, along with a measure of collegiality and other variables. We develop a measure of interpretive philosophy using two sources of data: judicial opinions, which we code for certain indicators of traditional interpretive approaches (i.e., the use of interpretive tools); and a survey of former judicial clerks. The critical question is whether judges with similar interpretive philosophies are more likely to agree with one another in deciding cases. Our general finding is that ideology and interpretive philosophy are not significant predictors of agreement. Instead, experience on the bench together is a significant predictor of agreement, supporting the conclusion that judging is more about solving problems pragmatically with an eye to maintaining a collegial work environment. While further testing of the importance of the legal model is certainly warranted, at the present time, interpretive theorists pursue their labors in the absence of any evidence that interpretive theory matters in the day-to-day activities of actual judging."

Some papers of interest

Brown and Candeub have written Ideology versus Partisanship: Regulatory Behavior and Cyclical Political Influence.  The Abstract:

"We employ a unique data set of 41,137 commissioner votes in 8,252 Federal Communications Commission (FCC) decisions and orders, reflecting every voted commission decision from 1976-2003, in order to examine those factors that determined commissioner voting at the FCC. We discover a cyclical relationship between commissioner party-line voting and the presidential election. We hypothesize a partisanship effect, tied to the presidential election cycle, which works separately from commissioners' ideological preferences."

Minna Kotkin (Brooklyn) has written Outing Outcomes: An Empirical Study of Confidential Employment Discrimination Settlements.  The Abstract is below the fold:

A Reprise of the Mismatch Argument - Sander 3

Previously: Part 1, Part 2

A comment to Monday's post prompted me to draw up an outline of various claims made in Systemic Analysis, identifying which claims are controversial and the principal critiques. The post that follows summarizes the three central arguments for the mismatch effect -- all of them often ignored by critics.

Students receiving large admissions preferences enter law school with lower credentials (GPA, LSAT) than their fellow students. The central argument in “Systemic Analysis” and “A Reply to Critics” is that these “mismatched” students have worse outcomes – in terms of graduation, bar passage, and the post-school salaries – than if they had gone to a less elite school where their credentials better matched their fellow students. I hypothesize this is because students experiencing mismatch actually learn less than they would otherwise – though that’s just an inference from the data, not something directly observed from experiments.

This mismatch effect is not caused by race. It can affect any group that receives a large preference. (For example, one of my students discovered that older white students – who receive admissions preferences at many law schools – appear to experience mismatch effects.) But it correlates to race because racial minorities are the most common beneficiaries of preferences. It correlates most strongly to blacks as a group simply because in contemporary legal education, blacks receive large and nearly universal preferences (at all but historically minority schools, where there is little evidence of a mismatch effect).

I have advanced three theoretical/empirical arguments so far that support the mismatch conclusion. These are:

"Extreme" and "Non-Extreme" Awards: More on Punies and Comps

Because my Cornell colleagues, Ted Eisenberg, Valerie Hans, and Marty Wells, are all too modest to shamelessly self-promote, the task falls to me to do so on their behalf (with neither their consent nor knowledge).  Their recent paper, The Relation Between Punitive and Compensatory Awards: Combining Extreme Data with the Mass of Awards, is now available on SSRN.  As the excerpted abstract, below, suggests, this paper intrigues for methodological and substantive reasons.  On the methods front, the paper cleverly exploits three separate data sets by integrating them into one. This move facilitates studying the compensatory:punitive damage awards relation throughout a broad range of trial awards.   Substantively, what they find, in part, is that even after accounting for "extreme" punitive awards, compensatory awards still explain the bulk of variation in punitive awards.

"This article assesses the relation between punitive and compensatory damages by combining two data sets of extreme awards with state court data from the National Center for State Courts (NCSC) for 1992, 1996, and 2001. One data set of extreme awards consists of punitive damages awards in excess of $100 million from 1985 through 2003, gathered by Hersch and Viscusi (H-V); the other includes the National Law Journal's (NLJ) annual reports of the 100 largest trial verdicts from 2001 to 2004.... Combining the data sets assists in observing the punitive-compensatory relation throughout the full range of trial awards....  The models indicate that the compensatory award explains more than 50 percent of the variance in the punitive award. We also find no increase in punitive or compensatory awards over time in any of the three data sets."

Are Black/White Disparities in Graduation and the Bar Getting Better, or Worse? - Sander 2

Previously: Introduction - Sander 1

Before delving into the debate on the mismatch effect, it is worth considering a prior question – have the terrible racial disparities documented by the Bar Passage Study improved since the mid-1990s?

The dismal statistics documented by the BPS and explored in Systemic Analysis are fairly familiar now: only 45% of blacks starting law school graduated and passed the bar on the first attempt; only 57% became lawyers after multiple bar attempts (compared to 79% and 83% for whites). 61% of blacks actually taking the bar in the BPS passed on their first attempt, compared to 91% of whites.

These are by far the best hard numbers we have on black outcomes – the only ones based on a large national sample of bar results – but they have a couple of weaknesses. They are ten years old now; and they are tainted to a degree by some creaming of the BPS sample. The overall pass rate of all participants in the BPS was 88%; the actual national average bar passage rate at the time was 82.3%.

There have been some encouraging developments for blacks since the BPS. As I noted in Systemic Analysis, the credentials gap between white and black students narrowed a little between the time of the BPS study and recent years. Fewer blacks thus entered law school with LSATs under 140.

On the other hand, following an argument laid out by Jeff Stake, it may be that law schools have widened their intra-school gaps between the races by focusing more on numbers (especially the LSAT) in admissions (Stake 2006, pp. 309-313). If intra-school race gaps expand and professors teach to the median student, then the "mismatch effect" I hypothesize would be worsened.

So what has actually happened? In the BPS sample, the black graduation rate was about 80.7% (19.3% of blacks did not graduate). The only source I know of for national figures on graduation rates is the ABA. The ABA maintains enrollment and graduation statistics on its website. The ABA data does not track individuals to determine graduation rates; it simply reports enrollment by race in each year of law school, and the number of J.D.s awarded by race. 

If we assume that black students entering in Year 0 will mostly graduate in Year 3, but some will graduate in Year 4 and a few in Year 5, we can estimate graduation rates for particular cohorts. Using that method, I estimated that 80% of the black students entering law school in 1991 earned J.D. degrees – a figure very close to the BPS number. 

Using similar methods for more recent cohorts, I estimate current black graduation rates are 78-79%.  This slight drop – which is paralleled by a slight drop in white graduation rates – may simply reflect the addition of several new schools and a ten percent overall increase in enrollment over the past ten years; the new schools probably tend to be low-prestige and have higher flunk-out rates.

Determining what has happened to black bar passage rates is much harder; the states generally don’t even keep bar passage data by race, much less disclose it. We do know that the national first-time bar passage rate was 75% in 2004. We also have very good bar passage data by race for California, and more fragmentary data from Texas and Colorado. What do these tell us?

Pass rate for first-time bar takers

	California, 2003-05 (ABA schools only)	Texas, 2004	Colorado, 1990-2004
All races	67.3%	81%	81.5%
Blacks	39.9%	53%	52.8%

In all three of these states, the first-time bar passage rate for blacks is about 28 points below the overall rate. Simple interpolation would suggest that if the national first-time bar passage rate is 75%, the national black first-time bar passage rate is close to 47% – unless these three states are somehow unrepresentative of national patterns.

(Recall, BTW, that in Systemic Analysis I argue that about half the black-white gap in bar passage is due to preferences and concomitant mismatch effects. Many studies show that, controlling for credentials and law school grades, blacks do not underperform on the bar.)

If the black first-time bar passage rate is only 47%, that is a large decline from the 61% rate observed in the BPS study. (But recall that the BPS figures were probably somewhat higher than the true 1994 numbers, and partly reflect a real decline over the past decade) Again, a substantial decline in black bar passage rates over the past decade is consistent with the available data.  In Colorado, for example, the overall bar passage rate fell from 76% in 1990-94 to 70% in 2000-04; for blacks, it fell from 42.8% to 34.5% – a larger absolute and relative decline.

Let’s assemble the data so far.  If 78% of black law students in the Class of 2004 successfully graduated from law school, 93% of black graduates took the bar (estimated from BPS data), and only 47% of those who do take the bar pass on the first attempt, then only 34% of the overall cohort graduated and passed the bar on the first attempt. The corresponding numbers for whites, using the same estimation procedures, are 90% graduation, 94% take the bar, and 78% pass on the first attempt – that is, 66% of whites starting law school graduate and pass the bar on their first attempt.

Using the BPS, we can estimate how a particular success rate for a cohort of first-time bar takers translates into ultimate bar passage. Again, the method is crude, but the point of this exercise is to gain some initial insight into where things stand now. A first-time bar pass rate of 47% (my guess of the 2004 black rate) translates into an ultimate pass rate of around 65%. (That is, cohorts in the BPS that had first-time bar passage rates in the 45-50% had ultimate bar passage rates about 18 points higher.) A first-time bar pass rate of 78% (my guess of the 2004 white rate) translates, by the same method, into an ultimate pass rate of around 90%.  This yields the following totals:

	BPS (1994 era)		2004 Era (my estimates)
	Whites	Blacks	Whites	Blacks
% of entering law students who graduate	92%	81%	90%	78%
% of graduates who take the bar	94%	93%	94%	93%
% of bar takers who pass on first attempt	91%	61%	78%	47%
% of bar takers who ultimately pass	96.5%	78%	90%	65%
% of entering law students who graduate and pass bar on first attempt	78.7%	45.1%	66%	34%
% of entering law students who ultimately become lawyers	82.7%	57.1%	76%	47%

Clearly, both whites and blacks are having worse outcomes today than in the BPS – mostly because of the decline in bar passage rates. But what about relative outcomes? In one sense, blacks are doing relatively better; the absolute declines for blacks and whites on first-time bar passage are similar, so the ratio of black-to-white failure rates has fallen from around four to around three.

On the other hand, relative chances of success for blacks have fallen much more sharply than for whites. The proportion of blacks graduating and passing the bar on the first attempt has fallen something like one-fourth ((45.1-34)/45.1) and the proportion of the black cohort becoming lawyers has fallen something more than one-sixth ((57.1-47)/57.1); the comparable declines for whites are one-sixth and one-twelfth.

These are seat-of-the-pants estimates, based on limited available data. But they leave no doubt that the mere passage of time has not cured the problem of racial disparities in legal education. The need to understand the causes of those disparities is more urgent than ever.

Satisfied and Unsatisfied Clients

The theoretical framework for my Law Firms as a Business Organization class is the Heinz-Laumann two-hemisphere theory, which posits that lawyers  fall into one of two distinct "hemispheres": (1) lawyers who serve organizational clients, such as corporations, institutions, and labor unions; or (2) lawyers that provide personal services to individuals and small businesses.   See Chicago Lawyers (1982) (detailed empirical study of ~700 lawyers in greater Chicago).

Jack Heinz (NWU Law/ABF) and Edward Laumann (Chicago Sociology) created the hemisphere typology because their research documented that these two groups had very little in common; they attended different law schools, hailed from different socio-economic and religious backgrounds, belonged to different bar organizations, and otherwise traveled in completely separate social and professional circles.

For the next two weeks, my class is focusing on the personal services hemisphere.  And let's face it--virtually all law professors either worked for, or were groomed to work for, the corporate hemisphere.  We don't have a lot of exposure (at least as working adults) to the gritty world of personal injury, family law, workers compensation, and other mainstays of solo and small firm practice.  Since I teach the Law Firms course, I have compensated over the last few years by attending the Annual ISBA Solo & Small Firm Conferences.   It was there that I first learned about Jay Foonberg, the author of the ABA's all-time best selling book, How to Start and Build a Law Practice (5th ed. 2004).  Literally tens of thousand of small firm lawyers have built highly successful practices by following his simple nuts and bolts advice.

In law school, one gets the impression that success in the legal profession is primarily a function of analytical ability.  But this snippet from an ABA survey, republished in Foonberg's magnificant volume, suggests that client service (a function of organization and emotional intelligence rather than book smarts)  is the real make or break category:

Although the importance of these attributes, upon reflection, is obvious, they are rarely discussed in law school.  Moreover, they reflect a skill set (indeed, an awareness) that many law students have not yet acquired.  Foonberg is a welcomed antidote to this deficiency.

Sander (1) - Introduction

I am grateful to Jason, Bill, and their fellow hosts for inviting me to write a bit this week.

            
The ELS movement is exciting and important.  For thirty years, the legal academy has been taking tentative steps toward the social sciences.  Many of us hope that legal scholars can adopt the rigor and sophistication that characterize the best social science work, while retaining the engagement with judges and legislatures that has traditionally been our strength.  ELS has signified a growing maturity in the academy and a readiness to step closer to legitimate social science status.

The success of legal empiricism partly depends on its ability to help the legal academy – and society more generally – deal with sensitive issues where data has often been trumped by emotion and politics. My “Systemic Analysis” of affirmative action  (mentioned by Bill in his introduction) raises important questions about a significant American social policy. But it raises equally important ethical questions about our academic community.

            
“Systemic Analysis” carefully documented that the use of racial preferences in legal education is far more intense and widespread than previously believed.  It also showed that at individual schools there are enormous gaps in the bar passage rates of blacks and whites. The first of these claims seems to be (quietly) conceded by all; the second is disputed by a few, but shouldn’t be, for reasons I will spell out in the course of the week.   

Advanced Conducting Empirical Legal Scholarship Workshop

For those of you who attended the helpful Empirical Legal Scholarship Workshop geared for beginners or who want more advanced information on doing empirical work, I pleased to make the following announcement:

On Friday, February 9, 2007 through Sunday, February 11, 2007 Lee Epstein and Andrew Martin will be conducting an Advanced Conducting Empirical Legal Scholarship workshop.  This workshop will be held in St. Louis, and is co-sponsored by Washington University and Northwestern University. There is more information available about the workshop here: http://cerl.wustl.edu. The workshop is designed for those who have some experience with empirical legal research, and an understanding of elementary statistics (at the level taught in the introductory workshop). Topics to be covered will include: multiple regression, regression models for limited dependent variables, presenting results from non-linear models, data visualization and graphics, and matching methods for causal inference. All logistical details will be available in mid-October.  In the interim, please feel free to contact Andrew Martin ([email protected]) with substantive questions, or Karma Jenkins ([email protected]) with logistical questions.

New Book - Roosevelt on Judicial Activism

The Myth of Judicial Activism: Making Sense of Supreme Court Decisions

by Kermit Roosevelt (Yale University Press)

Availability: Amazon / B&N / Yale

From the Description: “This carefully considered book is a welcome addition to the debate over ‘judicial activism.’ Constitutional scholar Kermit Roosevelt III offers an elegantly simple way to resolve the heated discord between conservatives, who argue that the Constitution is immutable, and progressives, who insist that it is a living document that must be reinterpreted in new cultural contexts so that its meaning evolves. Roosevelt uses plain language and compelling examples to explain how the Constitution can be both a constant and an organic document.”

Guest Blogger: Richard Sander

This week's guest blogger is Richard Sander, who is an economist and law professor at UCLA School of Law.    As I noted in an earlier post, Sander is best known for his empirical work on racial preferences in legal education (Systemic Analysis of Affirmative Action in American Law Schools, 57 Stan L. Rev. 367 (2004)) and corporate law firms (The Racial Paradox of Corporate Law Firms, 84 N.C. L. Rev. 1755 (2006)). These articles raise the possibility that racial preferences in legal education may have significant, unintended consequences that may work to the detriment of many aspiring minority lawyers.  Obviously, this is a very troubling prospect for those of us who work within the legal academy, especially since most law schools and law professors strongly supported the Supreme Court's decision in Grutter. 

Not surprisingly, over the last two years, Sander has garnered no shortage of critics.  Yet, Richard Sander's earlier scholarship and professional activities belie the perception that his work is motivated by politics or ideology.  During the last 20 years, Sander has been a ardent advocate of fair housing laws and worked on the successful campaign and transition team of Harold Washington, the first black mayor of Chicago.  Since joining the UCLA faculty in 1989, Rick has served as President of the Fair Housing Congress of Southern California, founded the Fair Housing Institute, and helped the City of Los Angeles design and implement what was, at the time, the nation's most ambitious living wage law. He also participated in the effort to develop outreach programs that sharply increased the local use of the Earned Income Tax Credit, generating tens of millions of dollars annually for LA's poorest working families.  In addition to scholarship on housing integration and living wage laws, Sander has done important empirical work on legal education, including effective strategies in academic support programs and how law students perceive the law school experience.

At UCLA, Sander teaches courses in Property, Quantitative Methods, Urban Housing, and Policy Analysis.  I look forward to his postings over the coming week. 

Funding for Empirical Research Projects

As I noted several months ago when I wrote ELS on the Cheap (i.e., without Graduate School), there are many organizations that are willing to fund empirical research on legal topics and legal institutions.  I recently ran across this excellent list of potential funders, which was compiled by the Charles B. Sears Law Library at SUNY--Buffalo. 

Political Science Data Directory

A terrific benefit of the ELS Blog is that it provides links to a great many databases that are of interest to scholars doing empirical research on legal issues.  One very useful site that is not yet linked on the ELS Blog, however, is the Data Directory of the Law and Courts Section of the American Political Science Association.

This site, which can be accessed here, contains links to numerous data holdings, including online data resources, a number of important databases, and several public domain datasets that are available for the replication and extension of published research. Unlike other sites that focus on a particular topic or dataset, this site enables researchers to access a wide variety of information and data.  It is well worth browsing.

The Gender Wage Gap: Spouses with "Intense Jobs"

Here at Indiana Law, I teach a course called The Law Firm as a Business Organization. In a nutshell, the course provides students with an overview of the historical, economic, and sociological factors that have shaped—and continue to shape—the modern legal marketplace.   

This year, thanks to the dedication and hardwork of my colleagues, Ken Dau-Schmidt and Jeff Stake, my course includes extensive data on Indiana Law graduates; in other words, current IU students get to study the career trajectories of IU alumni.  This has been an interesting experience as both a teacher and a researcher.  The first study, which was explicitly designed to facilitate comparisons with the famous Michigan alumni dataset, is now available on SSRN: "The Pride of Indiana": An Empirical Study of the Law School Experience and Careers of Indiana University Law Alumni. (Ken and Jeff will also be presenting these results at the CELS conference in Oct.)

As we moved through some of the data last week, I noted some interesting breakdowns associated with the variable "% Spouse has intense job."   This item reflects the percentage of respondents who are married to professionals with demanding jobs (e.g., other lawyers, doctors, business executives).  Here is a summary of the "intense job" data along with some relevant additional variables:

As shown in the table above [click to enlarge], male and female Indiana Law graduates in both the five- and fifteen-year cohorts are fairly comparable on some dimensions (shown in blue): % married, size of families, % of spouses who are homemakers, and total household income.  But there are consistent statistically significant differences along three dimensions (shown in green):

Where's the Theory?

Yesterday, I asked readers to consider the relative merits of aggregate v. individual level analysis. Regardless of the empirical approach, our analyses must be framed by theories about individual behavior. To the extent that we are concerned about developing an understanding of the choices of either individuals or organizations, we need to make certain explicit assumptions about what motivates choices.

Interestingly enough, it is here where I find most of our empirical work to be especially lacking. Despite our ability to generate a great many interesting empirical insights, we seem not to be as concerned as scholars in other disciplines with laying the building blocks that support our expectations.

I do think that we know a great deal about, say, the behavior of judges, especially appellate judges in the United States. Nevertheless, I’m not sure that we have anything like a “theory of judging.” Several years ago, Larry Baum wrote a wonderful book, The Puzzle of Judicial Behavior, in which he described what the various motives of judges might be and how their personal goals and professional goals might interact with what judges can do in different institutional settings. One of his conclusions is that we need to be more cautious about imputing motives to judges.

What are the consequences of our collective inattention to theory? If we simply want to predict behavior, then the ill-effects are not really that severe. If we want to understand and explain that behavior, then I think we should be more guarded. We may misdiagnose much of what we observe.

Perhaps one reason for our lack of theory is that, since many of us actually are what we study -- i.e., we are often lawyers studying the behavior of lawyers -- I suspect that the kind of formal training and informal socialization that takes place within the legal community leads to bit of overconfidence; that is, since “we know how lawyers really think and behave,” our empirical work can proceed according to long-internalized assumptions. Those assumptions may be quite correct, but they may also fail to consider other plausible options.

Imagine, for example, studies of congressional behavior that were authored by members of Congress. On the one hand, I would take very seriously what those analyses had to say, since they would be informed by the perspective of individuals who possessed a detailed knowledge of both the mechanics of legislative institutions and the various incentives and constraints that they face in making difficult decisions. On the other hand, I would almost certainly be circumspect about the results, since they would be based upon assumptions that were heavily informed by the personal experiences and biases that necessarily come with it.

Now, I am not a lawyer, and that surely limits what I am able to do. In fact, I’ve been humbled on several occasions by lawyer friends who have pointed out the limitations of some of my work by drawing upon a perspective that I simply don’t have. And this has saved me from embarrassing errors I might otherwise have made. At the same time, those of us who are not lawyers may have a more ready knack for generating a wider range of assumptions -- some plausible, some less so -- that might account for what we observe.

Despite this flexibility, none of us has had any greater success in specifying a general theory of why judges do what they do. To be sure, we often produce research with provocative and surprising results, but it is not at all clear that we share a basic core of ideas about the choices judges make. As a result, we probably have a harder time reconciling disparate findings or sorting out the relative importance of the causes that we think are at work.

I’m certainly not prepared to propose any general theories about judicial behavior, but I do think that we should exercise greater care and be more explicit about the assumptions that go into our empirical work. It will surely improve the discourse between various communities who study the law and enhance our understanding.

Lumpers v. Splitters

One of the interesting questions of empirical analysis is whether to approach a research question from a macro-level or a micro-level perspective. Should we aggregate data, ignore relatively small differences, and look for the big picture? Or should we look at the individual level, take account of particulars, and try to leverage the differences between cases in order to increase the sophistication of our explanations?

The approach one takes, I suspect, has a lot do with the type of question one hopes to address, but I’m particularly interested in the tradeoffs that exist between the two for those of us who are interested in formulating an empirical view of law and courts. Is one approach inherently preferable for what we study? Does one approach make quantification more problematic?

My impression is that those of us who do empirical work on the law have, on balance, been lumpers more often than splitters. Certainly my fellow political scientists do lots of convenient aggregating of data -- across judges, courts, states, and years, for example -- to test various notions about the intersection between law and politics. As a consequence, I think we have some very good evidence relating to, among other things, judicial decision making and the relationship between courts and other political actors.

In treating courts as political institutions, however, we are sometimes criticized for failing to consider more specific legal factors, such as doctrinal tests, legal customs, methods of statutory interpretation, and other variables that are regarded as significant for explaining what courts do. The consistency of these criticisms should, I think, give us some pause.

Given the growth of empirical legal scholarship among people with fairly diverse training and orientations, I am interested to see how both lumping and splitting will be brought to bear to offer some new insights. For instance, I can imagine that splitters, who see the need to test the kinds of individual differences that lumpers often ignore, will end up devising refined explanatory models with which the lumpers will have to come to terms. Indeed, an interesting body of work is already developing along those lines.

Obviously, I’ve oversimplified things a good deal, but my point, as I start off this week, is to ask others to think about which analytic approach will yield the biggest benefits and when. Which questions of law and courts are better handled from the macro-perspective, and which are more profitably addressed at the micro-level?

My thanks to Jason Czarnezki for inviting me to serve as a guest-blogger.  I look forward to participating throughout the week.

Guest Blogger: Kevin McGuire

This week's guest blogger is Kevin McGuire, an Associate Professor in Political Science at North Carolina at Chapel Hill.  Kevin teaches courses on the U.S. Supreme  Court and American constitutional law.  His current research interests include the influence of public opinion on the justices’ decisions, the content analysis of the Court’s written opinions, measuring the ideology of the Court’s case outcomes over time, and compliance with judicial policy.

Empirical Work on Transaction Costs

The Coase theorem holds that in the absence of transaction costs, all allocations of property are equally efficient because affected parties will privately bargain to correct any externality.  The familiar rejoinder by people who don't like law and economics is that transaction costs are never zero--but that was exactly Coase's point.  Since transaction costs distort or shape virtually all outcomes, effective social policy depends upon our understanding of real world contexts.  In short, the natural upshot of the Coase theorem is a comprehensive empirical research agenda on a wide range of legal and nonlegal disputes--enough work to keep several generations of researchers busy.  (Granted, those of us without a Nobel Prize under our belt have been somewhat slow to catch on.)

With this understanding of Coase in mind, this paper by Barak Richman (Duke Law) and Jeffrey Macher (Georgetown Business) caught my eye:  Transaction Costs Economics: An Assessment of Empirical Research in the Social Sciences.  Here is the SSRN abstract:

This paper provides a comprehensive review of the empirical literature in transaction cost economics (TCE) across multiple social science disciplines and business fields. We show how TCE has branched out from its economic roots to examine empirical phenomena in several other areas. We find TCE is increasingly being applied not only to business-related fields such as accounting, finance, marketing, and organizational theory, but also to areas outside of business including political science, law, public policy, and agriculture and health. With few exceptions, however, the use of TCE reasoning to inform empirical research in these areas is piecemeal. We find that there is considerable support of many of the central tenets of TCE, but we also observe a number of lingering theoretical and empirical issues that need to be addressed. We conclude by discussing the implications of these issues and outlining directions for future theoretical and empirical work.

Information Technology for Data Collection

Many of us doing empirical legal research are involved in data collection projects of various types. Until very recently, my research assistants and I wasted a tremendous amount of time duplicating work and coordinating amongst ourselves. Recently I have learned -- from some of our excellent IT staff and some savvy graduate students -- about useful systems we all can use to manage our data collection projects.

The traditional approach is to hire teams of law or graduate students to collect and and code data, perhaps reading cases and coding a number of variables. What I had done in the past is to have the students enter data in either Excel or Stata, and then return their portion of their data to me when complete. Then, a research assistant would create a master dataset from all of the various sources. This approach is highly inefficient, and provides many opportunities for human error.

A better way to perform this sort of data collection is to create a web-based data collection portal that research assistants can use to input coded data. One advantage of this is the ability for multiple research assistants located anywhere to read from and write to the same data set. Systems can also be built to allow for various levels of integrity checks, supervision, etc. And, when complete, it is straightforward to build systems to deliver data in whatever form you need.

Here at Wash U we have built a system for a project my colleague Margo Schlanger has undertaken called the Civil Rights Litigation Clearinghouse. You'll be hearing about the substance of this exciting project on this blog and elsewhere starting in about a month. A crucial facet of this project was creating an information system to catalog and subsequently disseminate a diverse collection of case documents. And, germane to this post, to allow research assistants and the data owner, Margo, to code various aspects of the cases, documents, and court officials.

Below are some screenshots of the administrative back-end of this site. Using this environment, research assistants enter data about a case using actual court documents collected by the data owner. In this case, entered data can include the case name, the document title, the document type, citation etc.

click to enlarge

The data owner plays a governing role in overseeing the work done by their research assistants. It is important that a workflow and auditing system is present so information entered into the collection may be reviewed and modified where necessary. In the below screen the data owner views the status of an existing case. Here the case summary is in progress; the approver may edit what the student wrote and when ready approve the changes. Such measures allow data to be phased through an identified process and allow the public display of items to be granularly controlled.

click to enlarge

This custom information system has made the full maturation of this very ambitious data collection project feasible. Without a consistent and stable method of collection and interpretation, large data sets often get obfuscated and disjointed, thus rendering them difficult if not impossible to share in any meaningful manner.

Obviously planning for and organizing such an intricate piece of technology requires the right sort of collaboration. It is vital that appropriate subject matter experts are consulted as to avoid missteps in the planning, which is so often the most crucial step to any technology undertaking.

In this case, the base-technologies were quite standard; apache webserver, php and a mysql back-end. In addition to the hard technologies, it is helpful to have people skilled in information architecture, site design, web usability, database management, and obviously web development. And this is of course all in addition to the data owner and their collection, without which, the endeavor would not be necessary.

Another common task the people frequently undertake is harvesting data from various websites. This can be done by humans using point-click-copy-paste. But that, too, is inefficient and opens up lots of opportunities for human error. It also makes it difficult to replicate studies conducted by others, and to update existing datasets. There are a variety of scripting languages, including perl (using LWP::UserAgent for fetching and HTTP::Request::Common for posting forms) and python, that can be used to harvest data from websites and turn it into useable datasets. The Snoopy php module also works well for these purposes.

Using these technologies is more expensive in terms of time and resources at the start-up phase of a research project. But the efficiencies that can be captured are quite significant.

What Makes a Paper "Interesting"?

Over the years I've sat on a number of search committees in political science departments and other departments in the college. This year I'm serving, for the first time, on the entry level appointments committee in the Law School. One conversation that oftentimes comes up in our discussions is whether a paper is "interesting" or not.

What do people in legal academia view as "interesting" work. I, personally, have standards I use to judge whether a paper is of theoretical or substantive importance as relates to the social scientific literature. And, editors always ask me to assess the research design, data, and methods of any piece I'm refereeing. But I'm having a hard time figuring out what makes a piece, whether empirical, theoretical, doctrinal, or something else "interesting" to the law review audience. Thoughts?

Warren Buffett as a Data Point

Shortly after Warren Buffett announced that he was giving away the vast bulk of his estate to the Bill & Melinda Gates Foundation, I heard a story on NPR that interviewed Buffett's granddaughter, Nicole Buffett.   Nicole recounted five specific pieces of advice that she received from her grandparents; and over the last several months I have wondered whether they are indeed the formula for a successful life (however one might define success). 

I ultimately concluded that the only way to empirically ascertain the benefits of these sayings was to follow them with earnest for a long period of time (many years, if not decades).   I passed along these sayings to my students on the first day of Corporations, so why not to ELS readers?  Here they are:

1. "Show up"
2. "Tell the truth"
3. "Pay attention"
4. "Do your best"
5. "Let go of outcomes"

I have found that if I do 1-4, #5 is much less difficult.    I may be able to provide more comprehensive data in a few years.

Measuring Case Complexity

Thanks a lot to Jason and the other editors for inviting me to blog this week. I plan to spend my time here tapping the collective wisdom of the blogosphere about what I view as a number of important questions. I'll also spend some time blogging about various tools we should all be using in our empirical work.

I'll start with this. One of things political scientists oftentimes include in statistical models of judicial decision making is a measure of "case complexity." In the Supreme Court context, the standard measure, as coded by Harold Spaeth, is to label a case "complex" if there was more than one legal issue or provision at play in the case. This may be not be an accurate measure of complexity. Many cases involve, say, a first amendment free speech issue as well as a due process or establishment of religion issue. Is this more complex than a tax case? Securities? Telecommunications? Surely not.

Any thoughts about how to systematically measure case complexity? Perhaps looking at the number of issues cited or the level of factual complexity might help. How could we do better?

A Return of the Wigmore Trial and Error Approach?

On Friday, law.com posted a story on a recently filed lawsuit against St. Thomas School of Law in Miami.   The plaintiff, a former student who is seeking to certify the suit as a class action, alleges that several St. Thomas administators violated federal anti-racketeering laws by admitting a disproportionately large 1L class with the intention of flunking out approximately 25 percent the following year (students with less than a 2.5 GPA).   

The school purportedly benefitted from this scheme by (a) collecting first year tuition from more 1L students, and (b) improving its anticipated bar passage rate for feature classes because students with low 1L grades are much more likely to struggle on the bar.   A St. Thomas official quoted in the story disputes the high attrition rate (only 40 of 330 1L's were dismissed) and called the revenue theory "illogical" because early dismissals reduce the school's income.

Setting aside the speculative nature of the lawsuit (and note that the ABA Section of Legal Education and Admission to the Bar has been named as a defendant on a negligent supervision theory), such a scheme would deliver clear economic benefits to many lower ranked schools.

Specifically, St. Thomas is a non-prestigious law school operating in Florida, which has lots of other law schools.   According data published in the ABA-LSAC Official Guide, the bottom quartile of entering class at St. Thomas is--relatively speaking--academically weak:   147 LSAT (12th lowest out of 190 schools) and 2.58 UGPA (lowest of out 190 schools).  Although these number bode poorly for bar passage statistics, this outcome can be mitigated by flunking out the bottom of the class after the 1L year.  After all, law school performance is a better predictor of bar passage than an LSAT/UGPA admission index.    Higher bar passage statistics could increase U.S. News rankings and applicant volume, thus setting in motion a virtuous upward cycle.  Moreover, the plan becomes revenue-neutral if the 1L entering class is significantly increased--a fact alleged by the St. Thomas lawsuit.

This system represents a return to the past, albeit the economic drivers are quite different.  As the modern law school hierarchy took shape in the second half of the twentieth century, one of the achievements celebrated by law school administrators was the very low 1L attrition.  The reason was simple; the LSAT had effectively identified those students who lacked the aptitude to become lawyers.  Prior to the LSAT, law schools relied upon the Wigmore method of trial and error, which winnowed classes based on first year exams.  See John H. Wigmore, Juristic Psychopoyemetrology--or How to Find Out Whether a Boy Has the Makings of a Lawyer, 24 Ill. L. Rev. 454, 463 (1929). 

Unfortunately, the Wigmore system seems destine to rise again as the number of ABA-approved law school continues to climb (+12 since the first complete U.S. News rankings in 1992), the applicant pool remains relatively flat or on the decline, and states continue to increase bar cut scores (at least a dozen states have done so in the last 15 years).  Indeed, I have heard of second-tier law schools meddling with 1L cut-offs in order to improve bar passage results.   Two observations:

1. Where is the disclosure?  One of the dangers here is that law schools don't bear the cost;  rather, it is borne by former students who incurred the time and expense of attending one year of law school before getting the boot.  At a minimum, academic attrition should be part of the data collected and disclosed by the ABA on an annual basis.  If the market is going to sort out this significant risk, it needs adequate information.
2. Can curricular innovation improve bar passage? An outsider to legal education might ask why curricular innovation is not the first line of defense for low bar passage.  Actually, there is data that this approach might actually play large dividends for struggling law students.  See Jellum & Reeves, Cool Data on a Hot Issue: Empirical Evidence that a Law School Bar Support Program Enhances Bar Performance, 5 Nev. L. J. 646 (2005) (exerpt reprinted in The Bar Examiner).

Perhaps the St. Thomas lawsuit will serve a catalyst for these two issues.

Guest Blogger: Andrew Martin

This week's guest blogger is Andrew Martin, Professor of Law and Professor of Political Science at Washington University in St. Louis.  Andrew is the founding Director of the Center for Empirical Research in the Law, and also directs the interdisciplinary Applied Statistics and Computation program.  Andrew specializes in political methodology, Bayesian statistics, and American political institutions. Much of his substantive research focuses on law and courts, particularly the Supreme Court and the Courts of Appeals.