Empirical Legal Studies

Spotlighting Student ELS Work

Advanced students increasingly display an appetite and aptitude for empirical legal scholarship. A recent paper by David K. Kessler (Harvard Law School, class of 2009), forthcoming in The Journal of Criminal Law and Criminology (Jan. 2009), illustrates this trend and presents results from an empirical test of the Supreme Court's standard for identifying a seizure under the Fourth Amendment.

Drawing on a survey of 406 randomly selected Boston residents, Kessler concludes that "people would not feel free to end their encounters with police. By the Court's standard, respondents would be seized in both scenarios.... The data also show that knowledge of one's legal right to end the encounter with the police would not make people feel free to leave, and that women and people under twenty-five would feel less free than would men and people over twenty-five. This initial empirical evidence suggests the need to re-think the current seizure standard."

The Limitations of the Spaeth Supreme Court Database

I have posted on SSRN my article, forthcoming in Hastings Law Journal, Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court.  This article examines the well-known and widely-used U.S. Supreme Court Database (created by Harold Spaeth) – and most recently mentioned here –  and addresses the Database’s limitations particularly for those interested in law and legal doctrine.
The key point of the Article is that the Database does not contain complete or accurate information about law and legal doctrine as they appear in Supreme Court opinions.  Given Harold Spaeth’s own purposes in creating the Database, these limitations may not be surprising -- although they do raise at least some challenges to his attitudinal model.  Unfortunately, however, they are frequently misunderstood.  Scholars all too frequently use the Database in ways that it simply cannot support, leading to the possibility of invalid or unreliable results.  This post summarizes the Article’s main arguments. 
The primary challenges presented by the Database involve the coding for the “issue,” “issue area,” and “legal provision” variables.  As the names of these variables suggest, they are frequently used by researchers interested in studying law and legal doctrine.  Yet, the coding protocols for these variables (as set forth in the Codebook are not conducive to such research.  Some of the limitations of these variables include:
(A) The “issue” variable is not, despite its name, designed to identify any legal issues in a case.  Rather, it is designed to identify the “public policy context” of a case.  A case like Schenck v. Pro-Choice Network of Western N.Y. is one example.  In Schenck, a group of abortion protesters challenged an injunction limiting their activities as violating the First Amendment.  The only legal issue in the case involves the First Amendment and the limits it places on judicial power.  But the Database codes the case as having an issue of “abortion” because that is the factual, or “public policy” context in which the case arises.
(B) The coding contains a strong presumption of assigning each case only a single issue.  So the Database does not add a First Amendment issue code to the coding of Schenck.
(C) The issue codes are quite underinclusive and somewhat dated.  For example, there are no codes for immunities, for sexual harassment, or for the dormant commerce clause.
(D) Each of the approximately 260 issue codes is classified into one, and only one, of 13 “issue areas.”  In some cases, the classification makes no sense.  For example, in Markman v. Westview Instruments, Inc., the Court addressed the question of whether patent claims construction is a question for the judge or the jury; that is, whether there is a 7th Amendment jury right.  The Database classifies Markman as a case about the right to a jury trial, but that code, which does not distinguish between civil and criminal jury rights, is located in the Criminal Procedure issue area.
(E) The legal provision code does not identify cases or judge-made legal doctrines.  It is limited to identifying statutes, constitutional provisions, and court rules.
(F) The coding protocols provide that only legal provisions mentioned in a case’s syllabus should be identified.  But the syllabus – a short summary of the case – is akin to headnotes.  It is not officially part of the case, it is not written by the justices or their law clerks, and it cannot be cited by lawyers or judges.
To some extent, misuses of the Database are likely due to differences in the ways that different disciplines (political science and law) use the same words.  To some extent, misuses stem from scholars failing to evaluate their research design in light of the Database’s coding protocols, which are described in the Database’s Codebook.  In my Article, I provide a series of examples of research project that fail to adequately take account of the Database’s limitations and that therefore produce results that may be inaccurate.
To further explore the limitations of the Database and to experiment with more legally nuanced types of coding, I undertook a Recoding Project of a random sample of 10% of the cases from the last Rehnquist natural court.  The details of the coding project are, of course, explained in the Article.  Among other things, I redefined “issue” to mean legal issue, I expanded and rearranged the lists of issues and issue areas, I put no limit on the number of issues that could be coded per case, I redefined legal provision to include seminal cases and legal doctrines, and I identified legal provisions by looking at the opinions themselves, not just the syllabi.
Some of the key findings of the Recoding Project include:
(1) I identified an average of 3.7 issues and 2.4 issue areas per case, rather than the single issue and issue area per case identified in the Database.
(2) I identified an average of 2.2 as many legal provisions per case as did the original Database.
(3) A surprising number of legal provisions that I identified should have been identified in the Database because they were mentioned in the syllabi.
(4) In both issue and legal provision coding, the “missing” codes – those that I identified but that the Database did not – disproportionately related to structural and jurisprudential issues, including procedure, the powers and operations of the federal and state governments, and the relationship between different branches of government.
These and other findings have a variety of implications for researchers working with the Database.  Chief among these is the importance of not drawing conclusions about the Supreme Court’s cases by looking at the numbers and types of issues, issue areas, and legal provisions coded.  Researchers all too often rely on such information to draw conclusions about case complexity or about the number of issue “dimensions” in the cases.  In other words, researchers sometimes point to the Database to justify their assumptions that most Supreme Court cases involve only a single issue.  But as the Article demonstrates, this single-issue coding is -- or at least may well be -- an artifact of a coding protocol that presumes that each case should be assigned only a single issue, so such conclusions are circular.  A second important implication is that the Database’s issues and issue areas do not accurately identify all cases involving particular legal issues and that not all cases with a particular issue or issue area code in fact involve the legal issues that a researcher might presume from the names of those codes.

Michigan's New Admissions Policy

Over at Moneylaw, rankings guru Tom Bell (Chapman) breaks an important story:  The University of Michigan Law School has introduced the new Wolverine Scholars program, which is a special admissions program for UM undergraduates.   It has two threshold requirements:  (1) your UGPA must be at least 3.80, and (2) you cannot have taken the LSAT. 

Studying the labor market for lawyers--including law school rankings, which is a important mechanism for mediating supply and demand--has made me sensitive to potential ulterior motives beyond the standard cant of "admitting the best students."   While I would love to see a world where the LSAT is assigned is its rightful limited place in law school admissions, I cannot overcome the perception that Michigan is really just upping the U.S. News gamesmanship.  The lofty rhetoric of the Wolverine Scholar program cannot be squared with the unnecessarily rigid admissions criteria.  In my opinion, the only rational explanation is that Michigan seeks a rankings payoff.   Here, an elite law school sets a new low in our obsession of form over substances--once again, we legal educators are setting a poor example for our students. 

Readers can judge for themselves.  Here is how Michigan pitches the Wolverine Scholars program:

The Law School’s in-depth familiarity with Michigan undergrad curricula and faculty, coupled with significant historic data for assessing the potential performance of Michigan undergrads at the Law School, will allow us to perform an intensive review of the undergraduate curriculum of applicants, even beyond the typical close scrutiny we devote ... . For this select group of qualified applicants, therefore, we will omit our usual requirement that applicants submit an LSAT score.

With the exception of the intensified review of the undergraduate academic record and the omission of the LSAT, our admissions review and philosophy is the same for the Wolverine Scholars Program as it is generally. Thus, our evaluation criteria are holistic, and comprise the wide range of relevant considerations that can’t be reduced to any mechanical formula. We look for highly intelligent people who welcome challenging experiences, who have demonstrated leadership and community service, who have shown determination and discipline, who are eager to outdo themselves, and who are creative and resilient in dealing with adversity. We pay attention to evidence of academic progress. So, too, we pay attention to considerations – working many hours, coming from an educationally deprived background, having primary care responsibilities for family members, and so on – that may provide a context for the formal record of academic achievement. ... We look for individuals with intriguingly different backgrounds, experiences, goals, and perspectives. Academic majors, work experience, extracurricular activities, distinctive moral and political outlooks, socioeconomic background, time living or working abroad, and more inform our admissions decisions.

Wonderful stuff.   But it is impossible to take the above excerpt seriously when we focus on the unreal application cutoff:   

"Application Eligibility.  UM undergraduates who have at least completed their junior year ... with at least six full-time semesters of attendance on the UM-Ann Arbor campus and a UM cumulative grade point average of ≥ 3.80 are eligible to apply." [Emphasis in the original -- yes the original!]

So, a priori, the following student need not apply:  a UM chemistry or engineering major with a 3.75 GPA who comes from a blue collar family in the rural Michigan who put himself through college by running a business and volunteers in an organization for the disabled.   For this person, "holistic review" requires an LSAT score.   He or she may not be one of the "highly intelligent people" UM is looking for.  But the privileged frat boy who majored in political science and earned a 3.81 gets a free pass.  It is hard to imagine a more misguided "mechanical formula."

If Michigan was being serious, they would set the benchmark at 3.4 or 3.5  and actually conduct a true holistic review.  So why didn't they?  One logical possibility:  if the true motive is a rankings grab (i.e., a "mechanical formula") toward the 3.8 crowd, a FOIA request could lay it bare.  Yet, if you set the program at 3.8 and admit 90%, few people can quibble with the merit criteria--they are all capable candidates. 

The rankings motive is further corroborated by the disqualification if the potential Wolverine Scholar has taken the LSAT.  If Michigan were just interested in the best people, it would just ignore the LSAT.  After all, Michigan has a better predictor of success:  the "significant historic data for assessing the potential performance of Michigan undergrads at the Law School."  If holistic review is really taking place, excluding folks who have taken the LSAT has no rational basis beyond a hit in the rankings.  Again, if these folks get into the applicant pool, a FOIA request would expose any selection patterns that looks like the "mechanical formula" the Law School has allegedly disavowed.

More seriously, there are terrible externalities from this alleged merit-based program.   It is impossible to deny that the Wolverine Scholars program will encourage students to (a) take easier classes and majors to avoid the need to take the LSAT to get into an elite law school, (b) discourage extracurriculars that will threaten the 3.8, and (c) make a lot of Michigan undergraduate professors miserable with complaints from students that their B+ or A- grade is going to blow their Wolverine Scholar application.   

From a rankings perspective, what happens when you get 20, 30, or 40 candidates with 3.8+ UPGA and no LSAT score?  From day 1 of admissions season, Michigan has much greater latitude to lock in higher median LSAT and UPGA numbers--because zero Wolverine Scholars are dragging down the LSAT and all are helping the UPGA numbers.   Further, because of the idiosyncrasies of the USNWR rankings formula, see Ted Seto's "Understanding the U.S. News Law School Rankings," at the upper ranges, small changes in UGPA have a much greater sway on rankings that a single LSAT point.  For example, in the simulation model that Andy Morriss and I created, a move from 3.64 to 3.66 has a greater effect than a move from 169 to 170.   If Michigan can get to a 3.80 UGPA, they could tie with NYU at #5.

A final point is the ABA accreditation requirement, which requires an admissions test:

Standard 503. ADMISSION TEST

A law school shall require each applicant for admission as a first year J.D. student to take a valid and reliable admission test to assist the school and the applicant in assessing the applicant’s capability of satisfactorily completing the school’s educational program. In making admissions decisions, a law school shall use the test results in a manner that is consistent with the current guidelines regarding proper use of the test results provided by the agency that developed the test.

Interpretation 503-1

A law school that uses an admission test other than the Law School Admission Test sponsored by the Law School Admission Council shall establish that such other test is a valid and reliable test to assist the school in assessing an applicant’s capability to satisfactorily complete the school’s educational program.

The Wolverine Scholar program requires a waiver to access a student's SAT and/or ACT scores.  I assume that is how Michigan would proffer these test scores as a "valid and reliable test" for determining whether a student can complete the Michigan JD program.  Why not?  Northwestern's JD/MBA programs accepts the GMAT as substitute for the LSAT, and it passed muster during NW Law's recent re-accreditation.   I doubt the ABA will try to draw a line here (and for the record, I don't think this is battle worth fighting--ignoring the LSAT for UM undergrads who have a 3.8 is not going to produce bad educational outcomes).

Spaeth's S.Ct. Coding (07 Term)

Harold Spaeth has completed his coding for the Supreme Court's 07 Term. The dataset ("allcourt") is available for public download at the University of South Carolina Judicial Research Initiative (JuRI). To access the dataset please visit www.cas.sc.edu/poli/juri.

The Lingering Tulane Law Review Mess

Tulane Law School, Tulane Law Review, and others remain stuck in an evolving mess stemming from the Tulane Law Review's recent publication of a flawed empirical study.  Eugene Volokh (UCLA) and Dave Hoffman (Temple) have comprehensive summary posts here and here, respectively.

Try To Get This Study Through Your University's IRB

Saw this in Time magazine and immediately thought about, among other things, challenges incident to research design and IRB.

Call for Papers -- 2009 Midwest Political Science Ass'n Annual Meeting

Jeff Segal (SUNY--Stony Brook, Poli Sci) reminded me about the following Call for Papers.

"Ask your favorite political scientist what the biggest political science conference is, and she'll tell you it's the American Political Science Association. Ask her what the best political science conference is, particularly for empirical work, and she'll tell you it's the Midwest Political Science Association meeting, held every April in the beautiful Palmer House in Chicago.

The Midwest Political Science Association, like most academic associations, charges higher conference registration rates for nonmembers than to members. Hoping to continue to increase attendance by people outside of political science and related fields at its annual meeting, the Association will begin charging registrants who 1) have academic appointments outside of political science or related fields (policy, public administration and political economy) and 2) do not have a PhD in political science or the same related fields, conference registration rates equal to member registration rates.

 In addition, the Association grants, on request, a substantial number of conference registration waivers for first time participants who are outside the discipline.

 The call for papers for the 2009 meeting, due October 10, is at http://www.mpsanet.org/~mpsa/index.html.

 Great seeing everyone at CELS, and hope to see you in Chicago."

More on CELS: Cognitive Illiberalism

Dave Hoffman over at Concurring Opinions wasn't the only blogger at CELS in Ithaca.  Although he and I apparently went to different panels, like him, I thought the conference was excellent and the quality of the papers and discussion extremely high.  Congratulations and thanks to all of the conference organizers.

One particularly interesting paper was, coincidentally (or not) co-authored by Dave Hoffman, and was previously blogged about here.  The paper, Whose Eyes Are You Going to Believe? Scott v. Harris and the Perils of Cognitive Illiberalism by Dan Kahan, David Hoffman, and Donald Braman (forthcoming in Harvard Law Review), takes advantage of a unique experiment made possible by modern technology.  In Scott v. Harris, the Supreme Court addressed whether summary judgment was appropriate in a claim of excessive force where a police officer rammed his car into the car of the fleeing suspect, who was rendred quadriplegic.  The Court held that the use of deadly force was reasonable under the circumstances, given the risk that the car chase posed to the public.  The Court rested its conclusion on the contents of a videotape, shot from the police car itself, that was entered into evidence and that the Court posted on its website.   Interestingly, however, despite the fact that the Court said that no reasonable juror could find the use of force excessive, one Supreme Court justice -- Justice Stevens -- concluded otherwise.

Taking advantage of the now publicly available videotape, the paper's authors showed the video to 1350 Americans.  As they explain, "a majority agreed with the Court's resolution of the key issues, but within the sample there were sharp differences of opinion along cultural, ideological, and other lines.  We attribute these divisions to the psychological disposition of individuals to resolve disputed facts in a manner supportive of their group identities."  So individuals who tend to see the world hierarchically (demographically more likely to be white, male, and from the South or West), were more likely to agree with the Court majority than were individuals who take a more egalitarian perspective (demographically more likely to be nonwhite, female, and from the Northeast).

Normatively, these observations suggest that judges should, at a minimum, be cautious about the claims they make about what a "reasonable juror" could conclude.  (Indeed, as the discussant, Neal Feigenson, pointed out, even if all members of the jury were completely average across all of the dimensions identified by the authors, there would still be a significant probability that at least one juror would believe that the police used excessive force. )  Assuming that one's own views are the only reasonable views, which is essentially what the majority did, "invested [the Court's] decision with culturally partisan overtones that detracted from the decision's legitimacy."  As the authors point out, when different sorts of people have predictably different perspectives, deliberation is particularly appropriate.

This project suggests an interesting take on whether the standard for judgment as a matter of law -- which is, the Court has emphasized, the same is the standard for summary judgment -- should be different.  Perhaps once a jury has in fact heard all the evidence, it should be allowed to render a verdict, and the fact of that verdict should help to inform the judge's ruling on the JML motion.  This comes up often in employment cases, where JML is often sought -- and apparently disproportionately granted -- by employer-defendants.  (See here and here for more of my thoughts on courts' overzealousness in granting summary judgment and judgment as a matter of law in employment cases.)

More on CELS "III"

More on CELS "III" from Dave Hoffman (Temple) (via Concurring Opinions) here.

Academics on the Attack (or "Vigorous Contests") at CELS

Dave Hoffman (Temple) provides some real-time blogging from CELS at Concurring Opinions. The post of interest involves, in Dave's apt words, "one of the more vigorously contested academic panels I've ever seen." Dave's post (and comment) speaks comprehensively for itself and is found here.

CELS 2008

After much hard work and preparation, it is great to see many ELS folk begin to gather at Cornell for this year's CELS.  A copy of the final program is here. For those who like to plan ahead, USC will host the 2009 CELS on Nov. 20-21, 2009.

Call For Papers -- International Conference on Empirical Legal Studies

My colleague, Ted Eisenberg (Cornell), and Talia Fisher (Tel Aviv University) are calling for papers for an international conference on empirical legal studies scheduled for 26 March 2009, at Tel Aviv University Law School in Israel. The conference will include three sessions. One session will focus on empirical analysis of criminal law and criminal procedure. Another session will dwell on empirical research on civil law and procedure. The final session will be devoted to methodological and conceptual issues regarding quantitative analysis and empirical research. All conference sessions will be in English.

For those interested in submitting papers for consideration, the deadline for paper submissions is 1 December 2008. Papers should be submitted to either:

[email protected]  or  [email protected]

Authors of accepted papers will be notified by 10 January 2009. The conference is open to all who wish to attend.

Trends in Civil Rights Litigation, 1990-2006

A new BJS report, Civil Rights Complaints in U.S. District Courts, 1990-2006, examines civil rights claims involving employment, housing, welfare, voting, or other civil rights issues litigated in U.S. district courts from 1990 - 2006. The study assesses trends in types of civil rights cases filed in federal district courts, the basis of federal court jurisdiction, case processing time, disposition of civil rights cases, and the types of trials that occur in the federal courts. In addition, the report examines who wins in civil rights trials and the estimated median monetary amount awarded to litigants. Notable findings include:

• Civil rights filings doubled in U.S. district courts from 1990 (18,922 filings) to 1997 (43,278 filings) and subsequently stabilized until 2003. From 2003 through 2006, the number of civil rights cases filed in U.S. district courts declined by 20%.
• During the period from 1990 through 2006, the percentage of civil rights cases concluded by trial declined from 8% to 3%.
• From 2000 to 2006 plaintiffs won just under a third of civil rights trials on average, and the median damage awards for plaintiffs who won in civil rights trials ranged from $114,000 to $154,500.

If Yale is #1 in U.S. News, Is it the Best Law School?

In a provocative post entitled "Is the End Near for Yale's Dominance", Brian Leiter reports one insider's assessment that Yale may lose three or four additional faculty members.  If that happens, surely Harvard, with its recent lateral hiring sprees, will be the best law school in the country--right? Brian thinks that predictions of Yale's decline are premature.  I agee.  But in the process Brian implicitly highlights an interesting problem: what exactly does it mean to be the "best" law school? 

Here are the vexing facts:  on a per capital basis, Yale places more people in academia and Supreme Court clerkships than any other law school;  Yale's acceptance rate is 7.3% versus 11.8% for Harvard; yet, over the last decade, the average U.S. News academic reputation score for the two schools are exactly--yes, exactly--equal: 4.840 for Harvard, 4.840 for Yale. 

Is it possible that Yale is #1 because, well, Yale is #1 -- and has been every year since USN began publication?  Brian refers to U.S. News "small school bias".  He is right.  Because of Yale's massive endowment and small student size, it enjoys a per-pupil expenditure that is roughly 1/3 its total tuition price.  According to a simulation model of the 2008 U.S. News rankings, which Andy Morriss and I recently constructed, Harvard would not overtake Yale even if: 

• Harvard's median LSAT climbed to 180 and its median UGPA hit 4.0;
• Harvard's academic and lawyer-judge reputation scores were both a perfect 5.0;
• Harvard's acceptance rate plunged to less than 5%.

In fact, even with these changes, Yale would still have a nice leadership cushion.  [Note:  Similar anomalous math was noted by Ted Seto in his classic essay, Understanding the U.S. News Law School Rankings, SMU L Rev (2007).]

Yet, Yale's dominance keeps things simple.  Applicants signal their elite status by enrolling at Yale.  Judges, in turn, derive prestige by hiring Yale graduates, even though they mockingly complain that Yale clerks know very little law.  And faculty favor Yale graduates because it validates our own sense of eliteness and institutional upward movement.  We can rationalize Yale's dominance in terms of scholarship, but the real endgame is the allocation of positional goods.  It is so easy to get too caught up on the hamster wheel of envy and prestige without realizing that the energy expended does not necessarily produce anything of lasting social value.

Of course, each of us is free to determine our own merit criteria.  I think  the "best" law school is the one where faculty are willing to make inordinate personal sacrifices for the benefit of the collective enterprise--and where aspiring lawyers leave the law school skilled, confident, ethical, and ensconced in a powerful professional network that opens doors and values public service.  In turn, alumnus are sufficiently grateful for the transformative experience they received that they are willing to underwrite the law school's mission and subsidize this opportunity for future generations.  This vision requires a greater focus on internal rather than external metrics.  For us human beings, that is no easy trick.