Empirical Legal Studies

Updated Judicial Common Space Scores

here. Courtesy of Profs. Lee Epstein, Andrew Martin, Jeff Segal, and Chad Westerland. JCS attempt to provide preference estimates for Supreme Court justices that are directly comparable to preference measures of Courts of Appeals judges, members of Congress, and the President. Our data extend through 2008 and correspond with the most recent version of Keith Poole's Common Space scores.

Hello!

Thanks to the Empirical Legal Studies crew for letting me guest blog here. Consistent with the motto of the blog, I hope to use my time with ELS to bring some method to the madness that seems likely to swirl around Sonia Sotomayor’s confirmation hearings this summer.

 

For the past two years I have been compiling a dataset capturing all of the statements made by senators and Supreme Court nominees at the confirmation hearings held by the Senate Judiciary Committee.  The dataset starts with the Felix Frankfurter’s hearing and ends with Justice Alito’s. I have coded by issue area all statements made by the nominees and senators. Each unit of analysis includes information about the political party of the questioning senator, the appointing president, and the committee chair. The dataset also captures instances where a court case was discussed by name, and includes variables indicating whether the statement being coded addressed constitutional interpretation, statutory interpretation, or federalism.

 

I hope to use this data to provide real-time commentary on the confirmation hearings this summer.  Because Supreme Court vacancies occur so infrequently, our discussion of the confirmation process often lacks historical perspective. I hope this data can help rectify that, by bringing concrete information to conversations about what has – and has not – been “the norm” in this process. 

 

I also hope that the ELS community will provide feedback about how to use and improve this dataset. This is in many ways a ‘test run’ of the data, so I look forward to hearing your comments about it. I’m also open to suggestions regarding what type of information you think would be interesting to pull out of the dataset.

 

First up: What makes a confirmation contentious?

The Dataverse

I just finished posting some data and documentation on the Dataverse Network.  This is a wonderful -- and remarkably easy to use -- resource that provides scholars with a virtual archive for their data.  It allows you to make your data publicly available in a permanent format with a stable URL -- useful if, for example, you change institutions.  You can even incorporate links to your data into your own website, branding the material as your own, without having to worry about hosting and maintaining it.   If other scholars use or rely on your data, there is a formal citation that they can use, giving you appropriate scholarly credit.

And of course the Dataverse is a great resource for scholars who might want to replicate other people's work or rely on their data.   Thanks to Harvard's Institute for Quantitative Social Science for creating and making available this resource.  You can read more about it here, in a post by Gary King at the Social Science Statistics Blog.

Judicial Pay Debate Data

Over at the Volokh Conspiracy Eric Posner brings logitudinal data to the judicial pay debate. The data come from a paper on the same topic, Are Judges Overpaid?: A Skeptical Response to the Judicial Salary Debate, by Stephen Choi (NYU), Mitu Gulati (Duke), and Eric Posner (Chicago).

Effect of "High Score" Rule on Volume of LSAT Repeaters

A couple of years ago, Paul Caron flagged an obscure ABA rule change that required law schools to report the highest LSAT score of an admitted student versus the prior practice of averaging.  Paul and Moneylaw blogger Tom Bell foresaw a likely surge in the number of repeat test-takers--but without any appreciable benefit.  Repeat score are, in fact, less accurate in predicting law school grades -- though, let's not kid ourselves, law schools are not looking at the LSAT anymore for predicting 1L performance.  It also costs a lot of time and money.  And if upper class white kids are better able to afford test preparation courses, it is likely to exacerbate the racial performance gap. 

Well, the data is in for the October 2008 cycle.  It is indisputable that applicants are figuring out the implications of a low LSAT score in the US rankings era (fewer admissions letters, few scholarship dollars) and the potential upside/no downside of taking the exam again. Despite a -1.7% drop in first-time takers, the repeater volume is up 16.8%.  In the Northeast, where the positional competition is the most intense, there has been a staggering 33.7% increase. 

From an individual perspective, I know it makes sense to take the test a second or third time. Indeed, it is comforting to many to have that option.  But in the aggregate, this policy really just opens the door for a protracted zero sum game.  If law schools were making better decisions because of the second or third scores, the additional time and expense could be justified.  The second scores, however, are less reliable than the first and second combined. 

Understanding these dynamics, the regulator (the ABA Section on Legal Education and Admission to the Bar) is supposed to set rules that are in the best interests of the students and the profession -- not law schools or testing agencies. Sam Stonefield of Western New England Law wrote a very detailed objection to this policy. He was prescient.  I appreciate the time he took to spell it all out.

More From BJS: State Civil Litigation Trends, 1992--2005

Supplementing one of the leading sources of data for state civil litigation activity in the United States, the Bureau of Justice Statistics (BJS) recently released a report, Civil Bench and Jury Trials in State Courts, 2005, which provides an important snapshot and illustrates litigation trends since 1992. The report discusses general civil cases (tort, contract, and real property) concluded by a bench or jury trial in a national sample of jurisdictions in 2005. Topics include the types of civil cases that proceed to trial, the differences between civil cases adjudicated by judges or juries, and the types of plaintiffs and defendants represented in civil trials. Also, the report covers plaintiffs.  Key findings include:

• In 2005 plaintiffs won in more than half (56%) of all general civil trials concluded in state courts. The plaintiff was significantly more likely to win in a bench trial compared to a jury trial. Among all plaintiff winners the median final award was $28,000. Approximately 4% of all plaintiff winners won $1,000,000 or more. Contract cases in general had higher median awards ($35,000) than tort cases ($24,000).

• The total number of civil trials declined by over 50% from 1992 to 2005 in the nation’s 75 most populous counties. Tort cases decreased the least (40%) while real property (77%) and contract (63%) cases registered the largest declines.

• In the nation's 75 most populous counties, some tort case categories have seen marked increases in their median jury awards. This was particularly the case for product liability trials, where the median awards were about 5 times higher in 2005 than in 1992 and for medical malpractice trials, where the median jury awards more than doubled from $280,000 in 1992 to $682,000 in 2005.

The Drift Toward Pure Numbers Admissions

Law schools are part of a production function for entry level lawyers.  Therefore, if law schools alter their admissions practices, the character and complexion of the hiring pool can shift in significant ways.   

On the input side, the data are crystal clear:   over the last fifteen years, the rankings arms race has pushed U.S. law schools toward a pure numbers approach to admissions.   The more interesting question, however, is whether prestige-conscious law firms are now, inadvertently, experiencing any fallout.   First the data.

Law schools operate in an environment of supply and demand and are famously counter-cyclical.  When Silicon Valley was booming in the late 90s, law school applicants plummeted.  When the economy faltered in the early 90s or after 9/11, applicants spiked.  Therefore, to examine how admissions practices have changed over time, it is important to pay attention to the underlying applicant pool.  Below are trend lines for median LSAT scores by USNWR rank for 1994 and 2007, which reflect classes that entered in the fall of 1993 and 2006 respectively.  During those two admissions cycles, the number of applicants was virtually identical:  89,600 (1994) and 88,700 (2007).

Obviously the blue line (2007) is higher than the orange line (1994).  In fact, despite slightly fewer law school applicants, the average median LSAT increased by 2.18 points (std. dev. of 1.99). For the record, only three schools fell out of Tier 1 between 1994 and 2007.  And it cannot be explained by the ABA policy shift that instructs law schools to no longer average LSATs for reporting 25th, 50th, and 75th percentile figures, thus slightly pumping up the volume of high LSAT scores.  That change was not enacted until the summer of 2006. 

Here is the same analysis for UGPA (1994 data came from the Princeton Review, 2007 from the ABA):

Although we might chock some of the higher UPGAs (avg. of +.17, std. dev. of +.12) on grade inflation between 1994 and 2007, it is likely that schools were also trying to maximize this number. More after the jump ... .

The Limitations of the Spaeth Supreme Court Database

I have posted on SSRN my article, forthcoming in Hastings Law Journal, Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court.  This article examines the well-known and widely-used U.S. Supreme Court Database (created by Harold Spaeth) – and most recently mentioned here –  and addresses the Database’s limitations particularly for those interested in law and legal doctrine.
The key point of the Article is that the Database does not contain complete or accurate information about law and legal doctrine as they appear in Supreme Court opinions.  Given Harold Spaeth’s own purposes in creating the Database, these limitations may not be surprising -- although they do raise at least some challenges to his attitudinal model.  Unfortunately, however, they are frequently misunderstood.  Scholars all too frequently use the Database in ways that it simply cannot support, leading to the possibility of invalid or unreliable results.  This post summarizes the Article’s main arguments. 
The primary challenges presented by the Database involve the coding for the “issue,” “issue area,” and “legal provision” variables.  As the names of these variables suggest, they are frequently used by researchers interested in studying law and legal doctrine.  Yet, the coding protocols for these variables (as set forth in the Codebook are not conducive to such research.  Some of the limitations of these variables include:
(A) The “issue” variable is not, despite its name, designed to identify any legal issues in a case.  Rather, it is designed to identify the “public policy context” of a case.  A case like Schenck v. Pro-Choice Network of Western N.Y. is one example.  In Schenck, a group of abortion protesters challenged an injunction limiting their activities as violating the First Amendment.  The only legal issue in the case involves the First Amendment and the limits it places on judicial power.  But the Database codes the case as having an issue of “abortion” because that is the factual, or “public policy” context in which the case arises.
(B) The coding contains a strong presumption of assigning each case only a single issue.  So the Database does not add a First Amendment issue code to the coding of Schenck.
(C) The issue codes are quite underinclusive and somewhat dated.  For example, there are no codes for immunities, for sexual harassment, or for the dormant commerce clause.
(D) Each of the approximately 260 issue codes is classified into one, and only one, of 13 “issue areas.”  In some cases, the classification makes no sense.  For example, in Markman v. Westview Instruments, Inc., the Court addressed the question of whether patent claims construction is a question for the judge or the jury; that is, whether there is a 7th Amendment jury right.  The Database classifies Markman as a case about the right to a jury trial, but that code, which does not distinguish between civil and criminal jury rights, is located in the Criminal Procedure issue area.
(E) The legal provision code does not identify cases or judge-made legal doctrines.  It is limited to identifying statutes, constitutional provisions, and court rules.
(F) The coding protocols provide that only legal provisions mentioned in a case’s syllabus should be identified.  But the syllabus – a short summary of the case – is akin to headnotes.  It is not officially part of the case, it is not written by the justices or their law clerks, and it cannot be cited by lawyers or judges.
To some extent, misuses of the Database are likely due to differences in the ways that different disciplines (political science and law) use the same words.  To some extent, misuses stem from scholars failing to evaluate their research design in light of the Database’s coding protocols, which are described in the Database’s Codebook.  In my Article, I provide a series of examples of research project that fail to adequately take account of the Database’s limitations and that therefore produce results that may be inaccurate.
To further explore the limitations of the Database and to experiment with more legally nuanced types of coding, I undertook a Recoding Project of a random sample of 10% of the cases from the last Rehnquist natural court.  The details of the coding project are, of course, explained in the Article.  Among other things, I redefined “issue” to mean legal issue, I expanded and rearranged the lists of issues and issue areas, I put no limit on the number of issues that could be coded per case, I redefined legal provision to include seminal cases and legal doctrines, and I identified legal provisions by looking at the opinions themselves, not just the syllabi.
Some of the key findings of the Recoding Project include:
(1) I identified an average of 3.7 issues and 2.4 issue areas per case, rather than the single issue and issue area per case identified in the Database.
(2) I identified an average of 2.2 as many legal provisions per case as did the original Database.
(3) A surprising number of legal provisions that I identified should have been identified in the Database because they were mentioned in the syllabi.
(4) In both issue and legal provision coding, the “missing” codes – those that I identified but that the Database did not – disproportionately related to structural and jurisprudential issues, including procedure, the powers and operations of the federal and state governments, and the relationship between different branches of government.
These and other findings have a variety of implications for researchers working with the Database.  Chief among these is the importance of not drawing conclusions about the Supreme Court’s cases by looking at the numbers and types of issues, issue areas, and legal provisions coded.  Researchers all too often rely on such information to draw conclusions about case complexity or about the number of issue “dimensions” in the cases.  In other words, researchers sometimes point to the Database to justify their assumptions that most Supreme Court cases involve only a single issue.  But as the Article demonstrates, this single-issue coding is -- or at least may well be -- an artifact of a coding protocol that presumes that each case should be assigned only a single issue, so such conclusions are circular.  A second important implication is that the Database’s issues and issue areas do not accurately identify all cases involving particular legal issues and that not all cases with a particular issue or issue area code in fact involve the legal issues that a researcher might presume from the names of those codes.

Spaeth's S.Ct. Coding (07 Term)

Harold Spaeth has completed his coding for the Supreme Court's 07 Term. The dataset ("allcourt") is available for public download at the University of South Carolina Judicial Research Initiative (JuRI). To access the dataset please visit www.cas.sc.edu/poli/juri.

If Yale is #1 in U.S. News, Is it the Best Law School?

In a provocative post entitled "Is the End Near for Yale's Dominance", Brian Leiter reports one insider's assessment that Yale may lose three or four additional faculty members.  If that happens, surely Harvard, with its recent lateral hiring sprees, will be the best law school in the country--right? Brian thinks that predictions of Yale's decline are premature.  I agee.  But in the process Brian implicitly highlights an interesting problem: what exactly does it mean to be the "best" law school? 

Here are the vexing facts:  on a per capital basis, Yale places more people in academia and Supreme Court clerkships than any other law school;  Yale's acceptance rate is 7.3% versus 11.8% for Harvard; yet, over the last decade, the average U.S. News academic reputation score for the two schools are exactly--yes, exactly--equal: 4.840 for Harvard, 4.840 for Yale. 

Is it possible that Yale is #1 because, well, Yale is #1 -- and has been every year since USN began publication?  Brian refers to U.S. News "small school bias".  He is right.  Because of Yale's massive endowment and small student size, it enjoys a per-pupil expenditure that is roughly 1/3 its total tuition price.  According to a simulation model of the 2008 U.S. News rankings, which Andy Morriss and I recently constructed, Harvard would not overtake Yale even if: 

• Harvard's median LSAT climbed to 180 and its median UGPA hit 4.0;
• Harvard's academic and lawyer-judge reputation scores were both a perfect 5.0;
• Harvard's acceptance rate plunged to less than 5%.

In fact, even with these changes, Yale would still have a nice leadership cushion.  [Note:  Similar anomalous math was noted by Ted Seto in his classic essay, Understanding the U.S. News Law School Rankings, SMU L Rev (2007).]

Yet, Yale's dominance keeps things simple.  Applicants signal their elite status by enrolling at Yale.  Judges, in turn, derive prestige by hiring Yale graduates, even though they mockingly complain that Yale clerks know very little law.  And faculty favor Yale graduates because it validates our own sense of eliteness and institutional upward movement.  We can rationalize Yale's dominance in terms of scholarship, but the real endgame is the allocation of positional goods.  It is so easy to get too caught up on the hamster wheel of envy and prestige without realizing that the energy expended does not necessarily produce anything of lasting social value.

Of course, each of us is free to determine our own merit criteria.  I think  the "best" law school is the one where faculty are willing to make inordinate personal sacrifices for the benefit of the collective enterprise--and where aspiring lawyers leave the law school skilled, confident, ethical, and ensconced in a powerful professional network that opens doors and values public service.  In turn, alumnus are sufficiently grateful for the transformative experience they received that they are willing to underwrite the law school's mission and subsidize this opportunity for future generations.  This vision requires a greater focus on internal rather than external metrics.  For us human beings, that is no easy trick.