Empirical Legal Studies

Of Interest from the MWPSA

Just returned from the Midwest Political Science Association conference and wanted to share some of "what I learned in Chicago."  Here goes.

First, we have discussed here issues of measurement, especially with respect to "the law" and "ideology."  A few presentations and comments of discussants are relevant to these discussions.  First, Sean Wilson's paper, entitled, "Modeling Justice Ideology Without Ecological Inference" argues that using career liberalism scores to predict votes is better than using the Segal and Cover scores in terms of model fit.  I suggested that using votes to predict votes is circular and the discussant (Brandon Bartels, soon-to-be assistant professor at SUNY StonyBrook) noted that use of the Martin-Quinn scores has been similarly criticized.  Martin and Quinn have a working paper on the question of when we can use their scores as independent variables that is interesting, though I remain unconvinced.  Seems to me that using percent liberal (or ideal points, which are based on votes) to predict votes shows us that judges are CONSISTENT, but not WHY they make the choices they do in the first place (which I find much more interesting).  Of course, this is all open to comment.  (And I do think Sean makes a good point that the attitudinal model operates differently in different issue areas.) Don Songer (South Carolina) brought up the issue of unanimous decisions during the discussion as well, and that's one area where the attitudinal model doesn't give us much.  We definitely need to better understand why the Court sometimes decides unanimously.  Thoughts?

Also interesting were a number of papers dealing with the influence of law on decision making, both on the Supreme Court and lower court levels.  Jason and I have already talked about our paper measuring the influence of legal interpretive strategies.  Bartels has an interesting paper that argues that the operation of preferences at the Supreme Court is influenced by law, using jurisprudential regimes to show that ideology has more room to influence outcome when, for example, rational basis review is appropriate than when strict scrutiny controls.   My coauthors (Jennifer Luse and Wendy Martinek) and I also use the notion of jurisprudential regimes to measure compliance by the Court of Appeals with the Supreme Court in our paper on Lemon, and Pam Corley (Vanderbilt) gave a very interesting paper that shows that TYPE of concurrence effects lower court compliance with the Supreme Court (and so, the Court has some control over its effect).   While I haven't read the papers or seen the presentations, papers by Cameron (Princeton) and Kornhauser (NYU), Jacobi and Tiller (Northwestern), and Lax (Columbia) look like they treat both law and the relationship between levels of courts in interesting ways.

Obviously, this barely scratches the surface in terms of what was presented at the conference and I encourage you to browse the MWPSA paper archive to read more!  (And would love for those of you who did attend, to use this opportunity to discuss the best work you saw presented.)

ELS and not be 'laughably beside the point'

Many thanks to the folks at ELS Blog.  I really appreciate the opportunity to guest blog. 

In a recent article in the Indiana Law Journal, Tracey George wrote that “[e]mpirical legal scholarship (ELS) is arguably the next big thing in legal intellectual thought.”  I agree. Yet, as I worked on a grant proposal last summer for a longitudinal study of the consumption, production and content of legal scholarship (discussed by one of my co-PI, Funmi Arewa, at Conglomerate back in March), I was surprised to learn that this is the second time that empirical studies was the next big thing.l During the 1920s and ‘30s, the sociological jurisprudence and legal realism movements embraced the social sciences as the primary vehicle for revamping legal scholarship (and, surprisingly, the law school curriculum). A key element of this agenda was empirical research by law professors.

Yet, the most prominent legal realists were much better at praising empiricism than actually doing the arduous task of data collection and analysis. High quality empirical work, it turned out, was time-consuming, expensive, and sometimes produced findings that were at odds with the ideological preferences of major funders or progressive colleagues.  There are several articles and books that chronicle this failure, but the most definitive is probably John Henry Schlegel’s American Legal Realism & Empirical Social Science (1995).   

War Stories From the Peer-Review Front

Stories about how student law review editors bungled, mis-handled, and ill-treated law professors and our work (especially empirical work) abound.  Some of the more outrageous are the stuff of urban legend.  (To be fair, I know student editors have more than a few stories of their own that feature misbehaving faculty.)  Indeed, a desire to have peers (presumably, knowledgeable peers) review and referee our work helps explain law professors' growing attraction to peer-reviewed journals.  The peer-review process, however, is far from perfect.  For those interested, the political science journal monitor includes an interesting thread on peer-review war stories.

Judge Kopf on Blogs and Law Reviews

3L Epiphany has an interesting Q&A with Judge Richard G. Kopf (D. Neb.) on the value of both blogs and law review articles for judges.

Guest Blogger: William Henderson

Bill Henderson is an Associate Professor of Law at Indiana.  Professor Henderson's primary research interests include the regulation of the financial markets, class action litigation, and the economics of the legal profession.  Professor Henderson is currently writing a series of empirical papers on large law firms.  The first paper, "Single-Tier versus Two-Tier Partnership Tracks at Am Law 200 Law Firms," will be presented this fall at a symposium on law firms at the University of North Carolina.

Professor Henderson also has a long-standing interest in education policy. In 2004, he published an innovative study in the Texas Law Review, "The LSAT, Law School Exams, and Meritocracy: The Surprising and Undertheorized Role of Test-Taking Speed," which presented evidence that the predictive validity of the LSAT may be partially attributable to the legal academy's heavy and undertheorized use of time-pressured exams. In 2002, he published an empirical study on the Cleveland public schools, which demonstrated the importance of demographic patterns in explaining, and ultimately predicting, educational outcomes at the K-12 level. The article was the first study to utilize GIS maps for visually depicting the relationship between a school district's socioeconomic composition and students' performance on state proficiency exams. Professor Henderson's recent contribution to the Indiana University Rankings Symposium (co-authored with Andrew Morriss) used multiple regression analysis to successfully model how a variety of factors influence law school competition for high LSAT students.

"Yet the nature and effect of legal doctrine has been woefully understudied."

Says Cross and Tiller's essay entitled What is Legal Doctrine?   The Abstract in its entirety:

Legal doctrine is the currency of the law. In many respects, doctrine is the law, at least as it comes from courts. Judicial opinions create the rules or standards that comprise legal doctrine. Yet the nature and effect of legal doctrine has been woefully understudied. Researchers from the legal academy and from political science departments have conducted extensive research on the law, but they have largely ignored the others' efforts. Part of the reason for this unfortunate disconnect is that neither has effectively come to grips with the descriptive meaning of legal doctrine. In this article, we attempt to describe the concept of legal doctrine and propound various theories of how legal doctrine may matter in judicial decision making and how those theories may be empirically tested.

Adjudicated empirical work in the courtroom

Hi Everyone,

I wanted to pose a question that relates directly to the issue of empirical work and the legal profession.   Cases are increasingly comprised of technical and scientific issues.  Judges are assigned the difficult task of determining whether evidence of this nature should be admissible.  Although Daubert provides greater guidance for judges in evaluating such evidence, the judge is still left with the unenviable - and largely unreviewable - task of make this determination.  And in many cases, the admission/exclusion of evidence largely determines the substantive outcome (e.g., toxic torts, voting rights, medical malpractice).

Judges are limited in how they can address deficiencies in their inability to evaluate such evidence.  They can hire special masters, but evidence suggests that they rarely exercise this option.  Judge can engage in independent research, but it is arguably contrary to the adversarial system to do so.   [By the way, for an excellent article describing these issues and drawing upon an original interviews with state judges, see Ed Cheng's INDEPENDENT JUDICIAL RESEARCH IN THE DAUBERT AGE, forthcoming in Duke Law Journal].

Thus, I am interested in your thoughts as to what, if anything, the judicial system should do to address this issue?  Should expert witnesses be drawn from a pool of experts  common to both parties?  Should judges be allowed to engage in independent research (with disclosure to both parties)?  Should there be specialized courts?

Do Rankings Matter?

The new issue of the Law & Society Review includes an article on the effects of law school rankings. Here is the abstract:

In recent years, there has been a tremendous proliferation of quantitative evaluative social measures in the field of law as well as society generally. One of these measures, the U.S. News & World Report rankings of law schools, has become an almost obsessive concern of the law school community, generating a great deal of speculation about the effects of these rankings on legal education. However, there has been no attempt to systematically ascertain what, if any, effects these rankings have on the decisionmaking of students and schools in the admission process. This article documents some of these effects by conceptualizing rankings as a signal of law school quality, investigating (1) whether students and schools use this signal to make decisions about where to apply and whom to admit, and (2) whether the creation of this signal distorts the phenomenon—law school quality—that it purports to measure. Using data for U.S. law schools from 1996 to 2003, we find that schools' rankings have significant effects on both the decisions of prospective students and the decisions schools make in the admissions process. In addition, we present evidence that the rankings can become a self-fulfilling prophecy for some schools, as the effects of rank described above alter the profile of their student bodies, affecting their future rank. Cumulatively, these findings suggest that the rankings help create rather than simply reflect differences among law schools through the magnification of the small, and statistically random, distinctions produced by the measurement apparatus.

Michael Sauder and Ryon Lancaster, Do Rankings Matter? The Effects of U.S. News & World Report Rankings on the Admissions Process of Law Schools, 40 Law & Society Review 105 (2006).

"Deal or No Deal"

An empirical analysis of contestant decision-making and risk-aversion in the TV game show "Deal or No Deal" (the Australian Version).  See here.

How does a district court judge get on the court of appeals?

Savchak, Hansford, Songer, Manning, and Carp have written Taking It to the Next Level: The Elevation of District Court Judges to the U.S. Courts of Appeals.  The Abstract:

We address an important aspect of judicial careers: the elevation of judges from the U.S. District Courts to the Courts of Appeals. We argue that the likelihood of a judge being elevated is a function of informational cues and signals regarding the nature of the judge and the judge's compatibility with presidential preferences. We also expect norms involving the intersection between geography and Senate politics to affect a judge's elevation chances. Using data on district court judges appointed between 1946 and 1995, we find that the likelihood of a judge being elevated is a function of the judge's ideological compatibility with the president, the judge's previous ABA rating, and Senate norms involving state "ownership" of appeals court seats. Blunt indicators of policy preferences trump direct signals when presidents decide whom to elevate, leaving judges little control over their career prospects and thus less incentive to slant their decisions in the direction of the president's preferences.