Empirical Legal Studies

Causal Inference, Research Design & Empirical Legal Studies

Insofar as I regard Prof. Underhill Moore as the (under-appreciated) intellectual "father" of the modern empirical legal studies movement (full disclosure: a small framed photograph of Moore adorns my office wall), I remain partial to those who recognize Moore's scholarly contributions. Among those includes Profs. Dan Ho (Stanford) and Donald Rubin (Harvard--Statistics) whose paper, Credible Causal Inference for Empirical Legal Studies, leverages Moore's famous New Haven Church Street Parking study.

Recognizing that "causal inference has always been central to the enterprise of empirical legal studies," Ho and Rubin's paper, which should interest many, reviews "modern developments in the statistics of causal inference, focusing in particular on matching methods and regression discontinuity. What unifies such approaches is the prioritization of research design to create--without reference to any data on outcomes--subsets of comparable units. In what might be considered a vindication (or even 'kindred soul') of Moore, modern approaches emphasize design over methods of analysis."

Does personality motivate job choice among lawyers and law students?

My thanks to Sara Benesh for her kind introduction and for allowing me to guest edit this post.

The government has been preparing itself for the retirement of baby boomers (the “quiet crisis”)--including legions of lawyers.  In 2007, Congress passed the College Cost Reduction and Access Act that, among other things, facilitates loan forgiveness to attract employees who have made 120 monthly payments on student loans while employed in public service.  Is the financial incentive enough to lure lawyers and law students to public service?

Beyond financial incentives, a coauthor and I have been interested in the impact of public service motivation--an intrinsic, measurable personality characteristic--on job choice.  Our articles are based on quantitative analysis of multiple data sets: (Article 1) primary data we collected from first year law students and (Article 2) analysis of secondary data “National Survey of Career Satisfaction” from the American Bar Association.  ELS readers might be particularly interested in the policy capturing design used in the first article.

Very briefly, we find that, among lawyers, public service motivation positively predicts whether one's second job is in the public sector (school debt is probably the guiding factor right out of law school). Among law students, we find that public service motivation is actually a better predictor of job-fit than sector-fit. In other words, regardless of which sector a student plans to enter, students with stronger public service motivation are more likely to accept jobs that emphasize service to others.

For government recruiters, the task ahead seems to be one of attracting applicants using service-oriented work, rather than financial perks like loan forgiveness. 

 

Identifying and Measuring Ideology in Judging

In a recent article, The Context of Ideology: Law, Politics, and Empirical Legal Scholarship, 75 Missouri L. Rev. 79 (2010), I argue for a new approach to identifying the ideological nature of Supreme Court cases.  The abstract follows:

In their confirmation hearings, Chief Justice Roberts and Justice Sotomayor both articulated a vision of the neutral judge who decides cases without resort to personal perspectives or opinions, in short, without ideology. At the other extreme, the dominant model of judicial decisionmaking in political science has long been the attitudinal model, which posits that the Justices‟ votes can be explained primarily as expressions of their personal policy preferences, with little or no role for law, legal reasoning, or legal doctrine.

Many traditional legal scholars have criticized such scholarship for its insistence on the primacy of ideology in judicial decisionmaking, even as empirical legal scholarship has grown in significance and influence in the legal academy. Recently, however, empirical scholars and traditional legal academics have begun to engage in serious discussions with each other about how to think about and evaluate the balance between law and ideology and about how to harness the powerful tools of quantitative analysis to study such questions. In this Article, I offer several contributions to this discussion.

First, the Article evaluates current efforts by empirical scholars to identify the ideological character of cases. These efforts generally assume that the ideological character of a case can be determined by reference to a single liberal-conservative spectrum, and they generally presume that all or most cases present only a single issue. Through a recoding and quantitative analysis of a random sample of recent Rehnquist Court cases, as well as through a qualitative analysis of many of the cases, I establish concretely some of the limitations of these efforts. Specifically, I demonstrate that these approaches are indeterminate and oversimplified, and often prevent scholars from identifying cases in which the Justices face issues that pull them in different ideological directions. At the same time, however, I identify their strengths, particularly the strengths of approaches that leverage information derived from the Justices‟ actual voting patterns.

Finally, I propose a new approach for empirical scholars interested in studying the role of ideology in Supreme Court cases. Rather than starting with the assumption that ideology is the most important factor in Supreme Court decisionmaking, my approach focuses on just how important ideology was to the Justices in a particular case. Measuring ideological salience, I argue, would allow empirical scholars to study the empirical question of when and how ideology – and other factors – affects decisionmaking. Doing so would also open the door to a wide variety of important and interesting research questions. Most importantly, this approach would allow empirical scholars to engage with more traditional legal academics in the important normative debates about when and how ideology should play a role in the work of the Supreme Court.

A Pre-1940 Bibliography of Empirical Legal Studies

Herb Kritzer (Minnesotal Law), a seminal figure in the Law & Society movement, has recently posted a paper that provides a very exhaustive survey of emprical legal studies in the pre-1940 era.  His paper, "Empirical Legal Studies Before 1940: A Bibliographic Essay", includes a full bibliography of the articles he located.  The introduction of paper is worth posting in full:

“Empirical Legal Studies” is a term that began to come into vogue around 2000. ELS built on and extended the law and society (socio-legal studies) approach and the law and economics approach, both of which have strong empirical elements. Of course empirical research on law and legal processes predate law and society and law and economics, with a number of well known studies that were conducted in the 1950s and early 1960s. These mid-century projects include the American jury project (Kalven and Zeisel 1971), the commercial arbitration study (Mentschikoff 1952; Mentschikoff 1961), the court delay study (Zeisel and Callahan 1963; Zeisel, Jr., and Buchholz 1959), the pretrial settlement conference study (Rosenberg 1964), studies of Supreme Court decision making (Kort 1957; Kort 1966; Pritchett 1948; Schubert 1959; Schubert 1963; Schubert 1965; Snyder 1958), studies of the legal profession (Carlin 1962; Carlin 1966; Johnstone and Hopson 1967; Smigel 1964; Zander 1968), studies of compensation for injuries suffered in auto accidents (Connard, Morgan, Pratt, Voltz, and Bombaugh 1964; Hunting and Neuwirth 1962; Linden 1965; Morris and Paul 1962), and studies of the lower courts (Dolbeare 1967; Green 1961; Jacob 1969). Many of these studies of the 1950s and 1960s have framed research agendas that continue to this day. While the mid-century studies are reasonably well-known, much more obscure is the body of research conducted before World War II. In the 1920s and 1930s, and in a few cases even earlier, one can find a wide range of empirically-oriented research on law. The specific topics of this early research include:

• Appellate courts and appellate decision making
• Automobile accident compensation and litigation
• Banking
• Bankruptcy
• Criminal courts
• Debt
• Divorce
• Judicial staffing and judicial selection
• Juries (both petit and grand)
• Jurisdiction
• Legal education
• Legal needs and legal aid
• Legal profession
• Litigation
• Procedure
• Small claims

A significant portion of the early work was linked one way or another to the legal realist movement, and one can find some description of the research in historical treatments of legal realism (Kalman 1986; Schlegel 1980). In significant part because legal realism was essentially an American movement, almost all of the empirical legal research of the period was done by Americans focusing on the United States. The purpose of the discussion that follows is to highlight the range of empirical research on law from the pre-World War II period. To locate that research, I examined the tables of content of all law reviews published prior to 1940, reviewed a large number of English government reports (command papers), and followed up on any citations I could find within those works to other sources. While I do not discuss every study I found, I have provided as complete a bibliography of that research as possible. My discussion of the research is organized around a series of topics. While most studies easily fall within a single topic, a few stretch across multiple topics, and appear more than once.

What Role Does Legal Academia Play in the Development of the Law?

A recent article posted on SSRN, Reproduction of Hierarchy? A Social Network Analysis of the American Law Professoriate, asks the intriguing question of to what extent do law professors influence the development of the law.  To begin to answer the question, the authors apply social network theory to the social structure of legal academia, drawing on information on each tenure-track professor employed at an ABA-accredited law school.  The authors tracked where each professor received his or her American law degree (they excluded faculty without such degrees) and where the professor now teaches and then used this data to generate a network analysis.  This analysis produces visual representations of the resulting networks, like the one diagram below, which illustrates the centrality of a small number of schools – and especially Harvard and Yale –  in this academic network.
The authors caution, however, that such visualizations are not sufficient to understand underlying structural relationships.  They therefore apply several statistical measures: outdegree (the number of arcs emanating from each node, or more concretely here, the number of faculty members produced by each law school), hubs (measuring the relative prestige of the origin and endpoint of each arc, that is, the relative prestige of each institution – “Hubs are the schools with a high degree of influence on other influential schools...”), and closeness (measuring the tightness of connections between different institutions).  Like the visualization, these measures demonstrate an extreme skew towards the influence of a small number of schools.
Finally, the authors use the network analysis to model the rate at which ideas spread from one institution to another – the institutions’ intellectual infectiousness.  Although they acknowledge that this model involves a number of simplifying assumptions that themselves need to be examined, they argue that they provide a first cut at demonstrating how quickly and effectively new legal or intellectual paradigms can spread from one school to another where they are then incorporated into the education and socialization of future lawyers.  This is one possible mechanism by which changes in the law become widely accepted.
The piece has five authors all from the University of Michigan:  Daniel Katz, Josh Gubler, and Jon Zelner, Eric Provins, and Eitan Ingall.

Exxon Shipping, Footnote 17

Today's New York Times includes,  "From One Footnote, a Debate Over the Tangles of Law, Science and Money,"  by Adam Liptak. 

While failing to give CELS it's full due as THE annual conference on empirical legal studies, it does mention Lee Epstein's keynote speech.  And the article includes pointed quotes from Jeff Rachlinski and Ted Eisenberg on the court's understanding of research on punitive damages. 

The Limitations of the Spaeth Supreme Court Database

I have posted on SSRN my article, forthcoming in Hastings Law Journal, Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court.  This article examines the well-known and widely-used U.S. Supreme Court Database (created by Harold Spaeth) – and most recently mentioned here –  and addresses the Database’s limitations particularly for those interested in law and legal doctrine.
The key point of the Article is that the Database does not contain complete or accurate information about law and legal doctrine as they appear in Supreme Court opinions.  Given Harold Spaeth’s own purposes in creating the Database, these limitations may not be surprising -- although they do raise at least some challenges to his attitudinal model.  Unfortunately, however, they are frequently misunderstood.  Scholars all too frequently use the Database in ways that it simply cannot support, leading to the possibility of invalid or unreliable results.  This post summarizes the Article’s main arguments. 
The primary challenges presented by the Database involve the coding for the “issue,” “issue area,” and “legal provision” variables.  As the names of these variables suggest, they are frequently used by researchers interested in studying law and legal doctrine.  Yet, the coding protocols for these variables (as set forth in the Codebook are not conducive to such research.  Some of the limitations of these variables include:
(A) The “issue” variable is not, despite its name, designed to identify any legal issues in a case.  Rather, it is designed to identify the “public policy context” of a case.  A case like Schenck v. Pro-Choice Network of Western N.Y. is one example.  In Schenck, a group of abortion protesters challenged an injunction limiting their activities as violating the First Amendment.  The only legal issue in the case involves the First Amendment and the limits it places on judicial power.  But the Database codes the case as having an issue of “abortion” because that is the factual, or “public policy” context in which the case arises.
(B) The coding contains a strong presumption of assigning each case only a single issue.  So the Database does not add a First Amendment issue code to the coding of Schenck.
(C) The issue codes are quite underinclusive and somewhat dated.  For example, there are no codes for immunities, for sexual harassment, or for the dormant commerce clause.
(D) Each of the approximately 260 issue codes is classified into one, and only one, of 13 “issue areas.”  In some cases, the classification makes no sense.  For example, in Markman v. Westview Instruments, Inc., the Court addressed the question of whether patent claims construction is a question for the judge or the jury; that is, whether there is a 7th Amendment jury right.  The Database classifies Markman as a case about the right to a jury trial, but that code, which does not distinguish between civil and criminal jury rights, is located in the Criminal Procedure issue area.
(E) The legal provision code does not identify cases or judge-made legal doctrines.  It is limited to identifying statutes, constitutional provisions, and court rules.
(F) The coding protocols provide that only legal provisions mentioned in a case’s syllabus should be identified.  But the syllabus – a short summary of the case – is akin to headnotes.  It is not officially part of the case, it is not written by the justices or their law clerks, and it cannot be cited by lawyers or judges.
To some extent, misuses of the Database are likely due to differences in the ways that different disciplines (political science and law) use the same words.  To some extent, misuses stem from scholars failing to evaluate their research design in light of the Database’s coding protocols, which are described in the Database’s Codebook.  In my Article, I provide a series of examples of research project that fail to adequately take account of the Database’s limitations and that therefore produce results that may be inaccurate.
To further explore the limitations of the Database and to experiment with more legally nuanced types of coding, I undertook a Recoding Project of a random sample of 10% of the cases from the last Rehnquist natural court.  The details of the coding project are, of course, explained in the Article.  Among other things, I redefined “issue” to mean legal issue, I expanded and rearranged the lists of issues and issue areas, I put no limit on the number of issues that could be coded per case, I redefined legal provision to include seminal cases and legal doctrines, and I identified legal provisions by looking at the opinions themselves, not just the syllabi.
Some of the key findings of the Recoding Project include:
(1) I identified an average of 3.7 issues and 2.4 issue areas per case, rather than the single issue and issue area per case identified in the Database.
(2) I identified an average of 2.2 as many legal provisions per case as did the original Database.
(3) A surprising number of legal provisions that I identified should have been identified in the Database because they were mentioned in the syllabi.
(4) In both issue and legal provision coding, the “missing” codes – those that I identified but that the Database did not – disproportionately related to structural and jurisprudential issues, including procedure, the powers and operations of the federal and state governments, and the relationship between different branches of government.
These and other findings have a variety of implications for researchers working with the Database.  Chief among these is the importance of not drawing conclusions about the Supreme Court’s cases by looking at the numbers and types of issues, issue areas, and legal provisions coded.  Researchers all too often rely on such information to draw conclusions about case complexity or about the number of issue “dimensions” in the cases.  In other words, researchers sometimes point to the Database to justify their assumptions that most Supreme Court cases involve only a single issue.  But as the Article demonstrates, this single-issue coding is -- or at least may well be -- an artifact of a coding protocol that presumes that each case should be assigned only a single issue, so such conclusions are circular.  A second important implication is that the Database’s issues and issue areas do not accurately identify all cases involving particular legal issues and that not all cases with a particular issue or issue area code in fact involve the legal issues that a researcher might presume from the names of those codes.

More on CELS: Cognitive Illiberalism

Dave Hoffman over at Concurring Opinions wasn't the only blogger at CELS in Ithaca.  Although he and I apparently went to different panels, like him, I thought the conference was excellent and the quality of the papers and discussion extremely high.  Congratulations and thanks to all of the conference organizers.

One particularly interesting paper was, coincidentally (or not) co-authored by Dave Hoffman, and was previously blogged about here.  The paper, Whose Eyes Are You Going to Believe? Scott v. Harris and the Perils of Cognitive Illiberalism by Dan Kahan, David Hoffman, and Donald Braman (forthcoming in Harvard Law Review), takes advantage of a unique experiment made possible by modern technology.  In Scott v. Harris, the Supreme Court addressed whether summary judgment was appropriate in a claim of excessive force where a police officer rammed his car into the car of the fleeing suspect, who was rendred quadriplegic.  The Court held that the use of deadly force was reasonable under the circumstances, given the risk that the car chase posed to the public.  The Court rested its conclusion on the contents of a videotape, shot from the police car itself, that was entered into evidence and that the Court posted on its website.   Interestingly, however, despite the fact that the Court said that no reasonable juror could find the use of force excessive, one Supreme Court justice -- Justice Stevens -- concluded otherwise.

Taking advantage of the now publicly available videotape, the paper's authors showed the video to 1350 Americans.  As they explain, "a majority agreed with the Court's resolution of the key issues, but within the sample there were sharp differences of opinion along cultural, ideological, and other lines.  We attribute these divisions to the psychological disposition of individuals to resolve disputed facts in a manner supportive of their group identities."  So individuals who tend to see the world hierarchically (demographically more likely to be white, male, and from the South or West), were more likely to agree with the Court majority than were individuals who take a more egalitarian perspective (demographically more likely to be nonwhite, female, and from the Northeast).

Normatively, these observations suggest that judges should, at a minimum, be cautious about the claims they make about what a "reasonable juror" could conclude.  (Indeed, as the discussant, Neal Feigenson, pointed out, even if all members of the jury were completely average across all of the dimensions identified by the authors, there would still be a significant probability that at least one juror would believe that the police used excessive force. )  Assuming that one's own views are the only reasonable views, which is essentially what the majority did, "invested [the Court's] decision with culturally partisan overtones that detracted from the decision's legitimacy."  As the authors point out, when different sorts of people have predictably different perspectives, deliberation is particularly appropriate.

This project suggests an interesting take on whether the standard for judgment as a matter of law -- which is, the Court has emphasized, the same is the standard for summary judgment -- should be different.  Perhaps once a jury has in fact heard all the evidence, it should be allowed to render a verdict, and the fact of that verdict should help to inform the judge's ruling on the JML motion.  This comes up often in employment cases, where JML is often sought -- and apparently disproportionately granted -- by employer-defendants.  (See here and here for more of my thoughts on courts' overzealousness in granting summary judgment and judgment as a matter of law in employment cases.)

Law Professor Free Agency and "School-Specific" Capital

[Update: Paul Caron (Cincinnati), Michael Madison (Pittsburgh Law), Jeff Lipshaw (Suffolk), Jim Chen (Louisville) have picked up on the analysis in the below post.  There seems to be some misunderstanding on my point of "long-term contracts."  In retrospect, I should have said "long-term commitments" (i.e., extra-legal and perhaps not committed to writing) to avoid what I think is an unproductive analysis of run-of-the-mill employment and commercial contracts. 

I am talking about this:  Academic X says, "I will stay here X number of years and ignore outside offers if you provide me with the resources to execute the following institutional plan [e.g., labor-intensive but high-yield teaching, public service, useful scholarship that will be noticed and solve a real world problem, etc.]."  Law School Y says, "I love this idea.  If you are right, it will grow our institution.  Because you have committed to building it here, School Y will fund it."  Because both Academic X and Law School Y have aligned personal and institutional agendas, their cooperation and commitment grows the institutional pie; both are made better off. Moreover, it becomes magnetic for other scholars and funders who share the substantive vision. 

So we are talking about communitarian norms here.  This type of approach is easy in small groups, which is what law faculty are.  Firm-specific capital in law firms is harder to grow/maintain because (a) they have gotten larger, (b) covenants not to compete are prohibited, and (c) there are liquidity constraints imposed by the ban on non-lawyer ownership.  On the other hand, law firms work harder at it because they increasingly operate in a competitive national marketplace--firm-specific capital can be huge competitive advantage.  Law schools, in contrast, are not subject to the same market pressures--the most elite have huge endowments and donors who want to give more to be associated with the elite brands.  Thus, in the legal academy, the free agent ethos is damn near ubiquitous.

No need to be abstract about all this.  I lay out a highly plausible counteractive approach in this comment.]

Several bloggers have noted Clayton Gillette's recent article, Law School Faculty as Free Agents, 17 J. Contemp. Leg. Issues 213 (2008).  See, e.g., Paul Caron, Larry Ribstein, Al Brophy, and Paul Secunda.  Gillette's essay provides the type of straight thinking needed to move the Moneyball-Moneylaw debate into a mode of institutional analysis that can produce actual results.   I will briefly lay out Gillette's analysis and then extend it to a concept I call "school-specific" capital--an analog to firm-specific capital.

Law Professor Free Agency

In a nutshell, here is Gillette's argument.  The lateral market for law professors is primarily based upon scholarship, which is an observable, coveted good.  Teaching and service, to be sure, are relevant goods, but they are hard to measure.   Further, faculty make hiring decisions; when they land a high profile scholar, they share equally in the school's reputational gain (albeit these gains are largely limited to opinions of other professors).  Yet, if new colleagues shirk committee work or are disengaged and uninspiring teachers, the costs borne by individual faculty members are negligible or non-existent. Hence scholarship becomes the focus of lateral hiring.  Clayton observes,

In 30 years of teaching, service as vice dean, and membership on appointments committees, I don’t believe I have ever heard a discussion of a candidate’s qualifications that included serious consideration of institutional service, except insofar as it related to scholarship. ... 

[H]iring schools tend to invest little in discovering teaching quality. The hiring decision is typically made after one or two faculty members at the hiring school attend one or two of the visitor’s classes, and that is done through a process (e.g., informing the visitor when faculty members will attend, and allowing the visitor to choose that time) that diminishes the likelihood that those classes will be representative. ... The result is that, as opposed to the meticulous, highly tailored criticism to which a candidate’s scholarship will be subjected, a candidate’s teaching will be evaluated largely to determine whether it is “good enough.” (pp. 228-29)

Gillette's key insight is that the lateral market in legal academia, unlike baseball (a crucial point), does not force the decision-makers [faculty] to internalize the benefits and costs of free agent activity:  Some costs potentially get externalized onto the students, alumni and law school administrators.   When  scholarship opens so many doors, Gillette suggests, it is easy to see how a more robust lateral market can skew institutional incentives and detract from overall educational quality.

To my mind, Gillette sets forth a very coherent and plausible analysis. [I suspect a lot of people will quibble with it, however, believing that their own lateral experience (or aspiration) reflects a more optimal outcome at the institutional level.  Listeners interested in the merits of this debate should weigh the critic's potential bias.]   It is an open question whether lateral mobility is really on the rise. At Indiana Law, we are building a law faculty universe database that covers 80 years of AALS schools.  See "Is Lateral Movement on the Rise? A Precise Answer is on the Way," ELS Blog (Dec. 21, 2006).  We see a lot of lateral movement in the 30s, 40s, 50s, 60s, and 70s. Eventually we will answer to the nagging empirical question of whether lateral movement is truly on the rise.

But one thing I can say with confidence--information published on the Internet (Leiter Faculty News and Concurring Opinions) has increased the perception of heightened movement.  And perception is all that is necessary to change behavior and institutional norms--possibly in the wrong direction.

"School-Specific" Capital

Gillette actually understates his argument.  Specifically, the proliferation of a free agency ethos not only undercut educational quality, it inhibits the cooperative, highly committed, selfless environments need to create truly exceptional institutions.   One of the major implications of more professor mobility is the diminution of "school-specific" capital--i.e., desirable law school attributes, such as innovative curriculum, public service reputation, alumni good will, that remains largely intact when a professor leaves.   So more free agency suggests fewer law schools that transform good human capital into great human capital.  On this score, the "best" law schools can, in fact, be pretty mediocre.   (I believe there is a way out of this box, which I will address below.)

More after the jump. ...

"Deal or No Deal": NYT Story on Forthcoming JELS Article re: Case Settlement Decisions

Last Friday's NYT (8.8.08) included a lead story in the Business section (at C1) on a forthcoming JELS article. The subject of the NYT piece--a forthcoming article in 5:3 JELS (Sept. 2008)--reports results from a study of the financial consequences of case settlement decisions. To better assess the financial cost of going to trial, the study analyzes cases in which a settlement offer was considered, but rejected in favor of proceeding to either arbitration or trial. The findings reveal the influence of contingency fee arrangements and the availability of insurance coverage on plaintiff and defendant settlement decisions, respectively.