Operationalizing "the Law"

One would hope that the now-more-common collaboration between Political Scientists and Law Professors would result in the increased ability of Political Scientists to "take the law seriously."  Empirical Legal Studies should certainly have as at least one of its foci the empirical study of the extent to which law matters or the extent to which law constrains judges.  Some political science scholarship and some legal scholarship has begun to address this issue.  I'd like to point out what we know in order that we might fruitfully discuss where to go from here.  How can one take something as nebulous and multifaceted as "the law" and operationalize it so that its effects might be quantitively studied?  While one anonymous reviewer has suggested to me that the debate between "the legal model" and "the attitudinal model" is a tired one, it is certainly not the case, it seems to me, that we are even close to knowing all we need to know about the effects of the law on judicial decision making at all levels.  The following sketches out what some political scientists have offered as ways in which to test the influence of law.  How have they done?  How could we do better?

First, Spaeth and Segal, in their book Majority Rule or Minority Will suggest that a fruitful way to study the effect of "the law" is to determine whether justices who dissent in one case will disregard their apparent disagreement with the majority and vote to uphold the disfavored precedent in that cases' progeny.  They find that, in fact, this rarely happens.  Rather, once a justice disagrees with a precedent, s/he continues to vote to dissent in any case in which that precedent is relied upon or expanded.  This means, they suggest, that justices are not constrained by the law, at least not in terms of their operationalization thereof.  (The article on which this book is based generated enough controversy to warrant a special issue of the American Journal of Political Science (Vol. 40, 1996).)

Of course, precedent is not the only potential influence of "the law."  Perhaps interpretive strategies matter?  Howard and Segal (2002) seek to determine whether justices base their decisions on their view of the "plain meaning" of the text or on the "intent of the framers." In order to systematically study the effect of these two interpretive strategies, they code the briefs filed with the Court for arguments related to plain meaning and intent of the framers and then determine whether those justices who profess an interest in deciding in accordance with each do so regardless of the ideological position taken by the party making the argument.  They find that justices only vote with the party making the plain meaning or originalist claim when they agree with their position.

Howard and Segal (2004) also examine the claim that role perceptions affect justices' votes by looking at "activism."  Here, they find that the ideological position of the party requesting the Court declare a law unconstitutional is the largest predictor of the justices' decision to either accept or reject the request; e.g., the propensity to engage in "activism" is not conditioned by the justices' role perceptions but rather by their ideological predilections.

A couple of other sets of authors have had more success in identifying legal influences on decision making.  Hausegger and Baum (1999), for example, find that justices sometimes invite Congress to overturn the Court's decisions, suggesting, then, for these justices in these cases, the law is constraining them to behave in a way that does not maximize their preferences.  Of course, this is a very rare behavior (found in only 7% of all statuory decisions).  Additionally, Richards and Kritzer (2002) find that "jurisprudential regimes" effectively structure the justices' decisions once they are created. Some have argued, though, that these regimes are attitudinally created themselves (Segal and Spaeth 2003).

Currently, I am working on two pieces of research in which I attempt to ascertain the extent to which "law matters."  In one paper, we model the decision of a justice who disagrees with the majority opinion to either dissent or to concur in order to determine whether that decision is driven by attitudinal divergence or a difference of legal opinion over what the case is about.  We find that both matter, but that justices rarely disagree over the legal provision, issue and/or authority for decision in a given case (Benesh and Spaeth 2006).  In another paper with Jason Czarnezki, we explore the role of interpretive strategies on the vote, arguing that, if legal interpretive strategies matter, they should NOT predict a liberal or conservative vote on the Court of Appeals.  We find, (preliminarily!!), that originalism statistically significantly decreases the likelihood of a liberal vote.  This suggests, as many before have probably suspected, that adherence to a particular interpretive strategy may indeed be influenced by desired ideological outcome rather than a "real" view of the law (we will present this paper at the Midwest Political Science Association meeting in April). 

So, what is the best way (or some good ways) to operationalize "the law" so that we might better understand the role law undoubtedly plays in judicial decision making? Barry Friedman has an interesting article forthcoming in Perspectives on Politics that addresses this issue in terms of what he sees as the shortcomings of political scientists engaged in testing for the influence of law. Perhaps I can draw him out to comment on this question? 

I look forward to your reactions.

Sara C. Benesh

Comments

[Mark Hall]

As an interesting piece of intellectual history, I just came across an excellent article from 1977, by Charles Haar at Harvard Law, that examines over 100 legal or factual factors relevant in zoning cases. Charles M. Haar, John P. Sawyer, Stephen Cummings, Computer Power and Legal Reasoning: A Case Study of Judicial Decision Prediction in Zoning Amendment Cases, 2 Am. Bar Found. Res. J. 651-768 (1977). The appendix, p. 754, presents a graphical model for how legal, factual, attitudinal, and political/social factors each potentially influence judicial decisions. This article is also a good example of how to employ content analysis to better understand hidden patterns in caselaw, in a manner potentially relevant to practicing lawyers. I call attention to this article because, to my knowledge, it has been largely overlooked.

[Sara Benesh]

These are great points being raised in response to my question and I'd like to add a comment of my own, especially with respect to the study of lower courts. I emphatically agree that, in studying the lower courts, one need concern oneself even more with the influence of law and Professor Cross has certainly done notable research in that regard, along with the research highlighted by Barry Friedman in his post. However, there is also A LOT of political science scholarship on the lower courts that Barry misses both in his post and in his forthcoming Perspectives article. As Professor Cross notes, many scholars (including me!) have found a substantial influence of Supreme Court precedent on Court of Appeals and state supreme court decision making. Scholars have also uncovered effects for the preferences of the circuit as a whole, suggesting that review en banc is a motivator for getting the law right with respect to circuit law. Klein's book is an excellent example of careful attention to the influence of the law and I commend it to anyone interested in the decision making of the federal circuit courts. And I could go on and on with a literature review of this work.

So, I suppose I should amend my initial query a bit: how do we operationalize law's influence on *Supreme Court justices* and are those working on decision making in the Courts of Appeals (which includes Songer, Wasby, Lindquist, Klein, Haire, Sheehan, Kaheny, Martinek, Hettinger, Scott, Ditslear, Hurwitz, me and many others) considering the influence of law in the "right" ways or to a "substantial enough" degree?

[Howard Gillman]

Thanks to Sara for raising an important question. One additional point to add to the mix: How one conceptualizes "law" in these projects might influence what empirical methods are most appropriate to the investigation. The more "positivist" the conception the more appropriate it is to adopt quantitative methods (since one is defining law in a way that is relatively straightforward to code). By contrast, if one adopts a more post-positivist conception of law -- where following law is a state of mind within the practice of judging (e.g., a good faith belief that a particular decision is the most legally defensible)-- then historical or ethnographic methods might be more appropriate, since one is trying to uncover a judge's state of mind.

This raises the question of whether, in these studies, the conceptualization of "law" is being driven by methodological considerations or jurisprudential ones. (Shameless plug: I address some of these issues in an essay a few years back entitled "What's Law Got to Do With It?", Law and Social Inquiry 26 (2001):465-504.)

Finally, and relatedly, Mike Klarman uses more historical/interpretive methods to investigate whether the justices in Brown were motivated by law or conventional policy preferences; see From Jim Crow to Civil Rights, and also the essay “Brown v. Board of Education: Law or Politics?,” in Peter F. Lau, ed., From the Grassroots to the Supreme Court: Brown v. Board of Education and American Democracy 198 (Duke University Press, 2004).

[barry friedman]

I thank Sara for the plug of my Perspectives piece, due out later this Spring. As the piece makes clear, I'm a big fan of this work. At the same time, I feel it has happened in isolation from the world of law, and that ought to change. Collaborative efforts may be the key.

I'm headed out of town and can't say much, but let me make three brief points

First, Sara is absolutely right - and Forrest makes this point as well - it is time to move beyond coding and analyzing votes to getting at the heart of legal reasoning - operationalizing the law. This is going to be a thorny task, but it is essential. Especially on the Supreme Court, the votes mean little and the opinions almost everything.

Second, to the extent we are interested in constraint of judges by law - and of course we are - it is virtually meaningless to continue studying the Supreme Court rather than the lower courts. No one in the legal academy really believes the Supreme Court is, or necessarily should, be strongly constrained by their precedents. That is precisely why so many legal academics fail to take this legal work seriously. What is interesting is how and to what extent rules constrain the lower courts. There have been some excellent projects of this sort - Frank Cross's work; an article by Nancy Staudt in the NYU law review, etc.

Third, and most important, perhaps it is time to move beyond the simple issue of constraint to more pressing and interesting questions. The attititudinal-legal debate is an artifact of history, tracing back to when our disciplines split in the 1940s. Meanwhile, there are numerous questions within the law that are important and susceptible to empirical analysis. That is why so many law faculty are engaging in this work now. My mantra to students is "you need to read to write". Political scientists would like us to read their work. Rightly so. But if empiricists want to do work that had bite in the law, they need to read law and the work of legal scholars. This is a lot of work, of course. Interdisciplinary scholarship always is, but there is no way around it.

Even if one is going to look only at law and courts material - i.e., not substantive law - there are a host of problems that get insufficient attention. To give but one example, Jeff Lax is doing terrific work on the operation of collegial courts. We know far too little about how interactions on multi-judge courts shape the law.

[gregTouchton]

Do the google-like search capabilities of West or Lexis aid in your research, or are word associations too elementary? If there was a technical aid that you would seek, what would it be? It seems to me that much of this work could be simplified by a del.icio.us style tagging system.
http://del.icio.us
http://en.wikipedia.org/wiki/Tags

[Forrest Maltzman]

I think Sara's post is very thought provoking and useful. I will make three points. First, many of the studies Sara cites are modeling the effects of various factors (e.g. precedents) on voting patterns. We need to be careful to not automatically equate votes on the merits with legal interpretations. The challenge is in moving beyond votes to the underlying opinions. Some scholars have recently begun to move in this direction.

Second, empirical legal scholars need to be careful not to make the claim that a pattern across the bench can explain an individual justice's positions. For example, some studies have suggested that justice's do not respond to changes in congressional preferences. Overall, this seems to be the case. But, when one limits one's analysis to justices in the middle of the bench, the evidence is much murkier. In other words, it might be the case that Scalia does not respond to congress, but O'Connor does. It is important to use the word "might" here since analyses on this point have not reached any definitive conclusions.

Finally, we need to think about ways of measuring the status quo. Many (if not most) of the models being tested have as their intellectual basis--rational choice. Hence, the heart of these models is judicial preferences. [This is why debates over measuring judicial preferences are so important.] But, rational choice models usually also take into account the reversion point or the status quo. I have no doubt that good lawyers and good judges try to take into account the legal state of affairs when making their arguments and decisions. Unfortunately, we lack a perfect measure of the status quo. Hence, judicial models frequently leave out this variable that may be as important as preferences.

[frankcross]

Well (shameless plug) my research on circuit court decisions found that the legal standard of affirmance deference was vastly more significant to outcomes than was ideology.

And there's lots of research showing that the circuit courts generally adhere to Supreme Court precedents.

If you want to limit it to the Supreme Court there are a variety of approaches that might be used. For example, the differential standards in equal protection cases could be used as a variable.

But I think that one expects to see much less legal effect at the USSC level, given their caseload control and interest in taking only the cases that are the hardest on the legal standard