Measures of Judicial Ideology

Both the existence and content of this blog show that empirical legal studies has made extraordinary progress in recent years in the field of applying empirical analysis to the effects of the law. However, the progress of empirical legal studies in analyzing legal institutions themselves -- courts, judges, and cases -- is more haphazard.

The central limitation is that the primary test of judicial ideology is inadequate. The traditional approach is to count the number of case outcomes categorized as liberal and contrast them to the number of cases labeled conservative. But case facts and the questions of law they present are not dichotomous; they represent the full breadth of legal inquiry. One case that is labeled liberal may be far more liberal than another case with the same label. Michael Bailey and Kelly Chang equated this mode of analysis to giving students a calculus exam and an addition quiz, and treating a score of 80% on each as equivalent.

In recent years, scholars have turned to proxies and indirect measures to capture judicial ideology. Notably, Michael Giles, Virginia Hettinger and Todd Peppers have pursued the use of the proxy of the ideological score of the appointing president to substitute for a judicial score. Another indirect measure is to use media reporting, using content analysis of newspaper editorials, as Segal and Cover have done.

One of the most exciting developments that provides a direct score of judicial ideology is that developed by Andrew Martin and Kevin Quinn. Analogous to Keith Poole and Howard Rosenthal’s measures of congressional ideology using rollcall votes, Martin and Quinn have developed a rank ordering of Supreme Court justices on a constant standard, using the composition of coalitions to develop their scores. This allows for historical comparisons of justices who served on different courts, as well as tracking the movement of individual justices over time.

My colleague Vanessa Baird and I have used Martin Quinn's measure of judicial ideology to develop a new measure of the ideological positions of case outcomes. Explored formally and in application, we propose that the mean of the majority coalition is a more nuanced and generally superior measure to the simple traditional test of the labels liberal and conservative.

These measures offer a promising way of continuing the progress in empirical legal studies generally in measuring judicial ideology and the ideological placement of case outcomes.

Comments

[Tonja Jacobi]

I think these are some very useful definitions, though the literature uses them largely interchangeably. In our formal model of rules vs standars, Emerson Tiller and I differentiate between partisanship and ideology, using the former to characterize judging that conditions outcome preferences on litigant status (e.g. blacks vs whites, Republicans vs Democrats, or in one famous case, arguably Bush vs not-Bush) and ideology to refer to a more consistent ordering of preferences, favoring directions of outcomes without regard for litigant status. See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=752284 . But equally, ideology vs atitudes captures that idea.
As Eileen defines it, surely ideology is MORE important to know, as predicting outcomes based on the more idiosyncratic 'attitudes' is far more difficult than predicting by ideology.

[Eileen Braman]

Tonja -- first sorry about your getting your name wrong in the post above -- what can I say I'm an idiot.

Second, and more importantly, what I meant to express is that your original post sets forth the problem I am struggling with beautifully. Ideology limited as a proxy for preferences for several reasons -- first so far its been treated as two dimensional when cases have more complex facts and multiple issues. This could be addressed by adding more dimensions as is currently the trend with congressional voting behavior - but may doing so may not solve the underlying problem.

Even in two dimensional space ideology is not a particularly accurate indicator of specific outcome preferences --even if our measures of ideology were perfect preferences are often based on other (sometimes idiosyncratic) factors like personal experience with particular policy or class of litigants. I think your original post raised all of these issues -- I just wanted to add my own two cents in hopes others will chime in.

[Eileen Braman]

Tonji -- nice post.

I'm gonna chime in here though because I've been somewhat concerned about recent postings concerning measurement of "judicial ideology" as an end in and of itself -- rather than what I think it is really meant to stand as -- a proxy for judicial preferences in particular cases --to help us explain and perhaps predict voting behaivor.

I've copied a footnote from the first page of my dissertation below where I grapple with defining some terms that have been used too loosely in the literature. I wonder if anyone would have a response to my characterization:

“Ideology,” “attitudes,” and “preferences” are all words that have been used to describe personal views judges have about cases they are called on to decide. The three terms are related and have, on occasion, been used interchangably in the literature on judicial decision-making. It is important to understand, however, that they refer to distinct concepts. I discuss them here, specifically, as they are used in the context of research on judicial behavior.

In the public law literature “ideology” refers to political ideology, a coherent set of beliefs individual judges have about the proper role government in society. Ideology is usually operationalized in dichotomous terms; both judicial actors and case outcomes are categorized as liberal or conservative. Very generally, liberal judges favor more government regulation of the economy and less state interference in the lives of everyday citizens in civil liberties matters. Conservative judges have the opposite views. Because, ideology involves a very general orientation toward the role of government, it covers a wide range of issues judges may encounter in their decision-making.

Attitudes are more specific than ideology. They describe positive or negative feelings judges have toward a particular target (McGuire 1985). Attitudes may be influenced by ideology (for instance, conservatives like tax cuts) but the relationship is not perfect. Judges can have particular attitudes that are inconsistent with their ideology. Moreover, attitudes can be shaped by factors other than ideology such as personal experience with a particular issue or class of litigant.

Preferences are outcomes judges favor. Like attitudes they are more specific then ideology; the term preference implies judges are choosing from a set of alternatives in a particular choice domain (Velleman 1994). Individual judges may have preferences with regard to a given policy (such as when someone prefers tax cuts to tax hikes), or dispute (as when a judge prefers the plaintiff prevail instead of the defendant). An underlying assumption in judicial research is that judges prefer outcomes consistent with their attitudes and/or ideology. Ideology is often used to approximate judges’ preferences in cases they are called on to decide. Sometimes researchers have more specific indicators of judges’ attitudinal preferences with regard to cases involving particular issues.

Am I missing something?? If not I'm not sure approximating ideology in and of itself is as important as getting better proxies for preferences in particular cases. I'd love to hear what others have to say about this!