Empirical Legal Studies

Intriguing Use of Content Analysis

Political scientists Brian Calvin, Paul Collins, and Pamela Corley (at Univ. North Texas, Univ. of North Texas, and Vanderbilt, respectively) report results from their study that compares Supreme Court opinion text with text from the relevant lower court opinions. Over at the Conglomerate David Zaring's post provides helpful context. The paper's excerpted abstract, below, summarizes.

"... We argue that lower court opinions will influence the content of the Court’s opinions based on a number of factors, including the prestige of the lower court opinion author, the published or unpublished nature of the lower court opinion, the ideological compatibility of the lower court opinion vis-à-vis the Supreme Court’s decision, the type of lower court opinion, and the lower court from which the opinion emanated. Utilizing plagiarism detection software to compare lower federal court opinions with the majority opinions of the Supreme Court during the 2002-2004 terms, we uncover support for our hypotheses, indicating that the Supreme Court systematically incorporates language from the lower federal courts into its majority opinions."

Yung on Activism (and Sotomayor)

Over at the Sex Crimes Blog, Corey Yung (John Marshall) discusses the preliminary results of a mostly sex crimes-free paper on judicial activism. He will be presenting the paper at Law and Society in Denver this Saturday.

For Yung, “Judges are ‘activist’ when they substitute their judgment in place of other constitutional actors when the formal model would predict otherwise.” For appellate judges, his operational measure of activism is tied to the standard of review on appeal. As the standard of review can be a hindrance to achieving a preferred outcome, activist judges should pay less attention to it. Yung therefore compares the reversal rates in cases (1) with and (2) without a deferential standard of review. The calculation for each judge is straightforward: Reversal Rate in Non-Deferential Cases minus Reversal Rate in Deferential Cases. The smaller the gap between the reversal rates in these two classes of cases, the more "activist" the judge.

Based on the (preliminary) data from 2008 for five circuits, the average reversal gap is 10.4%. Judges Calabresi and Posner have smaller reversal gaps at 3.7% and 4.3%, respectively. Judge Sotomayor comes in near the average at 11.7%. Judges Easterbrook and Wilkinson are at the more "restrained" end of the spectrum with reversal gaps of 22.1% and 35.2%, respectively.

For more, see Yung's post here.

Discussing "Bias and the Bar" II

Sara - We definitely need to think more about how we can operationalize some of the features of "judicial temperament." I personally need more time to really think this through, so I instead thought it might be good to write about the question that keeps running through my head as I try to operationalize judicial temperament. The issue I'm struggling with, and which I'd love to hear the thoughts from others, is how we guard against imputing too much to the obvious proxies we'd have to use. For example, to play devil's advocate, I'm interpreting your suggestion of using the Congressional staff variable to reflect the quality of "open-mindedness" as implying that the political nature of the job might actually point towards close-mindedness. The question is whether that is reading too much into what it means - or what the ABA perceives it to mean - when someone has served on a Congressional staff. We thought about it in two terms: (1) an obviously political position, but also (2) a not-always-good substitution for traditional legal experience, especially given the political nature of the job. Put another way, did the person get the nomination because of their political service, or due to their qualifications? However, is the acknowledgement of the political flavor of the appointment and the nominee's background the same as measuring whether a nominee will be viewed by the ABA as "open-minded"? Alternatively, by this definition, our inclusion of the Years as Public Interest Lawyer could be seen as another measure of potential "open-mindedness" or "commitment to equal justice" given the equal justice-nature of much public interest work. But, again, are we presuming too much to suggest that the ABA views such work in this manner? Finally, what are service as a government attorney? The vast majority of people in this category - which we found the ABA viewed positively - serve as some type of prosecutor. Is this a signal of substantial courtroom and/or litigation experience, or could one possibly somehow link such experience - either positively or negatively - to some of the suggested ways of understanding "judicial temperament" in the ABA's guidelines? More broadly, how do we make sure that even our operationalization of these terms does not in and of itself make judgments? For example, does it (or should it) matter who the staffer worked for in Congress? Does it matter what issues the staffer focused on?

Of course, much of this relates back to Sara's initial post about the impact of the ABA's investigations - it's not merely that someone was a judge, but that the ABA attempts to determine whether they really were a fair and impartial judge. The problem, however, is that we have little to work with on the outside. We could obviously attempt to look into the information brought forth by others (i.e., the media, interest groups, senators, etc.), but (a) most nominees receive little attention if any, and (b) the information we hear publicly about is likely to be somewhat biased. In fact, as a side note, a big problem is the lack of information on nominees -- when I was constructing a list of nominees who were opposed by interest groups, there were a surprising number who has zero articles written about their nominations, including passing mentions that they were nominated and/or confirmed. Which brings us back to the issue of how do we attempt to explain something - or some idea - for which we only have limited information and potential proxies?

Thoughts?

Finally, we will certainly work on creating a more nuanced ideology measure. The one question, Sara, is were you referring to the distance between the ABA committee and the nominee or the distance between the Judiciary Committee and the nominee? The issue is that, and forgive me if I’ve totally misunderstood your post, is that I'm just not sure what it would mean to hypothesize that the ABA Standing Committee's ratings are influenced by how ideologically distant the nominee is from the Senate Judiciary Committee. This would in turn have to mean that we believe that the ABA is more likely to rate nominees highly in times of unified government (when the ideological distance would be closer), and rate them less well in times of divided government (when the ideological distance would be farther). And, this would suggest the ABA's ratings are in part driven by the makeup of the Judiciary Committee, rather than based on the nominee's background.  Obviously, if we could create some measure of the ABA Committee's ideology that we could then compare to the nominee's ideology, that would be great.... Last thought: yes, determining whether the bias has switched over time is definitely a worthy question; the issue, as always, is getting the data.

Sara - thanks again for really pushing me/us to think about these issues!

Re: Chris's post yesterday, the only possible response is to say thank you for such a great suggestion, and the reminder to pay close attention to how one describes what their model actually does - and for providing us with all of the information on how to put that suggestion into action! We appreciate the guidance!! And, we'll let you know what we found as soon as we're done.

Discussing "Bias and the Bar"

Thanks for your post, Sara! You are obviously correct in that we cannot truly capture in our model the full parameters of the investigation the ABA conducts. Most importantly, we do not know what is learned in the background interviews, and the ABA does not share that information with anyone outside of the ABA committee, including the president and the Senate Judiciary Committee members. However, we would argue that our model does more than simply capture nominee qualifications per se since the crucial question is not how would others rate these nominees, but how did the ABA evaluate them? The issue is that what is considered "qualified" really does vary based on who you ask, and, as evidenced by external discussions of nominee ABA ratings, the ABA's ratings do not merely reflect the conventional wisdom as to whether a certain person is, objectively, qualified for the federal bench. We do examine information others can also find, but we chose these measures because these are the types of qualifications the ABA itself says it considers in its guidelines. Furthermore, even if others possess this same information, the reason the ABA is granted this role is its unique ability to evaluate how much these different background experiences matter. Senators thus rely upon the ABA to give them guidance about whether a particular nominee is, in fact, fit for the federal bench.

The other important issue is that Senators depend on the ABA because they lack the time, resources and staff to do their own investigations. Even if the senators have nothing more than the ABA rating, the belief that the rating reflects an in-depth evaluation heightens the importance of this rating. And, even if our model is merely capturing what factors are likely to lead to a determination that a nominee is qualified per se, since it is the ABA that sets the gold standard for this discussion, then even capturing that part of the process provides important insights.

As to whether the findings are time-bound, our reason for looking at nominees from 1985 onward was to deal with any potential time-based issues. Since 1985, and especially since the Bork nomination in 1987, criticisms directed at the ABA have been from conservatives. We therefore believe our findings explain the process as it exists today, and most likely, the process in the future, but agree that we cannot necessarily use them to explain the ABA's behavior in earlier periods. And, given important changes such as the increased politicization of lower court appointments, the increased politicization of the ABA House of Delegates, and changes in the evaluation standards themselves, we think it best to focus our attention on this more recent period.

Omitted variable bias is always of concern, and certainly our model cannot capture how the additional context provided by the ABA's committee's interviews influences the decision-making process. That said, our model does, we believe, reflect the influence of the major factors the ABA considers, and provides us with insights on how the ABA determines whether a nominee is considered "qualified' for the federal bench. If the factors we examine only point toward whether a nominee is qualified per se, that in fact suggests that we need to understand even more why we then find differences based on ideology and partisanship.

Forum: "Bias and the Bar"

I'd like to thank Rich and Amy, again, for agreeing to come on and discuss their work with all of us.  I'd like to start the discussion with the ABA's standing committee itself.  Why does it have the role it does, and how does it decide which rating to give to a given judicial nominee?

The authors agree that the ABA plays a vital role in the process, but I would argue that its vital role really isn't captured in this paper.  The models are couched as attempts to explain the ABA's ratings, but I think they are really attempts to ascertain what makes a judicial nominee qualified for the job per se, as it cannot be said that the models adequately capture the decision making of the ABA's committee.  Indeed, the measures in the models here are composed of information that is not exclusive to the ABA and would be easy for Senators or the President to obtain on their own:  judicial experience, clerkships, law professorships.  What the ABA brings that is valuable and that makes it relevant to Senators trying to make a decision on a given nominee is its extensive research on the nominee that includes interviews of colleagues and bosses and lawyers and judges who have personal knowledge of and information about the candidate.  It is quite likely that the information obtained through these interviews is highly related to the ABA’s rating, as is other information the ABA considers, including all published writings of the candidate; i.e., opinions and briefs and law review articles. 

Of course, the interviews are confidential and I do not know whether the reports the committee issues are available to anyone not serving on the committee, and so this sort of information would be extremely difficult to include in an analysis of the Committee ratings.  But I think it leaves the reader wondering how any model of this particular decision-making process could be fully specified and how much credence we can place on the findings reported here regarding the ABA's apparent preference for liberal potential jurists, given limitations on specification.  In addition, I wonder whether the results are time-bound, given that liberals used to claim that conservative jurists were favored.  What changed in the ABA's process between then and now?  I think the authors are onto something when they discuss the criteria the ABA says it uses in these evaluations, which includes judicial temperament.  (Indeed, reading the ABA's Senate Judiciary Committee testimony in the case of Michael Wallace (who received a Not Qualified rating from the ABA) raises all kinds of questions about how that criterion operates in practice. And, I would argue that one measure in the model -- service as a Congressional staffer -- may actually tap into that criterion (since open-mindedness is one component of judicial temperament).)  But, is that criterion new?  

I suppose that's enough for now.  Thoughts?

  

Discussing "Bias and the Bar"

Hello ELSblog readers! We are Richard Vining (Assistant professor, University of Georgia) and Amy Steigerwalt (Assistant professor, Georgia State University), and we have been asked to contribute here this week. We're also the authors of a recent study, "Bias and the Bar: Evaluating the ABA Ratings of Federal Judicial Nominees" (available at SSRN), that has gotten notice here and elsewhere. In that paper, we assess the influence of several variables on the ABA ratings of nominees to the U.S. Courts of Appeals from 1985-2008. The headline from the paper is that we find, all else being equal, the longstanding conservative complaint of ABA bias against conservative judicial nominees may have merit. Liberals/Democrats are in fact more likely, holding all else constant, to receive a Well Qualified rating than their conservative/Republican counterparts. Response to our study so far has included journalistic reports, scholarly interest, and enthusiasm. The op-ed page of the New York Times offered its take, while the Wall Street Journal op-ed page offered an alternative opinion.  Amy will also be interviewed by Fox News Channel on Monday (or possibly Tuesday, due to swine flu coverage), with excerpts to appear on Special Report with Brett Baier at 6PM EST.  Given the attention the paper has received, we welcomed the opportunity to discuss our methodology and findings with the readers of the ELSblog.

As many of you are probably aware, the American Bar Association’s Standing Committee on the Federal Judiciary has, since the 1940s, evaluated nominees to the federal courts as to their professional qualifications, integrity and judicial temperament. Traditionally, the ABA provided the president with evaluations of potential nominees, and these ratings were then forwarded to the Senate if the nominee was selected by the president. In 2001, President George W. Bush, removed the ABA from the presidential selection process, arguing that the ABA’s ratings were biased against more conservative nominees.

In "Bias and the Bar," we sought to address in a methodologically sophisticated way the frequently heard claim that the ABA is biased against conservatives. While seemingly a straightforward empirical question, the existing evidence was anything but. Many earlier studies contained numerous problems, such as only examining a small subset of nominees or studying only those who were ultimately confirmed. Some of these studies were also funded or supported by conservative organizations, which itself raised red flags. We conceived of this project in the summer of 2008, and believed that determining whether ABA ratings of judicial nominees are actually biased was important given the impending change in presidential administrations in January 2009. Whether John McCain or Barack Obama won the general election, the new president would be faced with a number of pending vacancies on the federal bench as well as the decision of whether to restore the ABA to its traditional role of evaluating nominees pre-presidential selection. It should be noted that when we began this study, we did not necessarily expect to find evidence of any type of bias. We were, however, curious to discover whether George W. Bush’s reasons for removing the ABA from the nominations process had merit.

To answer this question, we analyze the ratings of all nominations to the U.S. Courts of Appeals between 1985 and 2008; our sample consists of over 300 nominations. We present two competing theories, the political theory and professional theory, which reflect the rival views of the ABA's ratings process – i.e., ratings based on ideology vs. ratings based on professional qualifications and/or competence. We collected data on each nomination, including a number of professional history variables (e.g., years as a judge, years as a lawyer [and of what type], and whether the nominee had been a full-time law professor, Supreme Court clerk, or circuit court clerk) as well as indicators of political leanings (e.g., party of appointing president, Giles et al. [2001] ideology scores, and years as a congressional staffer) and demographic characteristics (gender and race). We chose to use an ordered logit model with an ordinal scale which ranges from Not Qualified to Well Qualified and takes into account split ratings by the ABA’s Standing Committee on the Federal Judiciary. We ran two separate models, one containing the party of appointing president and the other the nominees’ GHP scores. Since GHP scores are closely tied to the party identification of the president, the two measures are highly correlated. We ran both models, however, in order to ensure robust findings and to determine whether changes in ideology, rather than just mere partisanship, influenced the ABA’s ratings of nominees. Our models, therefore, “controlled” for all of these factors to determine their independent effects.

Most notably, we find that, all else being equal, Republican/conservative nominees are less likely to receive the ABA’s highest rating than Democratic/liberal nominees. These findings are robust, with very little difference between the results using the appointing party and those employing GHP scores. However, we also find significant support for the professional theory as well: all else being equal, nominees who have more experience as a judge are more likely to be rated highly, as are nominees who have more experience as a government attorney. Former circuit court clerks are more likely to receive higher ratings, while former congressional staffers, possibly due to the overtly partisan and political nature of such positions, are less likely to receive higher ratings. In sum, we find both that the ABA looks to professional experience when evaluating potential nominees to the federal circuit courts, and that conservative nominees receive lower ratings than liberal nominees.

We cannot explain exactly why Democratic/liberal nominees, all else being equal, are more likely to receive a Well Qualified rating. One possible conclusion is clear bias on the part of the ABA, though we believe this is unlikely. Alternatively, there may exist some unaccounted for factor which distinguishes Democratic nominees from Republican nominees. Finally, it is possible that the ABA’s evaluation procedure, while designed to be neutral, contains an inherent liberal bias as to what qualifications or types of experience are valued by the ABA. We believe the disparity may be due to the ABA’s rating criteria of “judicial temperament,” defined by the ABA as including open-mindedness, commitment to equal justice and freedom from bias. We agree that potential federal judges need to be open-minded, of the highest integrity, and unbiased in their approach to each case and litigant. However, if these words are more broadly interpreted to suggest, for instance, that nominees must hold certain views on public policy issues such as affirmative action or equal rights, then ideological biases may permeate the ABA’s evaluation process.

In the future we plan to continue this research by extending our analysis to the nominees to the federal district courts; one possibility is that partisanship and/or ideology may play a more prominent role in the vetting of circuit court nominees given the greater role of these judges in policy-making.

As is the custom here, we are happy to field questions regarding the study, its methodology and findings, and so forth. Thank you for your interest, and the ELSblog for the invitation.

More on Bankruptcy, Methods, and the Selection Effect

I previously posted on a series of bankruptcy papers that vigorously contest assessments of the recently reformed bankruptcy code. Over at Concurring Opinions Dave Hoffman (Temple) very helpfully weighs in on one of the more basic points in dispute--how to understand and account for selection effects. Dave's summary of the issue is well worth a read.

SSRN and Copyrights

While writing these blog entries and in dialogue with some commentators I realize that it is not just publishing venues that separate political scientists who specialize in public law and Empirical legal scholars and that is posting of manuscripts prior to and even after publication. While many political scientists post their research on their own websites and will post papers on conference proceeding websites before delivery the paper. I would venture to guess that most political scientists do not post manuscripts during the review process. Perhaps the “blind” peer review process inhibits academics from posting work under review, although we do attend conferences and present our research to panels before submission to a journal.

More than posting of work under review or under submission another issue that I now deal with as editor of the Justice System Journal is requests by law professors to post their accepted and published manuscripts on SSRN. Personally I do not care if someone does that. It disseminates the work and helps to publicize the Justice System Journal. Besides, most journal articles are available in some digital format. We are available on Westlaw and on Hein On Line, and the National Center for State Courts, the sponsor of our journal puts each article in a PDF file and posts in on line in the publications section of the research division of the National Center for State Courts.

However, publishers of the journals do care. For most academic journals publishers retain the copyright to the work. Before publication I am required to assign the copyright to the publisher. To reprint work in an edited volume or in a book that reproduces the words, tables or figures, I have to get written permission from the publisher.

Thus my dilemma with authors who want to post their work that is in the JSJ on SSRN. How do I satisfy the publisher’s copyright and the author’s desire for greater dissemination of his or her article?

Journal Rankings

In response to Alex’s question on journal rankings, there is no website that I know of but there is published information.

 Acceptance rates for journals comes from:

 Martin, Fenton, and Robert Goehlert. 2001. Getting Published in Political Science Journals. Washington D.C.: American Political Science Association.

Two studies by Jim Garand and Mike Giles have become very influential in journal rankings:

 James C. Garand and Micheal W. Giles. 2003. “Journals in the Discipline: A Report on a New Survey of American Political Scientists. PS: Political Science and Politics 36(2): 293-308.”

 Micheal W. Giles and James Garand, “Ranking Political Science Journals: Reputational and Citational Approaches.”PS: Political Science and Politics 40 (October): 741-751

Empirical Studies for Non empirical audiences

The journal that I edit, The Justice System Journal, is a peer reviewed public law and judicial politics journal that publishes articles on courts and court administration, very broadly defined. The journal is available online through Westlaw and Hein on Line. Unlike The Journal of Empirical Legal Studies, we operate on an exclusive submission basis.  We serve two audiences. First we have an audience of court administrators and court professionals. These readers want information both practical and theoretical about the state of courts and court administration as well suggestions as to how to improve performance. Our second audience, and the one most of you would be interested in, is that of social scientists and those in the legal profession.  We seek quality social science work, and will accept different methodological approaches.

The practice of some previous editors was to insist that submissions to the Journal forgo most statistical and methodological language that would, in their opinion, be incomprehensible to much of the audience of court administrators. While I understand the concern, that begs the question of where one draws the appropriate line in talking to our various audiences. The prior editor did not allow any use of numbers, data or the use of statistical of methodological language and analyses. Thus, authors could not discuss actual data or methods and could not present coefficients, only asterisks to represent statistical probabilities of occurrence.  There was a thought that academic authors could save their detailed data driven work for a second version and publish that in other journals. This “secondary market” of publication rarely, and I think appropriately rarely, occurred. Once submitted to us, authors were understandably reluctant to submit essentially the same piece to another journal.

I am not criticizing past practices for this, but I am trying to find the right mix. I will include methodological analyses and full reporting of data and results. I do ask, however, for greater clarity and greater explanation of data and analysis with the understanding of our dual audience. I am now trying to decide about formal theory analysis. Can they be explained so that a diverse audience can appreciate that?