The Trouble with Surveys

It’s interesting to me that, as Todd Peppers says, he “ha[s] worried less about low response rates than responses from law clerks who downplay their job responsibilities while exaggerating their employer’s intellectual abilities and moral virtues.” Certainly that’s a problem for survey research, not just surveys of Supreme Court law clerks, but in general. One challenge of surveys is that many people won’t participate. Another challenge is that people who do participate may not give objective responses.

It’s a problem faced both by academics and by journalists. Books like The Brethren and Closed Chambers were criticized as based on information from law clerks who allegedly exaggerated their roles. Now Todd expresses concern that law clerks may have understated their roles. Both cases lead me to question whether maybe we should be trying to find other ways to gather information than surveying law clerks.

Certainly surveys can provide valuable information about topics that can’t be studied any other way. And I’m sure Todd, Stephen Wasby, and Artemus Ward do not rely solely on surveys of and interviews with law clerks in their studies of court practices and procedures. Nonetheless, it seems to me worth thinking about other ways in which information about the inner workings of the Supreme Court (and other courts) might be gathered.

One obvious source of information is the Justices themselves. In a comment to my earlier post, Todd explained that while he was able to meet with Justices Stevens and Scalia, Justice Scalia would only speak off the record and none of the other Justices would speak with him. My reaction is that if the Justices won’t speak with researchers, is it any surprise that the Justices’ law clerks won’t either? That’s a pretty strong signal by the Justice that such matters are in fact confidential (or at least might be perceived as such a signal by a former clerk). But even if the Justices would regularly and openly speak with researchers, there would seem to be the same concerns about the objectivity of their responses. If in fact Todd’s Justice X would be willing to be interviewed, we certainly wouldn’t expect him to own up delegating all the work to his clerks (much less the problems in his personal life).

(As an aside: Todd suggests that it might be in the public interest for this sort of information to come to light. That was the same argument Eddie Lazarus, author of Closed Chambers, used to justify his journalistic exposé on the Court. It’s not the law clerk’s place to “go public”; it’s a matter for the Court as an institution to deal with.)

An alternative is to try to develop objective ways of examining how the chambers of Supreme Court Justices work. Textual analysis of opinions might be a way of determining the extent to which opinions are drafted by Justices or their clerks. The article by Timothy Johnson et al. on “The Influence of Oral Arguments on the U.S. Supreme Court” (described in the post below) uses data from Justice Blackmun’s papers to test whether oral arguments influence the outcome of cases. Perhaps information from bench memos might be used to similar effect (or information from cert pool memos might provide a basis for testing whether clerks influence agenda setting). Indeed, a team of political scientists and law professors has used a statistical model to predict the outcomes of Supreme Court cases more accurately than panels of legal experts were able to do. Theodore W. Ruger et al., The Supreme Court Forecasting Project: Legal and Scientific Approaches to Predicting Supreme Court Decisionmaking, 104 Colum. L. Rev. 1150 (2004). (Embarrassingly, the experts predicted fewer than 60% of the Court’s cases correctly, worse than simply picking “reverse” every time!) Maybe a variable could be added to the model based on the identity of the law clerk responsible for the case in a particular chamber. Some of these approaches may already have been tried, either with or without success. No doubt creative scholars could come up with many others.

Now don’t get me wrong: the research done by people like Todd, Stephen, Artemus, H.W. Perry (whose book I own), and others based on interviews with law clerks has been fascinating and invaluable. But that research methodology, like all research methodologies, has limits, and those limits aren’t unique to law clerks.

Comments

[Stephen Waby]

Indeed, Chris is right that I have used material other than survey/interview responses in my research. I have used opinions (as suggested), but, more important, and as hinted at in my original post, I have been able to see casefiles, which contain memoranda between clerks and judge; those are invaluable in determining what clerks actually do, and which suggestions by the clerk are adopted (on the influence by clerks point).
As to the difficulty of obtaining interviews with judges, I would suggest that some of the difficulty is trying to interview "the Supremes" -- it's like trying to interview members of Congress: there are so many demands on their time (and, in their case, so many others have already interviewed them), that one can't access to them. Now, if people would study other courts --e.g., the courts of appeals-- they would find judges much more willing to be interviewed, about PROCESS, not about specific cases, of course.
Steve Wasby