Less Public Sources of Information About Judges for Empirical Research

In yesterday’s guest blog, I listed a variety of publicly-available sources of information on federal and state judges and court decisions, where researchers may find ready-made databases or easily-accessible raw information for formulating appropriate variables for empirical testing of a research question regarding judicial decisionmaking. Nearly all of the sources I listed yesterday may be accessed directly from a researcher’s computer, by simply connecting to the internet.

Today, I want to supplement that list of public sources by discussing three sources of information that are not as readily available to researchers and thus to be tapped require greater effort. If researchers are to constantly push the envelope by exploring new dimensions of or influences on judicial decisionmaking, existing sources of information must be more thoroughly exploited or new sources of information must be found. In my limited experience as an empirical researcher and my greater experience as a consumer of research on the courts, I have found that the most interesting studies of judges in action tend to involve creative efforts by researchers to find and use new data. Sometimes researchers take information long available, but use the data to create new variables or to code common variables in a more precise or efficient way (or at least a manner more appropriate to the particular purpose of the study at hand). Even more exciting are those occasions in which a researcher has unearthed new information that had not previously been collected or fully exploited by others or that has become available only through new records.

Collections of Judicial Papers:

One quasi-public source of information, of particular value to those who study the United States Supreme Court, consists of the papers of deceased justices which have been donated to a library or other archive. Most recently, researchers have pored over the papers collected by Justice Harry Blackmun, producing multiple books, articles, and symposia. See Linda Greenhouse, Becoming Justice Blackmun: Harry Blackmun’s Supreme Court Journey (Henry Holt & Co. Times Books, May 2006); David J. Garrow, The Brains Behind Blackmun, Legal Affairs (May-June 2005); the responses by several leading legal figures to the Garrow essay; and various other articles and symposia examining the Blackmun Papers.

Although the papers of past Supreme Court justices are available to researchers, access is more limited and requires a significant investment of time (and travel expenses) so that a researcher may study physical documents at their archive location, given that only certain excerpts are available on-line or have been reproduced and distributed to other libraries. For those willing and able to examine the papers, the rewards may be great in learning more about how leading cases were decided. Translating what is learned from these sets of documents into something useful in quantitative empirical work is not straightforward, although the original documents and memoranda may indicate which factors consciously influenced the justices deciding a case or series of case and thus may assist in devising research questions or considering what case type variables should be included in an empirical study of judicial decisionmaking.

I’d be interested to learn whether other researchers on the courts have had success in finding and making effective use of papers gathered by deceased judges on the lower federal courts or state appellate courts.

Surveys of Judges:

One of the most frustrating experiences for a researcher is to develop a large data-set in which information is available for all but a handful of judges on a particular variable, such as practice specialties by a judge before assuming the bench. To a limited extent, this problem may be addressed by inserting a proxy, such as a mean, so as not to exclude that observation altogether because of missing data on a single point. It is much more preferable to fill in the gap in the data by obtaining more information.

A researcher should consider the possibility of sending a survey to judges for whom data is missing. Given the tendency of researchers to study relatively recent sets of decisions, most of the judges within the typical study are still alive and indeed still on the bench. The effectiveness of the survey method of gathering information may surprise many researchers. If the researcher devises a brief and easily-completed survey and makes it simple for the survey to be returned, the rate of return may be very high.

In our study of federal district court decisions ruling on the constitutionality of the federal sentencing guidelines, we found that standard biographical sources were not always clear as to whether a prior legal position involved prosecutorial duties. See Gregory C. Sisk, Michael Heise and Andrew P. Morriss, Charting the Influences on the Judicial Mind An Empirical Study of Judicial Reasoning, 73 New York University Law Review 1377 (1998). For example, while the listing of a judge’s previous position as an Assistant United States Attorney (AUSA) likely reflects criminal prosecution experience, some AUSAs handle civil litigation against the federal government. We sent a survey to all judges whose biographies listed service as an AUSA, service in the military JAG Corps, or municipal or county official positions, but where the description did not clearly include prosecutorial work. Of the 34 judges we surveyed, we received a response from 30.

Based on this experience and what I’ve heard from others, the secrets to a successful survey of judges directed to a particular variable appear to be three in number. A judge is more likely to positively respond if (1) the information sought is not personally-sensitive, (2) the survey is easy to complete in a few moments of time (we used a simple question requiring only a check-mark to answer), and (3) the survey is easily returned (we included a self-addressed and postage prepaid envelope).

Contacting Other Researchers:

If empirical researchers adhere to the replication standards of our disciplines, the necessary expectation is that data will be made available to other researchers, preferably through public archives or by other easily-accessible means (such through the researcher’s web page). That expectation is not, however, always satisfied, at least immediately. The reasons for the absence of an easily-accessible set of data underlying a recently published study may be at least threefold: First, the researcher may simply have neglected to make the data available or (less likely) be unaware of that expectation. Second, the researcher may not yet have completed a series of studies using the same data and thus be reluctant to share data gathered at some expense and effort until the researcher has had the first opportunity to fully exploit it. Thus, during an embargo period, the data is not made publicly-available. Third, the researcher may have gained access to information pursuant to a promise of confidentiality, thus precluding making all of the data generally available to the public.

In my experience, however, our colleagues are most generous in sharing data when contacted. Even if the information is being temporarily embargoed, the researcher may be willing to share data when specifically asked or subject to some limitations. Likewise, when data has not been made publicly-available because it includes confidential information, the researcher may be willing to share it with another researcher who agrees to abide by the same confidentiality restrictions.

Tomorrow, I’ll discuss a couple of variables that have proven “tricky” in recent studies of judicial decisionmaking, either because of difficult in finding pertinent information or controversy in measurement.

Greg Sisk

Comments

[Gozombas]

Thanks for the amazing post, Now I will read time to time that...