Of Empirical Work & Law Reviews: A Cautionary Tale from Stephen Wasby

[The author of the following post is not me but former guest blogger Steve Wasby. He offered to discuss an unusual experience involving the publication of empirical work in a student-edited law review. I am posting it on his behalf. – Bill]

What follows is a cautionary tale, with potentially profound implications for those who conduct empirical legal research -- whether of a highly statistical sort or using less-high-powered statistics.

Perhaps many of us who have published in law reviews, or who have attempted to do so, are familiar with the vagaries of the process. In particular, some of us -- myself included -- have encountered difficulties when we have used sources (such as interviews or unpublished documents) with which the law review editors, and particularly their cite-checking minions (whom I now refer to as “cite-checking munchkins”), are not familiar. Sometimes the process goes well, particularly if an articles editor or the editor-in-chief had some social science training, but more often it is a PITA. Somehow, while we “cuss” to each other about it, we live through it.

What I am about to report may be a “one-of-a-kind” instance, and I am not able to determine the source of the “idea” I report. However, given the possibility that it will arise elsewhere, I thought the readers of ELS Blog would find the story interesting. If it is sui generis or a “sport,” so much the better. If we have managed to defeat this incursion, so much the better still.

NOTE: The name of the law review in question is not used in what follows, nor are the names of the other players. This is done because the point is the story, not a tar-and-feather job on any individuals. Some redactions therefore appear in quoted material.

The story: I was involved with some others in a set of articles on a particular subject; these articles were to appear in an issue of a law review. A number of the articles were “social science-y” although, in recognition of the law review audience, the statistics used were not to be particularly sophisticated. During the law review editing process -- in which I was not involved -- word came to me that the law review had informed an author that the following message was to be attached to each article involving empirical analysis:

“Some of the conclusions and observations contained in this Article are based on the manipulation and analysis of data and are entirely the Author’s. The [X] Law Review played no role in compiling or testing the data, and therefore, is not responsible for the conclusions drawn therefrom.”

It will not surprise you to learn that this prompted a response, in the preparation of which I had fine assistance from a law professor who also publishes empirical studies of the law.

The response, in relevant part, called the disclaimer “inappropriate and unnecessary,” and pointed out that the law review required no comparable disclaimer for non-empirical work; in short, there was nothing which said, “X Law Review played no role in the doctrinal analysis undertaken here” -- even where the analysis might involve a technical area of law with which none of the Review’s editors would be familiar or for which they could vouch.

Matters did not immediately improve. Indeed, the law review’s editor-in-chief can be fairly said to have resisted making any change, although there was an apology for failing “to realize that such a disclaimer was either unusual on inappropriate”; the editor later noted, in what is certainly an understatement, that “perhaps common sense should have told me so.”

While claiming no “distrust any of the authors or because we are of the belief that their analyses will not stand up to scrutiny,” the editor said that law review editors “are inherently a risk averse group and make some of our decisions based on worst case scenarios,” and continued by saying, “I still fail to see how stating that the author’s work is his own and that the Law Review played no role in compiling or testing the data, and therefore, is not responsible for the conclusions drawn therefrom is either inappropriate or unusual. We are simply stating the truth. I do not believe that it would be entirely clear to some or most readers that the Law Review did not test or verify at least some of the data used in an article to ensure accuracy unless we specifically say so in a footnote right up front. Most readers know that we check everything we can and to some readers that probably includes data.”

The editor also spoke of the law review’s “concern that some people might have the expectation that we verified the accuracy of author assertions which rely on manipulated data.” (Notice how the editor can’t give up the notion of “manipulation.”) As to doctrinal analysis, the editor said similar disclaimers were not inserted there “because, even if our editors do not have experience in a particular area, we can, and do, go back to the case law, statutes, or whatever else is being used as support and verify that the author’s reading of or assertion based on those materials is reasonable and accurate.” While indicating a willingness “to adjust the language of the disclaimer to make it more amenable to you and the authors’ tastes,” the editor “[stood] by the decision to have some type of disclaimer stating that we did not verify the data used in the articles.”

Despite my recognition that law school deans often wish to avoid anything that could be seen as meddling in the work of student-run law reviews, my next step was a message to the dean, with a copy to the editor. Given what then happened, at least in this case, my experience would suggest that one ought not to be hesitant to communicate with a dean. The message to the dean said, in relevant part, that “to change the wording [of the disclaimer] is to concede the validity of the disclaimer, which [I] object to in principle”; particular objection was stated to “manipulation” as being grating as well as pejorative. As to the editor’s argument that law review editors would treat doctrinal materials in the same way, my response was that “all this should mean is that if the editors are concerned about the interpretation of the statistical data in any of the articles, they can and should ask the authors for explanation or greater clarity in presentation.” It was also noted that the disclaimer carried law review editors’ risk-adversity to an untoward extreme and that, moreover, the disclaimer seems to say, “We are publishing this material but holding our noses,” thus making the Law Review look foolish.

Particularly relevant to this blog, the message to the dean noted that the law school had indicated a wish to bring interdisciplinary work to the law school, but that “interdisciplinarity” (pardon the word) would not get very far if the law review editors were to treat social science material as guilty until proven innocent.

During this communication process, it was also learned that the disclaimer had not been required of all those who had undertaken empirical work in their articles. Whether this was a result of one editor imposing the disclaimer while others did not or resulted from the imposition of the disclaimer at a stage of editing not yet reached by all articles, I do not know. In any event, the apparent inconsistency further weakened the argument for the disclaimer.

The law review editor’s immediate response indicated irritation at the contact with the dean. Claiming no intent that the disclaimer “serve as an indictment of social science” and conceding that use of the phrase “manipulation . . . of data” has a negative connotation at best, the editor went on to “assure you that we did not mean to imply that we believe any of the authors have manipulated data in the unethical sense.” Recognizing that “absolutely no disclaimer will be permitted by you,” the editor asked if it would be “more amenable” if the disclaimer were removed from each article and replaced by “a short statement in the introduction to the . . . issue that ‘The views and analysis contained within the articles are those of the authors alone.’” 

The editor soon confirmed that the disclaimer would not appear in the individual articles, but the editor’s effort to seek agreement for use of the language in the introduction to the issue produced further objection. In particular, the objection was that it was one thing for an author to make a comment about the basis of his or her own analysis, but quite another, and “very different,” for the editors to say something about an author’s article, particularly when the result was to deprecate the very articles the law review had agreed to publish and to walk away from those articles, in addition to being “demeaning” to the authors. It was also noted that the law review did “not do this for any other type of article, even though it is as clear for those other articles as for these that the views are those of the authors.”

A further point conveyed to the law review during the exchanges was “some bafflement about the concern that apparently underlies” the proposed disclaimer. It was explained that “everyone who reads a law review knows that the signed articles reflect the research, scholarship, and conclusions of the authors. Indeed, if anything, a disclaimer would serve less purpose for social science research than for doctrinal analysis, because readers would probably assume that, while the editors would have engaged in cite-checking, they would never have replicated an author’s statistical analysis.”

It can be reported that, in due course, after considerable communication with the law review, the dean, and the authors engaged in the process -- they were strongly supportive of the objections that had been made -- and after the dean had met with the law review editor and the faculty adviser to the law review, the law review receded completely from its position; there will be neither disclaimers in the individual articles nor any “backing-off” language in the introduction to the issue. So, in the end . . . . all appears to be well.

Yet knowing the ability of difficulties, aided by Murphy, to disseminate themselves and to reappear when least expected, it seemed important to convey this tale to you, so that it could serve a cautionary purpose.

Stephen L. Wasby (Professor Emeritus, Political Science, University at Albany)

Comments

[Rick Lempert]

I fully understand Steve's reactions, but there is a culture clash here, and I have some sympathy for the law review editors. Law reviews pride themselves on checking each cite and often not just for form but also for substance in that at least some editors will look at the cited article and if they don't think it supports the point made will query the author to be sure the cite is correct. Putting aside my view that this is a pathology of law reviews - professionals should be responsible for the accuracy of all they write and shouldn't need students to check their work - consider the perspective of the law review editor who lacks the capacity to check data or analyses based on it. So while the editor would be willing to have his/her journal take responsibility if a cite is incorrect in form or misquotes something, etc, he she/does not want responsibility or blame if something is wrong with the analysis. Thinking (a bit of self-aggandizement here) that the law review will be held responsible for not catching statistical errors in the same way a reader would think less of not just the author but of the journal if the form of the citation is wrong or there is a misquote, the editor wants to be sure the journal is not blamed. An honest disclaimer might be something like, "We can barely understand the statistics in this article much less check the data and analysis, but we have done our usual fine job in changing "whiches" to "thats" and being sure the dates of all cited article are correct." But this isn't very dignified so Steve had to deal with various other versions. None of them, I am sure, were intended to suggest that his paper had any deficiencies (I expect it never occurred to the editor that they could be read this way), just that the law review should not be blamed if it did.
For what it's worth, I have never heard of anyone else who has published an empirically based article having to confront such silliness, but I mean it when I say that I have some sympathy for the editor because I appreciate his/her mind set. The real problem is not the editor's; rather it is a system which leaves it to students to determine what pieces are worth publishing, allows articles to be of almost any length, requires oodles of mind numbing and unnecessary footnoting and has no peer review. The bright side - law reviews often publish excellent work, including empirical pieces, some of which might not be published elsewhere.
Rick

[Al Brophy]

Thanks for the story. Yet more evidence that faculty are taking an increasing interest in what happens in their schools' law reviews. One minor point here: if others are going to go around the law review editors, it might be a nice courtesy to start with the faculty advisor.