ELS and the question of Refereed vs. Student Edited Law Review

I have seen this question discussed in general on a number of blogs, but I am particularly interested in what empirical legal scholars thought about the benefits and detriments of both types of publishing outlets. To some degree, your views may turn on what your institution rewards (law reviews or peer reviewed journals), but there is also the matter of what journals you find publishing in most rewarding personally and intellectually. While there is increasing cross-over in both publishing and reading in the two camps, I think that you do make an important decision about the potential audience for your work when you choose one over the other.

This being said, there are a number of what are reasonbly considered law review outlets that are peer reviewed or refereed.  In reviewing John Doyle's excellent law review rankings and submissions webpage, I count over 200 law outlets that are refereed, including the Supreme Court Review, Journal of Legal Studies, American Business Law Journal, Law & Society Review, and the Journal of Empirical Studies, among others.  Of course, there are a multitude of other referreed outlets in discplines such as political science, economics, criminology, public administration, and sociology, to name just a few.

I would be interested in hearing what others think about the relative advantages and disadvantages of publishing in student edited law reviews vs. peer reviewed journals. Here are what I believe to be some the advantages of each:

Student edited law reviews:

1) Simultaneous submission. Perhaps nothing is more annoying than having to wait six months or more for the slow wheels of the peer review to render a verdict - and then on to the next journal for another slow, nail-biting wait. The sequential process of peer review is especially hard on untenured people, for whom timing is very important.
2) A larger, and more diverse audience. This might be arguable, but I believe that most will at least agree that law review exposure usually brings you a new audience for your ideas, relative to the rather limited audience of some social science disciplines.
3) Greater academic freedom. There, I said it. For whatever reason, it seems that the peer review process somehow tends to limit the type of questions that people think are "interesting," and therefore worthy of publication.  It is my impression that law reviews are more open to different approaches and new research topics. I'm quite sure that many will disagree with me on this point.

Peer Reviewed Journals

1) A more rigorous review process. There, I said it. I think that good things can come from having peers review your work and I believe that many of my publications are much better because of the peer review process.  Further, with regard to ELS, some may question the capability of student editors to assess high level empirical methods. That being said, it is my understanding that informal networks of peer review in legal academia go a long way toward addressing this concern.
2) A blind review process. I've seen some discussion on the lack of blind review with regard to law review publishing and I think that the "academic celebrity factor" in law review publication decision making is a bit hard to deny. Balkin and Levinson's witty and insightful article "How To Win Cites and Influence People" is especially revealing on the law review publication process and gaining citation. To be sure, the blindness of the review process in refereed journals is perhaps open to question and the development of internet search engines hasn't helped this concern.
3) It keeps you humble. I really couldn't think of a good third advantage, but I wanted to mantain symmetry of the categories. However, peer review certainly does keep you grounded - nothing like a blistering (and insightful) review to remind you that you don't know everything.

I'm sure that you all will have plenty to say about this. If I have offended people from both sides equally ...  well then, my job is done. So, in the words of Saturday Night Live's Mike Myers, playing his mother-in-law, here's the topic ... "Refereed vs. Student Edited" .... talk amongst yourselves.

Comments

[Tony Smith]

I'm glad Jeff raised the issue of academic freedom. For the non-tenured in particular a disregard of the content in favor of the low-level rationality assesment (which kind of law review is it) means articles are judged on the package not the content. This preference for peer-reviewed over student edited also rests on a potentially false assumption that peer reviews are never relegated to graduate students. If the grad students are peers, it seems that the grad student editors would not be that different. It's likely that it's functionally blind review as well since few of the law review students would know or ahve any reason to care who political scientists are - nice topic Jeff, thanks for the push to think about these things more thoroughly -

[Margo Schlanger]

Responding to the Anon. post above, I think lots of untenured professors try FIRST to publish in law reviews, not peer-reviewed journals, because of the speed of publication and because there is always the risk that some (maybe the crucial few) members of their faculties discount non-law review publication--in part because peer reviewed articles are shorter, and also for other reasons. I certainly wouldn't count law review publication as even a noisy signal of low quality.

[Anon]

What does publishing an empirical paper in a student-edited law review do for your impression of the quality of a piece? For example, there is an empirical piece forthcoming in Michigan Law Review recently posted on SSRN by an untenured professor. Normally, that would sound great. It is so empirical, though, full of models and regressions, that one has to wonder whether it failed at some peer reviewed journals before getting to that point. The piece is probably excellent, but the short law tenure clock and the bias for general law review publications may have forced the author to move on quickly rather than endure the wait for other peer reviewed journals and possibly miss the deadline for closing the tenure file. If the author had published in the second or third peer-review try, no one would think twice about it (or even know). Because it went in a general student-edited law review, though, I wonder if people think less of it (of course, they should read the piece, but if you're not an expert in the field you may assume experts found flaws). It is plasuible that it was submitted exclusively to law reviews because of a bias on the author's faculty for general student-edited law reviews, but would that be your first assumption when seeing a rigorous empirical piece in a law review?

[Alex Robbins]

Honestly, for ELS work I think the greatest defect is the lack of ELS knowledge on the part of law students. The fact that most editorial board members aren't qualified to evaluate ELS work is only going to increase the prestige factor, because they have nothing else to go on (short of farming out to an ad hoc "referee," which involves a lot of extra effort). Even if there's a qualified econometrician on the staff, the way (in my experience) that law reviews assign board positions and editing assignments is generally not tailored to one's academic specialties. In other words, if you've got a good ELS resume and work at a top school you may be in good shape, but if you're a 3L like me who paid attention in undergrad econometrics courses it's going to be considerably harder to be taken seriously. (Speaking of which, if anyone out there has any advice on what I should do with my own empirical paper, http://papers.ssrn.com/abstract=941238, I'd be happy for the feedback!)

[Michelle Mello]

I publish primarily in peer-reviewed journals (medical, policy, and economics journals). While I begrudge the amount of time I spend responding to sometimes niggling, sometimes irrelevant, sometimes indecipherable, and sometimes just plain wrong reviewer comments, I have to acknowledge that I have never published a paper that was better at the beginning of this process than at the end.

A well-known problem with peer review is poor interrater reliability. This is a particular problem for interdisciplinary research, because what happens at many journals is that the editors invite reviews from representatives of each discipline (rather than from people who work across both). You then end up with reviews that are, at best, orthogonal to one another and at worst, demanding radically different approaches to issues in the paper and reflecting very different philosophies about methods, what constitutes adequate and reliable data, and what the "important" findings are. You also have to spend a lot of time explaining the basics of each discipline to the reviewer who isn't familiar with it. Hopefully, as the number of scholars trained in multiple disciplines grows, this problem will diminish over time.

On the plus side, peer reviewers make a huge contribution in identifying what constitutes cutting-edge research on a topic of importance. There may be a bias against things that cut against accepted scientific inventions, but it is very valuable for editors to receive informed explanations of how the scholarship advances the field (or doesn't) and how it responds (or doesn't) to topics of great policy significance or scholarly interest. The trivial and the banal have trouble finding a home in peer-reviewed journals.

Overall, I think the peer review system has many warts, but is an appopriate form of review for empirical research. As legal research comes to more closely resemble other forms of social-science research in its methods and standards, I think it will be hard to resist treating it as social-science and scientific research are treated in the publication process.

[Bert Kritzer]

Jeff refers to a number of peer-reviewed "law reviews"; however, I think there is an important distinction between law reviews and the broader category of law-oriented journals. I would not describe Law & Society Review, Law and Social Inquiry, Law and Policy, Journal of Empirical Legal Studies, or the various law and economics journals as law reviews.

It is important to keep in mind that from the student editors, the primary role of their work on the law reviews is professional socialization and training to be a lawyer. It is not to insure that high quality empirical work is published, or that the empirical elements in that work meets the standards of professional social science.

For empirically-minded scholars, the major problem with student-edited law reviews is that the probability of encountering a student editor who has any knowledge about or experience of empirical research is quite low. This means that the editor cannot really help an author make the empirical part of the article better. Sure, the editor can check citations and bibliographies, but frankly errors there are annoyances while errors in empirical analysis are much more serious.

In fact, I suspect that student editors who don't have a clue are probably more likely to make empirical articles worse than to improve them. Often this may involve demanding that authors do things that don't make sense to any knowledgeable empirical scholar. I once had a student editor ask me to provide a citation for the original data I had collected and was presenting in an article; it took a couple of rounds of email to convince the student that no cite was either needed or appropriate.

I have on several occasions been asked to do tenure reviews for law school faculty who have published empirical work in both student-edited journala snd peer-reviewed journals. The quality of the papers passing through the peer review process is almost always markedly superior to those that have been man or woman-handled by a student editor.

Finally, regarding Frank Cross's point that peer review is single blind, I heartily agree that for senior scholars submitting to peer-reviewed journals this is the case. However, I don't think peers are as affected by the "wow" factor as are student editors. If anything, I suspect that anonymous peer reviewers may be more likely to "stick it to" a senior eminence.

[Robert Justin Lipkin]

Jeff offers a good mix of the pros and cons of student-edited and peer-reviewed law journals. With respect to the wait that peer-reviewed journals in the greater academy sometimes require, I wonder whether anyone has ever documented the longest wait. I once published an article on Hume's ethics in the Australasian Journal of Philosophy--a pretty good placement, if I recall correctly--that took approximately ten years to place. I had several rejections, some from lesser journals, until the Australasian Journal of Philosophy accepted it. In this case, single journal submission was excruciatingly frustrating.

I think it important to also distinguish between mainstream articles and maverick, for want of a better term, articles. I suspect maverick pieces fare poorly in both student-edited and peer-reviewed journals unless the author is an established scholar. I suspect, unfortunately, that a maverick piece, even a good one, has a better shot in student-edited journals. If that's right, it's not a very flattering commentary of the cliquishness often associated with peer-reviewed journals.

[frank cross]

To the list of advantages of student publications, I would add speed in publication and more rigor in proofreading (a surprising number of peer reviewed pubs contain mistakes such as failure to include a source in the bibliography)

As for peer reviewed, bear in mind that many of the leading peer review law publications are only single blind, so that the academic celebrity factor is not removed.

[charlie Geyh]

One additional advantage of peer reviewed journals is that the quality of editorial assistance is more consistently good. Student journals feature editors with skills that range from the sublime (sometimes) to the ridiculous (often). Yet another advantage of peer reviewed journals is somewhat greater continuity on the editorial staff. Law review boards turn over completely every other year--a good experience one year can imply nothing for the next.