« Second Annual Conference on Empirical Legal Studies | Main | Family Leave in Law Schools »

23 January 2007

Comments

Tony Smith

I'm glad Jeff raised the issue of academic freedom. For the non-tenured in particular a disregard of the content in favor of the low-level rationality assesment (which kind of law review is it) means articles are judged on the package not the content. This preference for peer-reviewed over student edited also rests on a potentially false assumption that peer reviews are never relegated to graduate students. If the grad students are peers, it seems that the grad student editors would not be that different. It's likely that it's functionally blind review as well since few of the law review students would know or ahve any reason to care who political scientists are - nice topic Jeff, thanks for the push to think about these things more thoroughly -

Margo Schlanger

Responding to the Anon. post above, I think lots of untenured professors try FIRST to publish in law reviews, not peer-reviewed journals, because of the speed of publication and because there is always the risk that some (maybe the crucial few) members of their faculties discount non-law review publication--in part because peer reviewed articles are shorter, and also for other reasons. I certainly wouldn't count law review publication as even a noisy signal of low quality.

Anon

What does publishing an empirical paper in a student-edited law review do for your impression of the quality of a piece? For example, there is an empirical piece forthcoming in Michigan Law Review recently posted on SSRN by an untenured professor. Normally, that would sound great. It is so empirical, though, full of models and regressions, that one has to wonder whether it failed at some peer reviewed journals before getting to that point. The piece is probably excellent, but the short law tenure clock and the bias for general law review publications may have forced the author to move on quickly rather than endure the wait for other peer reviewed journals and possibly miss the deadline for closing the tenure file. If the author had published in the second or third peer-review try, no one would think twice about it (or even know). Because it went in a general student-edited law review, though, I wonder if people think less of it (of course, they should read the piece, but if you're not an expert in the field you may assume experts found flaws). It is plasuible that it was submitted exclusively to law reviews because of a bias on the author's faculty for general student-edited law reviews, but would that be your first assumption when seeing a rigorous empirical piece in a law review?

Alex Robbins

Honestly, for ELS work I think the greatest defect is the lack of ELS knowledge on the part of law students. The fact that most editorial board members aren't qualified to evaluate ELS work is only going to increase the prestige factor, because they have nothing else to go on (short of farming out to an ad hoc "referee," which involves a lot of extra effort). Even if there's a qualified econometrician on the staff, the way (in my experience) that law reviews assign board positions and editing assignments is generally not tailored to one's academic specialties. In other words, if you've got a good ELS resume and work at a top school you may be in good shape, but if you're a 3L like me who paid attention in undergrad econometrics courses it's going to be considerably harder to be taken seriously. (Speaking of which, if anyone out there has any advice on what I should do with my own empirical paper, http://papers.ssrn.com/abstract=941238, I'd be happy for the feedback!)

Michelle Mello

I publish primarily in peer-reviewed journals (medical, policy, and economics journals). While I begrudge the amount of time I spend responding to sometimes niggling, sometimes irrelevant, sometimes indecipherable, and sometimes just plain wrong reviewer comments, I have to acknowledge that I have never published a paper that was better at the beginning of this process than at the end.

A well-known problem with peer review is poor interrater reliability. This is a particular problem for interdisciplinary research, because what happens at many journals is that the editors invite reviews from representatives of each discipline (rather than from people who work across both). You then end up with reviews that are, at best, orthogonal to one another and at worst, demanding radically different approaches to issues in the paper and reflecting very different philosophies about methods, what constitutes adequate and reliable data, and what the "important" findings are. You also have to spend a lot of time explaining the basics of each discipline to the reviewer who isn't familiar with it. Hopefully, as the number of scholars trained in multiple disciplines grows, this problem will diminish over time.

On the plus side, peer reviewers make a huge contribution in identifying what constitutes cutting-edge research on a topic of importance. There may be a bias against things that cut against accepted scientific inventions, but it is very valuable for editors to receive informed explanations of how the scholarship advances the field (or doesn't) and how it responds (or doesn't) to topics of great policy significance or scholarly interest. The trivial and the banal have trouble finding a home in peer-reviewed journals.

Overall, I think the peer review system has many warts, but is an appopriate form of review for empirical research. As legal research comes to more closely resemble other forms of social-science research in its methods and standards, I think it will be hard to resist treating it as social-science and scientific research are treated in the publication process.

Bert Kritzer

Jeff refers to a number of peer-reviewed "law reviews"; however, I think there is an important distinction between law reviews and the broader category of law-oriented journals. I would not describe Law & Society Review, Law and Social Inquiry, Law and Policy, Journal of Empirical Legal Studies, or the various law and economics journals as law reviews.

It is important to keep in mind that from the student editors, the primary role of their work on the law reviews is professional socialization and training to be a lawyer. It is not to insure that high quality empirical work is published, or that the empirical elements in that work meets the standards of professional social science.

For empirically-minded scholars, the major problem with student-edited law reviews is that the probability of encountering a student editor who has any knowledge about or experience of empirical research is quite low. This means that the editor cannot really help an author make the empirical part of the article better. Sure, the editor can check citations and bibliographies, but frankly errors there are annoyances while errors in empirical analysis are much more serious.

In fact, I suspect that student editors who don't have a clue are probably more likely to make empirical articles worse than to improve them. Often this may involve demanding that authors do things that don't make sense to any knowledgeable empirical scholar. I once had a student editor ask me to provide a citation for the original data I had collected and was presenting in an article; it took a couple of rounds of email to convince the student that no cite was either needed or appropriate.

I have on several occasions been asked to do tenure reviews for law school faculty who have published empirical work in both student-edited journala snd peer-reviewed journals. The quality of the papers passing through the peer review process is almost always markedly superior to those that have been man or woman-handled by a student editor.

Finally, regarding Frank Cross's point that peer review is single blind, I heartily agree that for senior scholars submitting to peer-reviewed journals this is the case. However, I don't think peers are as affected by the "wow" factor as are student editors. If anything, I suspect that anonymous peer reviewers may be more likely to "stick it to" a senior eminence.

Robert Justin Lipkin

Jeff offers a good mix of the pros and cons of student-edited and peer-reviewed law journals. With respect to the wait that peer-reviewed journals in the greater academy sometimes require, I wonder whether anyone has ever documented the longest wait. I once published an article on Hume's ethics in the Australasian Journal of Philosophy--a pretty good placement, if I recall correctly--that took approximately ten years to place. I had several rejections, some from lesser journals, until the Australasian Journal of Philosophy accepted it. In this case, single journal submission was excruciatingly frustrating.

I think it important to also distinguish between mainstream articles and maverick, for want of a better term, articles. I suspect maverick pieces fare poorly in both student-edited and peer-reviewed journals unless the author is an established scholar. I suspect, unfortunately, that a maverick piece, even a good one, has a better shot in student-edited journals. If that's right, it's not a very flattering commentary of the cliquishness often associated with peer-reviewed journals.

frank cross

To the list of advantages of student publications, I would add speed in publication and more rigor in proofreading (a surprising number of peer reviewed pubs contain mistakes such as failure to include a source in the bibliography)

As for peer reviewed, bear in mind that many of the leading peer review law publications are only single blind, so that the academic celebrity factor is not removed.

charlie Geyh

One additional advantage of peer reviewed journals is that the quality of editorial assistance is more consistently good. Student journals feature editors with skills that range from the sublime (sometimes) to the ridiculous (often). Yet another advantage of peer reviewed journals is somewhat greater continuity on the editorial staff. Law review boards turn over completely every other year--a good experience one year can imply nothing for the next.

The comments to this entry are closed.

Conferences

July 2014

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Site Meter


Creative Commons License


  • Creative Commons License
Blog powered by Typepad