First, thanks to Bill and everyone for letting an empirical neophyte trespass on this blog for awhile. As a typical junior professor, I am a regular submitter to law reviews and have spent untold hours talking to colleagues about the submission process. However, lately these discussions have grown tiresome as I realize that as arbitrary and random as the process seems to be, unrelated anecdotes only make the process seem more random instead of creating some pattern of order. No one person's "N" is ever going to be big enough for that person to say with any authority "The best way to get your article accepted at top journals is to do X." However, these conversations seem always to dwindle to the point where Professor A says that doing one thing is important, but then Professor B will counter with the argument that the Professor A must be wrong because Professor B never does that one thing and always places well. And so on. So, I am pleased as punch to be moving from war stories to data. If nothing else, the student-run law review system produced the two authors of this study and perhaps honed their writing skills to the point that they were able to advance knowledge in this way. (I would like to hear about their own placement story, though, now that they are on the other side!)
However, as everyone else seems eager to point out, survey responses aren't the sort of data we need to conclusively dispel or confirm these anecdotes. (As an alumnus of the Lee Epstein/Andrew Martin Conducting Empirical Legal Scholarship workshops, I remember one of them saying that the best way to introduce bias into your project is to do a survey!) I won't belabor the point, but I think we all know that law review editors are smart enough to know what answers they should pick. (However, a survey of former Articles Editors might produce less self-conscious responses.) I think it's interesting that the "Tier 1" editors assigned lower importance scores to certain categories regarding author prestige and article topic, but those factors were still the most important in rank order. And, as the authors note, when asked virtually the same question in different form regarding author's place in the world, editors answered differently depending on wording. They don't care about how "notable" an author is, but they care whether the author is "highly influential in her respective field."
So, we know now that articles editors, even though they understand that they shouldn't say it's super important, admit that characteristics of the author are pretty much as important as characteristics of the work. So, that seems to jive with everyone's worst fears and conventional wisdom. And I think the objective data would support that. A few weeks ago, I updated some armchair empiricism on which authors get published in the Harvard Law Review (not a respondent to the survey). While my interest was in the gender breakdown of the authors published, I hasten to add that any casual legal scholar would recognize most of the names of those whose work was published in any given volume. I would suspect the same would be true of at least the top 5 journals.
So, what can I possibly add to this forum? Here are two small bits. First, I think the "hot topic" factor here is not treated in a precise way. Although the authors suggest that at least most journals do not look for hot topics, the factor was phrased as "The topic is one about which many articles are currently being written." Well, that sentence does not cry out for agreement, and one can think of many other ways in which the same could be worded. In fact, two factors that were ranked highly ("The article fills a gap in the literature" and "The topic would interest the general public") also describe hot topics. One can imagine that a legal topic that is currently in the news would spawn articles that both fill a (new) gap and that would interest the general public. Because of the timeliness, this topic may be the subject of many law review articles, but it's hard for an articles editor to know that at the beginning of the trend.
Second, one interesting aspect about the law review selection process is the agency problem. Editors are choosing articles that they will work on, perhaps personally. While the authors assume that these individual editors choose articles that will gain their law reviews citations and attention, in reality these editors will move on before this attention happens. So, editors may be more interested in choosing topics that they themselves are interested in and possibly choosing authors that they would like to get to know. I would be interested to know if any articles editors thought about the relationship between themselves and their authors. I definitely remember the authors that I worked with. And, I could definitely tell when I was getting an offer from an articles editor who really liked my topic.