Forum Post #1: Opening the Black Box of Law Review Selection

Periodically law professors convene forums to trash law reviews. The most virulent words are usually heaped upon the student editors who run these journals. For example, in a symposium published by the University of Chicago Law   Review in 1994, student editors were referred to “incompetents”; in a similar forum in the Stanford Law Review in 1995, they were called “dumb and stupid.”

Within the article selection process, more specific criticism includes the student editors’ (perceived) fixation with copious footnotes, excessive literature reviews, trendy topics, or an author’s institutional affiliation or prior publications (as opposed to the one in front of them). These last items are of particular concern for law professors because it stacks the deck against an objective evaluation of their work. And placement has significant collateral effects on pay, promotion, and the lateral market.

Fortunately, the Nance-Steinberg study opens the black box of article selection. In my estimation, it reveals a reasonably fair and objective process in which original, persuasive, and polished articles have at least a fighting chance of getting a good placement. Sure, the author’s letterhead matters. But these results suggest that this bias is actually less true at more elite journals—and that finding itself may be a bit of a mythbuster. Moreover, the study actually has some suggestions that could, at the margins, help with an article’s placement.

To get all of our readers onto the same page, after the jump I will briefly summarize the sample, methodology, and key findings of the Nance-Steinberg study.

Sample & Methodology

Drawing upon the experience of several Penn faculty members and former Penn Law Review Articles Editors, Nance and Steinberg developed a questionnaire that asked respondents to rate the positive or negative influence of 56 discrete factors on a 7 point (-3 to +3) Likert-type scale. [The survey instrument can be viewed here.] They also asked a second set of questions which asked respondents to ordinally rank the importance of seven attributes in the law selection process:

• Potential to influence scholarship
• Persuasiveness of arguments
• Originality of arguments
• Readability
• Timeliness of topic
• Potential to change substantive law
• Notability of author

Using an online commercial company to host the electronic survey, invitations were sent to editors at approximately 400 student-edited law journals. The instructions asked that the survey be completed only by editors with the authority to extend offers of publication. After two follow-up invitations, this process produced 191 responses from editors at 161 journals.

The authors do not discuss the survey response rate or whether the respondents appear to be evenly distributed through the law school hierarchy. They do, however, list the participating schools by nine “Tier” categories based on Washington & Lee impact factors [the responding journals, by Tier, can be view here.]. My own visual inspection revealed no obvious bias. There are quite a few elite and non-elite flagship law reviews and a wide array of specialty journals.

To check the internal consistency of the questionnaires and make the items more tractable, the authors used factor analysis. (For those unfamiliar with this technique, the most familiar application of factor analysis is I.Q.) This is an inductive process that, ideally, extracts and separates “factors” from the data (typically, the fewer the better) in way that supports the researchers’ theory. In this case, the reduction process separated into 18 factors that were theoretically coherent. The few judgment calls they had to make (discussed in footnote 79) are, to my mind, not very controversial. Only a handful of questionnaire items needed to be excluded.

Results

Every good empirical study should have one table that conveys the study's most significant findings. In the Nance-Steinberg study, that is arguably Table 3, which ordinally ranks the 18 factors from positive to negative influence in selecting an article for publication (3 = strong positive, 2= positive, 1 = weak positive, 0 = no influence, ... -3 = strong negative influence). As the authors note, the Cronbach α  statistic is used to evaluate the internal consistency of the factor vis-à-vis the variables used to produce it. 14 of 18 factors are .60 or higher, which is a commonly used threshold for evaluating reliability.

Each of the 18 factors is functionally a rubric that is derived from two to five questionnaire items, which are summarized in Tables 4 through 21. Assuming that law review editors are agents who are seeking to maximize the influence of their volume (and ultimately their law review franchise), this ordinal ranking makes a lot of sense. The most important criterion (based on five probative questions) is selecting articles that people will want to read.

The next two factors, Author Prestige and Peer Support, are arguably factors that affect the journal's long-term institutional interests. The Author Prestige construct is derived from an author’s prior publications (volume and placement) and institutional affiliation, and Peer Support includes review and unsolicited communications from faculty members (i.e., peers) in support of an article. But when these factors are broken down according to relative journal prestige, the influence at more elite journals tends to be less pronounced.

In my opinion, the authors were not sufficiently careful in generating their tables. When doing their tier analysis, histograms would have been much better than sprawling tables. And frankly, I don’t know why they use ANOVA on nine Washington & Lee impact tiers when it is far from clear that the continuum of prestige or impact supports that level of granularity.

Another way of stating this: what are the professional implications of publishing in W&L Tier 9 (456-540 in impact) versus Tier 4 (131-195)? I suspect every law professor would say “not much.” What about Tier 1 (1-25), which strongly resembles the top of U.S. News rankings, versus Tier 2 (26-65)? This second difference is arguably career making. A visual inspection of Table 23 (excerpted below), which breaks down the respondents' ordering of seven publication factors based on relative influence, suggests large and significant differences between W&L Tier 1 versus Tier 2.

Comparing Tier 1 to Tier 2, the data suggest:

• Tier 1 and 2 editors place similar emphasis on “potential to influence scholarship” and “readability of article”;
• Tier 1 editors place greater emphasis on (a) persuasiveness of arguments, (b) originality of arguments, and (c) likelihood to change the law;
• Tier 1 editors are less likely than their Tier 2 counterparts to be influenced by (d) timeliness of topic (which presumably includes trendiness), and (e) notability of the author.

In my opinion, the authors would have been better served by a simple independent sample t-test to compare differences between elite and non-elite law journals. Such a comparison would likely show strong and unambiguous differences between these two groups.

One way to read these results is that editors at Tier 1 journals are at the top of the foodchain (i.e., at the end of the expedite express) and thus have the luxury of thinking for themselves. Moreover, as the most accomplished students at the most selective schools, I presume that they also have the ability to evaluate original and persuasive arguments and the potential that a submission could change the law (if not now, then in two years, when they join a faculty themselves?). I doubt that Nance and Steinberg will quibble with that conclusion.

Comments

[buy kamagra]

It's really not fair to list this one because the site I'm referring to is still in development. It should launch in the next few weeks. I list it because it's going to be very "top 5" worthy when it launches.

[buy ativan]

Re Conor's point on this strange prestige-driven marketplace: I cannot justify why the editors of elite law journals have so much influence in determining merit, but few would dispute that they have it.

[buy viagra]

That said, I am interested in what the editors *say* they do because it is far from a foregone conclusion that there is a discrepancy. Very interesting post about Forum Post #1: Opening the Black Box of Law Review Selection thanks for sharing!!

[pharmacy viagra]

Hi! I read your post about "Forum Post #1: Opening the Black Box of Law Review Selection" and found it really interesting. Thanks!

[buy viagra]

I think we should at least have a discussion on this latter point before dismissing Nance's and Steinberg's interesting data. Law professors, in my opinion, are often a little too trigger happy in drawing conclusions.

[buy viagra]

The dose of sildenafil for erectile dysfunction is 25 mg to 100 mg taken not more than once per day between 30 minutes and 4 hours prior to sexual intercourse.

[generic valtrex]

amazing information about Forum Post #1: Opening the Black Box of Law Review Selection thanks for sharing!!!

[impotencesed]

Good post antiqueautos!! look up me... Treatment for impotence
Impotence is not an extremely grave thing, even though it can be extremely harmful to a man's self-esteem and confidence. More info at: http://www.impotencesed.com/

[antiqueautos]

Very good site, excellent content, I will recommend to my group of readers in the university, I found very interesting article on the internet, including this ... I will leave an input ----> The Antique Autos are considerate real treasures and many people pay a lot for them,? the quality of a good antique autos especially the classic ford cars represent many money, are a exception of the rule because can have higher prices in the market, the antique autos are a good option for some part of the society like the classic ford cars too, that meaning power, and position in a society that every day need it more of course, and the antique cars or classic ford cars like for example the Buick classic, the classic corvettes.More info at: http://www.antique-classiccars.com/

[Bill Henderson]

Michael raises a good point re the difference between what editors say and what they actually do. Chris Zorn will explore this point with more sophistication, so I will defer to him.

That said, I am interested in what the editors *say* they do because it is far from a foregone conclusion that there is a discrepancy. Positing the possibility of a discrepancy is what a clever law professor does; but using reasoning to think through the likelihood is another matter. I think we should at least have a discussion on this latter point before dismissing Nance's and Steinberg's interesting data. Law professors, in my opinion, are often a little too trigger happy in drawing conclusions.

Re Conor's point on this strange prestige-driven marketplace: I cannot justify why the editors of elite law journals have so much influence in determining merit, but few would dispute that they have it. I hope to pick up on Conor's market theme in a subsequent post. Elite editors are not the only king makers; there are other routes to success. bh.

[Conor Granahan]

Professor Henderson,

It seems strange to have so much of a professor's prestige decided by a handful of law students. I would like to see how external factors come into play, such as the pressure to publish from professors at one's own school.

But outside of this, how come a U of Chicago guy like yourself doesn't want the market to decide these things?

Shouldn't more law professors try to publish their work independent of journals run by students? I know have seen some law profs' books, but I would think there would be more if the competition is so fierce for journal spots. Perhaps I am not looking in the right places.

Rejected authors could combine their works to have a longer format or one could just publish and distribute his own work if no publisher will pick it up. Not very expensive these days.

I suppose the prestige comes from approval by others, but if the author believes her work can make an impact, that impact is impossible if it just sits on the shelf.

I am vaguely familiar with the academic publishing mindset. You research, write, publish, and move on. No promotion or focus on money. I realize this has some value in letting the work stand apart from the influence of sales. But I would think getting one's ideas out there is the next logical step if the prestige passes you by. And donate the money if it bothers you.

[Michael Heise]

Bill: Great idea for the Forum.

At the risk of nit-picking, the methodology used in the Nance-Steinberg study (survey self-reports) limits it to a study about what law review editors *say* they do in terms of article selection. A study about how article selection *actually* works would require data on the pool of articles submitted as well as those selected. Another possible study might compare what law review editors say they do with what they actually do in terms of article selection.