Paul Caron has a post an interesting forthcoming study by Andrew Oswald (Warwick Economics) in Econometrica on the relationship between journal prestige and citations. See Andrew Oswald, An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers. Since a single body in the UK makes funding decisions for university research, there is significant pressure to rely upon measures of productivity that take into consideration the relative prestige of faculty members' placements. Because these journals are peer-reviewed, and the hierarchy of journal prestige in each discipline is fairly established, placement -- so the argument runs -- should be construed as a strong signal of quality.
Oswald's study is important, though the abstract is -- in my opinion -- quite misleading. This claim caused me to the read the whole study:
The paper finds that it is far better to publish the best article in an issue of a medium-quality journal like the Oxford Bulletin of Economics and Statistics [a middle-of-the-pack journal] than to publish the worst article (or often the worst 4 articles) in an issue of a top journal like the American Economic Review.
Oswald used citation counts from articles in six economics journals over a twenty-five year period to evaluate the reliability of placement as a quality signal. But there is no implicit suggestion in the article that a scholar would be better off trading down.
Basically, it boils down to this: Placement is correlated with citation, but as a proxy for quality, there are large Type I and Type II errors. Articles in AER and Econometrica (#1 and #2 in prestige) did indeed get more citations. Yet, roughly 16% of the articles in the less prestigious journals garnered more citations than the median AER or Econometrica articles. The most cited articles in the less prestigious journals garnered 10 times or more the citations than the four least cited articles in each issue of AER or Econometrica.
In my opinion, here is the good news:
Peers read and cite important work in less prestigious journals, and ignore articles of lesser significance that nonetheless manage to place well. [I know the legal academy has lots of citation count skeptics, but in a discipline such as economics, which is almost entirely built on peer review, it is harder to argue that citation counts do not contain some measure of useful information.] At least in economics, placement is not destiny.
But here is the twist in Oswald's article: order of publication was a decent predictor of future citations. In other words, the
editors seemed to have some ex ante sense of the relative merit of the
articles they were publishing. If we can unpack the basis for these
judgments, Oswald observes, perhaps it is possible that UK
decisionmakers can do better than mechanical reliance on journal
prestige. In other words, since we are making or breaking careers here, we can and should move beyond heuristics.