"[E]mpiricism gone astray"

A very interesting and helpful post (and comments) by John Donohue (Yale) over at the Balkinization blog illustrates some all-too-common problems with empirical research. To illustrate some of these problems, John comments on a recent paper by Royce de Rohan Barondes (Missouri-Columbia), provocatively titled: Want Your Opinions Questioned or Reversed? Hire a Yale Clerk.

As John notes, "unfortunately, counterintuitive empirical results almost always turn out to be wrong if they are not based on an appropriate empirical methodology for the inquiry at hand. In my opinion, the methodology of the Barondes (paper) is flawed, and the conclusions drawn from this research are either incorrect or unfounded." (To be fair, Barondes responds to some of John's critiques in comments to John's post.)

Comments

[generic viagra]

amazing post about [E]mpiricism gone astray thanks for sharing!! I'm very interesting in this post

[Royce Barondes]

What I understand to be the primary criticism leveled by Prof. Donohue about my prior draft of my Yale clerks paper has now been withdrawn by Prof. Donohue, in a revision to his initial post that strikes large portions of his original criticism. (See http://balkin.blogspot.com/2008/04/why-id-stick-with-yale-clerks-some.html, dated 9/28/08, but posted 10/8/08) In this light, I think it should now be clear that the original post on this blog about my Yale clerks paper, in describing Prof. Donohue's post as a "helpful post (and comments) by John Donohue (Yale) over at the Balkinization blog illustrat[ing] some all-too-common problems with empirical research," is inaccurate.

I suppose it remains a matter of opinion whether, in light of the correction of that criticism on the Balkinization blog some 4-1/2 months after its original posting, Prof. Donohue's original posting remains "very interesting" (as the main posting here describes it), and precisely what is "illustrate[d]" by this sequence of postings.

Your readers are cordially invited to download from SSRN the current version of the paper, in which the data set has been expanded and the models have been revised to address various other issues that have been raised.

[Michael Heise]

Although I'm loath to intrude into a debate between others, Royce raises a good point in his reply to John Donohue's criticisms of his (Royce's) paper. (Readers interested in the substantial debates that have emerged should look at the extensive discussions on the Balkinization and Volokh blogs, among others.) Specifically, to the extent that John criticizes Royce's paper for failing to cluster at the judge-level, on this matter I think Royce's response carries the day. Simply put, as Royce notes, his models were estimated using Stata's clogit command which, if I take John's point correctly and understand Stata's manuals, plausibly (if not persuasively) addresses John's concern. (Moreover, in unreported supplemental analyses, Royce estimated alternatively-specified models using the logit command and including various judge-level dummy variables. Royce reports that his results are robust across various alternative model specifications.)

That said, on other matters John (and, now, others) raises important questions that illustrate limitations to Royce's paper. FWIW, my admittedly quick read of Royce's paper prompted three general responses. One is that modeling case reversals is a tricky, nuanced, and complicated task. I'm not yet persuaded that Royce's models address these complexities in a satisfactory manner. Second, at a practical level it is surely the case that judicial chambers--and how judges deploy their clerks--vary across judges. More to the point, relations among a judge, any particular opinion (whether reversed or not), and a clerk (whether a Yale grad or not) is far from clear in my mind. Consequently, problems of causal influence lurk. Third, Royce's model appears undertheorized. If so, the findings might be spurious and fall victim to the "correlation in search of a theory" critique.

In any event, the paper certainly warrants continued good faith debate.

[Royce Barondes]

Let me briefly address what I understand to be the basic criticism provided by Prof. Donohue. (There are other issues I won't address at the moment, because it is most helpful to focus on the basic point.)

He criticizes the paper by analogy:
"Barondes’ error is a bit like concluding that because the death penalty is almost nonexistent in the Northeast, which has the lowest murder rate in the country, and widespread in the South, which has the highest murder rate, this means that the death penalty causes murder."

As to my model, he writes:
"Instead, his model simply correlates the negative signal rate with the presence of Yale clerks, which leaves us with the same problem of correlating high executions with high murder rates."

The models were estimated using Stata's clogit estimator. Here's the first line of the output for these models:

"Conditional (fixed-effects) logistic regression"

I won't bog this post down with various textual descriptions of the estimator, although in response to the various posts on other blogs, I've quoted from a variety of sources. In sum, it would be my view that this fundamental criticism is not well-founded.

I'm not saying my model is the only way to try to control for the identities of the individual judges. Rather, given the data set, this seemed to me to be a way consistent with other scholars' approaches.

For your audience, my sense is that the paper may be helpful for purposes of showing it is practical--since I did it myself--to extract Lexis "topics" to use for controls instead of a single coding variable provided by some other database. (I have no objection to someone citing the working paper for this purpose.)

This paper was posted on SSRN as part of trying to get it included in a conference. Presenting a paper at a conference, and receiving comments, is one way one's research is improved. In a comment made a couple of hours after his post, I invited Prof. Donohue to provide the Stata syntax of an estimation that he believes properly estimates the relationship. Of course, suggestions from any of your readers also would be welcome. If you would be so kind as to do so by e-mail (there are too many blog posts about this for me to keep track) and put "Yale Clerks" in the subject line, I will be able to identify them from the hundreds of spam messages I receive each week. I'm barondesr --at-- missouri.edu.

Royce Barondes

[frankcross]

I wouldn't have thought that criticisms of Yale grads would be considered counterintuitive