In an intriguing paper forthcoming in the Journal of Empirical Legal Studies (2012), Does the Judge Matter? Exploiting Random Assignment on a Court of Last Resort to Assess Judge and Case Selection Effects, Ted Eisenberg (Cornell), Talia Fisher (Tel Aviv), and Issachar Rosen-Zvi (Tel Aviv), exploit a database of criminal cases appealed to the Israel Supreme Court. In the course of seeking to assess individual judge effects on case outcomes, and controlling for the usual complexities that contribute to non-random case assignment to justices, the authors lever a factor that distinguishes the Supreme Courts in Israel and the U.S. Specifically, the Israel Supreme Court's docket includes an important cohort of mandatory cases. The paper notes how "studies limited to discretionary jurisdiction case outcomes can distort perceptions of judges’ preferences." The abstract follows.
"We study 1410 criminal law cases appealed to the Israel Supreme Court in 2006 and 2007 to assess influences on case outcomes. A methodological innovation is accounting for factors — case specialization, seniority, and workload — that modify random case assignment. To the extent one accounts for nonrandom assignment, one can infer that case outcome differences are judge effects. Individual justices cast 3986 votes and differed by as much as 15 percent in the probability of casting a vote favoring defendants. Female justices were about two to three percent more likely than male justices to vote for defendants but this effect is sensitive to including one justice. Defendant gender was associated with outcome, with female defendants about 17 percent more likely than male defendants to receive a favorable vote on appeal. Our data’s full samples of mandatory and discretionary jurisdiction cases allow us to show that studies limited to discretionary jurisdiction case outcomes can distort perceptions of judges’ preferences. Justices’ ordinal rank in rate of voting for defendants or the State was uncorrelated across mandatory and discretionary jurisdiction cases. For example, the justice who sat on the most criminal cases was the fourth (of 16 justices) most favorable to the State in mandatory jurisdiction cases but the twelfth most favorable in discretionary jurisdiction cases. This result casts doubt on studies of judges on discretionary jurisdiction courts, such as the U.S. Supreme Court, in which votes in the selection process are not observed."