Empirical Legal Studies

Ted Eisenberg (1947-2014) & Israel ELS

Yet another datum illustrating my late colleague's astonishing scholarly reach and influence is found in next week's Ted Eisenberg Memorial Empirical Legal Studies Conference, hosted by Tel Aviv University's Buchmann Faculty of Law. Ted's ties to Israel and Israeli scholars were numerous, deep, and significant. So much so that Tel Aviv University graciously decided to host an international conference on empirical legal studies in Ted's honor.

Coding Disputes, The Supreme Court Database, and Judicial Ideology

While generally not news, academic squabbles over coding decisions, including coding decisions involving leading data sets, become news when they spill into the academic literature. A current example involves the Supreme Court Database and its use in a paper by Lee Epstein (Wash U), Christopher Parker (Centenary College), and Jeff Segal (SUNY Stony Brook).

In Do Justices Defend the Speech They Hate? In-Group Bias, Opportunism, and the First Amendment, Epstein et al. exploit the Supreme Court Database, use a "two-level hierarchical model of 4,519 votes in 516 cases," and find that "although liberal justices are (overall) more supportive of free speech claims than conservative justices, the votes of both liberal and conservative justices tend to reflect their preferences toward the speakers' ideological grouping, and not solely an underlying taste for (or against) the First Amendment." (The paper's findings made their way into a New York Times article.)

Another scholar, Todd Pettys (Iowa), however, took issue with Epstein et al., particularly their reliance on the some of data set's coding decisions. In Free Expression, In-Group Bias, and the Court's Conservatives: A Critique of the Epstein-Parker-Segal Study, Pettys digs deeply into the underlying data set and writes: "In a recent, widely publicized study, a prestigious team of political scientists concluded that there is strong evidence of ideological in-group bias among the Supreme Court’s members in First Amendment free-expression cases, with the current four most conservative justices being the Roberts Court’s worst offenders. Beneath the surface of the authors’ conclusions, however, one finds a surprisingly sizable combination of coding errors, superficial case readings, and questionable judgments about litigants’ ideological affiliations. Many of those problems likely flow either from shortcomings that reportedly afflict the Supreme Court Database (the data set that nearly always provides the starting point for empirical studies of the Court) or from a failure to take seriously the importance of attending to cases’ details."

Responding to Pettys' critique of their work, Epstein et al. write: "The upshot is that our coding procedures reject over 80% of the author's allegations, meaning his critique reduces to about 2% of our coding decisions, extrapolating over his non-random audit percentage. Although we think a few of these are debatable, we are happy to concede, and have corrected the dataset to reflect the changes he desires. We have also rerun the analysis. The results do not change in any substantively or statistically significant way (see Appendix A), nor do the results of the summary of our study reported in the New York Times and other outlets (see Appendix B)."

Regardless of who has the better of this particular argument, greater attention to coding conventions and the underlying accuracy of any data set (particularly influential data sets such as the Supreme Court Database) is welcome.

ELS at HLS

While I never quite know what to make of "in-house" promotional publications, the current issue of the Harvard Law School alumni magazine (Spring 2015) includes an interesting piece on The New Empiricists at HLS. In an introductory note, Dean Martha Minow notes that HLS is "increasing" its "investment in empirical research." The small cadre on the HLS faculty engaged in empirical research includes Jim Greiner whose work includes interesting efforts to randomize field experiments (previously discussed here, here, and here). Somewhat surprising, to me anyway, is the reported growing demand for ELS instruction and courses from HLS students.

More On Bayesian's Costs and Benefits

I'm seeing increased discussion of the comparative advantages (and disadvantages) of "Bayesian versus 'frequentist'" approaches toward inference. A recent post by Andrew Gelman (here), and the comments it triggered, illustrate.

More On Marginal Effects

A recent Statalist discussion (here) includes a helpful illustration of how to execute and understand margins output (as well as common mistakes). As well, Richard Williams (Notre Dame--Sociology) hosts a webpage that includes a rich array of resources, including an extremely helpful PowerPoint discussion of the margins command (see "Selected other highlights" at the bottom of the page).

The Absence of ELS from the Emergence of Behavioral Insight Teams Around the World.

In recent years, various countries have started the process of establishing behavioral insight teams ("BIT") that will advise the government on usage of knowledge from psychology and behavioral economics. Both the UK and U.S. governments rely increasingly on insights from behavioral sciences, especially with regard to non-deliberative choice by individuals. The EU, a comparative late-comer to this game, has recently established a task force, which published a guideline report, describing how policy makers should use psychology when implementing behaviorally based legal policy in various areas.

In the last year, I have been to a few meetings with academics and practitioners who are involved in BIT in the U.K, U.S., and Israel as well as the EU, which made me wonder about the missing role of ELS in what seems to be an emerging regulatory trend. Moreover, in the first BIT international conference which will take place in London in September, there is a highly impressive roaster of some of the world's leading psychologists and behavioral economists, but with the exception of references to the work of Cass Sunstein, I didn’t recognize any scholars associated with ELS or even with any law school.

One might have thought that the BIT which is, by definition, an evidence-based approach to law-making could have become a natural hub for empirical legal studies. Many of their recommendations are built on very intense field experiments, which lie at the heart of some of the work done by ELS scholars. It might be the case that simply by path dependency ELS scholars are late to a new party and hence I am not discussing here anything which is sustainable, nonetheless, there is a chance that this might tell us something about both ELS and BIT.

The first point is highly related to the argument regarding the role of legal theory, I have discussed in this blog few weeks ago, based on the work of Dagan et al. and Fishman, on the role of legal theory in ELS. Presumably one might argue that without legal theory, ELS might lose some of their advantage relative to psychology and economics. Indeed, much of the previous discussion around the "nudge" approach came from legal theorists and philosophers and not from ELS scholars. Furthermore, as I also mentioned in a previous post, when legal theory is missing many in the social sciences might feel that methodology-wise their line of research is more technically advanced.

Looking at this question from the reverse perspective, I wonder whether it makes sense that the scholars who are involved in BIT initiatives should care about issues such as fairness, autonomy, liberalism and paternalism, and instrument choice or institutional constrains. I tend to believe that they should and that, in the long-run, as BIT will evolve, the need for ELS involvement will prove helpful to the success of these projects. I hope that with greater theoretical interest and interaction between the two communities, both will bring to the table their relative advantages and create a more coherent research paradigm which would allow behavioral knowledge a proper integration with state efforts to modify human behavior.

Inaugural SELS Global Junior Empirical Legal Scholars Workshop

The first international workshop for junior empirical legal scholars will be held on December 17th and 18th 2015, at the Hebrew University of Jerusalem. The meeting is co-sponsored by the Society of Empirical Legal Studies (SELS) and The Center for Empirical Studies of Decision Making and the Law.

The workshop aims to foster the research of scholars who have recently completed their PhD or have recently been appointed to a tenure track position, by providing them with a forum in which they can present their research and receive feedback from fellow empirical researchers. Approximately ten scholars from around the world (submission criteria specified below) will be invited to the workshop. Each scholar will be assigned a commentator who will discuss their paper. The rigorous academic environment at the workshop will assist junior scholars to develop their ideas. The workshop is also intended to establish a sense of community among empirical legal scholars and help build cooperative relationships between participants.

Topics: Papers may touch on any legal field and can employ any empirical methodology.

Participation Criteria: To be eligible, an author must be an untenured faculty member at a research university in a tenure-track position; or a tenured faculty member if it has been less than five years since the initial entry level appointment; or a full time post-doctoral fellow or a visiting associate professor at a research university. In addition, the committee will consider the work of authors who do not fit these criteria, but are new to the field of empirical legal studies.

Travel: The costs of airfare (coach) and accommodations of authors who will be invited to participate in the workshop will be covered by the Center for Empirical Studies of Decision Making and the Law.

Paper Submission: Submissions should be sent to Ms. Ayelet Gordon ([email protected]) with the subject line: “Workshop for Junior Empirical-Legal Scholars.”

The deadline for submission is June 30th, 2015. Only drafts of complete papers will be considered.

Program Committee:
David Abrams – University of Pennsylvania Law School
Bernie Black – Northwestern University School of Law
Valarie Hans – Cornell Law School
Doron Teichman - Hebrew University
Keren Weinshall-Margel - Hebrew University
Eyal Zamir - Hebrew University

Research Design & Endogeneity

A fascinating paper exploring the effect of partition on violence does a wonderful job illustrating how creative efforts (and rich data) can potentially dampen (though not eliminate) endogeneity concerns. In Which Side Are You On? Political Violence and Partition in Ireland 1920-1921, Elissa Berwick (MIT--Poli Sci) analyzes a quasi-natural experiment produced by the partition of Ireland in 1921. Exploiting a truly unique data set, Berwick finds that "although partition decreased violence against civilians on Northern Ireland's side of the border as compared to the Irish Free State side, violence against civilians in the border areas as a whole significantly increased."

Along with her interesting findings Berwick's research design also warrants note, especially on how it addresses endogeneity concerns. As Berwick concedes, "Partition is usually provoked by conflict, and yet its effect on conflict is the outcome of interest." Notwithstanding this tension, however, Berwick goes on to observe that:

"As for endogeneity, the initial proposal of partition came from a backbench Liberal in June 1914, years before the start of any violent civil conflict. Although the partition itself occurred in a context of civil war, its original justification was not to end conflict between northern Protestants and southern Catholics. Instead, British legislators were concerned by their inability to coerce Northern Unionists, a point driven home by the Curragh Mutiny of March 1914, in which the British Army refused to disarm the Ulster Volunteer militia. Thus the partition of Ireland was not intended to separate two sides, but instead to forestall action by a minority."

While whether Berwick's methodological optimism is warranted, of course, requires further study, her transparent and helpful discussion of the issue deserves praise.