Empirical Legal Studies

Lott v. Levitt in the Chronicle of Higher Education

The Chronicle of Higher Education published a lengthy article today on the suit between Lott and Levitt. Of particular interest is that the author interviewed Gary King and an economist on the meaning of replication. King acknowledged that replication is used to mean different things, but the article discusses what might become Lott's amended argument. Although the complaint speaks of the meaning of replication "in the world of academic research and scholarship," Lott may argue he is referring only to its meaning in the world of economics research and scholarship. The article contains this response to King's broader view of the term:

Florenz Plassmann, an associate professor of economics at the State University of New York at Binghamton who has collaborated with Mr. Lott, disagreed. Mr. King's perspective might hold true for political science, Mr. Plassmann said, but not for economics. "I cannot recall a single instance in applied economic analysis," he wrote in an e-mail message to The Chronicle, "where 'replicate' was used in any way other than 'repeat the analysis with the same data and the same method and check whether you get the same results.'"

This argument is echoed in a comment posted at Overlawyered by Benjamin Zycher, a senior fellow at the Manhattan Institute. In part, Zycher said, "In the context of refereed economics journals, 'replicate' has one meaning only: The use of an author's data and model to ensure that falsification of findings is not an issue."

As the Chronicle explains, even if the narrow meaning of replication is the exclusive one in economics, Levitt can still plausibly argue he was using the term in a broader sense, given the context in which he used it. In another post at Overlawyered, however, Ted Frank has already found examples of replication being used in the economics literature to mean more than just verifying someone's results with the same data and same analysis. I'll add one more example to the list.

I should note that the Chronicle article also discusses the second count of the lawsuit, which is not tied to anything in Freakonomics, but to an e-mail Levitt sent to Professor John J. McCall. Thus far, most of the blogosphere's attention has been on the first count.

Law Schools, Law Reviews, and More Rankings

Al Brophy (Alabama) has an interesting paper on SSRN exploring the empirical relation between a law school's US News ranking and its main law review's citation count rank.  His paper finds "some support for a hypothesis that as law schools improve (or decline), there is a corresponding improvement in the quality of their main law journals (as measured by citations in other journals)."  The paper's punchline:

    "[I]f you want to know where a law school is
    heading, in addition to the glossy material that the school sends
    out - to announce new hires, student successes, faculty
    publications, and talks sponsored by the school - one should
    spend some time studying the scholarship their law review
    publishes."

Second Annual Conglomerate Junior Scholars Workshop

See here.

Bar Passage Rates

Hi Everyone,

Thanks for the responses to my initial post - empirical reserach on the legal profession.   I received some interesting responses, both submitted to the blog and to me directly.

I have a follow-up query that poses a specific  question about the legal profession.  One person responded that one area where empirical research would be helpful would be in the area of bar passage.

Bar passage rates vary across schools and across different demographics of students.  And almost all law schools grapple with how to improve bar passage among its graduates.   Based on conversations with colleagues at my own school and with professors at other schools, theories abound on the causal reasons for differential performance in bar passage, ranging from race, to class, to pedagogy. 

Rigorous research on this issue are surprisingly sparse.  There is an dangerous tendency to rely on studies that draw conclusions on simple comparisons across groups, without thinking more deeply about either alternative explanations or confounding factors.

My question, thus, is this:  1) what  are the factors that explain differences in bar passage rates?, and 2) how might we go about testing this (both practically obtaining data as well as methodological approaches to tease out what we believe might be the causal factors)?

Thanks very much.

AHY

"Empirically Bankrupt"

Bob Rasmussen (Vanderbilt), one the the nation's leading bankruptcy scholars, has a working paper on SSRN, provocatively titled: Empirically Bankrupt.  The abstract:

    Empirical legal scholarship endeavors to resolve disputes that
    are indeterminate at the level of theory. The nature of empirical
    claims, however, requires that consumers of this work bring a
    healthy dose of skepticism to any of these projects. Three recent
    works in the area of corporate reorganizations illustrate how a
    project that appears on its face to settle scholarly debate can
    rest on choices that the researcher made rather than on the data
    itself. One of these works seeks to discredit proposals to make
    bankruptcy law a default rule rather than a mandatory rule, but
    it draws its data from a sample half of which is made up of
    individuals, who by definition are outside the reach of the
    proposed reform. Moreover, the entire sample omits publicly held
    corporations, the main target of the reform being examined. The
    second article discredits prior reorganization practice, but only
    by establishing a standard that no bankruptcy system has ever
    satisfied. The third piece concludes that competition for large
    Chapter 11 cases has corrupted our bankruptcy system, but the
    empirical basis for this conclusion rests on combining
    fundamentally different types of bankruptcy cases. For empirical
    work to be credited, at a minimum, it has to look in the right
    place, ask the right question and draw the right inferences. When
    empirical work fails to cross this threshold, it conclusions must
    be rejected.

Setting aside the question of whether Rasmussen's specific critiques succeed, from a methods perspective his paper serves as a helpful reminder about important limitations to empirical legal scholarship and its claims.

More on Lott’s Replication Standard(s)

In my post on Saturday, I argued that John Lott’s complaint against Steven Levitt defines replication too narrowly. According to the complaint, Levitt’s reference to replication in Freakonomics can mean only one thing: analyzing “identical data” to Lott’s data and analyzing it in the same way that Lott analyzed it. (Compl. ¶ 12.) Again, this meaning of replication is valid. It matches what Paul Sniderman described as the first of three common definitions of replication, the verification of reported results using the same data and the same analysis. See Paul Sniderman, “Evaluation Standards for a Slow-Moving Science,” 28 PS: Political Science and Politics 464, 464 (1995). But as before, the point is that there are other valid meanings of replication. Last time, my examples were based primarily on the writings of political scientists. This time, I'll provide examples from Lott’s own writings, which I found after posting my initial comments.

Center for Student Empirical Studies

The Wake Forest Law Review has created the Wake Forest Center for Student Empirical Studies.  This is an initiative that I must applaud.  Wake Forest Law Review members have the traditional option of writing a Note or Comment, but a group of two or three students can also write an Empirical Study under the guidance of a faculty advisor and the faculty coordinator (Prof. Ron Wright) for the program.  Each student Empirical Study is published by the Wake Forest Law Review in hard copy or online.  Perhaps a hybrid empirical studies journal or singular issue is closer than I envisioned.  It seems law students themselves are part of the ELS Movement.  Is there any additional advice we should give law students interested in empirical research over and above our suggestions to faculty conducting empirical research for the first time? 

The Legal Profession and Empirical Research

Hi Everyone,

Thank you for the invitation to guest-blog this week.

I read the recent discussion generated by John Pfaff -- Federalism and Empirical Research -- with great interest.  It is not surprising that empirical legal research is drawn in part to where data is more easily available, arguably at the risk of overlooking different levels of government (e.g., as in the case with sentencing guidelines), or ignoring areas of law that would benefit from such inquiries.  It is in the latter case that I would like to pose my question:

A brief context for this question: while I was  serving on the judicial clerkship committee this year, many students asked whether it was a worthwhile experience.  A few were convinced with my simply responding "yes," but many wanted a fuller explanation.  Specifically, they wanted some statistics: what was the short- and long-term returns to clerking?  This of course is an empirical question that could be examined in a number of ways (e.g., propensity score matching examining similarly situated law students distinguishable by whether they clerked), but one that I had difficulty providing an answer, even for Northwestern graduates who had clerked.  The reason is that most law schools keep sporadic records on who among their graduates clerked, and if so, where. 

Judicial clerkships is only one such example of our inability to understand our own profession.  There are several aspects of the legal profession that are poorly understood (I have some thoughts).

It is fairly easy to explain why these questions are difficult to answer.  The legal profession is bound by strict rules of confidentiality (e.g., attorney-client privilege) but also opaque.  Unlike public corporations, law firms disclose little, if any information.  The government collects vast information on its attorneys, but makes little of it available to the public.

Questions about the legal profession track my own interest in labor and employment issues generally.  But more significantly, I believe in the value of better understanding our own profession, since at the end of the day, the central mission of law schools is to prepare its graduates to become lawyers.

Thus, I have two questions:
1. What areas of the legal profession warrant greater empirical examination?
2. How might this occur?

How to Conduct Field Experiments?

Yale and ICPSR are holding the 6th Annual Workshop on Designing, Conducting, and Analyzing Field Experiments (July 19-21, 2006 in New Haven, Connecticut).

Do Voters Vote Sincerely?

Degan & Merlo have written Do Voters Vote Sincerely?   The Abstract:

In this paper we address the following questions: (i) To what extent is the hypothesis that voters vote sincerely testable or falsifiable? And (ii) in environments where the hypothesis is falsifiable, to what extent is the observed behavior of voters consistent with sincere voting? We show that using data only on how individuals vote in a single election, the hypothesis that voters vote sincerely is irrefutable, regardless of the number of candidates competing in the election. On the other hand, using data on how the same individuals vote in multiple elections, the hypothesis that voters vote sincerely is potentially falsifiable,and we provide general conditions under which the hypothesis can be tested. We then assess whether the behavior of voters is consistent with sincere voting in U.S. national elections in the post-war period. We find that by and large sincere voting can explain virtually all of the individual-level observations on voting behavior in presidential and congressional U.S. elections in the data.