Empirical Legal Studies

Empirical Research for Law & Policy: A Cautionary Tale Through Three Case Studies

A recently circulated paper by Richard Lempert (Michigan), Empirical Research for Public Policy: With Examples from Family Law and Advice on Securing Funding, will interest many.  In it Lempert explores practical challenges that confront many empirical legal scholars through three case studies.  The paper "discusses the use of empirical research for public policy and suggests a number of cautions that should be exercised in generalizing from empirical studies to public policy with a focus on the role of theory and both internal and external validity."

As Prof. Lempert notes in his paper: "There are, however, many dangers in relying on empirical work, rather than normative arguments, to influence policy. Empirical work is social science, and the standards of social science as well as what it determines can change. Methods, data availability and, to some extent, data quality are improving all the time. What was state of the art research yesterday may be perceived as hopelessly flawed tomorrow, and reanalyzing even the same data using more modern methods may reveal that the earlier advice was not just unsupported by the data but contrary to the policies that the data, properly analyzed, suggest."  Of course, Prof. Lempert goes on to note that, amid this uncertainty "policy makers must act" and, as well, judges must decide cases.

New York State Judge Sacked

By a 9 to 1 vote, the New York State Commission on Judicial Conduct recently removed Judge Robert Restaino from his position as a judge of the Niagara Falls City Court. According to the commission:

“In an egregious and unprecedented abuse of judicial power, respondent committed 46 defendants into police custody in a bizarre, unsuccessful effort to discover the owner of a ringing cell phone in the courtroom. In doing so, he inexplicably persisted in his conduct over some two hours, questioning the defendants individually about the phone before committing them into custody, and ignoring the pleas of numerous defendants who protested that his conduct was unfair and pleaded that he reconsider. Respondent’s conduct, which resulted in the unjustified detention of the defendants for several hours and the incarceration of 14 defendants in the County Jail, caused irreparable damage to public confidence in the fair and proper administration of justice in his court.”

The commission’s decision is dated November 13, 2007, but the press is only now reporting the story.  The commission’s decision, which includes both a concurrence and a dissent, is here.

New ICPSR Data Releases

New ICPSR data releases here.   One that caught my eye is the 2004 update for the National Judicial Reporting Program Series.  Although the 2004 sample is a cross-section, the series enables longitudinal analysis from 1986 to 2004.  Here is a description for the 2004 dataset:

This data collection provides detailed information on the sentences and characteristics of convicted felons based on data collected from state courts. The 2004 survey was based on a sample of 300 counties selected to be nationally representative. The collection contains sociodemographic information such as age, race, and sex of the felon. Types of offenses committed include homicide, rape, and robbery. Data can be analyzed at the national level or by individual counties.

There is a lot of county-level demographic data available from the Census Bureau, including shape files for GIS mapping software.   All of these data sources can be linked together in a matter of hours thanks to Census Bureau FIPS codes.   For a criminal law scholar interested in empirical research, this is great (and free) starter data set. 

More From BJS: Pretrial Release of Felony Defendants in State Courts

Pretrial Release of Felony Defendants in State Courts presents findings on the pretrial release phase of the criminal justice process using data collected from a representative sample of felony cases filed in the 75 largest U.S. counties in May during even-numbered years from 1990 to 2004. It includes trends on pretrial release rates and the types of release used. Pretrial release rates are compared by arrest offense, demographic characteristics, and criminal history. Characteristics of released and detained defendants are also presented. Rates of pretrial misconduct including failure to appear and rearrest are presented by type of release, demographic characteristics, and criminal history.  Findings from the report include:

• About 3 in 5 felony defendants in the 75 largest counties were released prior to the disposition of their case.
• Surety bond surpassed release on recognizance in 1998 as the most common type of pretrial release.
• Defendants on financial release were more likely to make all scheduled court appearances.

Launch of International Data Resource Center (IDRC)

The Inter-University Consortium on Social and Political Research (ICPSR) has just launched a new data initiative with an international focus.  From the ICPSR website:

ICPSR is pleased to announce the launch of the International Data Resource Center (IDRC) Web site. The IDRC serves as a central entry point into ICPSR's international data holdings. Dr. Denise DeGarmo, Associate Professor of Political Science at Southern Illinois University, Edwardsville, developd the site while in residence at ICPSR as an ICPSR Official Representative Sabbatical Fellow.

International data available through the IDRC includes conflict data, economic data, data on electoral systems and political behavior, environmental data, health data, data pertaining to the human dimension of international relations, public opinion data, and data on international organizations. Searches for international data can be conducted using several different methods, including subject searches, series data searches, or the IDRC's interactive map interface.

The IDRC website is very attractive and easy to navigate.  It also includes access to several foundational datasets. 

Methodological Fit

The most recent issue of the Academy of Management Review contains a must-read for the legal empiricists who lack formal social science training (e.g., me):  "Methodological Fit in Management Field Research." The article, by Amy Edmondson (Harvard Business School) and Stacey McManus (Monitor Executive Development), provides a remarkably accessible and constructive framework for research design, including the respective roles of qualitative and quantitative methods.  Here is the abstract:

Methodological fit, an implicitly valued attribute of high-quality field research inorganizations, has received little attention in the management literature. Fit refers to internal consistency among elements of a research project—research question, prior work, research design, and theoretical contribution. We introduce a contingency framework that relates prior work to the design of a research project, paying particular attention to the question of when to mix qualitative and quantitative data in a single research paper. We discuss implications of the framework for educating new field researchers.

This article is chock full of useful reference charts that can be relied upon to develop a coherent and effective research plan.  For example, this chart [click on to enlarge] summarizes the four key elements of a field research project:

This graphic [click on to enlarge] depicts the iterative nature of good field research:

Two more graphics after the jump:

Testing the Conventional Wisdom of College Football

Empiricism can also used to test hypotheses that matter to actual people.  See, for example, this recent NBER working paper by Trevon Logan (Ohio State Economics):  "Whoa, Nellie! Empirical Tests of College Football's Conventional Wisdom."  Here is the abstract:

College football fans, coaches, and observers have adopted a set of beliefs about how college football poll voters behave. I document three pieces of conventional wisdom in college football regarding the timing of wins and losses, the value of playing strong opponents, and the value of winning by wide margins. Using a unique data set with 25 years of AP poll results, I test college football's conventional wisdom. In particular, I test (1) whether it is better to lose early or late in the season, (2) whether teams benefit from playing stronger opponents, and (3) whether teams are rewarded for winning by large margins. Contrary to conventional wisdom, I find that (1) it is better to lose later in the season than earlier, (2) AP voters do not pay attention to the strength of a defeated opponent, and (3) the benefit of winning by a large margin is negligible. I conclude by noting how these results inform debates about a potential playoff in college football.

Hat tip to Danny Sokol (Missouri Law).

New Book - Tushnet on Judicial Review

Weak Courts, Strong Rights: Judicial Review and Social Welfare Rights in Comparative Constitutional Law

by Mark Tushnet (Princteon 2007)

Availability: Amazon / B&N / Princeton

"[R]ecent innovations in constitutional design in other countries suggest that [social welfare] rights can be judicially enforced-not by increasing the power of the courts but by decreasing it."

Cornell Copyright Duration Chart

Suppose you wanted to assign Herman Pritchett's The Roosevelt Court (1948) in your class. It's out of print. Can you just pirate copy the whole book? In answering this question, a good place to start is the recently updated Cornell Copyright Duration Chart.

There are a couple of simple rules: (1) works published before 1923 are in the public domain, but (2) works published by Disney are not and never will be. For all other works, see the chart.

Empirical Scholarship in the news: mortgages and bankruptcies

Recent stories on American Public Media's Marketplace and in the New York Times highlight the empirical scholarship of Katie Porter of the University of Iowa Law School and Tara Twomey, a Lecturer at Stanford Law School.  The two have been conducting a study, described here, of mortgage claims in Chapter 13 bankruptcy proceedings.  Given the current mortgage crisis, this work has important and immediate real-world implications.

The study, which involved a sample of more than 1000 Chapter 13 bankruptcy filings, found that there were substantial questionable fees charged by mortgage holders.  From the New York Times article:

In an analysis of foreclosures in Chapter 13 bankruptcy, the program intended to help troubled borrowers save their homes, Ms. Porter found that questionable fees had been added to almost half of the loans she examined, and many of the charges were identified only vaguely. Most of the fees were less than $200 each, but collectively they could raise millions of dollars for loan servicers at a time when the other side of the business, mortgage origination, has faltered.

Additionally, the study determined that in about 70 percent of the cases they reviewed, the creditor claimed that the debtor owed more than the debtor thought he or she owed.

This kind of scholarship is, in my view, a particularly exciting example of how empirical legal scholarship can illuminate important matters about how the judicial system actually operates.  And in the current economic climate, this particular study may provide valuable information to regulators, bankruptcy judges, and bankruptcy lawyers alike.

Presuming Reasonableness By The Numbers After Booker

Alex Robbins, one of the authors (along with Lynda Lao) of The Effect of Presumptions: An Empirical Examination of Inter-Circuit Sentencing Disparities After United States v. Booker, was among the many up-and-coming scholars who participated in the poster session at last weekend's Second Annual Conference on Empirical Legal Studies. (The CELS poster session continues to be one of the conference's highlights.) The paper, previously noted in a recent post at the Sentencing Law and Policy Blog, will certainly interest criminal law and sentencing law scholars. A snippet from the paper's abstract follows:

"In the two years since United States v. Booker, the circuits have divided over how to use the Federal Criminal Sentencing Guidelines when reviewing sentences imposed by district courts. Seven circuits have held that a sentence within the Guidelines range is entitled to a presumption of reasonableness on appeal; five have held that it is not.  Although the Supreme Court's recent holding in Rita v. United States allows the courts of appeals to adopt a presumption of reasonableness for within-Guidelines sentences, it does not require them to, and so the circuit split remains.

Using this circuit split as a natural experiment, we undertake what we believe to be the first statistically robust analysis of the effect of a presumption of reasonableness on sentences imposed at the federal district court level.  Specifically, using 145,047 individual-level observations recorded by the United States Sentencing Commission (comprising all recorded federal sentences in all twelve circuits for a one-year period beginning in November 2004 and ending in October 2006), we perform a multivariate regression analysis to determine how a circuit's adoption or rejection of a presumption of reasonableness for within-Guidelines sentences affects the frequency with which district courts impose below-Guidelines sentences.  We find that a circuit's adoption of a presumption of reasonableness decreases the frequency of below-Guidelines sentences by less than one percent, although this result is statistically significant."

Ribstein's rundown of papers at CELS

Larry Ribstein has a nice post that summarizes some of the papers at the Second Annual Conference on Empirical Legal Studies.

Getting Data: My Experience with the California Bar

Today I am in California for a hearing before the Board of Governors, which is the elected body that oversees the state bar exam.   For the past 18 months, I have been part of a research team (with Richard Sander, Vik Amar, Doug Williams, and Stephen Klein) seeking California data to test a possible mismatch effect in law schools.  The substance of the hearing today (see agenda here) was to determine whether we would be permitted to obtain regression results, via the Bar's own longtime psychometrician, from the Bar's rich archive of historical data.

After hearing approximately 50 minutes of  prepared testimony from those for and against our study, the Board voted to deny our data request.  Based on the tenor of the meeting, which was set by a well-organized opposition led by Cheryl Harris (UCLA Law) and Michele Landis Dauber (Stanford Law), the vote turned on legal uncertainties surrounding candidate confidentiality and consent. 

Acceding to these concerns has, in my opinion, three major flaws: (a) 0ur research team would only have access to regression tables with results aggregated among groups of schools, so no individually identifiable information would ever be released to researchers or the public;  (b) we would, at all times, be subject to university IRB protocols; and (c) the broad construction given to consent in this context suggests that much of the research sponsored by the California Bar over the last 20 years has been unlawful; moreover, this decision casts doubt on the scope of any future research by the California Bar. 

[Note: Richard Lempert (Michigan Law and formerly chair of the NSF Law & Social Science Program), wrote this memo on the IRB issue, which he graciously forwarded to our research team in advance of the hearing. Rick's assessment, however, is quite different from my own experience in obtaining IRB approval at three different institutions for projects which raised, in my opinion, much more significant human subjects issues.  Regardless, no research would occur without IRB approval; that should be obvious to all parties involved.]

Of course, the Board of Governors is not an adjudicative body -- rather, in this case, it is the regulatory unit that would be dragged into court if it approved our proposal.  Thus, on one level, I can understand the institutional reasons for taking a cautious approach.  (For a snapshot of the public debate leading up to this decision, see this National Law Journal op-ed by Cheryl Harris and Walter Allen, and our NLJ response; in the spirit of transparency, all documents related to our projects are posted on this website.)

Although I am disappointed with the outcome, I am proud of the testimony we put on today.  The basic theory underlying the mismatch effect is that recipients of large preferences (regardless of race) tend to learn less because the classroom instruction is pitched toward a modal student with significantly higher entering credentials.  Honest researchers can argue over the methodology and inferences of Sander's Systemic Analysis study.  But existing data provide little comfort that our current system is working well.  Minority students are disproportionately clustered at the bottom of their class.  Since LGPA is the single best predictor of bar passage, it is not surprising that these same students struggle on the bar examination. Sander's first choice/second choice analysis (in his 2005 Stan. L. Rev. response essay) suggests that law school performance and bar passage prospects actually increase when a student opts for a less elite law school. In my opinion, if this theory is wrong, it is imperative that it be debunked empirically rather than through efforts to withhold data.

As a lifelong Democrat and ardent supporter of racial diversity, I don't need to apologize for raising these issues.  As a said in my prepared remarks, it does not follow that evidence of a mismatch effect requires the dismantling of racial preferences -- indeed, in my opinion, this would be a very bad idea. 

But if we fail to diagnosis factors that contribute to low minority bar passage, we have no basis to formulate effective policy or educational strategies.  Regardless of which way the data cut, our study would have guaranteed one of two favorable outcomes:

1. Using the more refined California Bar exam (i.e., a continuous dependent variable rather than pass/fail and schools put in analytically useful clusters, unlike the LSAC-BPS data), the mismatch theory would have little or no empirical support.  So a contentious academic theory would be put to bed, at least in the law school context.
2. A mismatch effect would be supported, which would pressure law schools to take concrete steps to help current or prospective students.  These might include: (a) disclosures that reveal bar passage prospects for past students with similar entering credentials; (b) creation of rigorous academic support programs (such as this one) that increase the bar passage for students in the bottom 1/2 of the class; and (c) identify curricular and teaching strategies that produce higher bar exam scores.

As legal educators, we should want more than the status quo.

Lott v. Levitt: 50% Settled, 50% Appealed

Lott v. Levitt is not yet fully resolved. Here are two belated updates: 

First, the dismissal of Count I, the one involving the meaning of replication, is now on appeal before the Seventh Circuit. According to the most recent docket entry (October 3, 2007), Lott’s opening brief is due on December 17th. The notice of appeal is here. I haven't found any discussion elsewhere about Lott having filed this appeal.

Second, Lott discussed the settlement of Count II on his website some time ago. His post includes a copy of Levitt’s clarification/correction letter to John McCall and also links to coverage of the settlement on various blogs. A thorough discussion can be found on the Chronicle of Higher Education blog.

UPDATE: In another Chronicle of Higher Education blog entry, David Glenn said Lott filed an appeal on July 27th of this year. Glenn must have been referring to Lott’s motion for reconsideration, which Lott filed on the 27th. (Judge Castillo denied the motion on July 31st.) Lott filed the appeal on August 30th, not July 27th.

BJS Study of Trends in State Court Organization

A recently released BJS report presents trend data from state court organization data collections covering the years 1987-2004. The report examines changes in the organization and operations of the Nation's state trial and appellate courts over this time period. Topics include the selection and educational requirements of judges, regulations of criminal and civil juries, the development of unified court systems, and adjustments in court management and staffing to address growing caseloads.  Findings include:

• Total trial court case filings increased by approximately 45% in limited jurisdiction courts and 43% in general jurisdiction courts.
• Specialty jurisdiction or problem-solving courts, such as drugs, family, mental health, and domestic violence courts, became more common.
• The ratio of trial judges to the population nationwide decreased slightly from 10 to 9 judges per 100,000 persons.

Conference on the International Criminal Court (with an empirical twist)

Passing this along from Will Lowe (Nottingham):

The Human Rights Law Centre and the Methods and Data Institute at the University of Nottingham, UK announce a multi-disciplinary conference, "The International Criminal Court and the State."  The conference will take place on Friday 9th November, 2007 in the University's Senate Chamber.

The conference brings together experts from the fields of law, politics and international relations as well as practitioners from the Court itself, civil society, states and the International Criminal Court Legal Tools Partners.

The keynote address will be given by Sir Adrian Fulford, a judge at the International Criminal Court. Other speakers include Morten Bergsmo, (PRIO, and former Senior legal Advisor and Chief of Legal Advisory Service, ICC Office of the Prosecutor), William Wiley, (International Law Advisor to the Iraqi High Tribunal), and Diego Cango-Fano (Council Secretariat of the EU).

As one of the four partners of the International Criminal Court Legal Tools Project, the Human Rights Law Centre and the Methods and Data Institute are developing a database of national implementing legislation. The conference will set in context our work with the Court, and the other partners, focusing on issues of implementation and the impact of new technologies in the Legal Tools Project on the study and practice of international criminal justice.

For more details, see the conference website:

http://www.iccandthestate.org

or send an email to [email protected]

ELSers in the U.K. and its environs should check it out.