Empirical Legal Studies

Survey Research

I am currently in the process of designing a survey on financial and billing information for the Indiana State Bar Association (ISBA).  In preparation for this task, I have read and reviewed lots of surveys by bar associations in other jurisdictions.  There are a surprising number, often conducted by local academics (I blogged about a really terrific survey of Pittsburgh lawyers here). 

What I lack, however, is a good reference guide for interpreting survey data, particularly a methods book or article that discusses how to check for a representative sample and various weighting techniques for either correcting sample bias or imputing missing values based on known characteristics of the population (information I have because the survey is sponsored by the ISBA). 

Any suggestions in the comments or by email would be greatly appreciated.

Closed Med Mal Insurance Claims

The folks at BJS recently released an interesting report on data from seven states that require med mal insurance carriers to report information on closed claims. The report, Medical Malpractice Insurance Claims in Seven States, 2000 - 2004, focuses on:

"... medical malpractice claims that were closed in Florida, Illinois, Maine, Massachusetts, Missouri, Nevada, and Texas from 2000 to 2004. In these states medical malpractice insurance providers are required by law to submit information on closed medical malpractice claims. These claims are typically submitted to the Department of Financial and Insurance Regulation, which provided the data to BJS. The report provides an in-depth analysis of medical malpractice claims, including the number of claims closed with or without payouts, types of medical providers named in claims, facilities where alleged injury occurred, severity of alleged injury, court disposition of claims, amounts paid to compensate claimants, costs of defense counsel, and claim processing time."

Vault Top 25 Underrated Schools: What Does It Mean?

Vault, Inc, which is a publisher that collects industry information on various professions, recently released its listing of the Top 25 Most Underrated Law Schools.   (Hat tip: Paul Caron & Volokh.)  The findings are derived from a survey of  512 law firm recruiting managers, hiring partners, and corporate counsel, who were asked "to name law schools that, based on their experience as hiring managers, are underrated."

As I read the list, I wondered, what is the practical implication of a law school being underrated by the people who make hiring decisions for entry level lawyers?  Presumably, it means that graduates of certain law schools tend to perform better than their school's U.S. News ranking would suggest; thus, legal employers are more likely to hire them. 

If this is true, what is the source of superior performance? Here are two possibilities:

1. Stronger Students.   Some schools may enroll a stronger student body than their rank might suggest.
2. Better Education.  Some law schools may equip graduates with more or better skills than other schools of comparable rank

Based on some preliminary statistical analysis, there is fairly clear quantitative evidence for the first hypothesis.  There is also some qualitative evidence for the second--enough to warrant some additional research. 

These research questions are really important to the ranking debate because their answers could reveal a mechanism by which some students will discount USN.  If an "underrated" school offers a better entree to coveted legal employers, why go to the higher ranked school, especially if the underrated school costs significantly less?  Law schools that understand these dynamics are in a better overall competitive position.

After the jump, I elaborate on these findings, including methodology.

A Serious Debate on Empiricism and Theory in Bankruptcy

Last April, we posted the abstract of Robert Rasmussen's (Vanderbilt) working paper, "Empirically Bankrupt," which critiques three empirical papers in the bankruptcy area.  The abstract concluded that "[f]or empirical work to be credited, at a minimum, it has to look in the right place, ask the right question and draw the right inferences. When empirical work fails to cross this threshold, it conclusions must be rejected."

One of the articles being critiqued was Elizabeth Warren's (Harvard) & Jay Lawrence Westbrook's (Texas) recent study, Contracting Out of Bankruptcy: An Empirical Intervention, 118 Harv. L. Rev. 1197 (2005).  A few months ago, Warren & Westbrook posted a rejoinder, "The Dialogue Between Theoretical and Empirical Scholarship," that not only addresses Rasmussen's criticism, but also tries to articulate an optimal relationship between theory and empiricism.   The entire exchange between Rasmussen and Warren & Westbrook is worth reading. Here is the abstract of the rejoinder essay:

In this essay we offer brief reflections on the best process for critiquing empirical work in law and sustaining an engagement between theoretical and empirical approaches. We emphasize the importance of theoretical work in helping to shape the scholarly agenda, but we urge that theory should be more closely tied to fact. We illustrate our argument by responding to a recent critique of our own empirical work by Professor Rasmussen. His principal claim is that our work should be discounted because we reported on all business bankruptcies, both those of entrepreneurs and those in corporate form. In response, we reanalyze our data, separating the individuals from the corporations; in every case the re-analyzed data support the conclusions of our original paper to the same extent or more strongly.  Similarly, his other claims about our work are shown to be incorrect.

Judicial Decisions as Data Points & Precedent

Over at the Social Science Statistics Blog Jim Greiner helpfully reminds us about the salience of precedent for those endeavoring to model court decisions. Jim notes (here):

"It's tempting to think that one can code appellate decisions or judicial opinions pursuant to some neutral criteria, then look for trends, tease out inferences of causation, etc. Here's a note of caution: they're not i.i.d. They're probably not i.i.d. given X (whatever X is). Precedent matters."

Judicial Ideology & Case Selection Effects

Ahmed Taha (Wake Forest) circulated an interesting paper, Judges' Political Orientations and the Selection of Disputes for Litigation," focusing on the influence of judges' ideology on settlement behavior. Taha's paper explores various case types and assesses whether (perceptions of) judge ideology influences which cases settle and which go forward. The study finds that:

"... at least for some types of cases, judges' political orientations have significant effects on the number of cases filed. Thus, the political orientations of even trial court judges have important effects on the outcomes of the federal justice system: fewer potential plaintiffs seek relief in federal courts if they believe the judges are less likely to be sympathetic to their cases. This finding also helps explain the surprising findings of other researchers that a federal district judge's political orientation generally does not affect the probability that a case ends in a judgment for the plaintiff or a settlement. This paper's findings suggest that those results are based on a selection bias: because parties believe that judges' political orientations matter, they are likely settling cases on more favorable terms for the plaintiffs if the cases are assigned to Democratic judges rather than to Republican judges."

Justices Who Change Over Time

This week, the Colloquy published two pieces as it begins a series of responses to an article  forthcoming in the print Northwestern Law Review by Profs. Lee Epstein, Andrew D. Martin, Kevin M. Quinn, and Jeffrey A. Segal.  In it, Epstein et al. perform an empirical analysis of Supreme Court Justices' voting patterns and find that almost all Justices exhibit some degree of shifting ideological preferences during their time on the Court. 

Colloquy posts the introduction to the full piece by Epstein et al (see here), and follows with a reflective piece by Linda Greenhouse, the Supreme Court correspondent for the New York Times and author of Becoming Justice Blackmun (see here).  In her piece, Ms. Greenhouse draws on her time covering the Court and her study of Justice Blackmun, the exemplar of shifting Justice ideology, to provide anecdotal support for the conclusions of Epstein et al. 

Judges Use Law Reviews "Like Drunkards Use Lampposts: More For Support Than for Illumination"

Paul Caron  links to a devastating (and devastatingly funny) article in the New York Lawyer, entitled "Federal Judges Discuss Law Reviews Usefulness."   Paul excerpted some the sharpest lines:

Kicking off a recent friendly-but-frank discussion about the relevance of contemporary law review articles, Dean David Rudenstine of the Benjamin N. Cardozo School of Law offered one description: "useless blather puffed up with self-indulgence" ...

Judge Sack returned to the not-so-funny problem under consideration: today's highly theoretical articles are largely ignored and seldom cited by judges, a dramatic turnabout from a generation ago when the mostly practical content of law reviews was a significant element of judicial decision-making. Judge Sack was sorry to say that the bench now uses law reviews "like drunkards use lampposts, more for support than for illumination." ...

"If the academy wants to change the world, it must decide if it wants to be a part of the world." ...

Although Judge Sotomayor agreed that brainy law professors should not trouble themselves by contemplating reactions from the bench to their writings, she leveled a sharp gaze at the Cardozo Law faculty and declared, "If you think that judges are not as capable of creative thought as you are, I beg to differ." She added, "My question to academics: do you really think you're serving some function to someone?"

Wow, that is some really straight talk.  For the record, law reviews had a sketchy record 40 years ago, see Lowell J. Noteboom & Timothy B. Walker, The Law Review–Is it meeting the needs of the Legal Community?, 44 Denver L. J. 426 (1967).

Revisiting State Supreme Courts

A recent paper by Herbert Kritzer (Wisc.--poli sci) et al., The Business of State Supreme Courts, Revisited, sets out to update Kagan et al.'s earlier study of the docket composition of state supreme courts. I am partial to such undertakings (updating, revisiting, and/or replicating important past work) and welcome attention to the comparatively understudied state courts. Kritzer and his colleagues find:

"... many of the patterns of change described by Kagan et al. continued through the 20th century: debt and real property continued to decline and criminal continued to increase. However, other patterns of change either reversed or halted. Specifically, neither torts nor family cases have continued to increase; torts have stabilized and family cases, rather than increasing, have declined. The most surprising shift is the sharp increase in "other contract", which had no particular pattern in the earlier data, but which represented five percent or less of the courts' business; in the 1990s, "other contracts" had grown to a level approaching that of public law, and exceeding real property and family and estate cases."

LSA Preliminary Program Now Available...

...see here.