Empirical Legal Studies

Updated Burger Court Dataset

Professor Harold Spaeth (Mich. St., Poli Sci) recently announced an updated Burger Court dataset, with attendant documentation.  This dataset is managed by the good folks at the University of Kentucky's S. Sidney Ulmer Project for Research in Law and Judicial Politics.  According to Harold:

"This version contains a 3.64 percent random sample, stratified by term, of petitions (mostly cert) denied by the Supreme Court.  The inclusion of these data will now make it possible for researchers to easily avoid selection on the dependent variable, the bane of most previous studies of the Court's agenda setting.  This database, then, provides users for the first time with data necessary to investigate the Court's agenda setting behavior over a substantial number of sequential terms: here the entire Burger Court, 1969-1986."

Updates to Federal Judges Dataset

An earlier post by guest blogger Greg Sisk (St. Thomas, MN) discusses and links to a helpful and important federal judges dataset, Attributes of U.S. Appeal Court Judges, 1801-1994 (compiled by Gary Zuk, Deborah J. Barrow, and Gerard S. Gryski).  Greg mentioned that this dataset was recently expanded. A brief description (and link) follows:

"... an update to the Attributes of Lower Court Judges dataset. Gerry Gryski (along with Gary Zuk) made a major expansion to their popular dataset. The updates include two new data files: one contains attribute data for Appeals Court judges serving from 1801-2000, and the other dataset contains attribute data for District Court judges serving from 1789-2000. Both of these datasets are archived at the University of Kentucky's S. Sidney Ulmer Project for Research in Law and Judicial Politics ...."

Is Lateral Hiring on the Rise? A precise answer is on its way!

Brian Leiter recently posted ruminations on his colleague's perception that lateral hiring at law schools has dramatically increased over the last 20 years.  His colleague is sure that it has, but Brian is skeptical, at least as to the degree.

A few months ago, when I was discussing Jim Chen's Moneyball/Moneylaw theories, I posted an excerpt from a 1957 Journal of Legal Education article that was directly on point.  Opining on the distinction between national and local law schools, Columbia law professor Harry Jones wrote:

Twenty years ago, it was common usage to discuss "local" and "national" in terms of analogy to "minor leagues" and "major leagues."  The analogy never held in individual cases, since "local" schools had their Mickey Mantles and every "national" school its full share of .200 hitters; but there was much to it: the average salary of a "national" law school professor was about twice the average salary at a typical "local" school, and the "nationals" regularly replenished their faculties by a raiding process comparable to the major league draft. The most striking legal development since World War II has been a narrowing of the "national"-"local salary gap [due to taxpayer funding of public law schools, which presumably reduced the incentives to leave].

Jones, Local Law Schools vs. National Law Schools: A Comparison of Concepts, Functions, and Opportunities, 10 J. Leg. Educ. 281, 289-90 (1957).

Note that Brian's colleague has raised an empirical question that has a clear factual answer.  The only thing holding us back is collection of the data.   Several months ago at The Conglomerate, Funmi Arewa (Northwestern), discussed our  joint longitudinal analysis of legal scholarship project.  Our c0-PI's are Andrew Morriss (Illinois) and Ken Dau-Schmidt (Indiana). 

More on IRBs

Though not itself empirical, David Hyman's (Illinois) recent piece (Institutional Review Boards: Is This The Least Worst We Can Do?, forthcoming in the Northwestern Univ. L. Rev.) will interest and inform anyone who has had the "pleasure" of dealing with an IRB incident to an empirical project.  David provides important perspective and context on the issues as well as helpful suggestions to improve the status quo.

Associate Satisfaction by Metropolitan Area

David Zaring and I are finishing up a book review on two novels that feature large law firm associates (Kermit Roosevelt's In the Shadow of the Law and Nick Laird's Utterly Monkey).  In a nutshell, we compare the fictional accounts of misery in large law firms with associate satisfaction data published by The American Lawyer.

The editors at Michigan Law Review have imposed a strict 10,000 word limit.  As a result, it looks like the table below, which analyzes various dimensions of associate satisfaction by metropolitan area, is headed for the cutting room floor.  It suggests that desirable work conditions in large corporate law firms are partially a function of market size/geography.

Some methodological notes:  In this table, associates (years 3-5) at 160+ law firms rated their firm on a 1 to 5 scale (5 = top/best category).  The geographic category was broken down by the top 10 cities in terms of lawyers in the sample: NYC (1,415 associates); DC, Chicago, LA, San Fran (1,834 associates); Boston, Atlanta, Houston, Dallas, Philadelphia (955 associates).  This breakdown tracks the "Global Cities" typology I have discussed in previous posts (see here and here).   The firm means are calculated at the branch office level; in turn, the metropolitan means are weighted by the number of mid-level respondents at each firm.  A large number of firms are represented in all three geographic categories.   

The most striking feature of this table is that, in ALL categories, working conditions  appear to be better in the smaller corporate law markets.   An interesting subsidiary observation is that the mean scores for "overall rating as a place to work" (in red) are typically higher than the working conditions.  This seems at odds with the intuition that these work conditions would be determinants of the "overall rating" score.  One possible explanation, which David and I explore in the Michigan essay, is that,  because of exposure to more high-end non-commodity legal work, associates at prestigious firms in large markets (a) enjoy much greater outplacement options than their smaller market counterparts, and (b) higher expected lifetime earnings.

I would be interested in hearing other alternative explanations for these results.

NOTE:  If you are an academic researcher interested in working with similar law firms data, see announcement for the Law Firms Working Group.

More Methodological Tools

Over at PrawfsBlawg, in a post about multidimensional scaling (MDS) Jeremy Blumenthal (Syracuse) helpfully reminds us all that "multiple regression isn't the be-all and end-all...."  With respect to MDS, Jeremy finds the methodology especially useful for "looking at lay intuitions about legal institutions, punishments, or cases; or to assess whether different punishments may have been proportionate; or any context in which (a) similarities among stimuli may be interesting or (b) we are interested in understanding what the basis might be for people's judgments or perceptions."  For those interested, a helpful MDS resource is found here.

Neglected Variables in Empirical Research into Judicial Decisionmaking

As a final guest post, I want to briefly discuss a few variables that arguably have been neglected in the recent body of empirical studies of judicial decisionmaking.  This list obviously is highly subjective, although it reflects not only my thoughts but those generously shared by other researchers that I’ve informally polled in recent days.  And given our natural human tendency to think that whatever we’re working on at the moment is the most important thing that could occupy the attention of any researcher, we are likely to regard whatever variable we are grappling with at the moment as the one most deserving of greater attention.  As demonstrated in my discussion of the religion variable at the end of this posting, I am no exception to that tendency.

If this list of supposedly “neglected” variables were to be periodically updated, it would also prove to be a very fluid one, changing constantly with new developments and new contributions to the literature.  In essence, what follows is one person's (my) impression of what lies on the cutting edge of empirical study of the courts today.

“Tricky” Variables in Empirical Research into Judicial Decisionmaking

Continuing my discussion about data-gathering and variable coding in empirical study of judges and court decisions, I want today to touch briefly, and necessarily incompletely, on a few variables that have proven “tricky” for researchers in recent studies. By “tricky,” I mean either that finding sources of information has been problematic or that translating raw data into an objectively measured factor that well estimates a variable is a particularly difficult and controversial task.

While every researcher can compile a list of troublesome variables that have troubled him or her, I’ve selected only three to discuss today, based on heightened attention given to the subject in academic debates studies and my informal poll of colleagues around the country.  Those three variables are: ideology, reputation of educational institutions, and judicial involvement at various stages of litigation.

Less Public Sources of Information About Judges for Empirical Research

In yesterday’s guest blog, I listed a variety of publicly-available sources of information on federal and state judges and court decisions, where researchers may find ready-made databases or easily-accessible raw information for formulating appropriate variables for empirical testing of a research question regarding judicial decisionmaking. Nearly all of the sources I listed yesterday may be accessed directly from a researcher’s computer, by simply connecting to the internet.

Today, I want to supplement that list of public sources by discussing three sources of information that are not as readily available to researchers and thus to be tapped require greater effort. If researchers are to constantly push the envelope by exploring new dimensions of or influences on judicial decisionmaking, existing sources of information must be more thoroughly exploited or new sources of information must be found. In my limited experience as an empirical researcher and my greater experience as a consumer of research on the courts, I have found that the most interesting studies of judges in action tend to involve creative efforts by researchers to find and use new data. Sometimes researchers take information long available, but use the data to create new variables or to code common variables in a more precise or efficient way (or at least a manner more appropriate to the particular purpose of the study at hand). Even more exciting are those occasions in which a researcher has unearthed new information that had not previously been collected or fully exploited by others or that has become available only through new records.

Data-Collection and Variable Coding in Study of Judicial Decisionmaking: Public Sources of Information for Empirical Research

In the beginning, for the empirical scholar, is the question and the hypothesis. Well, mostly and sort of.

In theory, the researcher wishing to explore a research question by quantitative methods formulates a hypothesis and then searches for evidence by which to test that hypothesis. The data to be sought and how variables should be measured and estimated from that data depends on the purpose of the experiment. A variable coded in one way for one study may be ill-suited for another study.

In practice, the researcher’s prior knowledge of sources of data, careful review of the existing literature in the field (which of course consists of earlier studies grounded on data collected and described by others), and ongoing interaction with data gathered for other related experiments may well influence the selection of a question worthy of exploration and contribute to the formulation of the hypothesis framed to examine that subject. To be sure, a scholar must be wary of fabricating a questionable hypothesis solely for the purpose of justifying a study based upon data that happens to be available (or left-over from a prior study) or excessively adjusting an ongoing study to fit the results. But the creative scholar naturally will find him or herself intrigued by the possibilities for further study that are posed by discovery of a new set of data or by identifying an aspect of that data that has been neglected in prior work. The scholar who remains abreast of developments in the field frequently will see implications and possibilities for empirical examination in the data collected and variables adopted by another researcher in a new study. After all, think of how often you find yourself thinking, as you read about a new study, hey, that gives me an idea!

Guest Blogger: Greg Sisk

This week's guest blogger is Greg Sisk (St. Thomas).  Greg is a nationally-recognized scholar on the subjects of civil litigation with the federal government and empirical (statistical) analysis of judicial decisionmaking; he also writes about federal courts, legal ethics, and constitutional law. He is the author of the casebook, “Litigation With the Federal Government,” which is published by Foundation Press. Sisk’s empirical work on court decisions was honored with the Article Prize from the Law and Society Association in 1999. 

Greg's publications include: Judges and Ideology: Public and Academic Debates About Statistical Measures, 99 Northwestern University Law Review 743 (2005); Signaling and Precedent in Federal District Court Opinions, 13 Supreme Court Economic Review 63 (2005); Searching for the Soul of Judicial Decisionmaking: An Empirical Study of Religious Freedom Decisions, 65 Ohio State Law Journal 491 (2004); Above the Rules: A Response to Epstein and King, 69 University of Chicago Law Review 135 (2002).

Puzzle 4: Why Are Small Business Bankruptcies So Rare?

Federal bankruptcy filings are increasingly rare among failing small businesses. For every hundred small businesses that fail, fewer than twenty file for bankruptcy. And bankruptcy filings have become more rare over time. Business failures have hovered around 650,000 since 1989, according to the Small Business Administration. But the number of business bankruptcy filings has fallen from about 60,000 in 1989 to 35,000 in 2003, a 42% drop. These statistics are taken from the federal government’s case filings database (PACER).

Why are business bankruptcies rare and increasingly so?

More on Propensity Score Matching

Bill asked me to identify a few articles that employ propensity score matching. In the legal literature, I've seen few applications, and most of these have appeared in the work of Daniel Ho (Stanford Law School). Interested readers might want to consult two of his papers: one is a short comment on Richard Sanders' controversial affirmative-action study; the other is an article on Supreme Court decisionmaking in wartime (co-authored with Lee Epstein, Gary King, and Jeffrey Segal). The second paper is particularly valuable because it takes time to explain PSM to a legal audience.

ELS on WSJ

The ELS Blog and my fellow ELS Blogger Bill Henderson has been cited (with a photo) on Wall Street Journal Online Law Blog.  Check it out!

Puzzle 3: Do Consumers Want Insurance Coverage for Pain and Suffering?

Have torts scholars put forth non-experimental, statistical evidence showing that consumers do or do not demand insurance coverage for pain and suffering? This kind of insurance is available through tort law, but is hard to find in private insurance markets. Generations of scholars have debated the sense (or nonsense) of this feature of the law. Some have pointed to the rarity of pain-and-suffering insurance in private markets and argued that consumers probably do not value such coverage (e.g., George Priest; Alan Schwartz; John Calfee and Paul Rubin). Others have pointed to anecdotal evidence consistent with a demand for insurance against pain and suffering (e.g., Steven Croley and Jon Hanson). And others have pointed to experimental evidence consistent with a demand for this kind of insurance (e.g., Ronen Avraham).

None of these papers, as far as I know, applies the kind of methodology usually employed when the effect of state law is uncertain: a comparison of (i) states with the law and (ii) states without it (or with less generous versions of it). The comparison usually exploits variation over time, across states, and (possibly) within states. Think, for example, of the many studies evaluating the effect of medical malpractice reform on damage awards (Ronen Avraham; Janet Currie and Bentley MacLeod; Catherine Sharkey and Jonathan Klick), physician supply (David Matsa, Jonathan Klick and Thomas Stratmann), defensive medicine (Daniel Kessler and Mark McClellan), choice of medical procedures (Janet Currie and Bentley MacLeod), …

Can something similar be done in studies of pain-and-suffering damages? Can a before-after or difference-in-difference methodology be used to test the demand for this kind of insurance?

Here's one idea ...

Yale Law School making bobblehead dolls of Dean Koh

I usually do not post items like this, but today I received the following email which I found odd enough to require a non-ELS related post (and, in the interests of fairness, I note that if asked I will post about any similar Yale Federalist Society fundraiser).:

Hello,

I am writing in hopes that you would be willing to put a link on the ELS blog to an interesting fundraiser going on at Yale Law School: our chapter of the American Constitution Society has commissioned custom-made bobblehead dolls of our Dean, Harold Hongju Koh (and yes, we definitely got the idea from Greenbag). They’re being sold as a fundraiser for ACS, and all proceeds will benefit our programming activities. We’re taking pre-orders for the next two weeks, and we have an online order form up at http://www.yaleacs.org/bobblehead.html (with pictures of the bobbleheads and more information about them and our organization).

I would be happy to answer any additional questions should you have them.

Thank you for your time and consideration,

An Empirical Analysis of Life Tenure

Stras and Scott have written An Empirical Analysis of Life Tenure: A Response to Professors Calabresi & Lindgren.  The Abtsract:

"Opposition to life tenure has been steadily mounting in the legal academy and Professors Steve Calabresi and Jim Lindgren are among those leading the charge. Crucial to their argument that life tenure is "fundamentally flawed" is an empirical claim that the increases in average tenure among Supreme Court Justices are both "dramatic" and 'unprecedented."

In this article, the authors respond to Calabresi and Lindgren by showing that their hypothesis of dramatic and unprecedented growth in average tenure has two fundamental flaws. First, it suffers from a period-selection problem. Rendering the data using longer or shorter periods blunts or eliminates the dramatic and unprecedented trend. Second, it suffers from a date-of-observation problem. Treating each Justice's tenure as an observation at date of swearing-in (or at the midpoint of a Justice's career), rather than at date of departure, eliminates the trend altogether. Instead, the authors show that, using an S-curve regression model, the most supportable hypothesis is that, despite short-term fluctuations, average tenure has been growing slowly and steadily over the long term, and we can expect more slow and steady growth in the future."

Puzzle 2: Propensity Score Matching

Question 1: How popular is Propensity Score Matching in the ELS literature?

Endogeneity and sample selection bias are major concerns in any empirical study of legal institutions. We rarely work with experimental data, in which subjects (people, firms, states, courts) are randomly assigned to "treatment" and "control" groups. Our data are instead taken from environments in which the subjects choose their own groups: young doctors choose whether to locate in states with ("treatment") or without ("control") medical malpractice caps; litigants choose whether to pursue claims to trial ("treatment") or settle out of court ("control"). Our data are also taken from environments in which the outcomes of interest (e.g., crime rates) may be driving the legal institutions (stringent laws or large police forces), not the other way around. These dynamics generate self-selection bias, sample selection bias, simultaneity bias (or “reverse causation”), etc.

There are, of course, many tools for addressing these biases. Common techniques include before-after estimators such as difference-in-differences (DD) and cross-section estimators such as instrumental variables (IV) and the Heckman selection estimator (Heckit). In recent years, DD (and triple DD) has become common in L&E papers and elsewhere.

In economics and other literatures, another technique is gradually gaining popularity: propensity score matching (PSM). This technique offers a method for structuring non-experimental data to look like experimental data. For every subject in the "treated" group, the researcher finds a comparable subject in the "control" group. Comparing the two groups, we can get an estimate of the effects of legal institutions on outcomes of interest.

Is this technique becoming popular in ELS circles?

Judge-Level State Supreme Court Database

A preliminary version of the Judge-Level State Supreme Court Database is now available at the State Supreme Court Data Project website.  To download, go to the website, and click "Judge Level Data Set."

Grant Funding for Empirical Work

I agree with Michael that it is worth checking out Ron Wright's provocative post at PrawfsBlawg on grant funding.   To backup some of Ron's ideas, I am once again linking to this terrific compilation of external funding sources (via the excellent law librarians at SUNY-Buffalo).  One of my suggestions in ELS on the Cheap was to apply for a grant.  It can be a great learning experience that forces a lawyer to "think like a social scientist."

Puzzle 1: How Inefficient is Tort Law?

Does the tort system generate significantly greater administrative cost than comparable first-party insurance, such as health insurance and workers' compensation? It is commonly thought so. In their 1987 book, Landes and Posner called the tort system “an exceedingly costly insurance mechanism.” Others have made the same criticism, noting that tort awards are much greater than amounts paid under comparable first-party insurance policies. 

There are two issues here. The "high costs" of the tort system, relative to first-party insurance, could be due to (i) its greater administrative cost or (ii) its broader coverage (or both). It is well known that tort law offers broader coverage than market insurance. Most notably, it compensates noneconomic harms, such as the pain and suffering arising from an accident. This undoubtedly raises the cost of the tort system relative to first-party insurance. But it’s a separate question whether tort law generates greater administrative expense, per dollar of compensation paid, relative to first-party insurance. Although it is widely believed that the tort system is administratively inefficient, what data support this claim?

Law Profs, Externally Funded Research, and ELS

Over at Prawfsblawg, Ron Wright (Wake Forest) has an interesting post about externally funded research, law schools, and the obvious implications for empirical legal scholars.  Well worth a look.

Puzzles and Methods

Thanks, Jason, for welcoming me to the ELS Blog. I will use my airtime this week for two purposes: (1) to highlight some “empirical puzzles” that have, well, puzzled me over the years and (2) to raise a few questions about popular empirical methods in the Law and Economics literature. I look forward to a fun discussion with ELS Blog readers.

With that short introduction, here goes …

 

Where did High-End Plaintiffs’ Lawyers go to Law School?

One of my favorite topics in my Law Firms as a Business Organization course  is the plaintiffs' bar, which runs the gamut from soft tissue personal injuries ("car wreck cases"), to product liability and mass torts, to class action litigation.  Law students and law professors know very little about this practice area.  From the student perspective, this gap may be due to the fact that firms that specialize in this area tend to be small and rarely recruit students through on-campus interviews.  Similarly, the corporate bar heavily recruits from top national law schools, where most law professors were educated.  So legal educators are top-heavy with corporate law practice.

In some respects, the plaintiffs' bar appears to reflect an idiosyncrasy in the market for legal talent.  On the one hand, successful plaintiff-side trial and class action lawyers often earn higher incomes than partners at the nation's largest and most prestigious law firms.  On the other hand, these lawyers tend to be graduates of non-elite law schools.   Below is graphic (generated for my class), which compares the educational backgrounds of members of the Inner Circle 100, which is the 100 best plaintiffs' side trial lawyers in the nation (and includes Joe Jamail, John Edwards, Phil Corboy, and, until his death, Johnny Cochrane), and a sample of five Am Law 200 law firms.
Granted, some (or possibly all) of this differential can be explained by the more stable earnings trajectory of corporate lawyers.  But I also wonder whether successful trial practice may entail attributes that are not strongly correlated with law school entering credentials (e.g., LSAT and UGPA).  For example, effective trial lawyers may have better affect recognition, and thus have a superior ability to size up their adversaries and communicate with a jury.

Guest Blogger: Ed Morrison

This week's guest blogger is Edward Morrison, an Associate Professor of Law at Columbia.  Ed received his J.D. and Ph.D. (Economics) from the University of Chicago, and clerked for both Seventh Circuit Judge Richard Posner and Supreme Court Justice Scalia. Ed's areas of expertise include bankruptcy, law and economics, and corporate reorganization.  His recent scholarly work includes "Bankruptcy Decisionmaking: An Empirical Study of Continuation Bias in Chapter 11"  forthcoming in the Journal of Law and Economics.

Data on Supreme Court Oral Argument

I've been invited to Carnegie Mellon to attend an NSF-sponsored workshop about how we can use the data available from Supreme Court Oral Argument.  This grows out of a broad expansion of the The Oyez Project (which has been substantially updated so check it out). 

The goal of the workshop is to get information from invited scholars in a number of fields (law professors, linguists, computer scientists, psychologists) about how to use oral argument data.  The data now available are XML-formatted transcripts of all Supreme Court oral arguments between 1979 and 2005, along with the associated audio recordings.  This format, using ProjectPad and Plone software, will allow for the creation of a structured codeable database for SCOTUS Oral Argument where coded categories can be exported into tab delimited files.  These audio/transcript formatted transcripts are aligned to the sentence/word level for arguments from 2000-2005.

This electronic database has been produced by the efforts of the Principal Investigators: Jerry Goldman (Political Science, Northwestern), Mark Liberman (Linguistics & Computer Science, Penn), Timothy Johnson (Political Science, Minnesota), and Brian MacWhinney (Psychology & Linguistics, Carnegie Mellon).  For more information on the data, contact Jerry Goldman.

UPDATE: Check out this great post by Mark Liberman over at Language Log that references the workshop and has some fun with Justice Co-Voting Percentages.