Empirical Legal Studies

NYT IRB Story

Thought readers may be interested in this story in today's New York Times given our previous coverage of the issue.  (Thanks for the correction, Mike.)

What do you want to know about the Supreme Court?

Having thrown out some musings about a potential future project, I want to raise a question about a project I am currently working on.  This project arises out of my desire to evaluate, across doctrinal areas and legal issues, under what circumstances the Supreme Court announces standards without applying them to the facts at hand.  The United States Supreme Court Database, is of course, the logical place to start for anyone doing quantitative empirical research on Supreme Court case law.  But the architecture and coding of the database are not conducive to the kind of doctrinally-focused analysis I want to do.  As a result, I'm engaged now in exploring the implications of the limitations of the database for scholarship like mine.  I'd be very interested in learning about other scholars' experiences and wish lists.  Are there doctrinally-focused empirical questions you've tried to answer or want to address but that are difficult or impossible to undertake with the database as it -- as well as its counterpart, the Justice-Centered database --  exists now? I’ve read dozens of published works – but what about the projects that have not taken off?

Fellowships for Aspiring Law Professors

One of the virtues of the blogosphere is that valuable research is often bundled together in a convenient form and freely shared with fellow travelers.  Paul Caron has done just that by aggregating an impressive list of teaching fellowships and readings on entering the legal academy. 

[New rule: if you do something communitarian, I post your picture.  Thanks Paul!]

Relationship between Journal Prestige and Citations

Paul Caron has a post an interesting forthcoming study by Andrew Oswald (Warwick Economics) in Econometrica on the relationship between journal prestige and citations.  See Andrew Oswald, An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers.  Since a single body in the UK makes funding decisions for university research, there is significant pressure to rely upon measures of productivity that take into consideration the relative prestige of  faculty members' placements.  Because these journals are peer-reviewed, and the hierarchy of journal prestige in each discipline is fairly established, placement -- so the argument runs -- should be construed as a strong signal of quality.

Oswald's study is important, though the abstract is -- in my opinion -- quite misleading.  This claim caused me to the read the whole study: 

The paper finds that it is far better to publish the best article in an issue of a medium-quality journal like the Oxford Bulletin of Economics and Statistics [a middle-of-the-pack journal] than to publish the worst article (or often the worst 4 articles) in an issue of a top journal like the American Economic Review.

Oswald used citation counts from articles in six economics journals over a twenty-five year period to evaluate the reliability of placement as a quality signal.  But there is no implicit suggestion in the article that a scholar would be better off trading down.

Basically, it boils down to this: Placement is correlated with citation, but as a proxy for quality, there are large Type I and Type II errors.  Articles in AER and Econometrica (#1 and #2 in prestige) did indeed get more citations. Yet, roughly 16% of the articles in the less prestigious journals garnered more citations than the median AER or Econometrica articles.  The most cited articles in the less prestigious journals garnered 10 times or more the citations than the four least cited articles in each issue of AER or Econometrica.

In my opinion, here is the good news: 

What Judges Don't Say: The Challenge to Content Analysis

First of all, many thanks for the opportunity to guest blog here.  I've found ELSblog to be a valuable resource for information and an invaluable resource for thought-provoking reading.  I'm generally a "lurker," but guest blogging will force me out from the shadows.

Like many others, I was trained as a lawyer and practiced law before I became an academic.  My interest in empirical legal scholarship arises in part from experiences I had and things I observed as a law clerk and as a litigator. 

One of the great frustrations lawyers sometimes experience is the sense that judges are not paying attention to the specific facts and evidence of the case. And in some areas of law, at least, academics also take note.  In the world of employment discrimination law, for example, one need not look hard to find lengthy discussions and critiques of courts' inconsistent, unpredictable, or just plain wrong application of the summary judgments standards to prevent plaintiffs from taking their evidence to a jury or, in some cases, to grant judgment as a matter of law and reverse jury verdicts in plaintiffs' favor.   But these are complaints about the application of law to fact, not about inaccurate reporting of the facts themselves.

At the same time, numerous academics have undertaken important efforts to identify the facts that appear to matter to judges in different contexts. To name but two examples:  Jeffrey Segal's well-known work on search and seizure looks at the relationship of certain facts (whether there search was warrantless, for example, or whether it was a search of a home or a car) to outcome in the Supreme Court's 4th Amendment case law.  Lauren Edelman and others are studying how trial and appellate court judges in employment discrimination cases refer to employers' anti- discrimination policies and programs.  But what these efforts do not account for, by definition, is what the judges do not say, what they do not mention in their summary of the facts.

Of course, some might say, who cares what the judges don't say.  What they do say is what was salient to them, and what they do say is what often becomes important in the case law.  (This latter point is one of the arguments of Edelman et al.)  What they don't say may matter to the disappointed party and lawyers, but it is of no consequence otherwise.

But it seems to me that the question of whether judges -- consciously or unconsciously -- leave out (arguably) relevant facts is directly relevant to several points of great interest to scholars as well as to lawyers.  First, it is directly relevant to the question of how judges decide cases.  Second and relatedly, it may shed light on the extent to which they are political in their decisionmaking.  Third, it should, I think, force hard thinking about the normative question of what we want judges to do.  Do we want them to make predictions about how juries will decide?  Do we want them to announce rules of law applicable in the future?  Do we want them to focus scrupulously but narrowly on the case before them?  Are these goals inconsistent with each other?

The answers to these questions may vary depending on the case, the court, and the issue, of course.   Chief Justice Roberts has recently made some news with his call for more narrow, focused decisions -- and evoked the criticism that the Supreme Court's role requires a broader, less case-specific approach.  In fact, in my view, the question of whether judges accurately report the facts is more important in the trial and appellate courts than in the Supreme Court. 

If I'm overlooking something, of course I'd welcome references to work that does try to address the question of what gets left out of judicial opinions.   But as we in empirical legal scholarship are struggling with the question of how to adequately take into account of what is in judicial opinions (how to operationalize the law), let's not forget that opinions must distill mountains of evidence and piles of briefs into a few pages of summary.  Things must fall through the cracks.  I'm wondering -- is there a pattern to what gets left out and do those things matter?

Guest Blogger: Carolyn Shapiro

Our guest blogger this week is Carolyn Shapiro, Assistant Professor of Law at the Chicago-Kent College of Law.  Professor Shapiro earned a B.A. with general and special honors in English from the University of Chicago, an M.A. from the University of Chicago Harris Graduate School of Public Policy and a J.D. (high honors) from the University of Chicago Law School, where she was articles editor of the University of Chicago Law Review and a member of the Order of the Coif.             

After graduation, Professor Shapiro was a law clerk for Chief Judge Richard A. Posner of the U.S. Court of Appeals for the Seventh Circuit and  for Justice Stephen G. Breyer of the U.S. Supreme Court.             

Professor Shapiro's scholarly interests include federal courts and labor and employment law.  She teaches professional responsibility, employment law, and legislative process.

The Impact of Wikipedia & Citation Counts

Over at Concurring Opinions, Dan Solove writes about Cass Sunstein's op-ed in the Washington Post.  Dan finds the following statistic provided by Sunstein to be "quite amazing":

"In the past year, Wikipedia, the online encyclopedia that 'anyone can edit,' has been cited four times as often as the Encyclopedia Britannica in judicial opinions, and the number is rapidly growing."

This statistic is accurate (well, maybe he should have said nearly four times as often).  Lexis shows the following hits:

"encyclopedia britannica" and date(geq (2/23/06) and leq (2/23/07)) = 21 Hits

wikipedia and date(geq (2/23/06) and leq (2/23/07)) = 81 Hits

Sunstein declares that "Wikipedia has become the most influential encyclopedia in the world, consulted by judges as well as those who cannot afford to buy books."  While citation counts are often useful and I have used them in my own research, this provides a forum to recognize their limitations.  For example, in Gashi v. U.S. AG, 2007 U.S. App. LEXIS 423, the hit comes from footnote one which states: "Wikipedia is a free internet encyclopedia that is collaboratively written by its readers and can be edited by anyone."  Is this the type of citation that shows wikipedia has "influence"?  For the most part, however, most of the citations to wikipedia arise from the court providing a citation for factual information.

Marquette Law Hallows Lecture

I was fortunate enough to attend the Marquette Law School Hallows Lecture earlier this week and hear Judge Carolyn Dineen King (5th Circuit, Carter appointee) lecture on federal judicial selection and its ramifications for judicial independence.  In a nutshell (you can watch the speech or read the transcript here), Judge King argued that the politicization of the judicial selection process has lead to judges who feel, consciously or not, bound to make decisions in an ideological fashion and has resulted in some judges on some courts who engage in "clique voting."  She argues that this is MORE problematic at the circuit court level because the heavy workload and the lack of accountability (to each other or to the public) provides the potential for one or two "hard-wired" judges to make a real impact on the law.  She argues that this causes problems for the rule of law and the legitimacy of the court system. 

I bring this to your attention both because it was an interesting speech and because I think Judge King's comments raise an interesting question:  which comes first?   Does politicization of appointments cause ideological voting, OR did ideological voting prompt the politicization of judicial appointments?  Seems to cry out for some empirical research, no?  (Hope to get at this soon, so any reactions/thoughts/data sources are welcomed!)

Female Attorneys Before the Supreme Court

I just read a very interesting paper by a colleague and wanted to bring it to the attention of the readership because of its very important findings.  In "Have We Come a Long Way Baby:  Female Attorneys before the United States Supreme Court," John Szmer, (UNC-Charlotte), Tammy Sarver (Benedictine) and Erin Kaheny (UW-Milwaukee) seek to determine whether attorney gender matters to U.S. Supreme Court justices in their decision making.  Here's their abstract:

While the impact of an attorney's sex has been examined with respect to trial court processes (e.g., jury decision making), no one has previously studied its effects on appellate court decision making. In this article, we argue that the application of gender schemas by some justices results in a devaluing of the arguments made by women litigators. Our findings suggest that women orally arguing attorneys are less likely to receive a favorable vote by a justice than are the male counsel they oppose and that conservative justices are more likely than their liberal counterparts to vote against litigants represented by female counsel at oral argument. This suggests that the ideology of elites influences whether they apply gender schemas in a negative fashion. We also find that justices are more likely to side with female lawyers in women's issues cases, indicating that the justices' perceptions of female lawyer expertise are enhanced in those cases. These findings persist even after controlling for multiple factors, including attorney expertise, the sex of the justice, amicus participation, party capability, and judicial ideology.

An extremely significant contribution to both judicial decision making AND questions of the role of gender in American politics.  I commend it to you.

*NOTE:  This link has been updated to point to the published version of the article rather than the draft, posted originally.  SCB 02.17.15

 

 

Using Empirical Methods to play Moneyball/Moneylaw

I am an avid reader of the MoneyLaw blog, which aspires to apply Moneyball principles to the legal academy. 

For those readers who are unfamiliar with Moneyball, it refers to the use of statistical methods to identify and exploit inefficiencies in the market for baseball talent.   In a widely read review essay on its potential implications for law schools, Paul Caron and Rafael Gely assembled statistical evidence showing that the hiring heuristics used by most law schools (e.g., law school attended, fancy clerkship) were poor predictors of quantity/placement of future scholarship.  Rather, the best predictors of future productivity were pre-academy productivity and the publication of a student note.

Aside from Caron & Gely's preliminary work, however, the collective ruminations of the Moneylaw contributors have not (yet, anyway) articulated a theoretically coherent basis for how Moneyball/Moneylaw principles can produce a more successful law school franchise. (Jim, Jeff, Ted, Paul, Nancy, Al, I say this with love in my heart.)  Since we don't have wins and losses or a World Series title, the most obvious theoretical shortcoming is how success is measured, both in the short and long term.  (In an earlier post, I suggested money as the best longterm metric.) Once this theoretical work is in place, empirical methods can be brought to bear to generate the appropriate chess moves.

So I was quite surprised to receive a phone call the other day from an administrator at a non-Tier 1 school whose primary charge is identifying data-driven ways to improve the functioning of the school.  For the past several years, the law school has paid for this administrator to acquire sophisticated statistical training and build the requisite datasets.  (Jim Chen, you have some serious competition!)  The approach is pretty simple: pick an outcome that matters and generate an inductive theory through data mining.  This "methodology" is not very academic, but it is how Moneyball was invented; it also reflects the approach of many successful hedge funds.

So what outcomes matter?  Here are a few suggestions, some borrowed from the Moneyball/Moneylaw administrator:

Nw. U. L. Rev. Colloquy & Survey Methodology

Northwestern University Law Review's on-line blog, Colloquy, features a debate this week about empirical support for the belief that white Americans hold negative stereotypes associated with viewing Asian Americans as a "model minority." According to Colloquy's editors:

"Robert Chang and Rose Cuison Villazor respond to McGowan and Lindgren's prior piece, Testing the "Model Minority Myth," which argued that empirical data do not support the belief that white Americans hold negative stereotypes associated with viewing Asian Americans as a 'model minority.' Chang and Villazor argue that McGowan and Lindgren's argument is flawed because it relies heavily on data produced by asking white respondents questions about their racial attitudes in face-to-face interviews.  Chang and Villazor assert that this methodology is unlikely to yield true data about racial attitudes, as it is infected with problems of response falsification, is unable to ferret out subtler forms of racial bias, and is limited by the 'perpetrator perspective' on racial discrimination."

Statistical Analysis of Sup. Ct. Clerk Selections and Claims of Justice Bias

In a paper that has already received well-deserved public attention, "Where Have All the Women Gone? 'Random Variation' in the Supreme Court Clerkship Lottery," Professors David Kaye (Law, ASU) and Joseph Gastwirth (Statistics & Econ., George Wash. U.) hold Supreme Court Justices' clerk selections up to empirical light in an effort to assess recent claims of gender disparity.

Some may recall that the New York Times gave front page coverage "to some disturbing facts about this year's clerks - only seven out of the 37 - a mere 19 percent - are women. This outcome, moreover, represents a shocking 50 percent drop from preceding years. Yet, two Justices portrayed this year's percentage as the result of 'random variation,' a claim that strikes many observers as incredible."  The Kaye and Gastwirth paper carefully takes up this claim. An excerpted abstract follows.

"This essay applies standard statistical reasoning to answer two questions - what do the numbers prove, and how strongly do they prove it? We show that this year's decline in women is not at all improbable. Likewise, if the percentage of women applying for these clerkships is in the range of what one Justice suggested, then the small proportion of women is about what one would expect.

The situation seems different, however, when one examines statistics on each Justice. Some Justices hire considerably fewer women than would be expected by chance, while others hire somewhat more. There are many possible explanations for this pattern. We marshal data to assess the plausibility of some of them, but in the end, the available records do not allow a definitive conclusion."

Law, Psychology, and Insurers

Chris Guthrie (Vanderbilt) and Jeff Rachlinski (Cornell) recently circulated an interesting paper, "Insurers, Illusions of Judgment & Litigation," that presents results from original research on insurers' abilities to resist indulging in cognitive illusions that pose a threat to legal decision making.  An excerpted abstract follows.

"Once litigation materializes, insurers provide representation, pay legal fees, and often play a central role in resolving disputes through settlement or adjudication. In this paper, we explore empirically how these key litigation players make important decisions in the litigation process, like evaluating a case, deciding whether to settle, and if so, on what terms. We find that insurers, though not entirely immune to the effects of cognitive illusions that have been shown to distort litigation decision making, appear to make decisions in a more economically rational fashion than other litigation players. This finding, though preliminary, casts new light on litigation theory and practice."

New Book - Mertz on Legal Education

The Language of Law School: Learning to "Think Like a Lawyer" by Elizabeth Mertz  (Oxford 2007)

Availability: Amazon/ B&N/ Oxford

From the description: "Anyone who has attended law school knows that it invokes an important intellectual transformation, frequently referred to as learning to think like a lawyer. This process, which forces students to think and talk in radically new and different ways about conflicts, is directed by professors in the course of their lectures and examinations, and conducted via spoken and written language. Beth Mertz's book is the first study to truly delve into that language to reveal the complexities of how this process takes place."

Altman Weil Report on "Diversity Managers" in Large Law Firms

I am on a subscription list for Altman Weil, Inc., which is one of the major consulting groups for law firms.   This report on "diversity managers" in large law firms caught my eye.   Drawing upon recent survey data, the report provides various breakdowns of firms with diversity managers; and among those that have them (~ 50%, with another 11+% expecting to add such a position in 2007), what is their educational background, salary, responsibilities, and status within the firm. 

The sample suffers from a low response rate (72 of 196 responses = 37%); firms with diversity managers are almost certainly overrepresented.  Nonetheless, the numbers (and high salaries and, for partners, reduced billing requirements) suggests the emergence of a potentially large trend.

My own research in this area causes me to separate out the concept of a diversity manager, which sounds appealing and substantial, from changes to law firms' underlying institutional incentives.  In other words, do these diversity manager have any muscle  vis-a-vis firm management (e.g., work assignments, mentoring, who makes partner, etc.) ?  As the chart below suggests [click on to enlarge], the answer is usually no.

Review - Supreme Conflict

Jack Rakove reviews Jan Crawford Greenburg's Supreme Conflict in this morning's Chicago Tribune book section .

Conducting Empirical Legal Scholarship: The Advanced Course

I am at Washington University for the Advanced Course in Empirical Legal Scholarship.  For the last few years, Lee Epstein and Andrew Martin have offered a basic course (see here for an earlier endorsement).  This is the first advanced course they have offered, and I think it is a tremendously time and cost effective way to cover approximately 95 percent of the skills needed to conduct and interpret empirical scholarship.   Lee and Andrew are extremely well prepared instructors who offer a very good mix of  lecture and class exercises.   Their use of PowerPoint also made a big difference in organizing and simplifying the material.

The topics covered so far include the basic assumptions of linear multivariate regression, including diagnostic methods and corrective measures; effective presentation of regression results; application of the Monte Carlo principle; a fairly in-depth treatment of probit and logit methods for ordinal and nominal dependent variables, including interpretation; and a fairly brief but useful explanation of panel data, time series, survival analysis, and several other methods.    Tomorrow we are going to cover nonparametric matching techniques, which offer (I am told) a better way to deal with endogeniety than instrumental variables in a linear model.

Approximately 20 people are in the workshop, including Conglomerate's Christine Hurt and Concurring Opinions' Dave Hoffman.   It is a very collegial group.  I have really enjoyed myself. 

Empirical Analysis of the Biglaw Tournament

Lately there has been a lot of buzz in the blogosphere on the associate-partner "pay gap" (Jeff Lipshaw, titled "The Days of Whine and Roses"), the application of Marxist economic theory to corporate law firms (David Luban), and what all the huge salaries and profits augur for the future of legal profession and legal education (Brian Tamanaha). 

All of it is worth reading, especially Tamanaha's insights on the growing fissure between elite and nonelite law schools.  (For a scholarly treatment of this topic, see Randolph Jonakait, The Two Hemispheres of Legal Education and the Rise and Fall of Local Law Schools, NYLS L Rev (forthcoming 2007)).

David Zaring and I recently posted our "Young Associates in Trouble" essay, which provides some new empirical evidence on the BigLaw tournament.   The essay reviews two recent novels that portray elite law firms (one in DC, the other in London) as an unending marathon of boring and substantively unfulfilling work.  In turn, we compare these fictitious accounts with findings from a unique dataset based on the AmLaw Midlevel Survey, which includes questionnaire responses from several thousand 3rd, 4th, and 5th year associates. 

So what is the key takeaway?  As shown in the table below, virtually every principle of enlightened management and social responsibility is either (a) associated with lower firm profits, or (b) irrelevant to the bottomline.

The essay also includes some multivariate regression models that reveal that firms actually make more money when associates work on less interesting work or report a greater likelihood of leaving the firm within the next two years.  In contrast, family friendliness, training and guidance, and feedback appear to have no effect on profits.

More empirics and another graph after the jump.

Empirical Study on Effects of Buckhannon

The economics of class action litigation is a fascinating topic.  This terrain shifted substantially in 2001 when the Supreme Court's decision in Buckhannon made it harder for lawyers to collect attorneys' fees under the civil rights fee-shifting statutes.   In essence, this decision limited fee-shifting to cases that result in a judicial judgment or consent decree.  Prior to Buckhannon, virtually every circuit had adopted the "catalyst theory", which permitted recovery when the lawsuit in question had produced some or all of the relief requested by the plaintiff class.

Without this potential pool of money available to pay for legal services, we might expect the number of civil rights class actions to taper off.  At the time, the Supreme Court rejected this line of reasoning as entirely speculative. Fortunately, Catherine Albiston (UC Berkeley Law) and Laura Beth Nielsen (ABF / Northwestern Sociology)  have recently posted "The Procedural Attack on Civil Rights: The Empirical Reality of Buckhannon for the Private Attorney General," which is forthcoming in the UCLA Law Review.

Here is the abstract:

In 2001, in Buckhannon Board & Care Home, Inc. v. West Virginia Department of Health and Human Resources, the Supreme Court rejected the catalyst theory for recovery of attorneys' fees in civil rights enforcement actions. In doing so, the Court dismissed concerns that plaintiffs with meritorious but expensive claims would be discouraged from bringing suit, finding these concerns “entirely speculative and unsupported by any empirical evidence.” This article presents original data from a national survey of more than 200 public interest organizations that call into question the Court's empirical assumptions. These data indicate that organizations that take on paradigmatic public interest cases, such as class actions seeking injunctive relief against government actors, are the most likely to be negatively affected by Buckhannon. In addition, our respondents report that Buckhannon encourages “strategic capitulation,” makes settlement more difficult, and discourages attorneys from representing civil rights plaintiffs. We argue that these far reaching effects herald a shift away from private rights enforcement and toward more government power, both to resist rights claims and to control the meaning of civil rights.

Criminal Pro Se Defense

In a forthcoming article Erica Hashimoto (Georgia) explores an interesting corner of criminal law from an empirical perspective.  Although principally descriptive in its treatment of the data, Defending the Right to Self Representation: An Empirical Look at the Pro Se Felony Defendant finds:

"The data clearly refute both the assumption that most felony pro se defendants are ill-served by the decision to self-represent and the theory that most pro se defendants suffer from mental illness. Somewhat surprisingly, the evidence establishes that pro se felony defendants in state court do just as well as represented felony defendants, and the vast majority of pro se felony defendants - nearly 80% - displayed no signs of mental illness. The results of the study also provide an alternative explanation for the pro se phenomenon, suggesting that at least some defendants choose self-representation because of legitimate concerns about counsel. In short, the data in this Article expose the fallacy of the prevailing view of pro se felony defendants and demonstrate that the right to self-representation in fact serves a vital role in protecting the rights of criminal defendants."

Longitudinal Changes in the Promotion to Partnership Tournament

The National Law Journal has a story discussing the disparity between associate and partner pay in large law firms.  See Leigh Jones, Pay: It's All Relative.  Until the recent starting salary bump to $160,000 per year, new associates were making 11.7% of the amount partners pulled in for 2005.  According to the story, this is the smallest percentage in the last 10 years.  Last week, the NLJ reporter called me up to ask for possible explanations for this pattern.

My initial reaction was changes in large law firm leverage.  Profits per partner are determined by dividing total firm profits by the number of equity partners.  Perhaps large law firms are relying more heavily on associates, staff attorneys, of counsel, and nonequity partners, thus spreading profits over a smaller pool of rainmaking partners.

The data largely corroborated my theory, but I was quite surprised by the magnitude of the trends.  The table below provides a quick snapshot.

Methodological notes: My sample is limited to firms listed on the NLJ 250 in 1995 and 2005.  The NLJ 250 is defined as the 250 largest firms based on the number of lawyers.  This list substantially overlaps  the Am Law 200, which is the 200 largest firms based on revenues.  I generated the change variables only from firms who provided the necessary data for both 1995 and 2005.   Between 1995 and 2005, the number of firms willing to provide a breakdown between equity and nonequity partners fell from 192 to 168.  So some of the change figures above are limited to 168 firms. 

This reticence in reporting partner breakdowns is a telling development, but first the major trend (after the jump).